KR102531742B1 - Method and apparatus for null value correction of sensor data - Google Patents

Abstract

The present invention relates to a missing value correction method and a missing value correction device for correcting missing values in sensor data composed of tables. The missing value correction method detects a specific column in which a cell value is missing from data collected from a sensor and , Correct the missing cell value of a specific column using the valid column except for the specific column.

Description

Missing value correction method and missing value correction device of sensor data {METHOD AND APPARATUS FOR NULL VALUE CORRECTION OF SENSOR DATA}

The present invention relates to a missing value correction method of sensor data and a missing value correction device, and more particularly, to an apparatus and method for deriving high-quality data by automatically correcting missing values included in sensor data collected from a sensor. it's about

In general, table data is data composed of columns and rows, and is an expression format mainly used in relational databases, and is a method of displaying a large number of items having multiple attributes. Each row of the table data is a data item, and each cell of the table can be viewed as an attribute value of the item as a column as an attribute of each item.

In the case of table data in which some of the data in each cell of the table is missing, the burden of handling exceptions in order to process the missing data occurs when analyzing data or using training data for machine learning. In this case, there may be cases in which entire row data cannot be used due to omission of one cell.

When exception handling is provided, all corresponding data items are excluded or replaced with average values or simple inference values. However, when all data items are excluded, values of other valid columns are excluded from data analysis. In addition, when the average value or a simple inference value is substituted, there is a problem that the missing data is biased to a specific value.

The present invention detects a specific column in which a cell value is missing in data composed of a table, and corrects the missing cell value using a valid row according to a valid column excluding the specific column, so that a missing value due to an error in a specific column is corrected. Provided is an apparatus and method for preventing all rows included from being excluded from data analysis.

In the present invention, after learning all rows of an effective column except for a specific column including a missing cell value, and then correcting the missing cell value of a specific column based on the learning result, a specific value according to the correlation between rows constituting a table. It provides an apparatus and method for preventing bias.

The present invention provides an apparatus and method for correcting missing cell values according to characteristics of time and place by considering spatial similarity and temporal similarity in the process of collecting data when correcting missing cell values of a specific column. do.

A missing value correction method according to an embodiment of the present invention includes generating test data composed of tables using target data collected from sensors installed at a plurality of observation points; determining a specific column in which a cell value is missing among cells forming a table by analyzing the test data; and correcting missing cell values of the specific column by using valid rows according to valid columns excluding the specific column from the test data.

The generating step according to an embodiment of the present invention may include collecting target data in a column unit over time from sensors installed at the plurality of observation points; and generating test data composed of tables for correcting missing cell values by sequentially collecting the collected target data in units of columns.

The determining step according to an embodiment of the present invention may include pre-processing a display format of a data type represented by cells forming the table; and determining a specific column in which a cell value is missing among cells of the table in which the display format is preprocessed.

Correcting a missing cell value of a specific column according to an embodiment of the present invention may include extracting a collection time of a specific column having a missing cell value from the test data; extracting at least one effective column collected at the same time as the collection time extracted from the test data according to time similarity; and correcting missing cell values of a specific column using cell values of valid rows according to the at least one extracted effective column.

Correcting a missing cell value of a specific column according to an embodiment of the present invention may include extracting an observation point of a sensor that collects a specific column having a missing cell value from the test data; Determining different observation points installed within a certain radius from the observation point of the extracted sensor according to spatial similarity; extracting at least one effective column collected from sensors installed at different observation points from the inspection data; and correcting missing cell values of a specific column using cell values of valid rows according to the at least one extracted valid column.

In the correcting of the missing cell value of the specific column according to an embodiment of the present invention, the missing cell value of the specific column may be corrected based on a data pattern for valid rows according to the valid columns.

In the missing value correction device including a processor according to an embodiment of the present invention, the processor generates test data composed of a table using target data collected from sensors installed at a plurality of observation points, and the test data is analyzed to determine a specific column in which a cell value is missing among cells forming a table, and the missing cell value of the specific column can be corrected using a valid row according to valid columns excluding the specific column in the inspection data. there is.

The processor according to an embodiment of the present invention collects target data in column units over time from sensors installed at the plurality of observation points, and corrects missing cell values by sequentially collecting the collected target data in column units. It is possible to create inspection data composed of tables for

The processor according to an embodiment of the present invention may perform pre-processing on the display format of the data type indicated by the cells forming the table, and determine a specific column in which the cell value is missing among the cells of the table in which the display format is pre-processed. there is.

The processor according to an embodiment of the present invention extracts the collection time of a specific column in which the cell value is missing from the test data, and selects at least one effective column collected at the same time as the collection time extracted from the test data according to time similarity. , and missing cell values of a specific column may be corrected using cell values of valid rows according to the extracted at least one effective column.

A processor according to an embodiment of the present invention extracts an observation point of a sensor that collects a specific column in which a cell value is missing from the inspection data, and extracts an observation point of the extracted sensor according to spatial similarity and different sensors installed within a certain radius. An observation point is determined, at least one effective column collected from sensors installed at different observation points is extracted from the inspection data, and a cell value of a valid row according to the at least one extracted effective column is used to determine a particular column. Missing cell values can be corrected.

The processor according to an embodiment of the present invention may correct a missing cell value of a specific column based on a data pattern for a valid row according to the valid columns.

A missing value correction method according to an embodiment of the present invention detects a specific column in which a cell value is missing from collected data composed of a table, and corrects the missing cell value using a valid row according to an effective column excluding the specific column. , it is possible to prevent the entire row containing missing values due to an error in a specific column from being excluded from data analysis.

A missing value correction method according to an embodiment of the present invention learns all rows of valid columns except for a specific column including missing cell values, and then corrects missing cell values of a specific column based on the learning result, It is possible to prevent bias to a specific value according to the correlation between rows constituting .

In correcting the missing cell value of a specific column, the method for correcting missing values according to an embodiment of the present invention considers spatial similarity and temporal similarity in the process of collecting data, Cell values can be corrected.

1 is a diagram illustrating an overall operation for correcting missing values of data composed of tables according to an embodiment of the present invention.
2 is a diagram for explaining the detailed operation of the missing value correction device according to an embodiment of the present invention.
3 is a diagram for explaining a process for predicting a missing value of data according to an embodiment of the present invention.
4 is a diagram for explaining a process of correcting a missing value of data according to an embodiment of the present invention.
5 is a flowchart illustrating a missing value correction method according to an embodiment of the present invention.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be changed and implemented in various forms. Therefore, the form actually implemented is not limited only to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Although terms such as first or second may be used to describe various components, such terms should only be construed for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of reference numerals, and duplicate descriptions thereof will be omitted.

1 is a diagram illustrating an overall operation for correcting missing values of sensor data composed of tables according to an embodiment of the present invention.

Referring to FIG. 1 , the missing value correcting device 101 may automatically correct missing cell values in sensor data composed of tables. In detail, the missing value correcting device 101 may extract similar values of the missing rows by learning a pattern of sensor data for all rows having valid values of a specific column with missing cell values. Here, a valid value of a specific column with a missing cell value may mean a valid value of the same row as a cell missing from other valid columns except for a specific column with a missing cell value. The missing value correcting device 101 may correct missing cell values of a specific column using valid values of the row.

To this end, the missing value correcting device 101 may generate test data composed of tables using target data 102 collected from sensors installed at a plurality of observation points 103 . The missing value correcting device 101 may perform a pre-processing of the display format of the data type indicated by the cells forming the table. Here, the preprocessing process may refer to a process of converting the data type of each column into a numeric value when the data type is not a numeric value. For example, if the data type of each column is categorical data, possible categorical strings can be converted into sequential numeric values. Also, if the data type of each column is text, the corresponding text can be converted into a numerical value based on the individual identifier number written in the term list. In addition, all numerical values can be converted into values between '0' and '1' by normalization. In this case, columns that cannot be quantified are excluded.

The missing value correction device 101 may determine a specific column in which a cell value is missing among cells forming a table by analyzing the test data. The missing value correcting device 101 may correct missing cell values of a specific column by using valid rows according to valid columns excluding the specific column from the test data.

2 is a diagram for explaining the detailed operation of the missing value correction device according to an embodiment of the present invention.

Referring to FIG. 2 , the missing value correction device 201 may include a processor 202 for correcting missing values. The processor 202 may detect a specific column in which a cell value is missing from data collected from the sensor, and correct the missing cell value of the specific column by using valid columns excluding the specific column. To this end, the processor 202 may receive target data 205 collected from sensors 204 installed at a plurality of observation points. The processor 202 may generate examination data 203 composed of one table by collecting the received target tables 205 .

The processor 202 may collect target data 205 in units of columns over time from the sensor. The processor 202 may sequentially collect the collected target data 205 in units of columns to generate examination data 203 configured as a table for correcting missing cell values. The processor 202 may generate the examination data 203 configured as a table by sequentially collecting the target data 205 from the sensor in order based on the collection time.

The processor 202 may determine a specific column from which a cell value is missing among cells forming a table by analyzing the test data. The processor 202 may perform a pre-processing process on the display format of the data type indicated by the cells forming the table. The processor 202 may determine a specific column in which a cell value is missing among cells of a table in which a display format is preprocessed.

The processor 202 may correct missing cell values of a specific column by using valid rows according to valid columns excluding the specific column from the examination data.

3 is a diagram for explaining a process for predicting a missing value of data according to an embodiment of the present invention.

Referring to (a) of FIG. 3 , the missing value calibrating apparatus may determine a specific column 301 from which a cell value 302 is missing among cells forming a table by analyzing test data.

In detail, the missing value correcting apparatus may perform a pre-processing of the display format of the data type indicated by the cells forming the table. Here, the preprocessing process may be a process of correcting the data type represented by each column of the inspection data composed of a table. The missing value correcting device may perform an operation of identifying a data type of each column and converting the identified data type into a numerical value. For example, when the data type of a column is a string type, the missing value correction device converts a string expressed as a string type into a sequential numeric value or a numeric value based on an individual identifier number. there is.

The missing value calibrating device may determine a specific column 301 in which a cell value 302 is missing among cells of a table whose display format is preprocessed. The missing value correcting device may detect a cell expressing a blank value or a numeric value of a character string among cells of the table, and determine that the corresponding cell is missing a value.

Referring to (b) of FIG. 3 , the missing value correcting device uses the valid row 304 according to the valid columns 303 excluding the specific column 301 from the test data to obtain the missing cell value of the specific column 301. (302) can be corrected. The missing value correction device may correct missing cell values of a specific column by using a forward propagation network (FFN) / back propagation network (BPN) or a learning model through regression analysis.

The missing value correcting device may apply a forward propagation network (FFN) / back propagation network (BPN) or a learning model through regression analysis to each cell value of the valid row 304 according to the valid columns 303 . The missing value correcting device may correct the missing cell value of a specific column based on the result thereof.

At this time, the missing value correcting device may correct the missing cell value according to the characteristics of time and place by considering spatial similarity and temporal similarity in the process of collecting data when correcting the missing cell value of a specific column. there is.

① time similarity

The missing value correcting device may extract a collection time of a specific column having a missing cell value from examination data when a specific column having a missing cell value is determined among cells forming a table. In this case, the collection time may be a time when data is collected according to a predetermined period from a sensor installed at an observation point. The missing value correction device may extract at least one effective column collected at the same time as the collection time extracted from the test data according to time similarity. Here, the extracted effective column is previously collected data, and may refer to a column having a valid value that is used to correct the missing cell value and does not have a missing cell value. For example, the missing value correcting device may extract at least one effective column collected at the same time on a daily, weekly, monthly, or yearly basis based on the present.

Also, the missing value correcting apparatus may correct missing cell values of a specific column by using cell values of valid rows according to at least one extracted valid column. The missing value correction device may determine a difference value, a median value, and an average value between cell values included in at least one valid column. The missing value correcting device may correct missing cell values of a particular column that are optimal from the determined values.

② Spatial similarity

When a specific column with missing cell value is determined among cells forming the table, the missing value correcting device may determine an observation point where a sensor that collects a specific column with missing cell value is installed in the examination data. The missing value correction device may determine different observation points installed within a certain radius from the observation point of the extracted sensor according to spatial similarity. The missing value correction device may extract at least one effective column collected from sensors installed at different observation points from the inspection data.

For example, when the observation point where a specific column with missing cell values was collected is Gangnam, and the adjacent observation points are Seocho and Eunpyeong, the missing value correction device selects a valid column corresponding to the data collected in Seocho and Eunpyeong from the test data. can be extracted.

4 is a diagram for explaining a process of correcting a missing value of data according to an embodiment of the present invention.

Referring to FIG. 4 , the missing value correcting apparatus may correct missing cell values of a specific column by using valid rows according to valid columns excluding the specific column from the test data. To this end, the missing value correction device may utilize a feed forward network (FFN) and a back propagation network (BPN). In addition, the missing value correction device may utilize a learning model through regression analysis.

The missing value calibrating device may determine a data pattern for a valid row according to valid columns except for a specific column according to the forward propagation network and the reverse propagation network. The missing value correcting device may correct missing cell values of a specific column based on a data pattern.

① How to use forward propagation network and reverse propagation network

The missing value correcting apparatus may correct missing cell values for rows having missing cells by utilizing forward propagation networks and back propagation networks according to the number of missing cells. Here, the number of inputs of the forward propagation network may be the number of valid columns excluding a specific column, and the number of outputs of the back propagation network may be the number of missing cells. In the present invention, test data may be used as training data for learning a forward propagation network and a back propagation network, and the training data may include the number of all rows having valid columns.

An input of the forward propagation network may be an attribute value for a valid column of a row including a cell that is missing from the inspection data, and an output of the back propagation network may be a value predicted as a missing cell value of a specific column.

② How to use a learning model through regression analysis

The missing value correcting device may predict and correct missing cell values of a specific column by using regression analysis used to determine the effect of an independent variable on a dependent variable.

The missing value compensating device may determine an item indicated by a cell having a missing value, and classify items other than the item determined from the test data. The missing value correction device may set the remaining items and rows of each item as independent variables, and set the rows of the missing cells as dependent variables. The missing value correction device may generate a learning model through regression analysis according to an independent variable and a dependent variable. The missing value correcting device may predict and correct missing cell values of a specific column by using a learning model.

5 is a flowchart illustrating a missing value correction method according to an embodiment of the present invention.

In step 501, the missing value correction device may generate test data composed of tables using target data collected from sensors installed at a plurality of observation points. Here, the missing value correction device may collect target data in a column unit over time from sensors installed at a plurality of observation points. The missing value correction device may generate examination data composed of a table for correcting missing cell values by sequentially collecting the collected target data in units of columns.

In operation 502, the missing value correcting apparatus may analyze the test data to determine a specific column in which a cell value is missing among cells forming a table. The missing value correcting apparatus may perform a pre-processing of the display format of the data type indicated by the cells forming the table. The missing value correcting device may determine a specific column in which a cell value is missing among cells of a table in which a display format is preprocessed.

In step 503, the missing value correcting device may correct missing cell values of a specific column by using valid rows according to valid columns excluding the specific column from the test data. The missing value correcting device may consider temporal similarity of object data collected from a sensor or spatial similarity of a location where a sensor that collects object data is installed to generate inspection data.

The missing value correcting apparatus may correct missing cell values of a specific column by using cell values of effective rows according to at least one extracted effective column. The missing value correcting device may perform learning through a regression model on cell values of valid rows according to at least one extracted valid column, and correct missing cell values of a specific column according to the learning result.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination, and the program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in the art of computer software. may be Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

The hardware device described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

101: missing value correction device
102: target data
103: observation point

Claims (12)

generating test data composed of tables using target data collected from sensors installed at a plurality of observation points;
determining a specific column in which a cell value is missing among cells forming a table by analyzing the test data; and
correcting missing cell values of the specific column by using valid rows according to valid columns excluding the specific column from the test data;
including,
The generating step is
collecting target data in column units over time from sensors installed at the plurality of observation points; and
generating test data composed of tables for correcting missing cell values by sequentially collecting the collected target data in units of columns;
Missing value correction method comprising a. delete generating test data composed of tables using target data collected from sensors installed at a plurality of observation points;
determining a specific column in which a cell value is missing among cells forming a table by analyzing the test data; and
correcting missing cell values of the specific column by using valid rows according to valid columns excluding the specific column from the test data;
including,
The determining step is
pre-processing a display format of a data type represented by cells forming the table; and
determining a specific column in which a cell value is missing among cells of a table in which the display format is preprocessed;
Missing value correction method comprising a. generating test data composed of tables using target data collected from sensors installed at a plurality of observation points;
determining a specific column in which a cell value is missing among cells forming a table by analyzing the test data; and
correcting missing cell values of the specific column by using valid rows according to valid columns excluding the specific column from the test data;
including,
The step of correcting the missing cell value of the specific column,
extracting a collection time of a specific column in which a cell value is missing from the inspection data;
extracting at least one effective column collected at the same time as the collection time extracted from the test data according to time similarity; and
correcting missing cell values of a specific column using cell values of valid rows according to the at least one extracted valid column;
Missing value correction method comprising a. generating test data composed of tables using target data collected from sensors installed at a plurality of observation points;
determining a specific column in which a cell value is missing among cells forming a table by analyzing the test data; and
correcting missing cell values of the specific column by using valid rows according to valid columns excluding the specific column from the test data;
including,
The step of correcting the missing cell value of the specific column,
extracting an observation point of a sensor that has collected a specific column in which a cell value is missing from the inspection data;
Determining different observation points installed within a predetermined radius from the observation point of the extracted sensor according to spatial similarity;
extracting at least one effective column collected from sensors installed at different observation points from the inspection data; and
correcting missing cell values of a specific column using cell values of valid rows according to the at least one extracted valid column;
Missing value correction method comprising a. According to claim 1,
The step of correcting the missing cell value of the specific column,
A missing value correction method for correcting missing cell values of a specific column based on data patterns for valid rows according to the valid columns. In the missing value correction device including a processor,
the processor,
Generate inspection data composed of tables using target data collected from sensors installed at a plurality of observation points,
Analyzing the inspection data to determine a specific column in which a cell value is missing among cells forming a table;
Correcting the missing cell value of the specific column using valid rows according to valid columns excluding the specific column from the test data;
In generating the inspection data,
Collecting target data in column units over time from sensors installed at the plurality of observation points,
A missing value correcting device that sequentially collects the collected target data in units of columns to generate test data composed of tables for correcting missing cell values. delete In the missing value correction device including a processor,
the processor,
Generate inspection data composed of tables using target data collected from sensors installed at a plurality of observation points,
Analyzing the inspection data to determine a specific column in which a cell value is missing among cells forming a table;
Correcting the missing cell value of the specific column using valid rows according to valid columns excluding the specific column from the test data;
In determining the specific column,
pre-processing the display format of the data type indicated by the cells forming the table;
A missing value correction device for determining a specific column in which a cell value is missing among cells of a table in which the display format is preprocessed. In the missing value correction device including a processor,
the processor,
Generate inspection data composed of tables using target data collected from sensors installed at a plurality of observation points,
Analyzing the inspection data to determine a specific column in which a cell value is missing among cells forming a table;
Correcting the missing cell value of the specific column using valid rows according to valid columns excluding the specific column from the test data;
In correcting the missing cell value,
extracting the collection time of a specific column in which the cell value is missing from the inspection data;
Extracting at least one effective column collected at the same time as the collection time extracted from the inspection data according to time similarity,
A missing value correcting device for correcting missing cell values of a specific column using cell values of effective rows according to the extracted at least one effective column. According to claim 7,
the processor,
Extracting observation points of sensors that have collected a specific column in which cell values are missing from the inspection data,
Determining different observation points installed within a certain radius from the observation point of the extracted sensor according to spatial similarity;
Extracting at least one effective column collected from sensors installed at different observation points from the inspection data;
A missing value correcting device for correcting missing cell values of a specific column using cell values of effective rows according to the extracted at least one effective column. According to claim 7,
the processor,
A missing value correcting device for correcting missing cell values of a specific column based on data patterns for valid rows according to the valid columns.

Images