KR102530794B1 - Data Pre-Processing System - Google Patents

Abstract

A data pre-processing system according to an embodiment of the present invention includes a data management unit for storing collected data collected by a data collection device; and a project management unit that creates a project that is a preprocessing workspace and provides a plurality of preprocessing modules in which the preprocessing process of the collected data is modularized into unit functions, wherein the project management unit configures the preprocessing process of the collected data. It is characterized in that unit history data corresponding to the pre-processing modules is stored, and pre-processing of the collected data is performed according to the pre-processing modules corresponding to the unit history data.

Description

Data Pre-Processing System {Data Pre-Processing System}

The present invention relates to a data pre-processing system, and more particularly to a data pre-processing system for data analysis.

Data analysis may generally be performed in the order of data collection, data exploration, data preprocessing, modeling, modeling verification, and model distribution. At this time, the data preprocessing process consumes the most time and cost among the data analysis processes.

Conventional data pre-processing systems require input of programs directly coded based on the analyst's knowledge and experience. Therefore, the preprocessing result may vary greatly depending on the analyst's knowledge and experience. In addition, even if the analyst has knowledge and experience, considerable time is consumed to perform preprocessing suitable for the purpose of data analysis.

The present invention is to solve the above problems, and its technical feature is to provide a data pre-processing system capable of reducing pre-processing time by using pre-processing history data suggested by a high-level analyst.

In addition, the technical feature of the present invention is to provide a data preprocessing system capable of generating data of uniform quality.

A data pre-processing system according to an embodiment of the present invention includes a data management unit for storing collected data collected by a data collection device; and a project management unit that creates a project that is a preprocessing workspace and provides a plurality of preprocessing modules in which the preprocessing process of the collected data is modularized into unit functions, wherein the project management unit configures the preprocessing process of the collected data. It is characterized in that unit history data corresponding to the pre-processing modules is stored, and pre-processing of the collected data is performed according to the pre-processing modules corresponding to the unit history data.

According to an embodiment of the present invention, through the preprocessing history presented by the advanced analyst, the preprocessing time can be shortened and novice analysts or ordinary people can perform the preprocessing task.

In addition, according to an embodiment of the present invention, it is possible to save time and cost required for data preprocessing by reducing trial and error and repetitive work by using the preprocessing history.

In addition, according to an embodiment of the present invention, preprocessed data may have uniform quality by using the stored preprocessing history for other data preprocessing tasks.

1 is a schematic block diagram of a data pre-processing system according to an embodiment of the present invention.
Figure 2 is a block diagram schematically showing a project history management unit according to an embodiment of the present invention.
3 is a diagram showing unit history data according to an embodiment of the present invention.
FIG. 4 is a diagram showing history data connected to unit history data shown in FIG. 3 .
5A to 5F are diagrams illustrating a history data management method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The meaning of terms described in this specification should be understood as follows.

Singular expressions should be understood to include plural expressions unless the context clearly defines otherwise, and terms such as “first” and “second” are used to distinguish one component from another, The scope of rights should not be limited by these terms.

It should be understood that terms such as "comprise" or "having" do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

The term “at least one” should be understood to include all possible combinations from one or more related items. For example, "at least one of the first item, the second item, and the third item" means not only the first item, the second item, or the third item, but also two of the second item, the second item, and the third item. It means a combination of all items that can be presented from one or more.

1 is a diagram showing a data pre-processing system according to the present invention.

As shown in FIG. 1 , the data preprocessing system 1000 according to the present invention includes a data management unit 100 and a project management unit 200 .

The data management unit 100 stores and manages pre-processing target data. Here, the preprocessing target data may be data collected by a data collection device (not shown). In one embodiment, the data collection device may collect micro data generated in the course of various processes. Here, micro data refers to raw data as data itself collected through various sensors and the like. Hereinafter, for convenience of description, micro data will be referred to as collected data.

The data collection device includes various instruments, sensors, and actuators for collecting micro data. The data collection device may further include a P/C, a Programmable Logic Controller (PLC), a Distributed Control System (DCS), and the like that integrates or controls data collected by measuring instruments, sensors, actuators, and the like.

For example, the data collection device may collect data generated in a continuous process. A continuous process refers to a process in which a plurality of processes for producing a finished product using raw materials are continuously performed, and the outputs of each process are mixed with each other or the state of the output of a specific process is changed and supplied to a subsequent process. Steel processing is a typical example of such a continuous process. The steel process may be composed of various processes such as an iron making process, a steel making process, a continuous casting process, and a rolling process. When the data collection device is applied to a steel process, micro data generated during various processes such as an iron making process, a steel making process, a continuous casting process, and a rolling process may be collected.

The data management unit 100 includes a data storage unit 110 and a data information editing unit 120 .

The data storage unit 110 stores preprocessing target data, for example, collection data collected by a data collection device.

The data information editing unit 120 edits information such as the format and authority of collected data stored in the data storage unit 110 . The data information editing unit 120 may edit information such as the format and authority of collected data according to user input.

The project manager 200 creates a project that is a preprocessing workspace and manages at least one project. The project management unit 200 includes a project creation unit 210 , a project information editing unit 220 and a project history management unit 230 .

The project generating unit 210 creates a project that is a preprocessing workspace. The project creation unit 210 may create a new project according to a user's request. The project generator 210 may provide various functions so that the user can easily perform data pre-processing in the created project.

In one embodiment, the project generation unit 210 may perform data pre-processing on the collected data according to the history data edited by the project history management unit 230 to be described later. For example, the history data corresponding to the preprocessing modules sequentially operated in the first project by the project history management unit 230 is copied to the second project, and the project generation unit 210 copies the first copied to the second project. Data pre-processing can be performed according to the historical data of the project and the result can be calculated. Alternatively, the project generation unit 210 may perform data preprocessing according to the history data added, modified, deleted, inactive, or order changed by the project history management unit 230 to recalculate the result. Accordingly, it is possible to reduce the time and cost required for data preprocessing by reducing trial and error and repetitive work by using the data preprocessing history of a skilled analyst. In addition, it is possible to generate data of uniform quality.

The project information editing unit 220 edits information such as the name and authority of the project created by the project creation unit 210 . The project information editing unit 220 may edit information such as the name and authority of a project according to user input.

The project history management unit 230 manages history data of sequentially operating preprocessing modules in a project.

The project history management unit 230 will be described in detail with reference to FIGS. 2 to 5F. Figure 2 is a block diagram schematically showing a project history management unit according to an embodiment of the present invention. 3 is a diagram showing unit history data according to an embodiment of the present invention, and FIG. 4 is a diagram showing history data connected to the unit history data shown in FIG. 3 . 5A to 5F are diagrams illustrating a history data management method according to an embodiment of the present invention.

As shown in FIG. 2, the project history management unit 230 includes a history generation unit 231, a history addition unit 232, a history correction unit 233, a history deletion unit 234, a history deactivation unit 235, A history copying unit 236 and a history order changing unit 237 are included.

The history generation unit 231 generates history data for sequentially operating preprocessing modules. Specifically, the history generator 231 generates unit history data corresponding to the preprocessing module as shown in FIG. 3 whenever the preprocessing module is operated, and as the preprocessing module operates sequentially, FIG. 4 And as shown in FIG. 5A, the generated unit history data 301 is sequentially connected to generate history data 300. At this time, the history data 300 includes at least one unit history data 301 .

Referring to FIG. 3, the unit history data 301 is composed of a head part H, a data part D, and a tail part T.

Information of unit history data 301 preceding the corresponding unit history data 301 is stored in the head part H. For example, an address at which the unit history data 301 preceding the corresponding unit history data 301 is stored may be stored in the head unit. Alternatively, when the corresponding unit history data 301 is the first unit history data, 'Null' is stored in the head.

Information of the unit history data 301 is stored in the data unit D. Specifically, the data unit (D) is a first data unit for storing the status code (Status) of the corresponding unit history data 301 and request data (Request Data) for data pre-processing using the corresponding unit history data 301. ) and a second data unit for storing response data (Response) and a third data unit for storing a preprocessing module number (Function_id) corresponding to the corresponding unit history data (301). At this time, the state code of the unit history data 301 stored in the first data unit may be a code indicating the state of each unit history data 301 as shown in Table 1. In addition, as shown in Table 2, the preprocessing module number (Function_id) stored in the third data unit may be a preprocessing module number corresponding to each preprocessing module operated in the project. In particular, the preprocessing module list is stored in the project information editing unit 220, and the preprocessing module number (Function_id) of the third data unit is a preprocessing module of any one of the preprocessing module lists stored in the project information editing unit 220. number can match.

status code situation 0 Ready One Running 2 Done 3 Error 4 Delete 5 Disable 6 Initialized

Function_id Functions of the preprocessing module One data load 2 Change variable name and type 3 Automatic quality diagnosis report output 4 Selective quality diagnosis 6 Percentage of characters in variable 7 Percent blanks in variables 8 Check and eliminate data duplication 9 Correlation Coefficient Reporting 10 data visualization 11 Data cleansing automation 13 Remove missing values 14 remove outliers 15 delete variable 16 normalization transformation 17 Create cumulative sum variable 18 variable split 19 Combining variables 20 Categorization of variables 21 create dummy variable 22 Create an interaction variable 23 Create maximum correlated lag variable 24 outlier replacement 25 imputation of missing values 26 merge data 27 Influencing factor detection 28 Large category clustering 29 stratification variable suggestions

In the tail part T, information of unit history data 301 following the corresponding unit history data 301 is stored. For example, the address at which the unit history data 301 following the corresponding unit history data 301 is stored may be stored in the tail part T. When the corresponding unit history data 301 is the last unit history data 301, 'Null' is stored in the tail part T.

In one embodiment, the plurality of unit history data 301 is connected in a linked list structure sequentially connected through the head part (H) and tail part (T) of each unit history data to obtain the history data. make up

As shown in FIG. 5B , the history adding unit 232 may add unit history data 304 to the generated history data 300 . Specifically, the history adding unit 232 may add new unit history data 304 to the generated history data 300 . The history adding unit 232 may connect new unit history data 304 that the user wants to add to the last unit history data 303 of the generated history data 300 .

As shown in FIG. 5C, the history correction unit 233 may edit the function of the preprocessing module by modifying the data unit D of the unit history data 301, 302, 303, and 304. Specifically, the history correction unit 233 may modify the detailed function of the preprocessing module corresponding to the unit history data 303 by modifying the data unit D of the unit history data constituting the history data 300 .

In one embodiment, the project management unit 200 may perform data pre-processing on the collected data by operating a pre-processing module corresponding to each unit history data based on the history data including the unit history data of which the data unit has been modified. .

The history deletion unit 234 may delete unit history data constituting the history data. Specifically, the history deletion unit 234 modifies the first data unit of the unit history data to be deleted with a status code (see Table 1) corresponding to Delete, and as shown in FIG. 5D, the history deletion unit (234) releases the connection between the unit history data 302, the unit history data 301 preceding the corresponding unit history data 302, and the unit history data 303 following, and the unit history data 302 The unit history data 301 preceding the ) is connected to the unit history data 303 following the corresponding unit history data 302.

In one embodiment, the project management unit 200 may operate a preprocessing module corresponding to each unit history data based on the history data from which at least one unit history data has been deleted to perform data preprocessing on the collected data.

The history deactivator 235 may deactivate unit history data constituting the history data. For example, when the user needs to compare the result of preprocessing according to the current history and the result of preprocessing according to the history of deleting the unit history data, the result can be compared by performing deactivation on the corresponding unit history data. Specifically, the history deactivator 235 modifies the first data part of the unit history data to be deactivated with a status code corresponding to Disable, and deactivates the corresponding unit history data as shown in FIG. 5E. When data pre-processing is performed according to the corresponding history data, the function of the pre-processing module corresponding to the deactivated unit history data is not performed, but the function of the pre-processing module corresponding to the succeeding unit history data is performed.

After pre-processing according to the history data, the inactive unit history data can be restored by modifying the first data part to a status code corresponding to Done.

The history copy unit 236 may copy the history data for the preprocessing process performed in one project to another project. Specifically, the history copying unit 236 may generate second history data by copying the first history data generated as the preprocessing modules sequentially operate in the first project to the second project. Accordingly, it is possible to save time and cost required for data preprocessing by reducing trial and error and repetitive work by using preprocessing history data that has already been performed.

In one embodiment, the project management unit 200 may perform data pre-processing on the collected data by operating a pre-processing module corresponding to each unit history data based on the copied history data.

The history order changing unit 237 may change the order of unit history data constituting the history data. For example, as shown in FIG. 5F, the history order changing unit 237 may change the order of second unit history data and third unit history data. Disconnect the second unit history data from the third unit history data, connect the second unit history data to the fourth unit history data, connect the third unit history data to the first unit history data, and then connect the third unit history data to the second unit history data. You can change the order of second unit history data and third unit history data by connecting the data.

In one embodiment, the project management unit 200 may perform data pre-processing on collected data by operating a pre-processing module corresponding to each unit history data based on the history data in which the order of unit history data is changed.

According to an embodiment of the present invention, the project management unit 200 pre-processes the collected data according to the pre-processing module corresponding to each unit history data for the history data including the unit history data edited by the project history management unit 230. can do. Accordingly, the pre-processing time is shortened by using the pre-processing history data presented by the advanced analyst, and novice analysts or ordinary people can perform the pre-processing task.

In addition, according to an embodiment of the present invention, data preprocessed by using the stored history data in another project can have uniform quality.

Those skilled in the art to which the present invention pertains will be able to understand that the above-described present invention may be embodied in other specific forms without changing its technical spirit or essential features.

Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

All disclosed methods and procedures described herein may be implemented, at least in part, using one or more computer programs or components. This component may be a series of programs via any conventional computer readable medium or machine readable medium including volatile and nonvolatile memory such as RAM, ROM, flash memory, magnetic or optical disks, optical memory, or other storage media. may be provided as computer instructions. The instructions may be provided as software or firmware, and may be implemented in whole or in part in hardware configurations such as ASICs, FPGAs, DSPs, or any other similar device. The instructions may be configured to be executed by one or more processors or other hardware components, which upon executing the series of computer instructions perform all or part of the methods and procedures disclosed herein, or make it possible to do

1000: data pre-processing system 100: data management unit
110: data storage unit 120: data information editing unit
200: project management unit 210: project storage unit
220: Project editing unit 230: Project history management unit

Claims (9)

a data management unit that stores collection data collected by the data collection device; and
A project management unit that creates a project, which is a preprocessing workspace, and provides a plurality of preprocessing modules that modularize the preprocessing process of the collected data into unit functions;
The project management department,
Storing unit history data corresponding to preprocessing modules constituting the first preprocessing process of the collected data;
Performing a second pre-processing of the collected data according to pre-processing modules corresponding to at least some of the unit history data;
At least some of the unit history data constitutes history data for a second pre-processing process of the collected data. delete According to claim 1,
The unit history data data pre-processing system, characterized in that for storing state information indicating the state of the pre-processing module corresponding to the unit history data and module number indicating the function of the pre-processing module. According to claim 1,
The project management department,
Data pre-processing system, characterized in that by editing the unit history data constituting the history data, modifying the function of the pre-processing module corresponding to the unit history data, and pre-processing the collected data according to the pre-processing module whose function is modified. According to claim 1,
The project management department,
A project history management unit for generating unit history data corresponding to the sequentially operated preprocessing modules as the preprocessing modules sequentially operate on the collected data in the project, and generating history data using the generated unit history data. A data pre-processing system comprising: According to claim 1,
The project management department,
The first history data generated as the preprocessing modules sequentially operate in the first project are copied to the second project to create second history data, and each of the unit history data constituting the second history data in the second project A data pre-processing system, characterized in that for pre-processing the collected data according to corresponding pre-processing modules. According to claim 1,
The project management department,
The data pre-processing system, characterized in that adding, deleting, and changing the order of the unit history data by using the connection between the unit history data. According to claim 1,
The history data is composed of unit history data corresponding to the preprocessing module,
The unit history data,
a head unit for storing information in which unit history data preceding the corresponding unit history data is stored;
a data unit for storing information about a preprocessing module corresponding to the corresponding unit history data; and
A data pre-processing system comprising a tail unit for storing information of unit history data following the corresponding unit history data. According to claim 8,
The data pre-processing system, characterized in that the history data is connected in a linked list structure using a head part and a tail part of unit history data corresponding to the pre-processing module to form the history data.

Images