JP6567720B1 - Data preprocessing device, data preprocessing method, and data preprocessing program - Google Patents

Abstract

Provided is a technique that enables even a non-expert to perform data preprocessing. A clustering server classifies high-dimensional data into clusters, calculates a score for an arbitrary two-dimensional combination of high-dimensional data, and a visualization server 30 displays a score for the two-dimensional combination in a tabular format. Display and accept selection, and plot high-dimensional data divided into clusters on a two-dimensional plane with each dimension of the selected two-dimensional combination as an axis, and accept a preprocessing device for the high-dimensional data. [Selection] Figure 2

Description

  The present invention relates to a technique for preprocessing data in data analysis.

  In recent years, business analytics that analyzes business data and uses it to formulate business strategies has become popular. Real business data is high-dimensional and has many noises and defects, and lack of necessary label information. Preprocessing such as cleansing, labeling, and feature selection is indispensable. There is a technique described in Patent Document 1 as a technique that can reduce the trial and error and man-hours required for the pre-processing work.

JP 2012-243013 A

  Data analysis is said to have 90% pre-processing. The operation of the data scientist was under pressure due to the pre-processing, and there was a problem that sufficient operation could not be used for model construction that was originally a business. The shortage of data scientists is also serious.

  The present invention has been made in view of the above, and performs data preprocessing even for non-experts who have domain knowledge of data but do not have basic knowledge of data science such as probability and statistics. The purpose is to provide technology that can be used.

A data preprocessing device according to the first aspect of the present invention includes a clustering unit that classifies high-dimensional data into clusters, calculates a score for any two-dimensional combination of the high-dimensional data, and a plurality of the two-dimensional combinations A global analysis screen display for displaying a global analysis screen for displaying a score for and accepting selection of the two-dimensional combination, and when the two-dimensional combination is selected on the global analysis screen, the selected two-dimensional a local analysis screen display unit that each dimension of combining the high dimensional data onto a two-dimensional plane having axes drawn divided into clusters, and displays the local analysis screen for accepting an operation on the high-dimensional data of the high dimensional A history management unit for making the data invariant and managing a history of operations on the high-dimensional data, and on the local analysis screen When the operation for the high-dimensional data is accepted, the history of the operation for the high-dimensional data is registered in the history management unit, and the clustering unit reflects the operation indicated by the operation history in the high-dimensional data. When recalculating the score using, and shifting from the local analysis screen to the global analysis screen, the global analysis screen display unit displays the recalculated score and accepts the selection of the two-dimensional combination It is characterized by.

A data preprocessing method according to a second aspect of the present invention is a data preprocessing method executed by a computer, classifying high-dimensional data into clusters, and calculating scores for arbitrary two-dimensional combinations of the high-dimensional data. Displaying a score for a plurality of the two-dimensional combinations, displaying a global analysis screen for accepting selection of the two-dimensional combination, and selecting the two-dimensional combination on the global analysis screen. Displaying a local analysis screen for accepting an operation on the high-dimensional data by drawing the high-dimensional data in a cluster on a two-dimensional plane with each dimension of the selected two-dimensional combination as an axis; The high-dimensional data is unchanged, and an operation on the high-dimensional data is received on the local analysis screen. If registered, the history of the operation for the high-dimensional data is registered in the history management unit, the score is recalculated using the latest data reflecting the operation indicated by the operation history in the high-dimensional data, and the local analysis is performed. When the screen is shifted from the screen to the global analysis screen, the recalculated score is displayed on the global analysis screen, and the selection of the two-dimensional combination is accepted .

  According to a third aspect of the present invention, there is provided a data preprocessing program for operating a computer as each unit of the data preprocessing apparatus.

  According to the present invention, even an unskilled person can perform data preprocessing.

It is a figure for demonstrating the outline | summary of the data pre-processing system of this embodiment. It is a functional block diagram which shows the structure of the data pre-processing system of this embodiment. FIG. 3A is a diagram illustrating an example of actual business data before conversion, and FIG. 3B is a diagram illustrating an example of data after conversion. It is a figure which shows the example of the database structure of DB server. It is an example of the global analysis screen which visualized global analysis. It is an example of the local analysis screen which visualized local analysis. It is a figure which shows the example which removes an outlier in a local analysis. It is a figure which shows the example which attaches a label in a local analysis. It is a float chart which shows the flow of the data conversion process by the data pre-processing system of this embodiment. It is a float chart which shows the flow of the global analysis process by the data pre-processing system of this embodiment. It is a float chart which shows the flow of the local analysis process by the data pre-processing system of this embodiment. It is a sequence diagram which shows the flow of the data conversion process by the data pre-processing system of this embodiment. It is a sequence diagram which shows the flow of the global analysis process by the data pre-processing system of this embodiment. It is a sequence diagram which shows the flow of the local analysis process by the data pre-processing system of this embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  As shown in FIG. 1, the data preprocessing system of the present embodiment uses a global analysis screen, and the score calculated for an arbitrary two-dimensional combination of high-dimensional real business data is associated with the diameter of a circle. Visualize in tabular form and accept selection of circles representing scores. As a method for visualizing the score, other methods such as color shading and numerical value notation can be selected instead of the circle diameter. Further, if the size of the circles is significantly different, the difference in display can be reduced by logarithmization or the like. Moreover, the data preprocessing system of this embodiment can plot and visualize data on a two-dimensional plane corresponding to the selected circle as a local analysis screen, and can perform removal and labeling of data outliers. Like that. As a global analysis, a score for an arbitrary two-dimensional combination is displayed and visualized, a selection of a two-dimensional combination is accepted, and data is drawn on a two-dimensional plane with each dimension of the selected two-dimensional combination as an axis. By accepting operations on the data, the high-dimensional data is visualized in an intuitive format that is easy to understand, so even a non-expert can perform data pre-processing. In addition, since it is possible to easily shift between the global analysis and the local analysis and repeat it, it is possible to execute the data preprocessing while overlooking the clustering result from various viewpoints.

  FIG. 2 is a functional block diagram showing the configuration of the data preprocessing system of this embodiment. The data preprocessing system shown in FIG. 2 includes a conversion server 10, a database (DB) server 20, a visualization server 30, a history management server 40, and a clustering server 50. Each device included in the data preprocessing system may be configured by a computer including an arithmetic processing device, a storage device, and the like, and the processing of each device may be executed by a program. This program is stored in a storage device included in each device, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or provided through a network. The functions of each device of the data preprocessing system may be realized by a single computer, or the functions of each device may be realized by being distributed by a plurality of computers. Hereinafter, each device will be described.

  The conversion server 10 executes data conversion processing of high-dimensional actual business data. 3A shows an example of actual business data (visit sales history) before conversion, and FIG. 3B shows an example of converted data. The conversion server 10 performs categorical variable removal, numerical conversion of continuous values (for example, dates), standardization for each column, and the like as data conversion processing. In the example of FIG. 3, the gender column is removed, the visit date is converted into a numerical value, and standardization is performed for each column. The ID is left for identification.

  The DB server 20 includes a post-conversion data table 21, a preprocessing history table 22, and a clustering result table 23.

  The post-conversion data table 21 holds post-conversion data converted from actual business data by the conversion server 10. The preprocessing history table 22 holds a history of each operation of preprocessing for the converted data. In this embodiment, the converted data is treated as an immutable (invariant) object, version management is performed for each preprocessing operation (outlier removal, labeling), and all change histories are version management systems such as Git Manage with. When the visualization server 30 and the clustering server 50 refer to the converted data, the latest data reflecting the change history of the preprocessing history table 22 to the converted data of the converted data table 21 (hereinafter referred to as “data”, “ Sometimes referred to as “high-dimensional data”). By managing the history of each operation on the post-conversion data without changing the post-conversion data, it is possible to restore the data at any point in time when an erroneous operation is performed.

  The clustering result table 23 holds the clustering result by the clustering server 50. The clustering result includes a result of classifying the high-dimensional data into clusters and a score for an arbitrary two-dimensional combination of the high-dimensional data.

  FIG. 4 shows an example of the database configuration of the DB server 20. Arrows in the figure indicate dependency relationships. In the pre-processing history table 22, a record of history ID and operation / value and a record of history ID and target ID are registered. For example, when the history ID is H0, deletion is registered as an operation, and D1 is registered as a target ID. This indicates an operation in which the data with the data ID D1 is deleted from the converted data. In the clustering result table 23, a record of cluster ID, history ID, axis 1, axis 2, and score and a record of cluster ID, target ID, and cluster number are registered. For example, in the record with the cluster ID C0, data with a history ID H0, axis 1 is age, axis 2 is contract days, and score is 256 is registered. This indicates that the score obtained when clustering with the axis 1 as the age and the axis 2 as the number of contract days is 256 for the data reflecting each operation up to the history ID H0 in the converted data. In the clustering with the cluster ID C0, the data of D1 belongs to the cluster with the cluster number 0, and the data of D2 is shown to belong to the cluster with the cluster number 1.

  The visualization server 30 uses the score information held in the clustering result table 23 as the visualization of the global analysis, and displays the score calculated for any two-dimensional combination constituting the high-dimensional data. More specifically, the visualization server 30 displays character strings representing arbitrary dimensions constituting high-dimensional data on the vertical axis and the horizontal axis, respectively, and is calculated for an arbitrary two-dimensional combination. Make the score correspond to the diameter of the circle and visualize it in a tabular format. FIG. 5 shows an example of the global analysis screen that visualizes the global analysis.

  In the global analysis, the analyst can select two dimensions of interest (a circle at the intersection of two axes) based on the visualized score. When any circle is selected in the global analysis, the visualization server 30 shifts to a local analysis in two dimensions corresponding to the selected circle, and visualizes the result of the local analysis.

  The visualization server 30 plots all data on the two-dimensional plane selected in the global analysis as visualization of the local analysis. The visualization server 30 draws data by changing the shape or color so that the cluster to which each data belongs can be known. All data belongs to one of the clusters. FIG. 6 shows an example of the local analysis screen that visualizes the local analysis. In the example of FIG. 6, data is plotted on a plane in which the number of inquiries is taken as the horizontal axis dimension and the number of days after the visit is taken as the vertical axis dimension. Each data is drawn with a shape and color indicating the cluster to which the data belongs. In the upper left of the screen, a legend describing the shape and color indicating each cluster, the cluster name, the shape of the center point of the cluster, and the shape of the farthest point of each cluster are displayed.

  When the visualization server 30 displays the result of the local analysis, the analyst visually confirms the cluster to which each data belongs, and removes outliers and labels the data.

  FIG. 7 shows how outliers are removed. In the removal of outliers, the user selects an arbitrary point or point set indicating data on the two-dimensional plane, and performs a deletion operation on the selected point or point set. The history of this deletion operation is recorded by the history management server 40.

  FIG. 8 shows the state of labeling. When labeling, the analyst selects the cluster name displayed as a legend and changes the selected cluster name to the desired label name. By this operation, labeling is collectively performed on data belonging to the cluster to which the label is assigned. Any method may be used as long as all data belonging to the cluster can be labeled. The labeling history is recorded by the history management server 40.

  The history management server 40 manages the history of preprocessing operations for data in local analysis. The history management server 40 records the operation history in the preprocessing history table 22 of the DB server 20.

  The clustering server 50 clusters the latest data obtained by applying the operation history recorded in the preprocessing history table 22 to the converted data, and calculates a score for an arbitrary two-dimensional combination constituting the high-dimensional data. The score for the clustering result and an arbitrary two-dimensional combination is recorded in the clustering result table 23.

  For clustering, k-means ++ is used. Other algorithms may be used. The analyst analyzes the data in several groups based on the knowledge of the data. The number of groups at this time is designated as the number of clusters k. For the calculation of the score, the following index called Caliski-Harabaz index consisting of the ratio of inter-cluster variance and intra-cluster variance is used. Other indicators may be used.

Here, SS _W is the variance within the cluster (sum of squared distances), SS _B is the variance between clusters (the sum of squared distances from the center point of all samples minus SS _W ), k is the number of clusters, and N is The total number of samples.

  The terminal 60 is a device operated by an analyst. The terminal 60 displays a global analysis screen and a local analysis screen created by the visualization server 30, receives a two-dimensional combination for local analysis, and inputs an operation for data.

  Next, the operation of the data preprocessing system of this embodiment will be described.

  FIG. 9 is a float chart showing the flow of data conversion processing by the data preprocessing system of this embodiment.

  The conversion server 10 reads the actual data (step S11), performs conversion processing of the actual data (step S12), and transmits the converted data to the DB server 20 (step S13).

  Upon receiving the converted data, the DB server 20 initializes the converted data table 21 with the received converted data (step S14).

  The clustering server 50 performs a clustering process on the converted data and calculates a score for any two-dimensional combination constituting the high-dimensional data (step S15). The clustering result and the score are recorded in the clustering result table 23.

  FIG. 10 is a float chart showing the flow of global analysis processing by the data preprocessing system of this embodiment.

  The visualization server 30 acquires a score from the clustering result table 23 and executes a visualization process for global analysis (step S21).

  The visualization server 30 accepts selection of any two-dimensional combination that constitutes the high-dimensional data. When any two-dimensional combination is selected (YES in step S22), local analysis is performed (step S23).

  Upon receiving the data output instruction, the visualization server 30 completes the preprocessing (YES in step S24), and applies the operation history recorded in the preprocessing history table 22 to the post-conversion data held in the post-conversion data table 21. Then, data is created and output (step S25).

  When the preprocessing is not completed (NO in step S24), the global analysis (step S21) and the local analysis (step S23) are repeated. If the outlier has been removed in the local analysis in step S23, the visualization server 30 executes the visualization process of the global analysis using the latest score information held in the clustering result table 23.

  FIG. 11 is a float chart showing the flow of local analysis processing by the data preprocessing system of this embodiment.

  The visualization server 30 executes a local analysis visualization process for the two-dimensional combination selected in the global analysis (step S31).

  When the analyst selects an arbitrary point or point set and performs a delete operation (YES in step S32), the selected point or point set is deleted, and the history management server 40 manages the history of the delete operation ( Steps S33 and S34).

  The clustering server 50 performs a clustering process using the latest data from which the selected point or point set has been deleted (step S35).

  When the analyst selects a cluster from the legend and changes the name (YES in step S36), the selected cluster is labeled, and the history management server 40 manages the history of the labeling operation (step S37). , S38).

  The order of performing outlier removal and labeling is not important. Outliers may be removed multiple times, or multiple clusters may be labeled.

  Next, the operation of the entire data preprocessing system of this embodiment will be described.

  12A to 12C are sequence diagrams showing the flow of processing of the data preprocessing system of this embodiment.

  First, the flow of data conversion processing will be described with reference to FIG. 12A.

  The conversion server 10 performs conversion processing of actual data (step S101), and transmits the converted data to the DB server 20 (step S102).

  The DB server 20 initializes the converted data table 21 with the received converted data (step S103).

  The DB server 20 creates the latest data by applying the operation history held in the preprocessing history table 22 to the converted data held in the converted data table 21 (step S104), and transmits the latest data to the clustering server 50. (Step S105). Since there is no operation history at this stage, the converted data held in the converted data table 21 is transmitted as the latest data.

  The clustering server 50 performs clustering processing using the latest data (step S106), and transmits the clustering result to the DB server 20 (step S107).

  The DB server 20 updates the clustering result table 23 with the received clustering result (step S108).

  Next, the flow of global analysis will be described with reference to FIG. 12B.

  The DB server 20 creates the latest data by applying the operation history held in the preprocessing history table 22 to the converted data held in the converted data table 21 (step S201), and visualizes the latest data and the clustering result. (Step S202).

  The visualization server 30 creates a global analysis screen using the latest data and the clustering result (step S203), and displays the global analysis screen on the terminal 60 (step S204).

  The terminal 60 receives the analyst's selection (step S205), and transmits the selected two dimensions (axis 1 and axis 2) to the visualization server 30 (step S206).

  Next, the flow of local analysis will be described with reference to FIG. 12C.

  The DB server 20 creates the latest data by applying the operation history held in the preprocessing history table 22 to the converted data held in the converted data table 21 (step S301), and visualizes the latest data and the clustering result. (Step S302).

  The visualization server 30 creates a local analysis screen using the latest data and the clustering result (step S303), and displays the local analysis screen on the terminal 60 (step S304).

  Upon receiving an outlier selection and deletion operation from the analyst (step S305), the terminal 60 transmits outlier data including the outlier data ID to the history management server 40 (step S306).

  When receiving the outlier data, the history management server 40 creates history data for deleting the data indicated by the data ID included in the outlier data (step S307), and transmits the created history data to the DB server 20 (step S307). S308).

  The DB server 20 registers the received history data in the preprocessing history table 22 (step S309).

  Since the operation for deleting the data has been performed, the DB server 20 creates the latest data by applying the operation history held in the preprocessing history table 22 to the converted data held in the converted data table 21 (step S310). ), The latest data is transmitted to the clustering server 50 (step S311).

  The clustering server 50 performs clustering processing using the latest data (step S312), and transmits the clustering result to the DB server 20 (step S313).

  The DB server 20 updates the clustering result table 23 with the received clustering result (step S314).

  Further, when receiving a labeling operation from the analyst (step S315), the terminal 60 transmits label data including a cluster number to be labeled and a label name to be given to the history management server 40 (step S316).

  When receiving the label data, the history management server 40 creates history data for assigning a label name to the cluster number included in the label data (step S317), and transmits the created history data to the DB server 20 (step S318). .

  The DB server 20 registers the received history data in the preprocessing history table 22 (step S319).

  In this embodiment, the analyst selects a two-dimensional combination for local analysis from the global analysis screen and selects an outlier to be removed from the local analysis screen. However, these processes are automatically processed. It can also be automated with a processing unit.

  For example, in the global analysis, the processing unit selects the two-dimensional combination having the maximum score from all the two-dimensional combinations.

  For removal of outliers in local analysis, the processing unit selects and removes outliers for each cluster according to the following criteria.

An outlier threshold θ _th (1 or more) and a maximum repetition number N _r are set, and the removal number N _d = 0, and data points X _n = {x _n1 , x _n2 , belonging to cluster n (1 ≦ n <k) .., X _nN }, the following processing is performed.

L ² distance (Euclidean distance) from the center point c _n = (c _n1 , c _n2 ) of cluster n of x _nm for all x _nm = (x _nm1 , x _nm2 ) (1 ≦ m <N) Is calculated by the following equation.

The farthest point x _np (1 ≦ p <N) having the maximum value d is obtained from all d.

The degree of abnormality is determined by the local outlier factor method (lof method) for x _np . If the degree of abnormality exceeds _θth , it is regarded as an outlier. If N _d <N _r , x _np is removed so that N _d = N _d +1, and if N _d ≧ N _r , the outlier is removed. finish.

  For labeling in the local analysis, a label may be automatically given to each cluster.

  As described above, according to the present embodiment, the clustering server 50 classifies the high-dimensional data into clusters, calculates a score for any two-dimensional combination of the high-dimensional data, and the visualization server 30 The score for the two-dimensional combination is displayed in a tabular format, and selection is accepted. On the two-dimensional plane with each dimension of the selected two-dimensional combination as an axis, high-dimensional data is divided into clusters and plotted. By accepting the pre-processing device, the high-dimensional data is visualized in an intuitive format that can be easily understood by non-experts, so that even non-experts can execute data pre-processing.

  According to the present embodiment, when the high-dimensional data is unchanged, the history management server 40 manages the operation history of the preprocessing for the high-dimensional data, and refers to the high-dimensional data, after the conversion of the post-conversion data table 21 By using the latest data reflecting the operation history in the data, it is possible to restore the data at an arbitrary time point when an erroneous operation is performed.

DESCRIPTION OF SYMBOLS 10 ... Conversion server 20 ... DB server 21 ... Post-conversion data table 22 ... Pre-processing history table 23 ... Clustering result table 30 ... Visualization server 40 ... History management server 50 ... Clustering server 60 ... Terminal

Claims (5)

A clustering unit for classifying high-dimensional data into clusters and calculating a score for an arbitrary two-dimensional combination of the high-dimensional data;
A global analysis screen display unit that displays a score for a plurality of the two-dimensional combinations and displays a global analysis screen that accepts selection of the two-dimensional combinations;
When the two-dimensional combination is selected on the global analysis screen, the high-dimensional data is divided into clusters and drawn on a two-dimensional plane with each dimension of the selected two-dimensional combination as an axis. A local analysis screen display unit that displays a local analysis screen that accepts operations on dimensional data;
A history management unit for making the high-dimensional data unchanged and managing a history of operations on the high-dimensional data,
When an operation on the high-dimensional data is accepted on the local analysis screen, the history of the operation on the high-dimensional data is registered in the history management unit, and the clustering unit performs an operation indicated by the operation history on the high-dimensional data. Recalculate the score using the latest reflected data,
The data pre-processing apparatus according to claim 1, wherein when the local analysis screen is shifted to the global analysis screen, the global analysis screen display unit displays the recalculated score and accepts selection of the two-dimensional combination .   The data according to claim 1, wherein the global analysis screen display unit displays a character string representing each dimension of the high-dimensional data side by side along a vertical axis and a horizontal axis, and displays the score in a table format. Pre-processing device. The high-dimensional data satisfying a predetermined distance between the center point of each cluster on the two-dimensional plane with each dimension of the two-dimensional combination as an axis and the high-dimensional data belonging to the cluster is removed as an outlier. The data preprocessing apparatus according to claim 1, further comprising a processing unit. A data preprocessing method executed by a computer,
Classifying the high-dimensional data into clusters and calculating a score for any two-dimensional combination of the high-dimensional data;
Displaying a score for a plurality of the two-dimensional combinations and displaying a global analysis screen for accepting selection of the two-dimensional combinations;
When the two-dimensional combination is selected on the global analysis screen, the high-dimensional data is divided into clusters and drawn on a two-dimensional plane with each dimension of the selected two-dimensional combination as an axis. Displaying a local analysis screen for accepting operations on the dimensional data , and
Making the high-dimensional data unchanged,
When an operation for the high-dimensional data is accepted on the local analysis screen, the history of the operation for the high-dimensional data is registered in the history management unit, and the latest data reflecting the operation indicated by the operation history is added to the high-dimensional data. To recalculate the score,
A data preprocessing method , wherein when the transition is made from the local analysis screen to the global analysis screen, the recalculated score is displayed on the global analysis screen, and the selection of the two-dimensional combination is accepted . A data preprocessing program for operating a computer as each unit of the data preprocessing device according to any one of claims 1 to 3 .

Images