JP2021039523A - Data preparation support system for data utilization and its method - Google Patents

Abstract

To provide a data preparation support system for data utilization and its method in order to promote utilization of data collected from a plurality of business systems.SOLUTION: The system determines a recommended analysis step for an analysis purpose (222), extracts a combination of data available for the recommended analysis step from target data, evaluates each of a plurality of extracted data combinations, creates a list in which each of the plurality of data combinations is ranked based on a result of the evaluation (223), and allows a user to select a predetermined data combination.SELECTED DRAWING: Figure 2

Description

The present invention relates to a system for supporting the utilization of data, particularly a system for supporting the preparation of data for the utilization of data, for example, from among a large amount of various types of data possessed by a plurality of business systems. It relates to a system suitable for selecting and extracting data according to the purpose of data analysis.

In recent years, various business issues have been improved by utilizing a large amount of data accumulated in a company. For example, Japanese Patent Application Laid-Open No. 2004-29971 easily extracts only the data necessary for the desired data analysis from the data of the production apparatus and the processing time in the production process, and obtains an analysis result effective for improving the yield. Therefore, an additional character string that identifies the category of the data item is added to the explanatory variable of the data, data cleansing is performed to replace or delete the abnormal value of the data with a specific value, and the characteristic information based on the fluctuation of the objective variable of the data is performed. Therefore, a data analysis method has been proposed in which a data category is recognized during analysis processing and data analysis is efficiently and automatically executed by setting conditions and an analysis procedure corresponding to the category.

Then, Japanese Patent Application Laid-Open No. 2016-181150 integrates the data warehouse that stores all the input data and the input data in order to perform data processing that can respond to the required changes in the analysis. After generating the integrated data, the integrated layer that stores the integrated data and the integrated data are aggregated for each combination of one or more non-addition items, and at least the number of addition items or the number of non-addition items is aggregated. After generating the data, one aggregated data is selected from the plurality of aggregated data based on the aggregated layer that stores a plurality of aggregated data and the conditions necessary for generating the analysis data set in the setting unit, and one more. A data processing system having an analysis layer for storing the analysis data after extracting the analysis data from the aggregated data has been proposed.

JP 2016-181150

In recent years, in many fields such as transportation, electric power, and industry, it is required to utilize business data collected across departments and businesses in order to solve various diversified and diversified problems. .. On the other hand, in order to handle a large amount of data, which is different for each of multiple business systems and has been constructed and operated separately, the user needs to understand the data and have experience and knowledge about the business, but the level is high. It varies from person to person, and this hinders the progress of data-based analysis of issues. For example, a user is often familiar with the department to which he / she belongs and the data related to his / her business, but often does not know the data related to other departments or other business, and as a result, multiple departments or It becomes difficult to prepare for analysis, such as selecting the data necessary for the analysis desired by the user and processing the data across tasks.

Therefore, even if the user lacks understanding and knowledge about the differences between multiple businesses and the data and systems for each business, the data preparation is done so that the users can start analyzing the data smoothly. It is desirable to reduce the workload.

The invention disclosed in Patent Document 1 is based on the premise that the user understands the target data, and for that purpose, the analysis purpose, the analysis means, and the data to be prepared for the analysis. It is necessary for the user to organize the contents in advance, and the data can only be used for a specific type of data for the purpose assumed by the user. On the other hand, the data that can be utilized for the invention disclosed in Patent Document 2 is limited to the data that can be integrated data. It is not always possible to uniformly integrate a wide variety of data from multiple business systems. In addition, in order to create analysis data suitable for the purpose from integrated data and aggregated data, the user needs to understand all the original data.

An object of the present invention is to provide a system and a method for supporting the preparation of data for data utilization in order to promote the utilization of data collected from a plurality of business systems.

In order to solve the above object, the present invention is a system that supports data preparation for data utilization, and includes a processing device and a storage device, and the processing device records in the storage device. By executing the executed program, business data is collected from each of a plurality of business systems, the business data is stored in the storage device at least temporarily as target data, and the purpose of analysis of the target data is performed from the user terminal. Receive, determine the recommended analysis steps for the analysis purpose based on the analysis purpose and the target data, and extract a combination of data available for the recommended analysis step from the target data. Then, a plurality of each of the extracted data combinations is evaluated, a list in which the plurality of the data combinations are ranked based on the evaluation result is created, and the list is output to the user terminal. , Allowed the user to select a predetermined data combination from the list.

According to the present invention, the load on the user in the data preparation work when advancing the utilization of these data for analysis such as problem solving based on a wide variety of a plurality of business systems and a large amount of data. Therefore, the utilization of data can be promoted quickly and with high quality.

It is a block diagram which shows an example of the system configuration which supports the data preparation for the utilization of data. This is an example of a sequence for realizing an operation in which a server supports data preparation for data utilization. This is an example of a server module configuration. This is an example of the configuration of analysis purpose information created by the user and reference information held by the server. This is an example of a data preparation table created by the server when the user specifies to grasp the whole picture as an analysis step. This is an example of a data preparation table created by the server when the user specifies "specific event extraction" as an analysis step. This is an example of a data preparation table created by the server when the user specifies "factor analysis / prediction" as the analysis step. An example of a management table that manages data combinations presented or proposed by the server to the user is shown. It is a flowchart which concerns on an example of the operation for the server to propose an analysis step to a user. This is the first example of a matrix for determining the suitability of an analysis step registered by a user. This is a second example of a matrix for determining the suitability of a user-registered analysis step. This is a third example of a matrix for determining the suitability of a user-registered analysis step. It is a flowchart which shows an example of the operation for presenting a combination of data which a server can use for grasping the whole picture as an analysis step to a user. It is a flowchart which shows an example of the operation for presenting the combination of data which a server can use for the peculiar event extraction as an analysis step to a user. It is a flowchart which shows an example of the operation for presenting a combination of data which a server can use for factor analysis / prediction as an analysis step to a user. This is an example of a graphical user interface provided by the server to the user terminal to support data utilization. It is another example of a graphical user interface provided by a server to a user terminal to support data utilization. It is yet another example of a graphical user interface provided by a server to a user terminal to support data utilization. In the flowchart of FIG. 8, it is a table for explaining the combination of a plurality of columns. It is a table for demonstrating the combination of a plurality of columns in the flowchart of FIG.

Next, an embodiment of the present invention will be described. FIG. 1 is a block diagram of an example of a computer system for realizing the present invention. This computer system includes a server 101 (computer) that supports data preparation for utilizing data collected from a plurality of business systems. The server 101 collates the purpose of data analysis input by the user who uses the system with the data of a plurality of business systems collected by the server 101, proposes the optimum analysis step to the user, and proposes the optimum analysis step. At the same time, we support data preparation for data utilization by proposing highly useful data that can be used for this analysis step.

The user can collect the data by simply registering the analysis purpose in the system, and by performing the analysis such as for solving the problem based on the data proposed or provided by the system. , Data analysis intended by the user can be performed without being forced to perform data preparation work such as extracting useful data from a large number of data.

A plurality of business systems 104, 105, 106 and a plurality of user terminals 102, 103 are connected to the server 101. The user terminal registers the analysis purpose desired by the user in the server 101, then receives an analysis step for realizing the analysis purpose and a proposal of data available for this analysis step from the server 101, and then receives a proposal from the server 101. After confirming these, the data analysis can be performed, or the server 101 can be made to perform the data analysis.

Each of the plurality of business systems provides the server 101 with data referred to by the server 101 (hereinafter, referred to as target data) as a target of data analysis desired by the user. It should be noted that the data is provided to the server 101 from each of the plurality of business systems not only when the data is provided by communication via the network 108, but also when the data is stored in the server 101 manually, for example, without going through the network 108. You may do it. As a plurality of business systems, for example, in the railway field, a train operation management system, a station management system, and a maintenance management system can be exemplified.

The data sent from the business system to the server 101 does not have to be particularly limited, and for example, the data table, other sensor information as binary data, the type of data, the data method, and the like are not limited. The user registers the purpose of data analysis in the server 101 in order to comprehensively search for the cause of a predetermined problem or problem across a plurality of business systems. For this purpose, taking the transportation industry as an example, there is, for example, "train delay", that is, the cause of the delay can be identified across multiple business systems, and thus the prevention and control of the delay can be achieved. I may try to help.

The server 101 and the plurality of user terminals 102, 103, ... Are connected to each other via the network 107. The server 101 and the plurality of business systems 104, 105, 106, ... Are interconnected via the network 108.

The main hardware configurations of the server 101 as a base for data utilization and data preparation are a storage device (main memory, hard disk, storage area of an external storage system) 111, a processing device (CPU) 112, and Communication device 113. The same applies to the user terminals 102 and 103. The processing device 112 records the target data at least temporarily on the non-temporary recording medium (storage device) 111.

FIG. 2 is an example of a sequence for realizing an operation in which the server 101 supports data preparation for data utilization. In addition, the user 201 is a person who is trying to find a problem, formulate a solution, etc. based on a predetermined analysis step for various data collected across a plurality of departments, or for that purpose. Application may be.

Each of the plurality of business systems 104-106 registers business data in the server 101 (211). The server 101 accesses the storage area of the business data, refers to the business data, and sets the data catalog and the data definition (221). Setting the data catalog or the like by the server 101 includes creating or updating the data catalog or the like. As will be described later, the server 101 creates an analysis mode (analysis step, candidate data list) based on the analysis purpose input by the user and the reference information (data catalog, data definition) created by the server 101. To the user.

For the data catalog and the reference information such as the data definition, the server 101 proposes the optimum analysis step to the user 201 based on the analysis purpose intended by the user 201 and the business data, and this analysis step. In order to propose highly useful data that can be used in the above, it may be understood as an example of a standard or an index for analyzing business data such as sorting and evaluating business data. Details of the data catalog and data definition will be described later.

The user 201 registers the desired purpose of data analysis in the server 101 via the user terminal 102 (231). The server 101 evaluates the analysis purpose registered by the user based on the business system data and the reference information (data catalog, data definition), determines, discriminates, determines, and determines the analysis step recommended to the user. It is selected and confirmed, and this is notified to the user terminal 102 (222).

The user 201 determines the analysis step by confirming the recommended analysis step presented by the server 101 or selecting a predetermined analysis step from the plurality of analysis steps, and confirms the analysis step, and confirms the analysis step. (232).

The server 101 extracts a plurality of candidate data suitable for the analysis step from the data collected from the plurality of business systems, and presents the candidate data to the user 201 (223). The user 201 refers to a list of a plurality of candidate data transmitted from the server 101, determines the data to be analyzed, sets it by selection or the like, and based on the determined analysis step (232). Analyze the data to be analyzed. In addition, this is notified to the server 101 (233). The server 101 updates the reference information (data catalog / data definition) based on the results of step 222, step 223, and step 233 (224).

The reference information such as the data catalog and the data definition may be created or updated by the administrator or the operator of the server 101. The server 101 may create or update a data catalog or the like by performing machine learning based on the user's analysis purpose and business data.

The purpose of the analysis (FIG. 4: 401 below) may be information that simply indicates the target of the data analysis, for example, "train delay". The user 201 can describe the analysis purpose by, for example, "analysis step", "KPI", and "data item of interest" and register it in the server. The "analysis step" and "KPI" may be essential information, and the "data item of interest" may be optional information. The KPI is an element that the user pays attention to in the analysis of data, for example, the above-mentioned "train delay". The KPI may be defined by a calculation formula in addition to the keyword.

The data item of interest is, for example, an element that should be focused as an axis for evaluating the KPI, or a name that should be monitored at the same time as the KPI. For example, when the KPI is "train delay", the data item of interest is at least one of "departure time", "arrival time", "operation date and time", and "delay time".

The analysis step represents the implementation stage in advancing the analysis for solving business problems, etc., and there are multiple types, mainly the mode for grasping the overall picture of the target data and the target data. An embodiment for extracting a peculiar event from the data, and an embodiment for analyzing a factor from the target data and predicting the result are exemplified. These are referred to as (1) grasping the whole picture, (2) extracting specific events, and (3) factor analysis / prediction.

User 201 can include a plurality of analysis steps for analysis purposes. There may be a distinction between superiority and inferiority in the plurality of steps such as priority and order.

Regarding each aspect of the analysis step, as a calculation method that can be used by the user 201, for example, trend analysis for grasping the whole picture, outlier detection for singular event extraction, and / or change point extraction, factor analysis / prediction. Has regression analysis / multiple regression analysis and / or classification / cluster analysis.

The server 101 has a data catalog and a data definition (221) in the storage device 111 as reference information (may be referred to as “definition information”). The data catalog (FIG. 4: 402) includes a data catalog of target data collected by the server 101 from a plurality of business systems 104 and a data catalog of processed data obtained by processing the target data. The processing data data catalog may include information on a calculation formula for obtaining processing data from the target data.

As a catalog related to the target data, for example, there is a catalog consisting of items such as "column name", "name / meaning", and "supplementary information". As an example, there are {arrival time, arrival time, ...}, {KRT, about km, ...}. The column name is a name recorded in the column of the target data (data table).

Catalogs related to processing data include items such as "data name", "calculation formula", "name / meaning", and "supplementary information". As an example, {delay, "actual time"-"planned time" , Delay time, ...}.

The data definition 221 described above includes, for example, a 4W dictionary (FIG. 4: 403) and a data transition pattern (FIG. 4: 404). The 4W dictionary is for grasping the tendency of the whole image of the target data from the viewpoint of 4W (When, Where, What, Who), and includes keywords for each 4W category. For example, 4W dictionary data consists of a combination of "keyword" and "category". As an example, {keyword, category} = {time, when}, {date, when}, {kilometer, where}, {Kilometrage, where}, {station code, where}.

The data transition pattern includes the transition of values in addition to the 4W dictionary. For example, "data name / type", "category", "data type", "value change range", "lower limit", and " It consists of "upper limit". Examples include {kilometers, where, integer, 1,0,300} and {arrival time (minutes), when, integer, 5,0,1440}.

FIG. 3 is an example of the module configuration of the server 101. The server 101 is equipped with the middleware 301 for realizing various functions described in FIG. 2 described above. The middleware 301 includes a plurality of functional modules realized by a processing device 112 that executes a program stored in the storage device 111 of the server 101. Then, the middleware 301 is provided in the storage device 111 with the target data 311 of the business system 104 and the processed data 312 processed by being extracted as a feature amount from the target data 311.

The middleware 301 includes a data management module 321 that manages the target data 311 and the machining data 312, and stores the data catalog 402, the data definition 315, and the matrix 313 for determining the suitability of the analysis purpose registered by the user 201. It is prepared for 111.

The middleware 301 has a reference information management module 322 that manages the above-mentioned reference information (data catalog, data definition), a user management module 323 that manages a user who accesses the server 101 and performs work related to data preparation, and definition information. It is provided with an analysis step proposal module 324 that references and / or proposes an analysis step and / or a work item to the user based on the user's analysis objective information 401.

The middleware 301 refers to the data definition 315, and has an analysis data proposal module 325 that proposes analysis data based on the user's analysis purpose information 401, and a proposal that updates and manages the proposal information for the user as a data preparation table 314. The execution management module 326, the interface module 327 that provides the user terminals 102 and 103 with an interface for accessing the functions of the middleware 301, the user terminals 102 and 103, and the business system 104 via the networks 107 and 108. Analysis with a data communication module 328 that communicates with ..., and an overall image grasp module 331 that executes processing such as trend analysis for proposing data for analysis when the analysis step is an overall image grasp. The analysis step includes, for example, a singular event extraction module 332 that executes a process of extracting singular points (for example, an outlier, a large change point, and a large bias) for data proposal for analysis when the step is a singular event. It is provided with a factor analysis / prediction module 333 for executing processing such as regression analysis and multiple regression analysis for data proposal for analysis in the case of factor analysis / prediction.

The reference information management module 322 described above manages reference information (data catalog 402, data definition 315, matrix 313). For example, this information may exist for each industrial field (railway, electric power, etc.), and the standard information management module 322 manages the standard information for each industrial field, and depending on the business data or the purpose of analysis, etc. Reference information in a predetermined field may be selected.

The module described above is a function realized by the processing device executing a program, and may be paraphrased as, for example, a part, a means, an element, a circuit, or a unit.

Next, a candidate list of analysis data created by the server 101 and presented to the user will be described. 501 of FIG. 5A is an example of the data preparation table 314 (candidate list) created by the server 101 when the user specifies "grasp the whole picture" as the analysis step. The data proposal module 325 for analysis creates or updates this table. This table contains rank 511, identification information 512, KPI 513, focus data 1 (When) 514, focus data 2 (Where) 515, focus data 3 (What) 516, focus data 4 (Who) 517, number of records 518, and , The output file 519 is provided.

The rank 511 stores information regarding the rank assigned based on the priority for the data combination (row) specified by the identification information 512. The data combination is the target data 311 including the record of the column included in the table of the target data 311 as the “KPI data” column name and the data of interest 1 (for example, the above-mentioned “time”) corresponding to “when”. The column name included in the table of 1), the similar record related to the attention data 2 corresponding to “where” (for example, the above-mentioned “kilo”), and the attention data 3 corresponding to “what”. It refers to a combination of a similar record related to (for example, the above-mentioned “column number”) and a similar record related to the data of interest 4 (for example, “driver ID”) corresponding to “who”.

If each of the data of interest 1-4 (514-517) has the same or similar data as the "data item of interest" of the user input constituting the analysis purpose information 401, this is described. If there is no corresponding data 1-4 (514-517), that is, there is no registration in the 4W dictionary and the data item of interest is not specified, a blank is recorded.

The KPI 513 stores the same or similar as the user-input "KPI" that constitutes the analysis purpose information 401.

The number of records 518 stores the number of records in the valid state in the data combination specified by the identification information 512. The output file 519 stores the path of the file that outputs the record of the data combination specified by the identification information 512.

FIG. 502 of FIG. 5B is an example of the data preparation table 314 created by the server 101 when the user specifies “specific event extraction” as the analysis step. In Table 502, which is also a singular event list, the description of the same code as in Table 501 is the same. Table 502 includes 520 singular points (outliers), 521 singular points (change points), and 522 singular points (bias).

The number of singular points (outliers) 520 stores information on the number of data combinations specified by the identification information 512 in which the type of singular points extracted from the KPI is "outliers".

The number of singular points (change points) 521 stores information on the number of data combinations specified by the identification information 512 in which the type of singular points extracted from the KPI is a "major change point". The singularity (bias) number 522 stores information on the number of singular points whose type is "large bias" extracted from the KPI in the data combination specified by the identification information 512.

503 of FIG. 5C is an example of the data preparation table 314 created by the server 101 when the user specifies "factor analysis / prediction" as the analysis step. In Table 503, which is also a combination list of objective variables and explanatory variables, the description of the same code as that of Table 501 is the same.

The objective variable / explanatory variable combination list 503 contains objective variable 530, explanatory variable 531, number of records 518, determination value 1 (533), determination value 2 (534), determination value 3 (535), and determination value 4 (536). Be prepared. The KPI included in the analysis purpose is stored in the objective variable 530. In the explanatory variable data 531, the name of the column selected by the user other than the column containing the KPI name is recorded.

The determination value 1 (533) stores the first determination value data used for calculating the rank 511 of the data combination specified by the identification information 512, and the determination value 2 (534) is 2 The third judgment value data is stored, the judgment value 3 (535) stores the information about the third judgment value data, and the judgment value 4 (536) stores the information about the fourth judgment value data. Is stored. These determination values will be described in detail with reference to FIG.

FIG. 5D is an example of a management table 504 that manages a combination of data presented or proposed to the user by the server 101. This table is created or updated by the data management module 321. The management table 504 stores information about the analysis purpose information 401 from the user, the reference information 221 and the data proposed to the user based on the target data 311 by the server 101.

The proposal execution management module 326 receives the analysis purpose information 401 from the user, recommends the analysis step, and creates or updates the table 504 when proposing the data to be used for the analysis. This table 504 includes identification information 541, KPI (513), data combination 543, number of uses 544, number of users 545, update date and time 546.

The identification information 541 stores information for identifying a data proposal to the user. The KPI (513) stores the KPI data in the data proposal specified by the identification information 541. In the data combination 543, the combination of data proposed to the user specified by the identification information 541 is stored when the analysis step is grasping the whole picture or extracting a peculiar event, and the analysis step is factor analysis / prediction. If so, the combination of explanatory variables is stored.

The usage count 544 stores information regarding the usage count of the data combination in the data proposal specified by the identification information 541, and is updated each time the user uses the data combination.

The number of users 545 stores information regarding the number of users of the data combination in the data proposal specified by the identification information 541. The number of users may be the total number of users or the number of different users. This information is updated each time the data proposal specified by the identification information 541 is used by the user. The update date and time 546 stores the date and time when the record in the candidate list 501 was updated.

FIG. 6 is a flowchart relating to an example of the operation for the server 101 to recommend the analysis step to the user. The server 101 determines whether or not the user's analysis purpose is appropriate based on the user's input information 231 and the reference information 221 and the target data 311 and outputs the determination result as an analysis step recommended to the user.

The server 101 starts the flowchart when the user has registered 231 of the analysis purpose information 401. In step 601 the analysis step proposal module 324 compares the names of the KPI and the data item of interest included in the analysis purpose information 401 from the user with the column names of the table of the target data 311 and compares the results of the comparison with the column names of the table of the target data 311. Calculated as the degree of agreement. It goes without saying that the same name is "match", but similar names may be included in "match". “Similar” means, for example, a synonym. The “matching degree” may be the cumulative total of matching, the percentage of matching, comparison with the threshold value, or the like. When there are a plurality of target data, the module 324 may calculate the degree of matching for each target data, and for example, accumulate the degree of matching for each target data to obtain the degree of matching for the plurality of target data.

Next, in step 602, the analysis step proposal module 324 compares the names of the KPIs and the data items of interest with the data item names of the data catalog 402, and calculates the degree of agreement described above.

Next, in step 603, the analysis step proposal module 324 compares the names of the KPIs and the data items of interest with the data item names in the data definition 315 (4W dictionary, data transition pattern dictionary), and determines the degree of agreement. Ask.

In step 601 the high degree of agreement indicates that the amount of target data is not insufficient and / or the quality of the target data is high for the purpose of analysis of the user, and the step In 602 and 603, a high degree of agreement indicates that the information for sorting, classifying, discriminating, or evaluating the target data is suitable for the analysis purpose of the user. The degree of agreement may be rephrased as, for example, goodness of fit, affinity, applicability, and the like.

In step 604, the analysis step proposal module 324 determines whether or not the analysis step in the analysis purpose information 401 from the user is “factor analysis / prediction”. When the analysis step proposal module 324 determines that step 604 is negative, in step 605, the degree of fulfillment of the reference information (data catalog 402, data definition 315) is calculated. The degree of fulfillment is an index showing the effectiveness, usefulness, or reliability of the reference information for the purpose of analysis, for example, an index showing the degree of abundance of information, which is, for example, data catalog 402, data definition. It may be determined based on the number of information registrations in 315 and the access frequency such as the number of references. The degree of fulfillment may be rephrased as the degree of effectiveness.

When the analysis step proposal module 324 determines affirmatively in step 604, the process proceeds to step 606 and calculates the degree of fulfillment of the machining data 312 as an explanatory variable candidate in "factor analysis / prediction" as the analysis step (step 606). ). This degree of fulfillment may be calculated from the number of column names in the processing data and the number of valid records.

In step 607, the analysis step proposal module 324 refers to the matrix 313 for determining the suitability of the analysis step 231 registered by the user, which is shown in FIG. 7A or the like, based on the results of steps 605 to 606, to the user. The analysis steps and / or work items that should be recommended are determined by making judgments, discriminations, and the like.

The analysis step proposal module 324 presents the determined content of step 607 to the user in step 608, and requests confirmation, selection, and the like of the user. As a result, the server 101 ends the flowchart. The analysis steps and work items may be collectively referred to as, for example, an analysis mode or an analysis method.

The analysis step proposal module 324 is a matrix 701 when the user selects "overview grasp" as the analysis purpose information (analysis step) as a matrix which is a standard for determining the analysis step to be proposed to the user. (Fig. 7A) is adopted, and when "specific event extraction" is selected, matrix 702 (FIG. 7B) is adopted, and when "factor analysis / prediction" is selected, matrix 703 (FIG. 7C) is adopted. adopt.

The determination matrix 701 (FIG. 7A) defines the correlation between the degree of agreement with the analysis purpose information in steps 601-603 and the degree of completeness of the data catalog and data definition in step 605. , The degree of agreement, and the degree of fulfillment may be compared with each predetermined threshold value to determine the level.

The “degree of agreement with the analysis purpose information” of the determination matrix 701 may be, for example, an addition or an average of the degree of agreement in each of steps 601-603. Of the matching degrees of each of the steps 601-603, the matching degree of a predetermined step may be prioritized. The “data catalog / data definition fulfillment” of the determination matrix 701 may be each fulfillment, for example, an addition, an average, or the like. One of the fulfillment levels may be prioritized.

If the degree of agreement is "high" and the degree of fulfillment is "high", and if the degree of agreement is "high" and the degree of fulfillment is "low", there is a sufficient amount of target data suitable for the user's analysis purpose. Therefore, the analysis step proposal module 324 recommends the "overall picture grasp" as the analysis step registered in the server 101 by the user as it is and presents it to the user.

On the other hand, if the degree of agreement is "low" and the degree of fulfillment is "high", the amount of target data may be insufficient, and it is not preferable to immediately carry out "overall picture grasp". The analysis step proposal module 324 first recommends "additional data" as a work item necessary to realize the "overall picture grasp" while maintaining the "big picture grasp".

Further, when the degree of agreement is "low" and the degree of fulfillment is "low", the quality of the data catalog 402 and the data definition 315 is not sufficient in the first place even if there is data suitable for the user's analysis purpose. Since there is a possibility, the analysis step proposal module 324 determines that it is not preferable to carry out "overall picture grasp" as an analysis step, and promotes data understanding by statistics, expands data catalog 402, and expands data definition 315 as work items. Is recommended.

The meanings of "concordance with analysis purpose information", "high", and "low" in the determination matrix 702 (FIG. 7B) are the same as those in the determination matrix 701. “Fulfillment of data matching the analysis purpose information” is determined in steps 601-603 that the degree of matching with the analysis purpose information 401 from the user is high, the data in the target data 311 or the data catalog 402, The degree of data enrichment in the target data 311 (or processing data 312) corresponding to the data item of the data definition 315 is shown.

This degree of fulfillment may be calculated from the large number of valid records of these data (tables). When the degree of agreement is "high" and the degree of fulfillment is "high", the analysis step proposal module 324 is suitable for the analysis purpose of the user, and the target data amount may be sufficient. "Extraction of peculiar events" is recommended as it is.

On the other hand, when the degree of agreement is "high" and the degree of fulfillment is "low", the number of data records as a peculiar event is relatively insufficient even if there may be sufficient data suitable for the user's analysis purpose. Since there is a possibility, the analysis step proposal module 324 determines that it is difficult to carry out "specific event extraction" as an analysis step, and recommends that the target data be added as a work item.

Further, when the degree of agreement is "low" and the degree of data enrichment is "high", or when the degree of agreement is "low" and the degree of enrichment is "low", the analysis step proposal module 324 is desired by the user. It is difficult to perform "specific event extraction" as an analysis step, judging that the target data is insufficient or the user may not fully understand the data, which is suitable for the analysis purpose. Therefore, it is recommended that the user first change to "Overview grasp", return to "Overview grasp", and start data analysis from now on. In this case, the analysis step proposal module 324 determines the suitability as "overall image grasp" based on the determination matrix 701 (FIG. 7A).

Regarding the "degree of agreement with the analysis purpose information" of the determination matrix 703 (FIG. 7C), the meanings of "high" and "low" are the same as those of the determination matrix 701. The "enrichment of processing data" in the determination matrix 703 is for the processing data, and the meanings of the "enrichment" and the "high" and "low" themselves are as described above.

When the degree of agreement is “high” and the degree of fulfillment is “high”, the amount of target data suitable for the user's analysis purpose is sufficient. Therefore, the analysis step proposal module 324 uses “factor analysis” as the analysis step specified by the user. / Prediction is recommended as it is.

When the degree of agreement is “high” and the degree of fulfillment is “low”, the analysis step proposal module 324 is suitable for the user's analysis purpose, and even if there may be sufficient target data, factor analysis / prediction can be performed. Judges that it is difficult to carry out "factor analysis / prediction" as an analysis step because the number of processing data records, which is an explanatory variable, may be insufficient, and recommends the user to expand the processing data as a work item. ..

If the degree of agreement is "low" and the degree of fulfillment is "high", the analysis step proposal module 324 may lack the target data itself, which is suitable for the user's analysis purpose. It is recommended to go back to "image grasping" and consider it.

If the degree of agreement is "low" and the degree of fulfillment is "low", the analysis step proposal module 324 is suitable for the user's analysis purpose, the target data itself may be insufficient, or the data understanding is sufficient. Because it may not be, and because the number of records of the data that is the explanatory variable may be insufficient to perform factor analysis / prediction, the user is asked to return to the “big picture”, and , It is recommended to expand the machining data as a work item.

FIG. 8 is a flowchart showing an example of an operation for the server 101 to present to the user a combination of data that can be used for “grasping the whole picture” as an analysis step. When the server 101 determines "overall image grasp" as an analysis step based on the determination matrix 701, the server 101 starts the flowchart of FIG.

In step 801 the analysis data proposal module 325 extracts the KPI and the data item of interest from the analysis purpose information 401 (231) received from the user 201. Next, in step 802, the analysis data proposal module 325 extracts a column corresponding to the KPI, that is, a column having the same or similar name as the KPI name from the table of the target data 311. In FIG. 12, the target data 311 includes two data (tables) 311A and 311B, and the column B of the table 311A is a column having the same name as the KPI.

In step 803, the analysis data proposal module 325 extracts a column corresponding to the data item of interest from the target data 311. In step 804, the analysis data proposal module 325 corresponds to each of the 4W (When, Where, What, Who) keywords of the 4W dictionary 221 from the target data 311 (same or similar to the keyword), and the target data. Extract 311 columns. The analysis data proposal module 325 excludes the columns extracted in step 802 and / or step 803 when they are extracted again in step 804.

In step 805, the data proposal module 325 for analysis refers to the record of each column in the target data 311 and corresponds to the transition pattern of the data value of each of the 4W keywords of the 4W dictionary 221 in the data transition pattern definition 404. The column is extracted from the target data 311. The analysis data proposal module 325 excludes the columns extracted in step 802 and / or step 803 when they are extracted again in step 805.

In step 806, the analysis data proposal module 325 sets the priority of the columns extracted in step 803, step 804, or step 805. The analysis data proposal module 325 sets the priority of the column extracted in the above three steps higher than that of the column extracted in only two steps or one step, for example.

In step 807, the data proposal module 325 for analysis presents a list of columns (4W candidate columns) of steps 804-805 to the user so that the priority related to step 806 can be known, and in step 808, the user presents the list. If requested, 4W candidate columns can be narrowed down and some columns can be excluded from the data analysis target.

In step 809, the analysis data proposal module 325 includes a column corresponding to KPI (KPI column) extracted in step 802, a column corresponding to the data item of interest extracted in step 803 (data item column of interest), and step 804. Create a combination consisting of a column (4W candidate column) according to −808. Hereinafter, this combination will be referred to as a "synthetic column combination".

The column corresponding to the data item of interest extracted in step 803 and the 4W candidate column according to step 804-808 are collectively referred to as an "axis candidate column" which is an axis candidate for the KPI column.

In FIG. 12, column C of table 311A is axis candidate column 1, column D of table 311A is axis candidate column 2, column G of table 311B is axis candidate column 2, and column H of table 311A is axis. Candidate column 3 (the names of column D and column G are the same or similar). Table 311C is a composite column combination.

The data proposal module 325 for analysis is a duplicate column when KPI columns with the same or similar names are duplicated in a plurality of data tables, and when axis candidate columns with the same or similar names are duplicated in a plurality of data tables. Create multiple composite column combinations by exchanging. Explaining according to the example of FIG. 12, in the analysis data proposal module 325, the table 311C as a composite column combination, the axis candidate column 2 is the column C of the table 311A, and the axis candidate column 2 is the table 311B. Create two of the columns G. That is, the analysis data proposal module 325 defines two combinations, a synthetic column combination 1 (columns B, C, D, H) and a synthetic column combination 2 (columns B, C, G, H). The number of KPI columns with the same name is "k", the number of axis candidates 1 with the same name is "m1", the number of axis candidates 2 with the same name is "m2", and the number of axis candidates n with the same name is "m2". If "mn" is set, the number of synthetic column combinations is "k * m1 * m2 * ... * mn". Alternatively, a composite column combination is created by combining tables 311A and 311B using axis candidate columns having the same or similar names as keys.

In step 810, the data proposal module 325 for analysis excludes the column combination in which the number of valid records is zero from the plurality of composite column combinations. A column combination in which the number of valid records is zero means that no data is recorded in all the records of the column combination.

Next, in step 811 the analytical data proposal module 325 determines the characteristics and attributes of the records of each combination, for example, the variation pattern of a plurality of data values contained in the records, for each of the synthetic column combinations. The analysis data proposal module 325 reflects this determination result in the creation of analysis data candidates.

The fluctuation pattern of the data value is, for example, a "continuous value", a "continuous repetition value", or a "discrete value". "Continuous value" is numerical data, and is a pattern in which the values are almost different in all records. The "continuous repetition value" is numerical data, and is a pattern in which the data value is repeatedly increased or decreased within a certain range for all records. The "discrete value" is a pattern in which the number of unique values is a certain percentage or less of the number of records in character string data or numerical data.

In step 812, the data proposal module 325 for analysis discriminates the data value fluctuation pattern for each composite column combination, and when it is determined that the pattern is a “continuous value”, the process proceeds to step 813, and each of the composite column combinations For, all synthetic column combinations are combined to calculate the total number of records.

When the data value proposal module 325 for analysis determines the data value fluctuation pattern in step 812 and determines that the pattern is a “continuous repetition value”, the process proceeds to step 814 to obtain data for each of the composite column combinations. The records are classified according to the repetition range, and further, the process proceeds to step 815, and the combined column combinations are combined for each repetition range to calculate the number of records.

When the data value proposal module 325 for analysis determines the data value fluctuation pattern in step 812 and determines that the pattern is a “discrete value”, the process proceeds to step 816, and the axis candidate column and 4W selected in step 811 One or more other axis candidate columns determined to be "discrete values" in the same classification are selected, and in step 817, the KPI column and the plurality of axis candidate columns selected in step 816 are combined to form a character. Calculate the number of records in each classification of column data and numerical data.

In step 818, the data proposal module 325 for analysis aggregates the number of records for each data value fluctuation pattern based on the result of steps 811-817, and in step 819, is the processing completed for all the composite column combinations? If it is determined whether or not it is, and if it is denied, it returns to step 811. If it is determined affirmatively, it proceeds to step 820.

In step 820, the data proposal module 325 for analysis evaluates the usefulness of each combination for each data value fluctuation pattern, for example, using the number of records (data amount) in the valid state as a key, and evaluates all combinations. A candidate list (501 in FIG. 5A) is created by sorting in descending order of values and presented to the user.

The data proposal module 325 for analysis stores the rank determined in step 820 in the rank 511 of the candidate list 501, and puts the KPI and the data item of interest in KPI 513 and the data of interest 1-4 (514-517). The number of valid records is stored in the number of records 518, and the file output destination of the data combination is stored in the output file 519.

In step 821, the analysis data proposal module 325 performs trend analysis for each column combination included in the candidate list 501, creates a graph of the results, and presents the results to the user. The user can specify and select a predetermined combination by referring to the graph. The user may select a desired combination from the candidate list. In this case, the graph may be created only for the selected combination.

In step 822, the analysis data proposal module 325 outputs the record of the column related to the combination specified by the user as a CSV file or the like. The CSV file can be used when the user performs a trend analysis of the target data again.

In step 823, the data proposal module 325 for analysis updates or adds the data catalog 402 and the data definition 221 based on the combination of the candidate list 501 created in step 820 and the combination specified in step 822. Therefore, as the data analysis progresses, the data catalog and data definition will be improved.

FIG. 9 is a flowchart showing an example of an operation for the server 101 to present to the user a combination of data that can be used for “specific event extraction” as an analysis step. When the server 101 determines "specific event extraction" as an analysis step based on the determination matrix 702, the server 101 starts the flowchart of FIG.

In the flowchart of FIG. 9, the steps of steps 801-819 of the flowchart of FIG. 8 are the same. In step 903, the analytical data proposal module 325 ranks each of all column combinations and extracts combinations whose rank is equal to or higher than the threshold value. The ranking may be based on the number of valid records in each of the plurality of combinations.

In step 904, the data proposal module 325 for analysis determines the presence or absence of a singular point in the data of one combination of the plurality of combinations extracted in step 903, and if the determination of the singular point is affirmed in step 905, the determination of the singular point is affirmed in step 906. , Calculate the number of singular points. The singularity may be, for example, an outlier, a large change point, a large bias, or the like. The analytical data proposal module 325 adds information about the singularity to the singular event list 502.

If the analysis data proposal module 325 denies step 905, it proceeds to step 907 without going through step 906, and determines whether or not steps 904, 905, and 906 have been applied to all the combinations described above. If a negative determination is made, the process returns to step 904, and if an affirmative determination is made, the process proceeds to step 908.

In step 908, the data proposal module 325 for analysis ranks the usefulness of each combination by, for example, sorting them in ascending order based on the number of singular points, and creates a singular event list 502. , Present this to the user. The number of singular points may be, for example, the sum of the number of outliers, the number of change points, and the number of biases, or an average.

The analysis data proposal module 325 stores the rank for each combination in the rank 511 of the singular event list 502, stores the KPI in the KPI 513, stores the data items of interest in the data of interest 1-4 (514-517), and stores the ranks of interest. The number of valid records for each combination is stored in the number of records 518, the number of singular points (outliers, large change points, large bias) for each combination is stored in the number of singular points 520-522, and the output destination of the data file for each combination is stored. Is stored in the output file 519.

In step 909, the analysis data proposal module 325 creates a peculiar event calculation graph for each combination and presents it to the user. The rest is the same as steps 822 and 823 of the flowchart of FIG.

FIG. 10 is a flowchart showing an example of an operation for the server 101 to present to the user a combination of data that can be used for factor analysis / prediction as an analysis step as a list as in the flowcharts of FIGS. 8 and 9. is there. When the server 101 determines "factor analysis / prediction" as an analysis step based on the determination matrix 703, the server 101 starts the flowchart of FIG.

When the analysis data proposal module 325 starts the flowchart of FIG. 10, the analysis data proposal module 325 extracts KPIs (KPI names, KPI calculation formulas) based on the analysis purpose information 401 received from the user in step 1001.

In step 1002, the analysis data proposal module 325 extracts the column names of the target data 311 and / or the processing data 312 corresponding to the KPI name, and uses this as the objective variable column. The data proposal module 325 for analysis may create an objective variable column based on the KPI calculation formula. In FIG. 13, column B of the target data 311C is an objective variable column corresponding to KPI.

In step 1003, the analysis data proposal module 325 selects a predetermined number of columns as candidates for explanatory variables from the table of target data 311 or the table of machining data 312. The columns that are candidates for the explanatory variables are the columns of the target data 311 table or the processing data 312 table, excluding the objective variable column selected in step 1002.

The analytical data proposal module 325 creates all one or more combinations of explanatory variable columns for the objective variable column. FIG. 13 shows that all columns of the target data 311 (table 311A) other than column B and all columns of the processing data 312 (table 311D) can be explanatory variable columns. For example, assuming that the number of explanatory variable columns to be selected is two, the analysis data proposal module 325 has two columns from all columns other than column B of the target data 311A and all columns of the processing data 311D. Create a plurality of all combinations, and combine each combination with the objective variable column to form a plurality of tables consisting of a composite column of the objective variable column and the explanatory variable column. Each of 311E-1 to 311E-4 ... In FIG. 13 is a table composed of synthetic columns.

In step 1004, the analysis data proposal module 325 calculates the correlation coefficient between the target data table and all the columns of the machining data table. In step 1005, the data proposal module 325 for analysis performs regression analysis or multiple regression for all combinations of the objective variable column extracted in step 1002 and the candidate columns of the number of explanatory variables selected in step 1003. Conduct learning such as analysis.

In step 1006, the data proposal module 325 for analysis calculates the usefulness of each combination of the objective variable column and the explanatory variable column, and creates a list in which all combinations are ranked based on the calculation result. To do. The user can select the desired combination.

The usefulness of each combination among the plurality of combinations is, for example, (1) the absolute value of the correlation coefficient between the objective variable and the explanatory variable (step 1004), and (2) the goodness (determination) of the learning result in step 1005. Variables, accuracy rate) (3) The total number of records including valid values in the objective variable column and the explanatory variable column, and (4) the absolute value of the correlation coefficient between the explanatory variables (step 1004). The analysis data proposal module 325 ranks the combinations in ascending or descending order of usefulness.

The data proposal module 325 for analysis determines the rank that is the sum of the ranks of usefulness (1) -usefulness (4), and sets this as the rank 511 of the objective variable / explanatory variable combination list 503 and KPI as the objective variable. In (KPI) 530, the name of one or more explanatory variables is set to explanatory variable 531, the number of valid records of all combinations is set to 518, and the usefulness (1)-(4) is set to usefulness 1-4 (533). -536), the output destination of the file of each combination record is stored in the output file 519, respectively.

In step 1007, the analysis data proposal module 325 creates a learning result graph for each combination of the objective variable and the explanatory variable included in the list created in step 1006, presents it to the user, and selects the desired combination by the user. It can be so. Here, it is also possible to create a graph containing only the combinations of the objective variable and the explanatory variable specified from the top of the list.

Then, in step 822, the analysis data proposal module 325 outputs a data record relating to the explanatory variable column and the objective variable column specified by the user who referred to the result of step 1007 to a CSV file or the like. This file is used by the user to perform factor analysis / prediction by himself / herself.

The data proposal module 325 for analysis changes the data catalog and the data definition 315 and adds information based on the result of creating the list 503 in step 1006 and the result of selection by the user from this list.

11A-11C are examples of graphic interfaces provided by the server 101 to the user terminals 102 and 103 to support data utilization. The proposal execution management module 326 causes the user terminal requesting the analysis purpose to display the graphic interface screen based on the analysis data proposal module 325 and the analysis method proposal module 324.

On the screen 1101 of FIG. 11A, it is displayed in the first area 1111 that the "overall image grasp" as the analysis step specified by the user and the server 101 recommending "addition of data" for the analysis step.

Further, a second area 1112 consisting of a degree of coincidence between the analysis purpose information 401 and the target data, a data catalog, and a degree of enrichment of data definition information, insufficient data for adding data, a storage destination of the data catalog, and , The third area 1113, which is the storage destination of the data definition, is displayed as supplementary information.

Screen 1102 of FIG. 11B is an example of the candidate list 501 in which the recommended analysis step is grasping the whole picture. For the data in which a plurality of columns are combined, the plurality of data (step 820) are listed in order of usefulness (1121). Region 1122 is an example of a graph obtained as a result of trend analysis for each data combination (step 821).

Screen 1103 of FIG. 11C is an example of the objective variable / explanatory variable combination list 503 in which the recommended analysis step is factor analysis / prediction. For data in which a plurality of columns are combined, a plurality of the data (step 1006) are listed in order of usefulness (1131). The area 1132 is a graph showing the learning result for each data combination.

According to the above-described embodiment, when the user 201 performs problem analysis or the like by utilizing a wide variety of a plurality of business systems and a large amount of data, if the user registers the analysis purpose in the server, the user A list of useful data for analysis can be presented to the user while reducing the burden on the user. The user will be able to quickly and easily realize the data preparation work before data utilization.

When the server 101 tries to present data related to the analysis purpose to the user who is trying to execute the analysis step recommended to the user, the server 101 analyzes the column related to the analysis purpose (KPI, data item of interest) and the analysis. In order to combine columns related to standard information that are compatible with the purpose and have a high degree of fulfillment and present the records of both columns to the user, the user only needs to register the analysis purpose in the server for analysis. A wide range of data combinations related to the purpose can be obtained.

101 Server 102, 103 User terminal 104-106 Business system

Claims (12)

It is a system that supports data preparation for data utilization.
Processing equipment and
Equipped with a storage device,
The processing device executes a program recorded in the storage device.
Business data is collected from each of a plurality of business systems, and the business data is stored in the storage device as target data at least temporarily.
Receive the analysis purpose for the target data from the user terminal and
Based on the analytical objectives and the target data, the recommended analytical steps for the analytical objectives are determined.
A combination of data available for the recommended analysis step is extracted from the target data and
Each of the plurality of combinations of the extracted data was evaluated and
Based on the result of the evaluation, a list in which multiple combinations of the above data are ranked is created.
The list is output to the user terminal, and the list is output.
Allowed the user to select a predetermined data combination from the list.
The system. The storage device includes reference information for the target data.
The processing device is
Determining the recommended analysis steps based on the reference information and
Determining the recommended analysis step is performed by determining one of the multiple types of analysis implementation stages for the target data at the analysis implementation stage.
Have,
The system according to claim 1. The processing device is
Calculate the degree to which the analysis purpose conforms to the target data or the target data and the reference information.
The degree of effectiveness of the reference information for the analysis purpose is calculated.
Based on the results of both calculations, the stage of performing any of the above types of analysis is determined.
The system according to claim 2. The analysis purpose includes an analysis performing stage desired by the user among the performing analysis stages of the plurality of types of analysis.
The processing device is
Based on the results of both of the above calculations, the user evaluates the stage of analysis desired, and this evaluation includes maintaining the stage of performing the analysis or changing to another stage of analysis.
Based on the results of the evaluation, the user is presented with the recommended stage of analysis.
The system according to claim 3. The analytical objectives include KPIs.
The reference information includes dictionary data for the target data.
The processing device is
The degree to which the analysis purpose is suitable for the target data is calculated by comparing the KPI with the column name of the target data.
The degree to which the analysis purpose conforms to the reference information is calculated by comparing the KPI with the dictionary data.
The system according to claim 3. The storage device includes reference information for the target data.
The processing device is
Extracting a combination of data available for the recommended analysis step from the subject data
Extracting data corresponding to the analysis purpose from the target data, and
Extracting the data corresponding to the reference information from the target data and
Run from
Based on the data corresponding to the analysis purpose and the data corresponding to the reference information, a plurality of combinations of the data are configured.
Evaluate each of the multiple combinations of data based on the amount of data.
The system according to claim 1. The processing device is
Extracting the data corresponding to the analysis purpose from the target data is executed by extracting the column corresponding to the analysis purpose from the target data.
Extracting the data corresponding to the reference information from the target data is executed by extracting the column corresponding to the reference information from the target data from the target data.
As a combination of the data, a plurality of combinations of a column corresponding to the analysis purpose and a column corresponding to the reference information are configured.
As the amount of data, the number of records for each of the plurality of combinations is calculated.
The system according to claim 6. The processing device is
For each of the plurality of types in the stage of performing the analysis, each of the plurality of combinations of the extracted data is evaluated.
The system according to claim 2. The processing device is
For each of the plurality of combinations, it is determined which data value fluctuation pattern is provided among the plurality of data value fluctuation patterns.
The number of records is calculated for each data value fluctuation pattern.
The system according to claim 7. The processing device is
The column of the target data is extracted based on each of the 4W information of the dictionary data.
The system according to claim 5. Give the user information about recommended analysis steps or work items for the user's registration of analysis objective information, as well as information about the combination of available data related to the user-specified KPIs and data items of interest in each analysis step. The system according to claim 1, wherein the system has an output device for presenting. It is a method to support data preparation for data utilization,
Computer
Business data is collected from each of multiple business systems, and the business data is stored in the storage device as target data at least temporarily.
Receive the analysis purpose for the target data from the user terminal and
Based on the analytical objectives and the target data, the recommended analytical steps for the analytical objectives are determined.
A combination of data available for the recommended analysis step is extracted from the target data and
Each of the plurality of combinations of the extracted data was evaluated and
Based on the result of the evaluation, a list in which multiple combinations of the above data are ranked is created.
The list is output to the user terminal, and the list is output.
Allowed the user to select a predetermined data combination from the list.
The method.