JP6229512B2 - Information processing program, information processing method, and information processing apparatus - Google Patents

Description

  The present invention relates to a technique for performing data processing using a processing flow in which processing components are combined in a computer.

  A tool that can create a processing flow that defines a series of data processing flows by combining processing components that perform data processing in small units such as data extraction and aggregation when performing data processing using a computer Is used. By using such a tool, the user can reduce the trouble of creating a data processing program from scratch. Also, the processing flow and individual processing components created by knowledge-rich users can be shared by a plurality of users.

  As an example of a conventional technique related to data processing, a technique for processing spreadsheet data and plain file data into a form desired by a user has been proposed. In this technique, input data is processed according to a scenario in which a series of data processing procedures such as data input and decomposition are defined. In this data processing, label data such as data item names attached to input data is analyzed with a predetermined pattern to extract item names and stored as schema information.

  In addition, as a technology example for supporting the work of combining processing parts, a technology for connecting components in a web application has been proposed. In this technique, a component has attribute information indicating a property type or the like that is data representing the property of the component. Then, other components having the same attribute information as the component selected by the user are presented as connection destination candidates.

  Furthermore, the following techniques have been proposed as other examples of techniques for providing support when combining processing components. That is, in this technique, component definition information of a component program having a function that belongs to a function type compatible with the component program corresponding to the component definition information set for the icon is searched. As a result of the search, the identification information of the hit component definition information and the identification information of the component program corresponding to the hit component definition information are displayed as a list on the display screen together with icons.

Japanese Patent Laid-Open No. 10-171636 JP 2010-176195 A JP 2011-008358 A

  Here, when a processing flow is generated by combining processing components, depending on the processing contents of the processing components, some components of the input data (for example, some columns in the case of tabular data or specific conditions) May be set as a parameter of the processing component. For example, it is a case where only some components of the input data are processed by the processing component. In this case, the user may specify a component as a parameter. At that time, in order to allow the user to correctly specify the component, it is desirable that the name of the component can be selected from candidates that can be specified, instead of manually inputting the name of the component.

However, for example, depending on the processing component, there are cases where the constituent elements of the output data change dynamically according to the contents of the data. For example, the table structure is converted according to the contents of the data, and a process for adding a new column to the input data is performed. In order to be able to select a candidate for a component of input data in a subsequent processing component that receives the output data of such a processing component, the component of the component corresponding to the data after processing by the preceding processing component is selected. Information needs to be extracted.
In view of the above, an object of one aspect of the present invention is to enable efficient extraction of data component information after processing by a processing component in a processing flow in which processing components are combined.

In one aspect of the present invention, a target process that performs processing for generating output data that is one target processing component included in a processing flow in which a plurality of processing components are combined and that includes components according to the content of input data Get input data for a part. Further, according to a plurality of preset sampling methods, a plurality of test data are generated by sampling a plurality of different sample numbers of data from the input data for each of the plurality of sampling methods. Further, for each of a plurality of sampling methods, while increasing the number of test data samples, processing by the target processing component that receives each test data is executed, and output data of each processing result is acquired. Then, among the multiple sampling methods, select the sampling method that increases the number of components of the output data according to the increase in the number of test data samples when the test data with the smallest number of samples is processed. To do.

  According to one aspect of the present invention, in a processing flow in which processing components are combined, information on data components after processing by the processing components can be efficiently extracted.

It is explanatory drawing which shows an example of an analysis flow. It is explanatory drawing of an example of an input data table. It is explanatory drawing of an example of a data profile and a process component setting screen. It is explanatory drawing of an example of the data which made input data item matrix. It is explanatory drawing of an example of correlation with the number of samples of input data, and the number of columns of output data. It is explanatory drawing which shows an example of a function structure and data structure of an analysis server. It is explanatory drawing of an example of an analysis flow table. It is explanatory drawing of an example of components database. It is explanatory drawing of an example of a sampling method table. It is explanatory drawing of an example of a test data table. It is explanatory drawing of an example of a data profile temporary storage file. It is explanatory drawing of an example of a data profile temporary storage file. It is explanatory drawing of an example of a data profile temporary storage file. It is explanatory drawing of an example of a column number table. It is explanatory drawing of an example of a sample data table. It is explanatory drawing of an example of a data profile table. It is a flowchart which shows an example of the whole process performed with an analysis server. It is a flowchart which shows an example of the data profile production | generation process performed with an analysis server. It is a flowchart which shows an example of the sampling method selection process performed with an analysis server. It is an example of the hardware constitutions of the analysis server in this embodiment.

[Background and Outline of this Embodiment]
In the present embodiment, in a tool that creates a series of processing flows by combining processing parts that perform data processing in small units such as data extraction and aggregation, information on data components after processing of the processing parts is efficiently used. A technique for enabling extraction will be described. In the present embodiment, description will be made using a data analysis tool which is an example of such a data processing tool. The data analysis tool analyzes the data contents according to the combination of processing parts and outputs the analysis result.

  FIG. 1 shows an example of an analysis flow constructed in an example of a data analysis tool. As shown in FIG. 1, the analysis flow includes, for example, input data, a combination of processing components that perform data processing in a small unit, and a final output of processing results by the processing components. Each processing component is connected to process the output data (or first input data) from the preceding processing component and pass the intermediate file of the processing result to the subsequent processing component (or output it last). Yes. Data can be analyzed by executing a series of processes in such an analysis flow.

  A user first constructs such an analysis flow using a data analysis tool. The user can create a processing component himself. In addition, a plurality of users can share and use a processing component once created so that the user does not have to describe the data analysis process from scratch. The user can construct an analysis flow by determining input data and connecting processing components that sequentially process the input data.

Here, depending on the processing content of the processing component, for example, only some constituent elements of the input data may be set as parameters. For example, there are cases where only some of the constituent elements are to be processed, or parameters for clustering analysis. In this case, the user may specify a component to be processed in the processing component.
For example, if the input data of the analysis flow shown in FIG. 1 is data in a table format as shown in FIG. 2, and the intermediate file 1 output by the processing component A has the same data structure as the data shown in FIG. To do. As shown in FIG. 2, this input data includes columns of “date”, “accounting time”, and “sales” as an example of the constituent elements. Here, in the processing component B, it is assumed that the user selects only a specific column from the input data shown in FIG. Here, when a normal database table or the like is used as input data, the table data profile generally includes column information indicating what columns are included in the table. For this reason, as shown in FIG. 3, on the setting screen in the processing component, selectable column name options can be displayed based on the column information of the data profile of the input data.

  However, for example, it is assumed that the processing component A preceding the processing component B performs a process of dynamically changing the constituent elements of the output data according to the contents of the data. For example, as shown in FIG. 4, within the time range in which the data exists in the “accounting time” of the input data shown in FIG. 2 (within the range of the minimum value and the maximum value), every time zone (every hour) In this case, the table structure is converted (item matrix) so as to indicate whether or not there is sales in positive and negative (T or F). Note that by converting the input data into such a data structure, for example, it can be used for processing such as clustering analysis and machine learning. In such a case, the column information of the data profile shown in FIG. 3 described above cannot be used even if it is attempted to display column name options in the subsequent processing component B that receives the output data of the processing component A.

  For this reason, in this embodiment, in order to be able to acquire the column information of the input data in the processing component B, a data profile of data after processing by the preceding processing component A is generated. Here, the most reliable method for generating a data profile is to process all data by the processing component A and generate a data profile from the processing result. However, in the analysis tool, it is assumed that a large amount of data becomes input data. For this reason, when all input data is processed in order to generate the data profile of the processed data of the processing component A, there is a possibility that the construction work of the analysis tool by the user is stopped for a long time.

  Therefore, in this embodiment, sample data is used to generate a data profile of data after processing by the processing component A. The sample data is at least a part of the entire input data processed by the processing component A. It is desirable that the sequence of output data after the processing component A processes the sample data is the same as the sequence of output data after the processing component A processes the entire input data.

  Here, how to extract such sample data (how to use the sampling method) becomes a problem. This is because the number of sample data required to generate the same column differs depending on the sampling method depending on the processing content of the processing component. In order to efficiently generate a column with a smaller number of samples, it is desirable to select a sampling method suitable for the processing content of the processing component. Here, for example, if it is possible to grasp what kind of processing is specifically performed by the processing component A, it is relatively easy for the user to select an appropriate sampling method. However, as described above, the data analysis tool may use processing components created by other users. In such a case, it is not always possible for the user to grasp the detailed processing inside the processing component, so it is difficult to select an appropriate sampling method.

  For this reason, in this embodiment, an appropriate sampling method of sample data for generating a data profile of data after processing by the processing component is automatically selected. The appropriate sampling method here is to extract sample data with a smaller number of samples and output data of components (here, column structure) closer to the output data after the processing component has processed the entire input data It can be done.

  Specifically, in the present embodiment, a process of generating output data that is one target processing component included in a processing flow in which a plurality of processing components are combined and that has a component according to the content of the input data is performed. Get input data for the target process component. Further, according to a plurality of preset sampling methods, a plurality of test data are generated by sampling a plurality of different sample numbers from the input data for each sampling method. Further, for each sampling method, while increasing the number of samples of test data, processing by the target processing component that receives each test data is executed, and output data of each processing result is acquired. Of these sampling methods, when the test data with the smallest number of samples is executed, the increase in the number of components of the output data according to the increase in the number of samples of the test data converges (specifically, a predetermined threshold value). Select the sampling method (below).

FIG. 5 shows an example of the correlation between the number of input data samples and the number of output data columns for each sampling method. In this example, whether or not the processing component A as an example of the analysis flow described above has been sold for each time period within the time range in which the data exists in the “accounting time” of the input data shown in FIG. 2 is positive or negative. This is an example of forming an item matrix as shown by. There are three sampling methods, for example, “random sampling”, “sampling including the maximum value and minimum value”, and “sampling including more differences”. A column used as a reference for sampling (a column from which maximum values, minimum values, differences, and the like are extracted) is set as “accounting time”.
In the example of the processing of the processing component A, a column divided every hour is added within the range of the maximum value and the minimum value of the accounting time. For this reason, the larger the range of the “accounting time” value of the sampled data, the more columns are generated.

In the case of such processing, as shown in FIG. 4, if the sampling includes the maximum value and the minimum value, the maximum value (21:48) and the minimum value (21:48) of “accounting time” at the time when the number of samples is two. 10:23). For this reason, in the output data of the processing result, the number of columns indicating the time zone of the accounting time (item column) is the same column as the output data of FIG. 4 obtained by processing all the input data of FIG. The number is 12. This number of columns is the maximum value, and even if the number of samples is increased thereafter, the number of columns does not change and converges. In this embodiment, “convergence” means that the number of columns reaches the number of columns in the output data as a result of processing the entire input data by the processing component or a number close to the number of columns, and the number of columns with respect to the increase in the number of samples. A state where the increase is less than a predetermined threshold (saturated state) is shown.
On the other hand, in the case of random sampling, even if the number of samples is increased, data that expands the value range is not always extracted. The same is true for sampling that includes more differences. For this reason, in these sampling methods, even if the number of samples of input data increases, columns are not generated efficiently, and the increase in the number of columns does not converge.

In this embodiment, the number of columns of output data according to the number of samples of such input data is specified, the convergence status of the increase in the number of columns is specified for each sampling method, and the efficiency of the increase in the number of columns converges earlier. A typical sampling method. For this reason, it is not necessary for the user to study an appropriate sampling method, and the work load is reduced. When the data profile is generated, only the sample data need be processed by the processing component instead of the entire input data, so that the data profile can be generated with a relatively small processing amount. Therefore, it is possible to efficiently extract column information of input data to be selected in subsequent processing components.
Hereinafter, specific modes of this embodiment will be described in detail.

<System functional configuration and data configuration>
FIG. 6 illustrates an example of a functional configuration and a data configuration of the analysis server 1 that is an example of a computer including a data analysis tool in the present embodiment. Note that the functional configuration described in the present embodiment is based on the assumption that each processing component in the analysis flow has already been created and is ready for use.

  As shown in FIG. 6, the analysis server 1 has its functions realized by executing a program. The analysis flow reading unit 11, the analysis flow change detecting unit 12, and the data profile generating unit 13 (test data generating unit 14). A test data processing unit 15, a sampling method selection unit 16, a sample data generation unit 17, a sample data processing unit 18), an analysis flow update unit 19, a column selection setting unit 20, and an analysis flow execution unit 21. In addition, the analysis server 1 includes an analysis flow table 31, a component database 32, an input data table 33, a sampling method table 34, a test data table 35, a data profile temporary storage file 36, a column number table 37, a sample data table in a storage unit. 38 and a data profile table 39.

The analysis flow reading unit 11 reads from the analysis flow table 31 information related to processing components included in the analysis flow and connection of the processing components. Further, the analysis flow reading unit 11 reads information on the processing contents and the like of each processing component included in the analysis flow from the component database 32. The analysis flow reading unit 11 reads input data of the analysis flow from the input data table 33.
Based on the information read by the analysis flow reading unit 11, the analysis flow change detection unit 12 detects that a processing component has been added or changed, input data has been changed, or the like in the analysis flow.

The data profile generation unit 13 generates a data profile of data after processing of a processing component, and includes a test data generation unit 14, a test data processing unit 15, a sampling method selection unit 16, a sample data generation unit 17, and sample data. A processing unit 18 is included.
The test data generation unit 14 samples a plurality of different sample numbers of data for each of a plurality of sampling methods stored in the sampling method table 34 from input data of a target processing component that is a target for generating a data profile. Generate test data. The generated test data is stored in the test data table 35.

For each sampling method, the test data processing unit 15 executes processing by the target processing component that receives the test data generated by the test data generation unit 14. At this time, the test data processing unit 15 executes a process of inputting test data of the same number of samples for each sampling method, and obtains output data of each processing result. Then, the test data processing unit 15 performs the same processing while increasing the number of test data samples.
The sampling method selection unit 16 selects a sampling method in which the increase in the number of columns of output data corresponding to the increase in the number of samples of test data converges earliest. Specifically, the sampling method selection unit 16 performs the sampling method in which the increase in the number of columns of the output data corresponding to the increase in the number of samples of the test data is equal to or less than a predetermined threshold when the test data having the smallest number of samples is executed. Select.

The sample data generation unit 17 generates sample data by sampling data from the input data of the target processing component using the selected sampling method, and stores the sample data in the sample data table 38.
The sample data processing unit 18 uses the sample data in the sample data table 38 as input data, executes processing by the target processing component, and acquires output data as a processing result. And column information is acquired from the output data of a processing result, and the data profile containing the said column information is produced | generated. Further, the generated data profile is stored in the data profile table 39.

The analysis flow update unit 19 reflects the addition or change of the processing component in the analysis flow, the change of the data profile, and the like in the analysis flow table 31, the component database 32, and the like.
The column selection setting unit 20 performs processing for allowing the user to set the component of the parameter of the processing component. As a specific example, the column selection setting unit 20 extracts column information of the target processing component from the data profile table 39, and uses the column name specified by the extracted column information as a column option to be processed on the setting screen. indicate. Then, the selection input of the column name by the user is accepted, and the selection content is stored in the analysis flow table 31.
Based on the operation input by the user, the analysis flow execution unit 21 executes an analysis flow using actual input data (all items, not sample data) as input, and outputs a data analysis result.

  Next, details of each data included in the analysis server 1 will be described with reference to FIGS. 2 and 7 to 16. For each table other than the analysis flow table 31 and the component database 32, only data related to the target processing component for which the data profile is generated in the specific example of the present embodiment is illustrated as a specific example.

  The analysis flow table 31 is a table in which information regarding the content of the analysis flow is stored for each analysis flow. For example, as shown in FIG. 7, the analysis flow table 31 identifies a component ID uniquely assigned to a processing component included in the analysis flow, a component name of the processing component, and a processing component connected immediately before the processing component. Connection source ID, connection destination ID for identifying the processing component connected immediately after the processing component, parameters specified in the processing component, input data processed by the processing component, and processing result by the processing component Output data, and a data profile column of data after processing by the processing component.

In the parts database 32, information related to parts that can be used in the analysis flow is stored in advance. For example, as shown in FIG. 8, the component database 32 includes a component name (indicating an outline of processing by the component), a parameter indicating a parameter that can be specified in the component, a variable name of input data, a data type of input data, and output data. Column of variable name, output data type, etc.
The input data table 33 is a table in which individual input data of the analysis flow is stored, and its structure differs depending on the contents of the input data. The input data in the specific example of this embodiment includes, for example, columns of date, accounting time, and sales as shown in FIG.

The sampling method table 34 is a table in which information related to various sampling methods is stored in advance. For example, as shown in FIG. 9, the sampling method table 34 includes a sampling method and a column of detailed contents of the sampling method.
The test data table 35 is a table (structure array) in which test data generated by each sampling method is stored. For example, as shown in FIG. 10, test data is stored for each sampling method and for each number of samples. ing.

The data profile temporary storage file 36 is a file in which data profile tables 39 generated as a result of processing test data by processing components are sequentially stored. For example, as shown in FIGS. 11 to 13, the data profile temporary storage file 36 stores a data profile table 39 including a column name of the input data and a column of the data type for each sampling number and each sampling method. Is done.
The column number table 37 sequentially stores, for each test data sampling method, the number of columns (the number of columns whose number varies depending on the number of samples) obtained as a result of processing the test data of each sample number by the processing component. It is a table to be. The column number table 37 includes, for example, a sampling method and a sample number column as shown in FIG.

The sample data table 38 is a table for storing sample data sampled by the selected sampling method after selecting the sampling method. The sample data table 38 in the specific example of this embodiment includes columns of date, accounting time, and sales as shown in FIG. 15, for example.
The data profile table 39 is a table for storing a data profile generated as a result of actually processing sample data by a processing component after selecting a sampling method. For example, as shown in FIG. 16, the data profile table 39 includes a column name of input data and a column of the data type of the column.

<Processes executed in the analysis server>
Next, processing executed in the analysis server 1 will be described with reference to FIGS.
First, the entire process executed by the analysis server 1 will be described with reference to FIG.

In step S <b> 1, the analysis flow reading unit 11 accepts designation of an analysis flow by the user, and reads information related to the content of the analysis flow from the analysis flow table 31. And the information regarding the component corresponding to the component name contained in the information read from the analysis flow table 31 is read from the component table. By these reading processes, the contents of processes to be executed in the entire analysis flow are specified.
In step S <b> 2, the analysis flow reading unit 11 reads input data of the first processing component of the analysis flow based on the information read from the analysis flow table 31.
For example, when a new analysis flow is generated, the processing in steps S1 and S2 is not necessary.
In step S3, the analysis flow reading unit 11 waits for an operation input by the user.
In step S4, the analysis flow reading unit 11 determines whether or not there is an operation input for instructing execution of the analysis flow. If there is no input, the process proceeds to step S5 (No). Proceed to S16 (Yes).

The following steps S5 to S15 are processes executed when the analysis flow is generated or changed.
In step S5, the analysis flow change detection unit 12 detects the addition when a new processing component is added to the analysis flow.
In step S <b> 6, the analysis flow change detection unit 12 detects the change or the like when a setting change is made to a processing component of the analysis flow or input data is changed.
In step S7, the analysis flow change detection unit 12 determines whether or not the column name setting screen for the column to be processed has been opened in any of the processing components of the analysis flow. When the setting screen is not opened, the process proceeds to step S8 (No), and when the screen is opened, the process proceeds to step S13 (Yes).

  In step S <b> 8, the analysis flow change detection unit 12 determines whether it is necessary to generate a data profile (including regeneration) in any of the processing components in accordance with the addition or change of the processing components in the analysis flow. Specifically, for example, when a processing component is added, it is necessary to generate a data profile of the processing component (data profile of data after processing by the processing component). In addition, for example, when the content of input data is changed or when the processing content of each processing component (or preceding processing component) is changed, it is necessary to regenerate the data profile of each processing component. It may become. If the data profile needs to be generated, the process proceeds to step S9 (Yes), and if not necessary, the process proceeds to step S12 (No).

The following steps S9 to S11 are processes executed when generating a data profile when creating or changing an analysis flow.
In step S9, the data profile generation unit 13 specifies a processing component that needs to generate a data profile.
In step S <b> 10, the data profile generation unit 13 generates a data profile of data processed by the processing component for a processing component that needs to be generated. The data profile generation process will be described in detail later.

In step S <b> 11, the data profile generation unit 13 stores the generated data profile in the data profile table 39.
In step S <b> 12, the analysis flow update unit 19 reflects the addition or change contents of the detected processing component, data profile change, or the like in the data of the analysis flow table 31 as necessary.

The following steps S13 to S15 are processes executed when the user sets column names to be processed by the processing component when the analysis flow is created or changed.
In step S <b> 13, the column selection setting unit 20 specifies a processing component in which the column name setting screen for the column to be processed is opened.
In step S14, the column selection setting unit 20 determines the data profile of the data to be input of the specified processing component, that is, the data profile of the data after processing of the processing component connected immediately before the processing component. Extract column information. Then, the column name specified by the extracted column information is displayed on the setting screen as a column name option to be processed.

In step S15, the column selection setting unit 20 receives a selection input of a column name to be processed by the user. Then, the column selection setting unit 20 stores information indicating the selected column name in the parameter item of the data of the corresponding processing component in the analysis flow table 31.
The following steps S16 to S17 are processes for executing the analysis flow.
In step S16, the analysis flow execution unit 21 executes the analysis flow specified by the user.
In step S17, the analysis flow execution unit 21 outputs the execution result of the analysis flow.

  Next, the data profile generation process (the process in step S10) executed in the analysis server 1 will be described with reference to FIG.

In step S <b> 21, the test data generation unit 14 refers to the analysis flow table 31, specifies input data to the target processing component that requires generation of a data profile, and acquires input data. If the target processing component is the first processing component in the analysis flow, input data can be acquired from the input data table 33. When the target processing component is a processing component in the middle of the analysis flow, an intermediate file after processing by the preceding processing component is input data of the target processing component.
In step S22, the sampling method selection unit 16 performs a process of selecting a sampling method of sample data from the input data. The sampling method selection process will be described in detail later.

  In step S23, the sample data generation unit 17 samples the data from the input data of the target processing component by the sampling method determined in step S22, and generates sample data. At this time, the sample data generation unit 17 may refer to the column number table 37 and sample data of the earliest stage sample number (the smallest sample number) in which the increase in the column number has converged. Specifically, from the number of samples in which the number of columns continuously increases (the number of columns corresponding to the increase in the number of samples) is less than or equal to the threshold, going back from the number of samples in which the number of columns was stored last. The data with the smallest number of samples may be sampled. Then, the sample data generation unit 17 stores the generated sample data in the sample data table 38.

In step S24, the sample data processing unit 18 inputs the sample data stored in the sample data table 38 to the target processing component, and executes the processing by the target processing component.
In step S25, the sample data processing unit 18 extracts columns from the processing result data in step S24, estimates the data type from the data in each column, and generates a data profile of the target processing component. Then, the sample data processing unit 18 stores the generated data profile in the data profile table 39.

  Next, the sampling method selection process (the process in step S22) executed in the analysis server 1 will be described with reference to FIG.

In step S31, the sampling method selection unit 16 determines a plurality of different sample numbers. For example, while the minimum number is 2, the number of samples can be obtained by sequentially adding the total number of input data by the number obtained by equally dividing the total number of input data. Then, an array of sample numbers (not shown) is generated.
In step S32, the sampling method selection unit 16 extracts one from the small number from the sample number array and sets it as the current sample number.

  In step S <b> 33, the sampling method selection unit 16 refers to the sampling method table 34 and samples data by the current number of samples according to each sampling method. Note that the column used as a reference for sampling (the column from which the maximum value, minimum value, difference, etc. are extracted) is a column in which the processing component generates a new column based on the data content (in the above example, “accounting time”). Column). In the present embodiment, it is assumed that at least information that can identify the column can be acquired from, for example, the analysis flow table 31 or the component database 32. Then, the sampling method selection unit 16 stores the sampled data in the test data table 35.

In step S34, the sampling method selection unit 16 executes processing by the target processing component that receives the test data of the current number of samples stored in the test data table 35 for each sampling method, and outputs each processing result. Get the output data.
In step S35, the sampling method selection unit 16 extracts a column from the output data of the processing result of each test data in step S34, estimates the data type from the data of each column, and generates a data profile of the target processing component. To do. Then, it is stored in the data profile temporary storage file 36.

In step S36, the sampling method selection unit 16 refers to the data profile temporary storage file 36, and among the columns extracted by the current number of samples, the column indicating the time zone of the accounting time in which the number of columns fluctuates (that is, fixed) The column number of “date”, “accounting time”, and “item column excluding“ sales ”” is stored in the column number table 37 for each sampling method. Here, the total number of columns included in the data profile may be stored.
In step S <b> 37, the sampling method selection unit 16 refers to the column number table 37 and reads the number of columns extracted by the previous and previous sample numbers for each sampling method.

In step S38, the sampling method selection unit 16 determines whether there is a sampling method that satisfies the following requirements.
(The number of columns in the previous number of samples−the number of columns in the previous number of samples) is less than or equal to the threshold, and
(Number of columns in the current number of samples-number of columns in the previous number of samples) is less than the threshold

  Note that this determination is to determine whether or not the increase in the number of columns of processing results accompanying the increase in the number of samples has converged. The threshold value used in the determination may be set in advance to an appropriate value for determining that the increase in the number of columns has converged according to the number and contents of input data. Here, for example, the determination can be made only with the criterion that “(the number of columns in the current number of samples−the number of columns in the previous number of samples) is equal to or less than a threshold”. However, depending on the sampling method and the processing content of the processing component, there is a case where there is no regularity in the increase in the number of columns of the processing result, and it may happen that the increase in the number of columns of the processing result is small. For this reason, in the determination in step S38, as described above, the determination criterion “(the number of columns in the previous sample number−the number of columns in the previous sample number) is equal to or less than the threshold value” is also added. Further, the determination may be performed based on an increase in the number of columns in the output data for the past several times.

By setting the threshold to “0”, it is possible to specify a state in which the increase is completely converged with the number of columns reaching the maximum value.
In this determination, there is a case where the increase in the number of columns converges while the number of columns is small (at least in a state where the increase in the number of columns is less than the number of columns in other sampling methods that have not converged). In principle, it is assumed that it will not occur. The same applies to step S41 described later.
If there is a sampling method that satisfies the above requirements, the process proceeds to step S39 (Yes), and if not, the process proceeds to step S40 (No).

  Note that the above steps S37 and S38 are executed only in the processing of the number of samples after the third time (because the number of columns in the previous and previous samples cannot be obtained from the first time to the second time). .

In step S39, the sampling method selection unit 16 selects a sampling method that satisfies the requirements determined in step S38. In other words, this processing selects a sampling method in which the increase in the number of columns has converged earlier (when the test data having the smallest number of samples is executed).
In step S40, the sampling method selection unit 16 determines whether or not the generation of the data profile using the test data of all the numbers of samples is completed. If completed, the process proceeds to step S41 (Yes). If not completed, the process returns to step S32, and the process proceeds to the next sample number (the number of samples increased from this time) (No).

In step S41, the sampling method selection unit 16 refers to the column number table 37, and selects a sampling method in which the increase in the number of columns of the processing result accompanying the increase in the number of samples is more converged. Specifically, as an example, a sampling method having the smallest value of the calculation result of the following equation is selected.
(Number of columns in previous sample number-Number of columns in previous sample number) + (Number of columns in current sample number-Number of columns in previous sample number)

  As in step S38, for example, the convergence state of the increase in the number of columns can be identified only by the value of “(number of columns at current sample number−number of columns at previous sample number)”. is there. However, considering the case where there is no regularity in the increase in the number of columns in the processing result, the value of “(number of columns in the previous sample number−number of columns in the previous sample number)” is added as described above. It shall be.

<Specific example of sampling method selection processing>
Here, the sampling method selection processing will be described by showing a specific example of data.
For example, in the sampling method selection process, when the input data of a certain processing component is the input data shown in FIG. 2, the sampling method stored in the sampling method table 34 shown in FIG. 9, that is, “random sampling”. An example of selecting one of “sampling including the maximum value and minimum value” and “sampling including more differences” will be described. Assume that the number of samples is 2, 3, and 4. In addition, here, a column as a reference for sampling (a column from which maximum values, minimum values, differences, and the like are extracted) is input data, and the processing component generates a new column based on the data contents. "

  In this specific example, the test data generated by the test data generation unit 14 with the number of samples in the array of the number of samples for each sampling method is the data content of the test data table 35 shown in FIG. Then, the test data processing unit 15 processes these test data by processing components in order from the test data having the smallest number of samples. The data profile generated from the output data of the processing result becomes the contents of the data profile temporary storage file 36 shown in FIGS. Further, for the data contents of the data profile temporary storage file 36, the test data processing unit 15 determines, for each sampling method, the number of columns of the data profile (that is, the number of columns of output data of the processing result) for each sample number. The number of columns indicating the time zone of the accounting time is stored in the column number table 37 as shown in FIG.

  Referring to the column number table 37 in FIG. 14, “sampling including the maximum value and minimum value” indicates that the number of columns is 12 when the number of samples is 2, and when the number is 4, The same. Here, it is assumed that the threshold in the determination in step S38 of the sampling method selection process is set to “1”. In this case, in the determination of step S38 when the number of samples is 4, (the number of columns in the previous sample number−the number of columns in the previous sample number) is (12-12) = 0, The number of columns in the number-the number of columns in the previous sample number) is also (12-12) = 0. For this reason, the requirements in the determination in step S38 are satisfied, and it can be seen that the increase in the number of columns of the processing results accompanying the increase in the number of samples has converged. Therefore, the “sampling including the maximum value and the minimum value” is selected as the most appropriate sampling method.

  Then, the sample data generating unit 17 generates sample data by sampling two pieces of data, which is the first sample number that the sample data has converged, in the “sampling including the maximum value and the minimum value”. 15 is sample data of the sample data table 38 shown in FIG. Further, the sample data processing unit 18 receives the sample data as input and executes processing by the processing component to generate the data profile shown in FIG.

  The increase in the number of columns as in the above specific example is, for example, as described above, within the time range in which the data exists in the “accounting time” of the input data shown in FIG. In this case, the table structure is converted so as to indicate whether or not there is sales in each time zone. The correlation between the number of samples of test data and the number of columns of output data in the specific example is as shown in FIG. Here, if the user can grasp in advance that the processing content of the target processing component is such content, it can be predicted that “sampling including the maximum value and the minimum value” is the most appropriate sampling method. However, as described above, the user does not always know the processing content of the processing component. In this embodiment, even in such a case, an appropriate sampling method can be selected by specifying the convergence state of the number of columns as described above. A data profile can be generated with a small amount of sample data.

<Effects of this embodiment>
According to the present embodiment, test data having a plurality of samples is generated for each of a plurality of sampling methods from input data processed by a processing component. Then, the test data is executed by the processing component, and an increase state of the number of columns of the processing result corresponding to the increase in the number of samples is specified for each sampling method. Then, the sampling method in which the increase in the number of columns converges at an earlier stage (that is, a small number of samples) is selected as an appropriate sampling method by the process of step S38. Therefore, an efficient sampling method can be selected, and a data profile of data after processing by the processing component can be acquired with a small processing amount. Since column information can be extracted from the data profile, it is possible to present to the user options for the column name to be processed on the subsequent processing component setting screen that receives data after processing by the processing component. . As a result, the user can select the column name correctly.

  Further, according to the present embodiment, in the process of step S38, not only the requirement that the increase in the number of columns in the current and previous sample number processes is less than or equal to the threshold value, but also the number of columns in the previous and previous sample number processes. Also consider the requirement that the increase in is below the threshold. Thereby, as described above, when there is no regularity in the increase in the number of columns of the processing result due to the sampling method and the processing content of the processing component, a state where the increase in the number of columns of the processing result happens to be reduced, The risk of discriminating that the increase in the number of columns has converged is reduced.

Further, according to the present embodiment, the sampling method candidates include, for example, “random sampling”, “sampling including the maximum value and minimum value”, and “sampling including more differences”. By using a plurality of sampling methods having different properties as candidates, it is possible to deal with various contents of input data and processing contents of processing components.
For example, even when a column is generated according to the data contents of the accounting time as described above, in the case of a processing component that performs processing that does not generate a column of a time zone without corresponding data, the “maximum value and minimum value” The column information can be acquired more efficiently by “sampling including more differences” than “sampling including values”. In the case of “random sampling”, since there is a possibility that a plurality of data of the same time is extracted, convergence of the increase in the number of columns is slower than “sampling including more differences”.

  As another example, the input data includes a column indicating importance, and the data in the column includes values such as “A +, A−, B +, B−, C +, C−”, for example. Suppose that Then, a processing component is assumed that adds a column (“A, B, C” in this data example) in which values included in the data of this column are grouped for each leading character to the input data. In such input data and processing parts, for example, in order to obtain all the column information, for example, in the case of “random sampling”, it is necessary to increase the number of samples until all the first characters are included. The same applies to “sampling including maximum and minimum values”. On the other hand, according to “sampling including more differences”, column information can be obtained efficiently.

  Furthermore, as another example, a process in which input data is a set of words included in predetermined text data, and a sequence of a predetermined number of words is created in order of appearance frequency of words in order to perform clustering analysis of the set of words. Assume parts. In such input data and processing components, for example, when “sampling including more differences” is selected, the appearance frequency of all words becomes “1”, and the number of columns only increases and does not converge. . On the other hand, according to, for example, “random sampling”, the data is close to the original word appearance frequency in the entire text data, and the increase in the number of columns is appropriately converged.

  In the above description, input data having a table structure is processed, and processing is performed based on the convergence state of the increase in the number of columns. However, data targeted by the technique described in this embodiment has such a structure. It is not limited to the data. Further, the structure of the data profile is merely an example, and any structure of data may be used as long as the number of components that vary depending on the content of the input data can be specified.

[Hardware configuration, etc.]
An example of the hardware configuration of the computer functioning as the analysis server 1 described above is shown in FIG. The computer includes a processor 101, a memory 102, a storage 103, a portable storage medium drive device 104, an input / output device 105, and a communication interface 106.
The processor 101 includes a control unit, an arithmetic unit, an instruction decoder, and the like. The execution unit follows the instructions of the program decoded by the instruction decoder, and performs arithmetic / logic using the arithmetic unit according to a control signal output from the control unit. Perform the operation. The processor 101 has a TLB that functions as a control register in which various information used for control is stored, a cache that can temporarily store the contents of the already accessed memory 2 and the like, and a page table cache of virtual memory. Prepare. The processor 101 may have a configuration in which a plurality of CPU (Central Processing Unit) cores are provided.

  The memory 102 is a storage device such as a RAM (Random Access Memory), for example, and is a main memory in which a program to be executed by the processor 101 is loaded and data used for processing of the processor 101 is stored. The storage 103 is a storage device such as an HDD (Hard Disk Drive) or a flash memory, and stores programs and various data. The portable storage medium driving device 104 is a device that reads data and programs stored in the portable storage medium 107. The portable storage medium 107 is, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a flash memory. The processor 101 executes a program stored in the storage 103 or the portable storage medium 107 while cooperating with the memory 102 or the storage 103. Note that the program executed by the processor 101 and data to be accessed may be stored in another device that can communicate with the computer. Note that the storage unit of the analysis server 1 described in the present embodiment indicates at least one of the memory 102, the storage 103, the portable storage medium 107, or another device that can communicate with the computer.

The input / output device 105 is, for example, a keyboard, a touch panel, a display, or the like, and receives an operation command by a user operation or the like, and outputs a processing result by a computer.
The communication interface 106 can include, for example, a radio frequency receiver and transmitter, and an optical receiver and transmitter in addition to a LAN (Local Area Network) card, for example. The aforementioned receivers and transmitters can be implemented to operate with one or more communication networks, such as a Wi-Fi network, a Bluetooth network, and long term evolution.
Each component of these computers is connected by a bus 108.

[Others]
Note that the functional configuration and physical configuration of the computer described in this specification are not limited to the above-described aspects. For example, the functions and physical resources are integrated and implemented, or conversely, are further distributed. It is also possible to implement.
In addition, in this specification, the description places “to be more than” and “to be less than” in comparison with the threshold value and the like are not limited to the description except for special cases, It can be appropriately replaced with “less than” or “less than (less than)”. The reverse is also true.

Regarding the above embodiment, the following additional notes are disclosed.
(Appendix 1)
Obtaining the input data for a target processing component that is included in a processing flow that combines a plurality of processing components, and that performs processing to generate output data having components corresponding to the contents of the input data ,
According to a plurality of preset sampling methods, for each of the plurality of sampling methods, a plurality of different sample number data is sampled from the input data to generate a plurality of test data,
For each of the plurality of sampling methods, while increasing the number of samples of the test data, the processing by the target processing component that receives the test data as input is executed, and the component in the output data of each processing result Identify the number,
Sampling in which an increase in the number of components of the output data in accordance with an increase in the number of samples of the test data is equal to or less than a predetermined threshold when the test data with the smallest number of samples is executed among the plurality of sampling methods An information processing program for causing a computer to execute a process for selecting a method.

(Appendix 2)
The process of selecting the sampling method is performed only when the increase in the number of components of the output data corresponding to the increase in the number of samples of the test data is equal to or less than the predetermined threshold value a plurality of times in succession. The information processing program according to appendix 1, wherein

(Appendix 3)
Sampling data according to the sampling method selected from the input data to generate sample data;
A process for executing processing by the target processing component having the sample data as an input, extracting information on the component from the output data of the processing result, and generating a data profile including information on the component is further performed on the computer The information processing program according to attachment 1 or 2 to be executed.

(Appendix 4)
Refer to the data profile of the output data of the target processing component, and the component of the output data of the target processing component is displayed on the parameter setting screen used in the processing by the subsequent processing component that receives the output data of the target processing component. The information processing program according to appendix 3, further causing the computer to execute a process of displaying the options.

(Appendix 5)
The information processing program according to any one of appendices 1 to 4, wherein the input data is data having a table structure, and the constituent elements are columns having a table structure.

(Appendix 6)
The information according to any one of appendices 1 to 5, wherein the plurality of sampling methods include at least one of random sampling, sampling including a maximum value and minimum value, and sampling method including more differences. Processing program.

(Appendix 7)
Obtaining the input data for a target processing component that is included in a processing flow that combines a plurality of processing components, and that performs processing to generate output data having components corresponding to the contents of the input data ,
According to a plurality of preset sampling methods, for each of the plurality of sampling methods, a plurality of different sample number data is sampled from the input data to generate a plurality of test data,
For each of the plurality of sampling methods, while increasing the number of samples of the test data, the processing by the target processing component that receives each of the test data is executed, and the number of components of each processing result is specified And
Sampling in which an increase in the number of components of the output data in accordance with an increase in the number of samples of the test data is equal to or less than a predetermined threshold when the test data with the smallest number of samples is executed among the plurality of sampling methods An information processing method in which a computer executes a process of selecting a method.

(Appendix 8)
Obtaining the input data for a target processing component that is a target processing component included in a processing flow that combines a plurality of processing components, and that performs processing to generate output data having components according to the contents of the input data A data reading section;
According to a plurality of preset sampling methods, for each of the plurality of sampling methods, a test data generation unit that generates a plurality of test data by sampling a plurality of different sample numbers of data from the input data,
For each of the plurality of sampling methods, while increasing the number of samples of the test data, the processing by the target processing component that receives each of the test data is executed, and the number of components of each processing result is specified A test data processing unit,
Sampling in which an increase in the number of components of the output data in accordance with an increase in the number of samples of the test data is equal to or less than a predetermined threshold when the test data with the smallest number of samples is executed among the plurality of sampling methods An information processing apparatus comprising a sampling method selection unit that selects a method.

DESCRIPTION OF SYMBOLS 1 ... Analysis server, 11 ... Analysis flow reading part, 12 ... Analysis flow change detection part, 13 ... Data profile generation part, 14 ... Test data generation part, 15 ... Test data processing part, 16 ... Sampling method selection part, 17 ... Sample data generation unit, 18 ... Sample data processing unit, 19 ... Analysis flow update unit, 20 ... Column selection setting unit, 21 ... Analysis flow execution unit, 31 ... Analysis flow table, 32 ... Parts database, 33 ... Input data table, 34 ... Sampling method table, 35 ... Test data table, 36 ... Data profile temporary storage file, 37 ... Column number table, 38 ... Sample data table, 39 ... Data profile table

Claims (8)

Obtaining the input data for a target processing component that is included in a processing flow that combines a plurality of processing components, and that performs processing to generate output data having components corresponding to the contents of the input data ,
According to a plurality of preset sampling methods, for each of the plurality of sampling methods, a plurality of different sample number data is sampled from the input data to generate a plurality of test data,
For each of the plurality of sampling methods, while increasing the number of samples of the test data, the processing by the target processing component that receives the test data as input is executed, and the component in the output data of each processing result Identify the number,
Sampling in which the increase in the number of components of the output data in accordance with the increase in the number of samples of the test data is equal to or less than a predetermined threshold at the stage of processing the test data with the smallest number of samples among the plurality of sampling methods An information processing program for causing a computer to execute a process for selecting a method.   The process of selecting the sampling method is performed only when the increase in the number of components of the output data corresponding to the increase in the number of samples of the test data is equal to or less than the predetermined threshold value a plurality of times in succession. The information processing program according to claim 1, wherein the information processing program is selected. Sampling data according to the sampling method selected from the input data to generate sample data;
A process for executing processing by the target processing component having the sample data as an input, extracting information on the component from the output data of the processing result, and generating a data profile including information on the component is further performed on the computer The information processing program according to claim 1, which is executed. Refer to the data profile of the output data of the target processing component, and the component of the output data of the target processing component is displayed on the parameter setting screen used in the processing by the subsequent processing component that receives the output data of the target processing component. The information processing program according to claim 3, further causing the computer to execute a process of displaying the options.   The information processing program according to any one of claims 1 to 4, wherein the input data is data having a table structure, and the constituent elements are columns having a table structure.   6. The method according to claim 1, wherein the plurality of sampling methods include at least one of random sampling, sampling including a maximum value and minimum value, and sampling method including more differences. Information processing program. Obtaining the input data for a target processing component that is included in a processing flow that combines a plurality of processing components, and that performs processing to generate output data having components corresponding to the contents of the input data ,
According to a plurality of preset sampling methods, for each of the plurality of sampling methods, a plurality of different sample number data is sampled from the input data to generate a plurality of test data,
For each of the plurality of sampling methods, while increasing the number of samples of the test data, the processing by the target processing component that receives the test data as input is executed, and the component in the output data of each processing result Identify the number,
Sampling in which the increase in the number of components of the output data in accordance with the increase in the number of samples of the test data is equal to or less than a predetermined threshold at the stage of processing the test data with the smallest number of samples among the plurality of sampling methods An information processing method in which a computer executes a process of selecting a method. Obtaining the input data for a target processing component that is a target processing component included in a processing flow that combines a plurality of processing components, and that performs processing to generate output data having components according to the contents of the input data A data reading section;
According to a plurality of preset sampling methods, for each of the plurality of sampling methods, a test data generation unit that generates a plurality of test data by sampling a plurality of different sample numbers of data from the input data,
For each of the plurality of sampling methods, while increasing the number of samples of the test data, the processing by the target processing component that receives the test data as input is executed, and the component in the output data of each processing result A test data processing unit for identifying the number;
Sampling in which the increase in the number of components of the output data in accordance with the increase in the number of samples of the test data is equal to or less than a predetermined threshold at the stage of processing the test data with the smallest number of samples among the plurality of sampling methods An information processing apparatus comprising a sampling method selection unit that selects a method.