JP6659618B2 - Analysis apparatus, analysis method and analysis program - Google Patents

Description

  The present invention relates to an analysis device, an analysis method, and an analysis program.

  In recent years, application examples of data analysis using machine learning have increased. On the other hand, acquiring statistics and machine learning knowledge, which are essential for data analysis, requires mid- to long-term education. Therefore, a technology that supports data analysis has been disclosed so that a non-expert can easily engage in data analysis without acquiring knowledge of statistics and machine learning.

  For example, there is known a method of evaluating the accuracy of each pipeline using a sequential optimization method (SMBO: Sequential model-based optimization) and searching for an optimal pipeline (for example, Non-Patent Document 1 and 2). Here, the pipeline is a series of processes for constructing a prediction model, and includes preprocessing for input data, learning of data based on hyperparameters, and the like. In addition, there is known a technique of presenting a user with a small number of pipelines suitable for data to be analyzed from a large number of pipelines designed in advance by experts.

  Also, in general, when the number of candidates for the setting contents related to the pre-processing of data constituting the pipeline and the prediction algorithm increases, while the possibility of finding a pipeline capable of constructing a prediction model with higher prediction accuracy increases, The time required for searching increases.

Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, Frank Hutter, “Efficient and Robust Automated Machine Learning”, NIPS'15 Proceedings of the 28th International Conference on Neural Information Processing Systems, December 2015, PP. 2755-2763 Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet Talwalkar, "Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization", arXiv: 1603.06560v3, cs. LG, November 2016

  However, the conventional technology for automating data analysis has a problem in that it may not be possible to construct a prediction model with high prediction accuracy while reducing the time required for searching for a pipeline. For example, it is conceivable to reduce the time required for searching a pipeline simply by reducing the setting contents serving as search candidates, but in this case, sufficient search is not performed, and the accuracy of the prediction model may decrease. .

  The analyzer of the present invention, when data is input, extracts a predetermined ratio of data as sample data from the input data, and when the predetermined ratio increases, the data is increased from the input data. An extracting unit for further extracting a predetermined ratio of data as the sample data, and each time the extracting unit extracts the sample data, a part of a predetermined process is performed using the sample data. Based on the required time, the time required for performing the predetermined processing using the input data is estimated, and if the estimated time is less than a preset time limit, the predetermined ratio And a candidate for a combination of setting contents applied to a plurality of processes executed when constructing a prediction model based on predetermined data, A search unit that searches for a combination of setting contents such that the prediction accuracy of a prediction model based on the sample data constructed when applied to a plurality of processes satisfies a predetermined condition; and a setting content searched by the search unit. And a selection unit that selects a combination of setting contents in which a prediction accuracy of a prediction model based on the input data constructed when applied to the plurality of processes is equal to or more than a predetermined threshold, from combinations of It is characterized by.

  According to the present invention, it is possible to construct a prediction model with high prediction accuracy while reducing the time required for searching for a pipeline.

FIG. 1 is a diagram illustrating an example of a configuration of the analyzer according to the first embodiment. FIG. 2 is a diagram illustrating an example of a data configuration of the setting information according to the first embodiment. FIG. 3 is a diagram illustrating an example of a data configuration of predictor information according to the first embodiment. FIG. 4 is a diagram for explaining an outline of a process performed by the analyzer according to the first embodiment. FIG. 5 is a diagram for describing a search for a pipeline according to the first embodiment. FIG. 6 is a diagram illustrating the cross-validation according to the first embodiment. FIG. 7 is a diagram illustrating a search for a pipeline according to the first embodiment. FIG. 8 is a diagram for explaining a promising area according to the first embodiment. FIG. 9 is a diagram for describing selection of a pipeline according to the first embodiment. FIG. 10 is a diagram for explaining a promising area according to the first embodiment. FIG. 11 is a diagram for explaining the synthesis of the prediction model according to the first embodiment. FIG. 12 is a diagram for describing the synthesis of the prediction model according to the first embodiment. FIG. 13 is a flowchart illustrating the flow of the process of the analyzer according to the first embodiment. FIG. 14 is a flowchart illustrating the flow of the process of the analyzer according to the first embodiment. FIG. 15 is a diagram for explaining the effect of the first embodiment. FIG. 16 is a diagram for explaining the effect of the first embodiment. FIG. 17 is a flowchart illustrating the flow of the process of the analyzer according to the second embodiment. FIG. 18 is a diagram illustrating an example of a computer that executes an analysis program.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited by this embodiment. In the description of the drawings, the same parts are denoted by the same reference numerals.

[First Embodiment]
[Configuration of analyzer]
The configuration of the analyzer 10 will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of a configuration of the analyzer according to the first embodiment. As shown in FIG. 1, the analyzer 10 is realized by a general-purpose computer such as a workstation or a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15. Prepare.

  The input unit 11 is realized using an input device such as a keyboard and a mouse, and inputs various kinds of instruction information to the control unit 15 in response to an input operation by an operator. The output unit 12 is realized by a display device such as a liquid crystal display, a printing device such as a printer, an information communication device, and the like, and outputs a result of data analysis and the like to an operator.

  The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between the control unit 15 and an external device such as a management server via an electric communication line such as a LAN (Local Area Network) or the Internet. .

  The storage unit 14 is realized by a semiconductor memory device such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. In the storage unit 14, a processing program for operating the analyzer 10, data used during execution of the processing program, and the like are stored in advance, or temporarily stored for each processing. The storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13. The storage unit 14 stores the setting information 141 and the predictor information 142.

  Here, the setting information 141 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a data configuration of the setting information according to the first embodiment. As shown in FIG. 2, the setting information 141 includes an execution order for each step and a setting content candidate. The setting content candidate is a candidate for the setting content of the setting item corresponding to each step.

  In the example of FIG. 2, the setting information 141 indicates that there are “missing value complementing method search”, “normalization method searching”, and “feature selection method searching” as steps. These steps correspond to data pre-processing.

  In the example of FIG. 2, the setting information 141 indicates that the step “feature selection method search” is the third step to be executed. Further, the setting information 141 indicates that there are “decision tree”, “L1 regularization”, “variance analysis”, and “no processing” as setting content candidates corresponding to the step “feature selection method search”. Is shown. In the example of FIG. 2, the setting item corresponding to the step “feature selection method search” is a method used in feature selection.

Next, the predictor information 142 will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of a data configuration of predictor information according to the first embodiment. As shown in FIG. 3, the predictor information 142 includes a default parameter for each predictor. The default parameter is a default value of the hyper parameter of each predictor. For example, the predictor information 142, the default value of the hyper-parameter N predictor A ₁ indicates that it is 100.

  As illustrated in FIG. 1, the control unit 15 executes a processing program stored in a memory by an arithmetic processing device such as a CPU (Central Processing Unit), thereby extracting an extraction unit 151, a search unit 152, an estimation unit 154, It functions as the synthesis unit 155 and the verification unit 156. The search unit 152 includes a step selection unit 152a, an accuracy calculation unit 152b, and a setting content determination unit 152c. These functional units may be implemented individually or partially in different hardware.

  When data is input, the extraction unit 151 extracts a predetermined ratio of data as sample data from the input data, and when the predetermined ratio increases, extracts the predetermined ratio of the increased predetermined ratio from the input data. The data is further extracted as sample data. As illustrated in FIG. 4, the extraction unit 151 extracts sample data from learning data, which is all input data. FIG. 4 is a diagram for explaining an outline of a process performed by the analyzer according to the first embodiment. Here, it is assumed that a predetermined ratio at which the extracting unit 151 extracts the sample data, that is, a sampling rate in sampling is set in advance.

  The search unit 152 converts a candidate of a combination of setting contents applied to a plurality of processes executed when constructing a prediction model based on predetermined data from a sample data constructed when applied to a plurality of processes. Search for a combination of setting contents such that the prediction accuracy of the based prediction model satisfies predetermined conditions.

  As illustrated in FIG. 4, the search unit 152 performs a search for each predictor. In addition, the search unit 152 performs a parameter search after determining the validity of the preprocessing, and selects a pipeline with the cross-validation accuracy by the cross-validation.

  Here, the processing of the search unit 152 will be described with reference to FIG. FIG. 5 is a diagram for describing a search for a pipeline according to the first embodiment. Note that the processing in the search unit 152 is performed by the step selection unit 152a, the accuracy calculation unit 152b, and the setting content determination unit 152c.

As illustrated in FIG. 5, first, the search unit 152 determines a pipeline necessary for constructing a prediction model by executing steps 1 to 3 for each predictor. For example, the search unit 152, after performing the steps 1 to 3 for the predictor _{A 1,} executes the parameter search.

  The step selecting unit 152a corresponds to a plurality of processes executed when constructing the prediction model, that is, corresponds to each of the pipelines, and in the step of sequentially determining the setting content of the corresponding process, every time the setting content is determined. , Select the next step to be performed. The setting content determining unit 152c determines the setting content of each step from the setting content candidates included in the setting information 141. At this time, the step selecting unit 152a selects the next step whose setting content is determined according to the execution order indicated in the setting information 141. If none of the steps has been executed, the step selecting unit 152a selects the step whose execution order is the earliest.

  For example, as shown in FIG. 5, since the next step after the step “normalization method search” is “feature selection method search”, when the setting content of the step “normalization method search” is determined, the step selection unit 152a As a next step, "feature selection method search" is selected.

  In addition, the steps “missing value complementing method search”, “normalization method searching”, and “feature selection method searching” in FIG. 5 are missing value complementing, normalizing, and normalizing data pre-processing of learning and analysis data, respectively. This is a step of determining the setting contents of the feature selection. The setting content candidates of the steps “missing value complementing method search”, “normalization method searching”, and “feature selection method searching” are methods used in missing value complementing, normalization, and feature selection, respectively.

  The accuracy calculation unit 152b performs the process of which the setting content is determined among the plurality of processes by applying the determined setting content, and performs the process corresponding to the step selected by the step selection unit 152a as the setting content. The prediction accuracy is calculated for each of the prediction models constructed when each of the candidates is applied.

  For example, when the step “feature selection method search” is selected by the step selection unit 152a, the steps “missing value complementation method search” and “normalization method search” whose execution order is earlier than the step “feature selection method search” Since the setting contents of have already been determined, each of the setting contents determined in the steps “Search for missing value complementing method” and “Search for normalization method” and the candidates for the setting contents in the step “Search for feature selection method” were applied. It is possible to construct a prediction model. At this time, since there are four candidates for the setting contents of the step “feature selection method search”, when the setting contents of the steps “missing value complementing method search” and “normalization method search” are determined to be one, At least four types of prediction models can be constructed.

  Then, the accuracy calculator 152b calculates the prediction accuracy for each of the predictable models that can be constructed. At this time, the setting contents of the steps “search for missing value complementing method” and “search for normalizing method” may be determined in a plurality of ways. For example, when two settings are made for the steps “search for missing value complementing method” and “search for normalizing method”, the number of predictable models that can be constructed is at least eight. The accuracy calculation unit 152b can calculate the prediction accuracy by performing cross-validation using learning data divided into a predetermined number. Here, the cross-validation will be described with reference to FIG. FIG. 6 is a diagram illustrating the cross-validation according to the first embodiment.

  As shown in FIG. 6, first, the accuracy calculation unit 152b divides the sample data into four pieces of divided data 20a, 20b, 20c, and 20d. Then, as the first process, the accuracy calculation unit 152b causes the predictor to learn the divided data 20b, 20c, and 20d using the prediction model, and measures the accuracy of the learned predictor using the divided data 20a. .

  Similarly, in the second process, the accuracy calculation unit 152b causes the predictor to learn the divided data 20a, 20c, and 20d, and measures the accuracy of the learned predictor using the divided data 20b. In the third process, the accuracy calculation unit 152b causes the predictor to learn the divided data 20a, 20b, and 20d, and measures the accuracy of the learned predictor using the divided data 20c. In the fourth process, the accuracy calculation unit 152b causes the predictor to learn the divided data 20a, 20b, and 20c, and measures the accuracy of the learned predictor using the divided data 20d. Then, the accuracy calculation unit 152b sets the cross-validation accuracy, which is the average value of the accuracy measured in the four processes, as the prediction accuracy. Note that the number of divisions in the cross-validation is not limited to four, and may be any number.

  The setting content determination unit 152c compares the prediction accuracy calculated by the accuracy calculation unit 152b, and determines the setting content candidate having the highest prediction accuracy among the setting content candidates in the step selected by the step selection unit 152a. Determine the settings for the corresponding process.

  For example, as shown in FIG. 5, in the step “normalization method search”, the accuracy calculation unit 152b calculates the prediction accuracy of the prediction model corresponding to the setting content “maximum / minimum” as 72%, and sets the setting content “standardization”. The prediction accuracy of the prediction model corresponding to the setting content is calculated as 78%, the prediction accuracy of the prediction model corresponding to the setting content “Z score” is calculated as 72%, and the prediction accuracy of the prediction model corresponding to the setting content “no processing” Was calculated as 70%. At this time, since the prediction model having the highest prediction accuracy in the step “normalization method search” is the prediction model corresponding to the setting content “standardization”, the setting content determination unit 152c corresponds to the step “normalization method search”. Set the settings of the setting items to "Standardized". That is, the setting content determination unit 152c determines the method used in the normalization which is the pre-processing of the data to the standardization.

  Then, as described above, the step selecting unit 152a selects a step to be executed next to the step whose setting content is determined by the setting content determining unit 152c. For example, when the setting content in the step “normalization method search” is determined by the setting content determination unit 152c, the step selection unit 152a selects the step “feature selection method search”.

  Further, the search unit 152 performs a parameter search. In the parameter search, the search unit 152 comprehensively calculates the prediction accuracy for each combination of the hyperparameters, and searches for the best parameter with the highest prediction accuracy.

The search unit 152 finally searches the pipeline such that the prediction accuracy of the prediction model satisfies a predetermined condition. For example, when the threshold of the prediction accuracy is set to 70% and only the pipelines whose prediction accuracy is equal to or more than the threshold are searched, as illustrated in FIG. 7, the search unit 152 sets the pipelines P _{1 to} P ₃ as the search result. . FIG. 7 is a diagram illustrating a search for a pipeline according to the first embodiment.

  The search result of the hyperparameter by the search unit 152 can be represented as an area as shown in FIG. FIG. 8 is a diagram for explaining a promising area according to the first embodiment. For example, the vertical axis of FIG. 8A represents the first hyperparameter, and the horizontal axis represents the second parameter. FIG. 8A shows that the higher the density of the pattern, the higher the prediction accuracy.

  At this time, the search unit 152 can set, as a search result, not only a pipeline having a prediction accuracy of 70% or more, but also a pipeline having a prediction accuracy of 70% and a value having no statistically significant difference. . That is, the search unit 152 sets the combination of the setting contents such that the prediction accuracy of the prediction model based on the sample data constructed when applied to a plurality of processes is 70% or more, and sets the prediction accuracy to 70% or more. A combination of setting contents that does not have a statistically significant difference from a certain prediction accuracy is searched. Note that 70% here is an example of the second threshold value. The area corresponding to the combination of the hyperparameters corresponding to the search result in this case is called a promising area.

  Here, FIG. 8A shows a promising region of the prediction accuracy when the sample data is used. FIG. 8B illustrates a promising region of the prediction accuracy when the learning data from which the sample data is extracted is used. As shown in FIG. 8, the two promising regions are similar. The promising area is used in the processing by the selection unit 153.

  The selecting unit 153 sets, from the combination of the setting contents searched by the searching unit 152, the setting contents in which the prediction accuracy of the prediction model based on the input data constructed when applied to a plurality of processes is equal to or more than a predetermined threshold value Select the combination of

  That is, while the search unit 152 evaluates the prediction accuracy of the prediction model using the sample data, the selection unit 153 evaluates the prediction accuracy of the prediction model using the learning data. At this time, the selection unit 153 further narrows down the prediction accuracy for each pipeline searched by the search unit 152 using a threshold.

Here, the selection unit 153 may simply select a pipeline in which each prediction accuracy is equal to or more than a threshold, or select a pipeline in which a value obtained by dividing each prediction accuracy by the maximum prediction accuracy is equal to or more than the threshold. Good. For example, as shown in FIG. 9, when the prediction accuracy of the pipelines P ₁ , P ₂ , and P ₃ is 85%, 70%, and 83%, the maximum prediction accuracy is 85%. FIG. 9 is a diagram for describing selection of a pipeline according to the first embodiment.

In this case, selection section 153 compares the 85/85 and the threshold for the pipeline _{P 1} is compared with 83/85 and the threshold for the pipeline _{P 2,} the 70/85 and the threshold for the pipeline _{P 1} The pipeline can be selected using the comparison result.

  Further, the selection unit 153 fixes the setting contents regarding the pre-processing of the data of the selected combination, and performs a hyperparameter search. Here, assuming that the hyperparameter with the highest prediction accuracy as a result of the search by the search unit 152 is the best parameter 210, the promising area can be represented as shown in FIG. FIG. 10 is a diagram for explaining a promising area according to the first embodiment.

  The shaded portions in FIG. 10 represent promising regions. The hatched portion in FIG. 10 is an area that is a target of the hyperparameter search by the search unit 152 but is excluded from a target of the hyperparameter search by the selection unit 153. Note that the search unit 152 can determine the presence or absence of a significant difference by a t-test using prediction accuracy in the case of the best parameter 210.

  Each time the extraction unit 151 extracts the sample data, the estimation unit 154 uses the input data based on the time required to perform a part of a predetermined process using the sample data. The time required for performing the predetermined process is estimated, and if the estimated time is less than a preset time limit, the predetermined ratio is increased. The processing by the estimation unit 154 is performed at an arbitrary timing during the execution of the processing of the search unit 152, before the execution of the processing of the search unit 152, or after the execution of the processing of the search unit 152, and when the processing of the selection unit 153 is executed. It may be performed before it is performed. For example, the estimating unit 154 can estimate the processing time of the searching unit 152 and the selecting unit 153 when the prediction accuracy of the “average value” for the step “missing value complementing method search” in FIG. 5 is calculated. .

First, it is assumed that the amount of increase in the amount of calculation for each predictor when the number of data increases is known. For example the increase of the calculation amount predictor A ₁ is assumed to be O (n ^2). It is also assumed that the sample rate is 20% and the time required for calculating the prediction accuracy of the “average value” in the step “missing value complementing method search” is 100 seconds. As shown in FIG. 5, steps are three, because each step has a candidate set of parameters for each four, estimation unit 154, the processing time of the search section 152 for the predictor A _1, wherein Calculate as in (1).
100 (seconds) x 3 (steps) x 4 (settings) = 1200 (seconds) ... (1)

Further, the selecting unit 153, 100/20 times the search section 152, that is, the 5 times the data is of interest, estimating section 154, the processing time of the selection unit 153 of the predictor _{A 1,} equation (2) Calculate as follows.
100 (seconds) x 5 ² (increase in calculation amount) x 3 (steps) x 1 (settings)
= 7500 (seconds) ... (2)

  Here, the subsequent processing differs depending on whether or not the processing time (total of (1) and (2)) calculated by the estimating unit 154 is shorter than a predetermined time limit. When the required time estimated by the estimating unit 154 is shorter than a predetermined time limit, the searching unit 152 performs a further search using the sample data extracted by the extracting unit 151 at a rate higher than a predetermined rate. .

  On the other hand, when the required time estimated by the estimating unit 154 is equal to or longer than the time limit, the searching unit 152 performs a search excluding a prediction algorithm having a similar prediction method. At this time, the search unit 152 excludes the prediction algorithm until the estimated time becomes less than the time limit.

  Specifically, first, the predictor information 142 stores a prediction algorithm, that is, a category for each predictor. Then, when the required time estimated by the estimating unit 154 is equal to or longer than the time limit, the searching unit 152 refers to the predictor information 142 and excludes any one of the plurality of predictors having the same category. For example, the search unit 152 excludes the predictors in descending order of the execution time based on information about the execution time set in advance.

For example, predictor _{A 1} a Random Forest in FIG. 3, the predictor _{A 2} the Extra Random Trees, predictor _{A 3} a Logistic Regression, the _{A 4} and Linear SVM. Then, the predictor information 142, "decision tree category" is set as the category of A ₁ and A _2, and the "linear predictor category" is set as the category of A ₃ and A _4. Further, it is assumed that the order of the execution time lengths is set to A ₂ , A ₁ , A ₃ , and A ₄ in the predictor information 142.

At this time, when the required time estimated by the estimating unit 154 is equal to or longer than the time limit, the searching unit 152 first sets A ₂ in which a plurality of predictors are set in the same category and the execution time is the longest. Exclude Here, if be excluded A ₁ is estimated to be required time is the time limit or more, the search unit 152, among the remaining predictor, a plurality of predictor is set to the same category, and, most execution time excludes a long a _3. The search unit 152 repeats the elimination of predictors until there is no category in which a plurality of predictors are set, or until the required time is not estimated to be longer than the time limit.

When a combination of a plurality of setting contents is selected by the selection unit 153, the combining unit 155 combines a plurality of prediction models corresponding to the respective combinations of the plurality of setting contents using a predetermined ensemble technique. For example, as shown in FIG. 11, if the two pipelines is selected by the selection unit 153, the two predictive models Model A ₁ and Model A ₃ is possible construction. FIG. 11 is a diagram for explaining the synthesis of the prediction model according to the first embodiment. In this case, the combining unit 155 by a predetermined ensemble technique, each synthesized weighted _{w 1} and the weight _{w 2} in Model A ₁ and Model A _3.

Furthermore, as illustrated in FIG. 12, the combining unit 155 sets the prediction model having the largest generalization performance among the Model A ₁ , the Model A ₃ , and the combined Ensemble Model as the best model. FIG. 12 is a diagram for describing the synthesis of the prediction model according to the first embodiment.

  The verification unit 156 verifies a prediction model constructed based on the pipeline selected by the selection unit 153. When a pipeline is selected by the selection unit 153, the verification unit 156 causes the predictor to learn learning data based on the determined pipeline, and constructs a prediction model. Then, the verification unit 156 uses the test data different from the learning data to measure the prediction accuracy of the constructed prediction model as the test accuracy. For example, the analyzer 10 may use the test accuracy measured here as a final output. In addition, by performing verification using test data different from the learning data, it is possible to confirm the overlearned state and the unlearned state.

[Processing of First Embodiment]
The processing flow of the analyzer 10 according to the first embodiment will be described with reference to FIGS. FIGS. 13 and 14 are flowcharts showing the flow of the process of the analyzer according to the first embodiment. As shown in FIG. 13, first, the analyzer 10 reads the learning data (step S101). Next, the extraction unit 151 extracts sample data of a predetermined sample size from the learning data (Step S102).

  Next, when there is an unselected predictor (Step S103, Yes), the search unit 152 refers to the predictor information 142 and selects the next predictor (Step S104). Next, the search unit 152 determines a pipeline of the selected predictor using the read learning data (step S105). The search unit 152 may determine a plurality of pipelines.

  Next, the process in which the search unit 152 determines a pipeline (step S105 in FIG. 13) will be described in detail with reference to FIG. As illustrated in FIG. 14, when there is an unselected step (Step S201, Yes), the step selecting unit 152a refers to the setting information 141 and selects the next step (Step S202). The next step is a step having the earliest execution order among unselected steps. In this embodiment, each step corresponds to each of the data pre-processing. On the other hand, when there is no unselected step (Step S201, No), the analyzer 10 searches for a hyper parameter (Step S207).

  When there is an unselected setting content among the setting content candidates of the step selected by the step selecting unit 152a (Step S203, Yes), the accuracy calculation unit 152b selects the next setting content (Step S204). On the other hand, when there is no unselected setting content (No at Step S203), the setting content determining unit 152c determines the setting content having the highest prediction accuracy calculated by the accuracy calculating unit 152b in the step selected by the step selecting unit 152a. The content is determined (step S206).

  When the setting content is selected, the accuracy calculation unit 152b calculates the prediction accuracy of the prediction model constructed based on the pipeline to which the selected setting content is applied (Step S205). At this time, the accuracy calculation unit 152b can calculate the prediction accuracy by cross-validation using the learning data divided into a predetermined number. Then, the accuracy calculation unit 152b repeats the processing of steps S203 to S205 until there is no unselected setting content.

  Returning to FIG. 13, when there is no unselected predictor (No at Step S103), the estimating unit 154 estimates the entire processing time (Step S106). If the estimated time is less than the time limit (step S107, Yes), the estimating unit 154 increases the sample rate (step S108). Then, the extraction unit 151 extracts the sample data again at the increased sample rate (Step S102). Thereafter, the analyzer 10 repeats the processing until the time estimated by the estimating unit 154 is not less than the time limit.

  On the other hand, when the time estimated by the estimating unit 154 is not less than the time limit (step S107, No), the searching unit 152 reduces the prediction algorithm (step S109). Then, the selecting unit 153 selects a pipeline whose prediction accuracy is equal to or larger than the threshold from the pipeline determined by the searching unit 152, and further performs a hyperparameter search for the selected pipeline (step S110). Then, the verification unit 156 constructs a prediction model based on the selected pipeline (Step S111), and verifies the constructed prediction model using the test data (Step S112).

[Effect of First Embodiment]
When data is input, the extraction unit 151 extracts a predetermined ratio of data as sample data from the input data, and when the predetermined ratio increases, extracts the predetermined ratio of the increased predetermined ratio from the input data. The data is further extracted as sample data. Each time the extraction unit 151 extracts the sample data, the estimation unit 154 uses the input data based on the time required to perform a part of a predetermined process using the sample data. The time required for performing the predetermined process is estimated, and if the estimated time is less than a preset time limit, the predetermined ratio is increased. In addition, the search unit 152 extracts a sample constructed when applied to a plurality of processes from a candidate of a combination of setting contents applied to a plurality of processes executed when constructing a prediction model based on predetermined data. A search is made for a combination of setting contents such that the prediction accuracy of the prediction model based on the data satisfies a predetermined condition. Further, the selecting unit 153 determines, from the combination of the setting contents searched by the searching unit 152, that the prediction accuracy of the prediction model based on the input data constructed when applied to a plurality of processes is equal to or greater than a predetermined threshold. Select a combination of settings. In this way, by roughly determining a pipeline with high prediction accuracy using sample data and selecting a final pipeline using all data, the time required for searching the pipeline is reduced. It is possible to construct a prediction model with high prediction accuracy while shortening.

  FIG. 15 shows the prediction accuracy of the prediction model constructed by the analysis of the same execution time when auto-skearn is used and when this embodiment is used. FIG. 15 is a diagram for explaining the effect of the first embodiment. As shown in FIG. 15, the prediction accuracy was higher in the present embodiment for eight of the twelve types of data. In addition, the same prediction accuracy was obtained for the remaining four types.

  FIG. 16 shows the execution time and the prediction accuracy in the case where the processing of the search unit 152 and the selection unit 153 is performed without extracting the sample (automation rule), and in the case of the present embodiment. FIG. 16 is a diagram for explaining the effect of the first embodiment. As shown in FIG. 16, in the present embodiment, it was possible to reduce the processing time by about 70% without substantially lowering the prediction accuracy.

  When the required time estimated by the estimating unit 154 is equal to or longer than the time limit, the searching unit 152 performs a search excluding a prediction algorithm having a similar prediction method. As a result, when there is not enough time, the processing time can be further reduced, and when there is enough time, a prediction model with higher prediction accuracy can be constructed.

  The search unit 152 is a combination of setting contents such that the prediction accuracy of a prediction model based on sample data constructed when applied to a plurality of processes is equal to or greater than a second threshold, and the prediction accuracy and the second threshold A combination of setting contents that does not have a statistically significant difference from the prediction accuracy described above is searched. As a result, it is possible to search for an optimal hyper parameter in a short time.

  When a combination of a plurality of setting contents is selected by the selection unit 153, the combining unit 155 combines a plurality of prediction models corresponding to the respective combinations of the plurality of setting contents using a predetermined ensemble technique. This makes it possible to construct a prediction model with higher prediction accuracy than in a case where each pipeline is selected alone.

[Second embodiment]
In the first embodiment, the case has been described where the estimation unit 154 estimates the processing time after the search unit 152 determines the pipeline. Here, as described above, the processing by the estimation unit 154 may be performed at any timing during the execution of the processing of the search unit 152 or before the execution of the processing of the search unit 152. In the second embodiment, the processing time is estimated by the estimating unit 154 before the processing regarding the determination of the pipeline by the searching unit 152 is started.

[Processing of the Second Embodiment]
The processing flow of the analyzer 10 according to the second embodiment will be described with reference to FIG. FIG. 17 is a flowchart illustrating the flow of the process of the analyzer according to the second embodiment. As shown in FIG. 17, first, the analyzer 10 reads the learning data (step S301). Next, the extraction unit 151 extracts sample data of a predetermined sample size from the learning data (Step S302).

  Next, when there is an unselected predictor (step S303, Yes), the estimating unit 154 refers to the predictor information 142 and selects the next predictor (step S304). Next, the estimating unit 154 actually measures the execution time in the sample data, and predicts the execution time in the learning data (step S305).

  Here, the estimating unit 154 actually performs the cross-validation using the sample data, and measures the time required at that time. Specifically, the estimation unit 154 performs the same processing as the processing by the accuracy calculation unit 152b as shown in FIG. However, since the estimation unit 154 aims to measure the time required for the processing, it is not necessary to calculate the prediction accuracy.

  Further, the accuracy calculation unit 152b performs pre-processing on the sample data, and then makes the predictor learn the sample data. On the other hand, the estimating unit 154 does not perform preprocessing on the sample data, and directly makes the learning device learn. Further, the estimating unit 154 estimates the execution time in the learning data using the above-described equations (1) and (2).

  The estimating unit 154 sequentially selects predictors, and executes step S305. When there is no unselected predictor (No at Step S303), the estimating unit 154 estimates the time of the entire process (Step S306). When the estimated time is less than the time limit (step S307, Yes), the estimating unit 154 increases the sample rate (step S308). Then, the extraction unit 151 extracts the sample data again at the increased sample rate (Step S302). Thereafter, the analyzer 10 repeats the processing until the time estimated by the estimating unit 154 is not less than the time limit.

  On the other hand, when the time estimated by the estimating unit 154 is not less than the time limit (No at Step S307), the searching unit 152 reduces the prediction algorithm (Step S309). Next, when there is an unselected predictor (step S310, Yes), the search unit 152 refers to the predictor information 142 and selects the next predictor (step S311). Next, the search unit 152 determines a pipeline of the selected predictor by using the read learning data (step S312). The search unit 152 may determine a plurality of pipelines. The process in which the search unit 152 determines a pipeline is the same as the process illustrated in FIG.

  If there is no unselected predictor (No at Step S310), the selecting unit 153 selects a pipeline whose prediction accuracy is equal to or larger than a threshold from the pipelines determined by the searching unit 152, and further selects a pipeline with respect to the selected pipeline. Is performed (step S313). Then, the verification unit 156 constructs a prediction model based on the selected pipeline (Step S314), and verifies the constructed prediction model using the test data (Step S315).

[Other Embodiments]
When the feature amount of the input data has a predetermined data characteristic, the search unit 152 performs a search from the setting content combination candidates that are previously associated with the predetermined data characteristic among the setting content combination candidates. You may. For example, in the case of data such as text data in which the proportion of zero in the data is large, that is, when the data is sparse, sufficient accuracy can often be obtained even with a linear predictor. Therefore, when the learning data or the sample data is sparse, the search unit 152 excludes a non-linear predictor from the predictors. For example, the search unit 152, a sparse feature quantity ratio of zero is equal to or greater than the threshold r _f, when the ratio of total is the threshold value r _a more, determines that the data is sparse. Threshold _{r f} and _{r a} can be, for example, 0.9.

[System configuration, etc.]
Each component of each device illustrated is a functional concept and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed / arbitrarily divided into arbitrary units according to various loads and usage conditions. Can be integrated and configured. Further, all or any part of each processing function performed by each device can be realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware by wired logic.

  Further, of the processes described in the present embodiment, all or a part of the processes described as being performed automatically can be manually performed, or the processes described as being performed manually can be performed. All or part can be performed automatically by a known method. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above-mentioned documents and drawings can be arbitrarily changed unless otherwise specified.

[program]
As one embodiment, the analyzer 10 can be implemented by installing an analysis program for performing the above analysis as package software or online software on a desired computer. For example, by causing the information processing device to execute the analysis program, the information processing device can function as the analysis device 10. The information processing apparatus referred to here includes a desktop or notebook personal computer. In addition, the information processing apparatus includes mobile communication terminals such as a smartphone, a mobile phone, and a PHS (Personal Handyphone System), and a slate terminal such as a PDA (Personal Digital Assistant).

  Further, the analysis device 10 can be implemented as an analysis server device that provides a terminal device used by a user as a client and provides the client with the above-described analysis service. For example, the analysis server device is implemented as a server device that provides an analysis service that inputs learning data and outputs a pipeline or a prediction model. In this case, the analysis server device may be implemented as a Web server, or may be implemented as a cloud that provides the above-described analysis services by outsourcing.

  FIG. 18 is a diagram illustrating an example of a computer that executes an analysis program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

  The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to the display 1130, for example.

  The hard disk drive 1090 stores, for example, the OS 1091, the application program 1092, the program module 1093, and the program data 1094. That is, a program that defines each process of the analyzer 10 is implemented as a program module 1093 in which codes executable by a computer are described. The program module 1093 is stored in, for example, the hard disk drive 1090. For example, a program module 1093 for executing the same processing as the functional configuration in the analyzer 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD.

  The setting data used in the processing of the above-described embodiment is stored as the program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 and the program data 1094 stored in the memory 1010 and the hard disk drive 1090 to the RAM 1012 as needed, and executes them.

  The program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read out by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (LAN, Wide Area Network (WAN), or the like). Then, the program module 1093 and the program data 1094 may be read from another computer by the CPU 1020 via the network interface 1070.

Reference Signs List 10 analyzer 11 input unit 12 output unit 13 communication control unit 14 storage unit 15 control unit 141 setting information 142 predictor information 151 extraction unit 152 search unit 152a step selection unit 152b accuracy calculation unit 152c setting content determination unit 153 selection unit 154 estimation Unit 155 Synthesis unit 156 Verification unit

Claims (8)

When data is input, a predetermined ratio of data is extracted as sample data from the input data, and when the predetermined ratio is increased, the increased predetermined ratio of data is extracted from the input data. An extraction unit for further extracting as sample data,
Each time the sample data is extracted by the extraction unit, using the sample data, for each prediction algorithm in which a category is set in advance , the execution time required when performing a part of predetermined processing is reduced. Based on the input data, the time required for performing the predetermined process is estimated, and when the estimated time is shorter than a preset time limit, the predetermined ratio is increased. An estimator,
In the case where the estimated time is equal to or longer than the time limit, the prediction algorithm is excluded from a plurality of prediction algorithms in which the same category is set, according to a preset order based on the execution time. In the above, the setting contents applied to a plurality of processes executed when constructing a prediction model based on predetermined data, wherein the plurality of processes are selected from candidates of a combination of setting contents including selection of the prediction algorithm. A search unit that searches for a combination of setting contents such that the prediction accuracy of the prediction model based on the sample data constructed when applied to the predetermined condition is satisfied,
From the combination of setting contents searched by the search unit, a combination of setting contents where the prediction accuracy of a prediction model based on the input data constructed when applied to the plurality of processes is equal to or more than a predetermined threshold value is determined. A selection section to be selected,
An analyzer comprising: When data is input, a predetermined ratio of data is extracted as sample data from the input data, and when the predetermined ratio is increased, the increased predetermined ratio of data is extracted from the input data. An extraction unit for further extracting as sample data,
Each time the sample data is extracted by the extraction unit, using the sample data, for each prediction algorithm in which a category is set in advance, the execution time required when performing a part of predetermined processing is reduced. Based on the input data, the time required for performing the predetermined process is estimated, and if the estimated time is less than a preset time limit, the predetermined ratio is increased. An estimator,
In the case where the estimated time is equal to or longer than the time limit, the prediction algorithm is excluded from a plurality of prediction algorithms in which the same category is set, according to a preset order based on the execution time. Above, the setting contents applied to a plurality of processes performed when constructing a prediction model based on predetermined data, each of the selection of the prediction algorithm, preprocessing for input data, and hyperparameters A search unit that searches for a combination of setting contents such that the prediction accuracy of a prediction model based on the sample data constructed when applied to the plurality of processes satisfies a predetermined condition, from candidates of the combination of setting contents. ,
From the combination of setting contents searched by the search unit, a combination of setting contents where the prediction accuracy of a prediction model based on the input data constructed when applied to the plurality of processes is equal to or more than a predetermined threshold value is determined. A selection section to be selected,
An analyzer comprising: The search unit, when the feature amount of the input data has a predetermined data characteristic, among the candidates for the combination of the setting contents, from the candidates for the combination of the setting contents previously associated with the predetermined data characteristic. analyzer according to claim 1 or 2, characterized in that to search.   When a combination of a plurality of setting contents is selected by the selection unit, a combination unit for combining a plurality of prediction models corresponding to each of the plurality of combination of setting contents by using a predetermined ensemble method. The analyzer according to any one of claims 1 to 3, characterized in that: The search unit,
In the step of sequentially determining the setting content of the corresponding process corresponding to each of the plurality of processes executed when constructing the prediction model, each time the setting content is determined, the step to be executed next is selected. ,
Among the plurality of processes, the setting content is determined by applying the determined setting content, and the process corresponding to the next step is performed by applying each of the setting content candidates. Calculate the prediction accuracy for each of the prediction models built when performed
The calculated prediction accuracy is compared, and among the setting content candidates, the setting content candidate having the highest prediction accuracy is determined as the setting content of the process corresponding to the next executed step. analyzer according to claim 1 or 2. An analysis method performed by the analysis device,
When data is input, a predetermined ratio of data is extracted as sample data from the input data, and when the predetermined ratio is increased, the increased predetermined ratio of data is extracted from the input data. An extraction step for further extracting as sample data ,
Each time the sample data is extracted by the extraction step, using the sample data, for each prediction algorithm in which a category is set in advance, the execution time required when performing a part of predetermined processing is reduced. Based on the input data, the time required for performing the predetermined process is estimated, and when the estimated time is shorter than a preset time limit, the predetermined ratio is increased. The estimation process;
In the case where the estimated time is equal to or longer than the time limit, the prediction algorithm is excluded from the plurality of prediction algorithms in which the same category is set according to a preset order based on the execution time. In the above, the setting contents applied to a plurality of processes executed when constructing a prediction model based on predetermined data, wherein the plurality of processes are selected from candidates of a combination of setting contents including selection of the prediction algorithm. A search step of searching for a combination of setting contents such that the prediction accuracy of the prediction model based on the sample data constructed when applied to the predetermined condition is satisfied;
From the combination of setting contents searched in the search step, a combination of setting contents in which the prediction accuracy of a prediction model based on the input data constructed when applied to the plurality of processes is equal to or more than a predetermined threshold is determined. A selection process to select;
An analysis method comprising: An analysis method performed by the analysis device,
When data is input, a predetermined ratio of data is extracted as sample data from the input data, and when the predetermined ratio is increased, the increased predetermined ratio of data is extracted from the input data. An extraction step for further extracting as sample data,
Each time the sample data is extracted by the extraction step, using the sample data, for each prediction algorithm in which a category is set in advance, the execution time required when performing a part of predetermined processing is reduced. Based on the input data, the time required for performing the predetermined process is estimated, and if the estimated time is less than a preset time limit, the predetermined ratio is increased. The estimation process;
In the case where the estimated time is equal to or longer than the time limit, the prediction algorithm is excluded from a plurality of prediction algorithms in which the same category is set, according to a preset order based on the execution time. Above, the setting contents applied to a plurality of processes performed when constructing a prediction model based on predetermined data, each of the selection of the prediction algorithm, preprocessing for input data, and hyperparameters A search step of searching for a combination of setting contents such that the prediction accuracy of a prediction model based on the sample data constructed when applied to the plurality of processes satisfies a predetermined condition, from candidates of the combination of setting contents. ,
From the combination of setting contents searched in the search step, a combination of setting contents in which the prediction accuracy of a prediction model based on the input data constructed when applied to the plurality of processes is equal to or more than a predetermined threshold is determined. A selection process to select;
An analysis method comprising:   An analysis program for causing a computer to function as the analysis device according to any one of claims 1 to 5.

Images