JP2021012734A - System for analyzing prediction data, and method and device related thereto - Google Patents

Abstract

To provide a system for analyzing prediction data, and a method and a device related to the system.SOLUTION: A technique for analyzing prediction data uses a statistical learning method to develop, select, and/or understand a prediction model for a prediction problem. The technique for analyzing predicted data may include a technique improved to generate a prediction model for a time-series prediction problem, a technique improved to determine a prediction value of an input variable ("feature"), and a technique improved to generate a secondary model of a primary prediction model.SELECTED DRAWING: Figure 1

Description

(Mutual Reference of Related Applications) This application was filed on October 21, 2016, and is entitled "Systems and Technologies for Determining the Presidentive Value of a Feature", US Patent Application No. 15 / 331,797 (agent). Control No. DRB-001C1CP), US Provisional Patent Application No. 62 / 411,526 (Agent Control No. DRB-002PR), filed October 21, 2016, entitled "Systems and Technologies for Predictive Data Analytics". In connection with, each of these is incorporated herein by reference to the maximum extent permitted by patent law.

The present disclosure generally relates to systems and techniques for data analysis. Some embodiments specifically relate to systems and techniques for using statistical learning methods to develop, select, and / or understand predictive models for predictive problems.

Many organizations and individuals use electronic data to improve their operations or to assist in their decision making. For example, many companies use data management techniques to improve the efficiency of various business processes, such as executing transactions, tracking inputs and outputs, or bringing products to market. As another example, many businesses use operational data to assess the performance of a business process, to measure the effectiveness of efforts to improve the process, or to determine how to adjust the process. ..

In some cases, electronic data can be used to predict problems or opportunities. Some organizations combine operational data that represents what happened in the past with assessments that represent subsequent values of performance metrics for building forecast models. Based on the outcomes predicted by the forecasting model, organizations can make decisions, regulate processes, or take other steps. For example, an insurer may attempt to build a predictive model that more accurately predicts future claims, or a predictive model that predicts when policyholders are considering switching to competing insurers. Automakers may try to build predictive models that more accurately predict the demand for new car models. Fire departments may attempt to build predictive models that predict high-risk days or structures at risk of fire.

Machine learning techniques (eg, monitored statistical learning techniques) may be used to generate predictive models from datasets containing pre-recorded observations of at least two variables. The predicted variable can be referred to as the "target," "response," or "dependent variable." The remaining variables that can be used to make predictions can be referred to as "features," "predictors," or "independent variables." Observations are generally divided into at least one "training" dataset and at least one "test" dataset. The data analyst then selects a statistical learning procedure and runs it on the training dataset to generate a predictive model. The analyst then tests the model generated on the test dataset to determine the predictive goodness of the model's target values against actual observations of the target.

Motivation for some embodiments

Data analysts can use analytical techniques and computational infrastructure to build predictive models from electronic data, including operational and evaluation data. Data analysts generally use one of two approaches to build predictive models. Using the first approach, organizations dealing with predictive problems simply use packaged predictive modeling solutions that have already been developed for identical or similar predictive problems. This "mass production" approach is inexpensive, but generally feasible only for a small number of forecasting problems common to a relatively large number of organizations (eg, fraud detection, revolving trading management, market response, etc.). Using the second approach, a team of data analysts builds a customized predictive modeling solution for predictive problems. This "expert" approach is generally expensive and time consuming, and therefore tends to be used for a small number of high-value forecasting problems.

The space for potential predictive modeling solutions for predictive problems is generally large and complex. Statistical learning techniques are based on many academic traditions (eg, mathematics, statistics, physics, engineering, economics, sociology, biology, medicine, artificial intelligence, data mining, etc.) and in many commercial disciplines. Affected by use (eg, finance, insurance, retail, manufacturing, medical, etc.). As a result, there are many different predictive modeling algorithms that can have many variants and / or adjustment parameters, as well as different pre- and post-treatment steps using their own variants and / or parameters. The amount of potential predictive modeling solutions (eg, combinations of pre-processing steps, modeling algorithms, and post-processing steps) is already quite large and is growing rapidly as researchers develop new techniques.

Given this vast space of predictive modeling techniques, expert approaches to generating predictive models tend to be time consuming and leave most of the modeled search space unexplored. Analysts tend to explore the modeled space in an ad hoc manner, based on their intuition or previous experience, and extensive trial and error testing. They may not pursue some potentially useful search tools, or may not adjust their search appropriately in response to the results of the initial effort. Moreover, the scope of trial and error testing tends to be limited by the analyst's time constraints, such that the expert approach generally explores only a small portion of the modeled search space.

The expert approach can also be very expensive. Developing forecast models through an expert approach often involves a large investment in computational resources and high-paying data analysts. Given these high costs, organizations can often be cheap, but include only a small portion of this vast forecast modeling space (eg, an acceptable solution to a defined forecast problem). In favor of mass production approaches, which tend to explore a priori (part of the modeling space), refrain from expert approaches. The mass production approach can generate predictive models that work imperfectly for unexplored options.

There is a need for tools to systematically and cost-effectively evaluate the space of potential predictive modeling techniques for predictive problems. In many respects, the traditional approach to generating predictive models is similar to exploring valuable resources (eg, oil, gold, minerals, gems, etc.). Exploration can lead to some valuable discoveries, but is far less efficient than geological surveys combined with carefully planned exploratory mining or drilling based on an extensive library of previous results. We recognize and understand that statistical learning techniques can be used to statistically and cost-effectively evaluate the space of potential predictive modeling solutions for predictive problems.

Time series forecast modeling

Many prediction problems are based on the value of one or more input variables (“features”) in one or more past times, and one or more in future times one or more. It raises the question of predicting the value of an output variable (“target”) that exceeds it. Such a prediction problem may be referred to as a "time series prediction problem", and a prediction model that models such a problem may be referred to as a "time series prediction model" or a "time series model".

Techniques are needed to rigorously and efficiently search the modeled search space for time series models. We find that rigorous and efficient exploration of time-series modeling search spaces, including efficient training, testing, and comparison of time-series models, is one aspect of time-series modeling procedures, such as models. The amount of training data used to train, the time interval between observations of input variables, the length of the time cycle covered by the training data, the recentity of the time cycle covered by the training data, provided to the model The time cycle between the time associated with the feature value to be associated with the target value predicted by the model and the time associated with it (“skip range”), and the time cycle in which the model predicts the target value (“expected range”). Recognizes and understands that can be facilitated by explicitly parameterizing.

In general, one innovative aspect of the subject matter described herein is (a) the step of retrieving time-series data, including one or more datasets, where each dataset is plural. Each observation, including observations, includes (1) an indication of the time associated with the observation and (2) a step and (b) the time of the time-series data, including the individual values of one or more variables. By (d) time-series data, a step of determining the interval, (c) identifying one or more of the variables as targets and identifying other variables of zero or more as features. The step of determining the expected and skip ranges associated with the predicted problem represented, where the predicted range indicates the duration of the cycle in which the target value is predicted, and the skip range is the oldest within the predicted range. A step that shows the time delay between the time associated with the prediction of and the time associated with the latest observation based on the prediction within the prediction range, and (e) the step of generating training data from the time series data. The training data includes a first subset of observations of at least one of the datasets, the first subset of observations containing training inputs and training output sets of observations, and within the training inputs and training output sets. The time associated with the observation corresponds to the training input time range and the training output time range, respectively, and the skip range separates the beginning of the training output time range from the end of the training input time range and the duration of the training output time range. Is a step that is at least as long as the expected range and (f) a step of generating test data from time series data, where the test data is a second subset of observations of at least one of the datasets. The second subset of inclusions and observations includes the test inputs and test validation sets of the observations, and the time associated with the observations within the test inputs and test validation sets corresponds to the test entry time range and the test validation time range, respectively. The skip range separates the start of the test validation time range from the end of the test input time range, and the duration of the test validation time range is at least as long as the expected range, steps and (g) training the predictive model. It can be embodied in a predictive modeling method that includes performing a predictive modeling procedure, including a step of fitting the data and (h) a step of testing the fit model on the test data.

Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform a method action. One or more computer systems are identified by having software, firmware, hardware, or a combination thereof installed on the system that causes or causes the system to take action during operation. Can be configured to perform the actions of. One or more computer programs, when executed by a data processing device, can be configured to perform a particular action by including instructions that cause the device to perform the action.

The aforementioned and other embodiments may optionally include one or more of the following features, alone or in combination. In some embodiments, the time interval of time series data is determined, at least in part, based on the time associated with at least a subset of observations contained in at least one of the datasets. In some embodiments, the step of determining the time interval of time-series data is, for each dataset, the step of determining the individual time interval of the dataset and the step of determining that the time interval of the dataset is uniform. And the step of setting the time interval of the time series data to the time interval of the dataset. In some embodiments, the step of determining the time interval of a dataset is a step of determining the individual time period between continuous observations and a continuous pair of continuous observations for a pair or more of the continuous observations contained in the dataset. It includes a step of determining that the time cycle between observations is uniform and a step of setting the time interval of the dataset to the time cycle between continuous pairs of observations.

In some embodiments, the step of determining the time interval of time series data is different for each dataset from the step of determining the individual time interval of the dataset and at least two time intervals of the dataset. The time interval of the time series data includes, at least in part, the individual proportions of observations contained in each of the datasets, and / or (2) each individual of the dataset. Judgment is based on the time interval. In some embodiments, the step of determining the individual time intervals of a dataset comprises determining the individual time period between each pair of continuous observations contained in the dataset, and the step of determining a plurality of pairs of continuous observations. If the time period between them exhibits multiple non-uniform durations, then the time intervals in the dataset are, at least in part, (1) individual pairs of continuous observations exhibiting each non-uniform duration. If the time cycle between multiple pairs of continuous observations is a uniform duration, then the time interval of the data set is the time cycle, as determined based on the percentage and / or (2) duration of the time cycle. The duration of each. In some embodiments, the time interval of the time series data is the shortest time interval, which is a multiple of an integer of each individual time interval in the dataset.

In some embodiments, the action of the method further downsamples the observations of the dataset, for each dataset, if the time interval of the dataset is shorter than the time interval of the time series data. Includes the step of converting the time interval of the time interval to the time interval of time series data. In some embodiments, the step of downsampling an observation of a dataset corresponds to a separate instance of the time interval of the time series data, for each instance of the time interval of the time series data within the time cycle corresponding to the dataset. A step that identifies all observations in a dataset, a step that aggregates the identified observations to generate an aggregated observation, and a step that replaces the identified observations in the dataset with an aggregated observation. And include. In some embodiments, the number of identified observations corresponding to an instance of the time interval of the time series data is equal to the ratio between the time interval of the time series data and the time interval of the dataset. In some embodiments, the step of aggregating the identified observations sets the value of each variable in the aggregated observation to (1) the corresponding variable value contained in the oldest of the identified observations. , (2) Corresponding variable values included in the latest of the identified observations, (3) Maximum values of the corresponding variable values included in the identified observations, (4) For identified observations To the minimum value of the corresponding variable value included, (5) the average of the corresponding variable values contained in the identified observation, or (6) the value of the function of the corresponding variable value contained in the identified observation. Includes steps to set.

In some embodiments, the time interval of the time series data is selected from the group consisting of the time intervals of the dataset. In some embodiments, the dataset comprises a first dataset exhibiting a first time interval and a second dataset exhibiting a second time interval that exceeds the first time interval. The time interval of is selected as the time interval of the time series data, and the action of this method further downsamples the observations of the first data set, thereby setting the time interval of the first data set to the time interval of the time series data. Includes steps to convert to time intervals. In some embodiments, the time interval of the time series data is different from each of the time intervals of the dataset.

In some embodiments, at least a group of observations of time-series data includes the individual values of the first variable, and the actions of the method further adapt the predictive model to the training data and the model to fit the test data. Prior to the step of testing, a step of determining that the value of the first variable contains a time value and a step of generating an individual value of the second variable for each observation in the group. The value of the second variable includes a step with an offset between the time value of the first variable and the reference time value, and a step of adding the value of the second variable to individual observations in the group. .. In some embodiments, the action of the method further comprises removing the value of the first variable from the observations in the group. In some embodiments, the reference time includes the date of the event. In some embodiments, the event involves birth, marriage, graduation from school, the start of employment for an employer, or the start of work in a particular position.

In some embodiments, the variables include a first variable and a second variable, and the actions of the method further include changes in the values of the first and second variables, which are the values of the first variable. With a time delay between the change in the value of the first variable and the correlation change in the value of the second variable, the step of determining the correlation, and the change in the value of the first variable and the second through the graphical user interface. Includes a step of displaying graphical content that indicates the duration of the time delay between the value of a variable and its correlation change.

In some embodiments, the expected range is, at least in part, (1) the time interval of the time series data, (2) the number of observations contained in the time series data, and (3) the time period corresponding to the time series data. , And / or (4) Nature selected from the group consisting of microseconds, milliseconds, seconds, minutes, hours, days, weeks, months, quarters, seasons, years, 10 years, centuries, and 1,000 years. Judgment is made based on the time cycle. In some embodiments, the expected range is an integer multiple of the time interval of the time series data. In some embodiments, the time period between the time associated with the continuous prediction within the prediction range is equal to the time interval of the time series data.

In some embodiments, the skip range is, at least in part, the latency in collecting time series data, the latency in communicating time series data, the latency in analyzing time series data, the analysis of time series data. Judgment is based on the latency in communication and / or the latency of implementing an action based on the analysis of time series data.

In some embodiments, the actions of the method further include, at least in part, the total number of observations contained in the time series data, the amount of variation in the value of at least one of the variables over time, at least of the variables. Determine the duration of the training input time range based on the amount of seasonal variation of one value, the consistency of variation of at least one of the variables over multiple time periods, and / or the duration of the expected range. Including steps. In some embodiments, the step of fitting the predictive model to the training data includes a step of fitting the predictive model to a subset of the training data corresponding to a part of the training input time range and is part of the training input time range. Starts at the time following the start time of the training input time range and ends at the end time of the training input time range. In some embodiments, the duration of part of the training input time range is a multiple of an integer of the duration of the expected range.

In some embodiments, the actions of the method further include downsampling the training data prior to fitting the predictive model to the training data. In some embodiments, the step of downsampling the training data includes removing from the training data all observations obtained from at least one of the datasets. In some embodiments, the steps of downsampling the training data include setting the downsampled time interval of the training data to be an integral multiple of the time interval of the time series data and the downsampled time of the training data. For each instance of the interval, the step of identifying all observations in the training data associated with the time corresponding to the individual instance of the downsampled time interval of the training data, and the aggregated observations of the identified observations. Includes a step to generate and replace the identified observations in the training data with aggregated observations. In some embodiments, the actions of the method further include downsampling the test data prior to testing the conforming model to the test data.

In some embodiments, the actions of the method further include performing cross-validation of the predictive model. In some embodiments, the training data is the first training data, the test data is the first test data, the fit model is the first fit model, and cross-validation of the predictive model is performed. The steps to be performed are (i) generating a second training data and a second test data from the time series data, the second training data being the third of the observations of at least one of the datasets. The second fit, which comprises a subset, the second test data contains a fourth subset of observations of at least one of the datasets, and (j) the predictive model is fitted to the second training data and the second fit. It includes a step of acquiring a model and (k) a step of testing a second conforming model in the second test data.

In some embodiments, the first subset of observations corresponds to a slide training window that covers the first range of training time, and each observation contained in the first subset corresponds to the first range of training time. A third subset of observations, associated with time within, corresponds to a slide training window covering a second range of training time, and each observation contained within the third subset corresponds to a second range of training time. The earliest time within the first range of training time, associated with the time within, is earlier than the earliest time within the second range of training time. In some embodiments, the second subset of observations corresponds to a slide test window that covers the first range of test time, and each observation contained in the second subset corresponds to the first range of test time. A fourth subset of observations, associated with time within, corresponds to a slide test window covering a second range of test times, and each observation contained within the fourth subset corresponds to a second range of test times. The earliest time within the first range of test time, associated with the time within, is earlier than the earliest time within the second range of test time. In some embodiments, the first test time range partially overlaps the second training time range. In some embodiments, the second test time range does not overlap any part of the first training time range and does not overlap any part of the second training time range.

In some embodiments, the actions of the method further include dividing the time series data into a plurality of partitions, including at least a first partition and a second partition. In some embodiments, the step of dividing the time series data into a plurality of partitions includes assigning each of the datasets to the corresponding partition. In some embodiments, the step of dividing the time series data into a plurality of partitions includes the step of dividing the time series data in time, each partition being in a separate part of the time cycle associated with the time series data. Corresponding, each observation contained in the time series data is assigned to a partition that corresponds to a part of the time cycle that matches the time associated with the observation.

In some embodiments, the first training data includes a subset of observations contained in the first partition of the time series data, and the first test data is all of the time series data except the first partition. The second training data contains a subset of the observations contained in the second partition of the time series data, the second test data contains the second partition, and the second test data contains a separate subset of the observations contained in the partition. Contains a separate subset of observations contained in all partitions of time series data except. In some embodiments, the first partition of the time series data contains the first and second training data and the first and second test data, and the second partition of the time series data holds. Including out data, the actions of the method further include the step of testing the holdout data with the first and second fit models. In some embodiments, no predictive model fits the holdout data.

In some embodiments, the actions of the method further include performing nested cross-validation of the predictive model. In some embodiments, the step of performing nested cross-validation of the predictive model includes the time series data, at least a first partition of the time series data and a second partition of the time series data. The step of dividing into a plurality of partitions of 1 and the first partition of the time series data, at least the first partition of the first partition of the time series data and the second partition of the first partition of the time series data. The training data includes the first partition of the first partition of the time series data, and the test data includes at least the steps of dividing the first partition of the time series data into multiple partitions, including a partition. , Includes multiple partitions of the first partition of time series data other than the first partition of the first partition of time series data.

In some embodiments, the training data is the first training data, the test data is the first test data, the fit model is the first fit model, and the predicted model is nested cross-validation. Further, the step of carrying out (i) is a step of generating a second training data and a second test data from the first partition of the time series data, and the second training data is of the time series data. The second test data includes at least a plurality of partitions of the first partition of the data set other than the second partition of the first partition of the time series data, including the second partition of the first partition. It includes (j) a step of fitting the predictive model to the second training data and obtaining a second fit model, and (k) a step of testing the second fit model on the second test data. ..

In some embodiments, the steps of performing nested cross-validation further include testing the first and second conformance models in the second partition of the time series data, and a second of the time series data. Includes a step of comparing the first conforming model with the second conforming model based on the results of the step of testing the first and second conforming models on the partition of.

In some embodiments, the actions of the method further include determining model-specific predictions for one or more of the features of the time series data with respect to the fitted model. In some embodiments, the action of the method further removes features from the time series data, at least in part, based on model-specific predictions of the features, two or more features in the time series data. The steps of creating features derived from and adding the derived features to time series data, mixing a predictive model with another predictive model, and / or the suitability of a predictive modeling procedure for a predictive problem. Includes the step of performing at least one action selected from the group consisting of steps of allocating resources during the evaluation process.

In some embodiments, the actions of the method further include, at least in part, the characteristics of the predictive problem and / or the attributes of the individual predictive modeling procedures of the multiple predictive modeling procedures for the predictive problem. The step of determining suitability and the step of selecting one or more predictive modeling procedures from multiple predictive modeling procedures based on the determined suitability of the selected modeling procedure for the predictive problem. And the step of performing one or more predictive modeling procedures.

In some embodiments, the step of performing one or more predictive modeling procedures is the step of transmitting the instruction to multiple processing nodes, where the instruction is for executing the selected modeling procedure. Contains a resource allocation schedule that allocates resources for processing nodes to, and the resource allocation schedule is, at least in part, based on the suitability of the selected modeling procedure for the prediction problem, with multiple steps and according to the resource allocation schedule. The step of receiving the results of the execution of the selected modeling procedure by the processing node of, the result of which is the predictive model generated by the selected modeling procedure and / or the time series data associated with the predictive problem. Includes a step that includes the score of the generated model for, and a step that selects a predictive model for the predictive problem from the generated model, at least based on the score of the partially selected predictive model.

In some embodiments, the actions of the method further include the step of generating a mixed prediction model by mixing the fitted model with another fitted model.

In some embodiments, the actions of the method further include the step of developing a conforming model. In some embodiments, the time series data is the first time series data, and the step of developing the fit model fits the second time series data representing one or more instances of the prediction problem. The first time series data does not include the second time series data, including the step of generating one or more predictions by applying. In some embodiments, the time series data is the first time series data, and the step of developing the fit model is to refresh the fit model, at least in part, based on the second time series data. Including. In some embodiments, the fit model is the first fit model, and the step of refreshing the fit model, at least partially based on the second time series data, is a predictive model to the second time series data. It involves performing a transformation procedure and generating a second fit model, and mixing the first fit model and the second fit model to generate a refreshed predictive model. In some embodiments, the step of refreshing the fit model, at least partially based on the second time series data, is at least a portion of the first time series data and at least one of the second time series data. Includes a step of performing a predictive modeling procedure on a third time series data, including parts, to generate a refreshed predictive model.

In some embodiments, the conformance model is deployed on one or more servers, and other conformance models are also deployed on one or more servers, predicting conformance models and other conformance models. The requests are, at least in part, (1) an estimate of the amount of time used by each of the fitted models to generate the prediction, and / or (2) an estimate of how often the prediction request for each fitted model is received. It is allocated between servers based on the value. In some embodiments, each predictive request is assigned to a separate thread, each predictive request has an associated latency sensitivity value, and the number of threads running on a particular server is at least partially , Judgment is based on the latency sensitivity value of the thread running on a particular server.

In some embodiments, the actions of the method further include a step of determining the value of the metric, which indicates the interaction strength of two or more of the features contained in the time series data, and the metric. If the value exceeds a threshold, it includes a step of generating a time series value of a new feature based on the value of two or more features and adding the new feature to the time series data.

In some embodiments, the actions of the method further include determining the time resolution of the time series data. In some embodiments, the target is identified based on user input.

Another embodiment of this aspect is a memory configured to store a machine-executable module that encodes a predictive modeling procedure, wherein the predictive modeling procedure has at least one preprocessing task and at least one preprocessing task. A memory and at least one processor configured to execute a machine-executable module, including a plurality of tasks, including one model-fitting task, the step of executing the machine-executable module is an apparatus. Includes a predictive modeling device, including a processor, which allows the device to perform a predictive modeling procedure. The steps to carry out the predictive modeling procedure are (a) the step of retrieving time series data, including one or more datasets, where each dataset contains multiple observations, each observation. A step that includes (1) an indication of the time associated with the observation, (2) an individual value of one or more variables, and (b) a step of determining the time interval of the time series data, and ( c) The step of identifying one or more of the variables as targets and identifying other variables of zero or more as features is associated with (d) the prediction problem represented by the time series data. The step of determining the prediction range and the skip range, where the prediction range indicates the duration of the cycle in which the target value is predicted, and the skip range is the time associated with the oldest prediction within the prediction range. It may include steps to perform a pretreatment task, including steps that indicate a time delay between the latest observations based on predictions within the prediction range and the associated time. The step of performing the predictive modeling procedure is (e) the step of generating training data from time series data, the training data containing a first subset of observations of at least one of the datasets of the observations. The first subset contains the training inputs and training output sets of the observations, the time associated with the observations in the training inputs and training output sets corresponds to the training input time range and the training output time range, respectively, and the skip range Separate the end of the training input time range from the start of the training output time range and generate test data from the steps and (f) time series data, the duration of the training output time range is at least as long as the expected range. The test data includes a second subset of observations of at least one of the datasets, the second subset of observations containing test inputs and test validation sets of observations, test inputs and tests. The time associated with the observation in the validation set corresponds to the test entry time range and the test validation time range, respectively, and the skip range separates the start of the test validation time range from the end of the test entry time range and the test validation time. The duration of the range is at least as long as the expected range, including (g) the step of fitting the predicted model to the training data and (h) the step of testing the fitted model to the test data. It may include steps to perform the task. The machine executable module may include a directed graph showing the dependencies between tasks.

Judge the predicted value of the feature

It is difficult to understand (1) the prediction problem itself and (2) how the concrete prediction model produces accurate prediction results, even when an accurate prediction model for the prediction problem is available. obtain. A measure of "feature importance" can facilitate such understanding. In general, a feature importance metric indicates a predicted value of a feature of a dataset for the purpose of predicting the outcome of the prediction problem represented by the dataset. Traditional techniques for measuring feature importance are generally applicable only to certain types of predictive models and are generally not well suited for use with other types of predictive models. Therefore, there is a need for tools that can measure feature importance with respect to arbitrary predictive models or with respect to various sets of predictive models. In addition, feature importance metrics are used to guide the allocation of resources to the evaluation of predictive modeling procedures, to feature engineering tasks, and to the mix of predictive models, thereby demonstrating potential predictions for predictive problems. There is a need for tools that can facilitate a cost-effective assessment of the space of modeling techniques.

In general, another innovative aspect of the subject matter described herein is (a) the step of implementing multiple predictive modeling procedures, each of which is associated with a predictive model and each model. The step of performing the transformation procedure determines the step and (b) the first individual accuracy score of each of the fit prediction models, including the step of fitting the associated prediction model to the initial data set representing the initial prediction problem. The first accuracy score of each conforming model is included in the step and (c) the initial data set, which represents the accuracy with which the conforming model predicts the outcome of one or more of the initial prediction problems. A step of shuffling feature values across individual observations to generate a modified dataset that represents a modified prediction problem, and (d) a second individual of each of the fitted prediction models. A step of determining the accuracy score, where the second accuracy score of each conforming model represents the accuracy with which the conforming model predicts the outcome of one or more of the modified prediction problems, and (e). The step of determining the individual model-specific prediction value of the feature for each fitting model, and the model-specific prediction value of the feature for each fitting model is based on the first and second accuracy scores of the fitting model. Can be embodied in methods, including.

Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform a method action. One or more computer systems are identified by having software, firmware, hardware, or a combination thereof installed on the system that causes or causes the system to take action during operation. Can be configured to perform the actions of. One or more computer programs, when executed by a data processing device, can be configured to perform a particular action by including instructions that cause the device to perform the action.

The aforementioned and other embodiments may optionally include one or more of the following features, alone or in combination. In some embodiments, the actions of the method are further based on the characteristics of the initial dataset, the characteristics of the initial prediction problem, and / or the characteristics of the characteristics, prior to the step of performing multiple predictive modeling procedures. Includes steps to select multiple predictive modeling procedures for predictive problems. In some embodiments, the plurality of predictive modeling procedures is selected from the group consisting of random forest modeling procedures, generalized addition modeling procedures, and support vector machine modeling procedures, two or more models. Includes a conversion procedure. In some embodiments, the plurality of predictive modeling procedures are a first modeling procedure selected from a first modeling procedure family and a second modeling procedure selected from a second modeling procedure family. Includes procedures.

In some embodiments, the actions of the method further refit the predictive model to a modified data set representing a modified predictive problem prior to determining the second accuracy score of the predictive model. Including steps. In some embodiments, the determined model-specific predictions of features for a particular fit model increase the difference between the first and second accuracy scores of the particular fit model. To increase. In some embodiments, the determined model-specific predictions of the features for a particular fit model are the first accuracy score and the second accuracy score of the particular fit model relative to the first accuracy score of the particular fit model. Includes the percentage difference from the accuracy score of.

In some embodiments, the actions of the method further include determining the model-independent predictions of the features based on the model-specific predictions of the features. In some embodiments, the step of determining the model-independent predictor of a feature comprises calculating a statistical measure of the center and / or diffusion of the feature's model-specific predictor. In some embodiments, the step of determining the model-independent predictor of a feature comprises calculating the statistical measure of the center of the model-specific predictor, where the central statistical measure is model-specific. It is selected from the group consisting of the mean, median, and mode of the predicted values of. In some embodiments, the step of determining the model-independent predictor of a feature comprises calculating a statistical measure of the spread of the model-specific predictor, and the statistical measure of the spread is model-specific. It is selected from the group consisting of the range, variance, and standard deviation of the predicted values of. In some embodiments, the step of determining the model-independent predictors of a feature comprises calculating a model-specific combination of predictors of the feature. In some embodiments, the step of calculating the model-specific prediction value combination includes the step of calculating the model-specific prediction value weighted combination. In some embodiments, the step of calculating a weighted combination of model-specific forecast values includes assigning individual weights to model-specific forecast values, including a particular model-specific forecast corresponding to a particular fit prediction model. The weight assigned to the value increases as the first accuracy score of the fit prediction model increases.

In some embodiments, the feature is the first feature, and the actions of the method further (c1) shuffle the values of the second feature across the individual observations contained in the initial dataset. Thereby, a step of generating a second modified data set representing a second modified prediction problem and a step of determining (d1) a third individual accuracy score of each of the fitted prediction models. The third accuracy score of each conforming model represents the accuracy with which the conforming model predicts the outcome of one or more of the second modified prediction problems, step by step, and (e1) for each conforming model. The step of determining the individual model-specific predicted values of the two features, the model-specific predicted values of the second feature for each fitted model, is based on the first and third accuracy scores of the fitted model. And include.

In some embodiments, the feature is a first feature, the initial dataset comprises a first feature and a plurality of second features, and the actions of the method are further feature-by-second. Includes a step of determining model-specific predictions of the second feature of the initial dataset by performing steps (c), (d), and (e).

In some embodiments, the actions of the method further identify model-specific predictions of the first and second features of the initial dataset, as well as the first and second features, through a graphical user interface. Includes steps to display graphical content. In some embodiments, the modeling procedure is a first modeling procedure that includes a particular modeling procedure associated with a particular prediction model, and the model-specific predictions of the first and second features are The actions of the method further include (a1) specific modeling procedures associated with the specific predictive model, including specific model-specific predictive values of the first and second features that are specific to the particular predictive model. Includes steps to perform a plurality of second predictive modeling procedures, including. In some embodiments, the step of performing a particular predictive modeling procedure is to perform feature engineering on the initial dataset based on the particular model-specific predictions of the first feature and the second feature. including.

In some embodiments, the steps of performing feature engineering include removing specific features from the initial dataset based on the particular features with low model-specific predictions. In some embodiments, the actions of the method are further based on model-specific predictions of specific features below the threshold and / or specific model-specific of the first and second features of the initial dataset. Includes a step of determining that the model-specific predictions for a particular feature are low, based on the model-specific predictions for a particular feature within the specified percentile.

In some embodiments, the step of performing feature engineering is to generate derived features based on two or more specific features in the initial dataset with high model-specific predictions. Includes a step of adding the derived features to the initial dataset, thereby generating a second initial dataset. In some embodiments, the actions of the method are further based on model-specific predictions of specific features above the threshold and / or specific model-specific of the first and second features of the initial dataset. Includes a step of determining that the model-specific predictive value of a particular feature is high, based on the model-specific predictive value of a particular feature within the prescriptive percentile of.

In some embodiments, the step of performing a particular predictive modeling procedure further includes the step of fitting the particular predictive model to a second initial dataset, and the actions of the method are further adapted to the specific. A step of determining the first accuracy score of a predictive model, the first accuracy score of a particular fitted model predicts the outcome of the particular fit model to one or more of the initial prediction problems. Shuffle the values of the first feature across the steps and the individual observations contained in the second initial dataset, representing the accuracy of the second, thereby representing the second modified prediction problem. The step of generating the modified data set and the step of determining the second accuracy score of the specific predicted model that was adapted, and the second accuracy score of the specific model that was adapted is determined by the conforming model. Determine the second model-specific predictions of the steps and the first feature for a particular model fitted, which represent the accuracy of predicting the outcome of one or more of the second modified prediction problems. The second model-specific predictions of the first feature for a particular fitted model are based on the first and second accuracy scores of the particular fitter model. including.

In some embodiments, the actions of the method are further based on the suitability of the selected modeling procedure for the initial prediction problem prior to the step of performing the plurality of second modeling procedures. The suitability of a particular predictive modeling procedure for an initial prediction problem, including the step of selecting two modeling procedures, is at least in part a high model-specific predictive value for the particular predictive modeling procedure. The determination is based on the characteristics of a particular feature that is one or more of the initial data sets it has.

In some embodiments, the action of the method is further a step of transmitting the instruction to multiple processing nodes, the instruction allocating resources for the processing node for execution of the second modeling procedure. The resource allocation schedule, including the schedule, is a second model with steps and multiple processing nodes according to the resource allocation schedule, at least in part, based on the suitability of the second modeling procedure for the initial prediction problem. The step of receiving the result of the execution of the transformation procedure, which results in the score of the prediction model generated by the second modeling procedure and / or the generated model for the data associated with the initial prediction problem. Includes a step to select a prediction model for the initial prediction problem from the generated model, at least based on the score of the partially selected prediction model. In some embodiments, the actions of the method further include a step of generating a mixed prediction model and a step of evaluating the mixed prediction model by combining two or more of the generated prediction models. And include.

In some embodiments, the predicted value of the feature for the modified prediction problem is less than the threshold predicted value.

In some embodiments, the initial data set is an initial time series data set, the initial prediction problem is an initial time series prediction problem, and the modified data set is a modified time series data set. The modified prediction problem is a modified time series prediction problem. In some embodiments, the fit prediction model includes one or more time series prediction models.

Another embodiment of this aspect is a processor that is configured to store processor executable instructions and is configured to execute processor executable instructions and executes processor executable instructions. The step is (a) a step of executing a plurality of predictive modeling procedures on the device, each of the predictive modeling procedures is associated with a predictive model, and the step of executing each modeling procedure is an associated predictive model. Is a step that includes a step of fitting to an initial data set representing an initial prediction problem, and (b) a step of determining the first individual accuracy score of each of the fit prediction models, the first of each fit model. The accuracy score of is a shuffle of feature values across steps and (c) individual observations contained in the initial data set, representing the accuracy with which the fitted model predicts the outcome of one or more of the initial prediction problems. Then, a step of generating a modified data set representing a modified prediction problem and (d) a step of determining each second individual accuracy score of the conformity prediction model, each conformity. The model's second accuracy score represents the accuracy with which the fitted model predicts the outcome of one or more of the modified prediction problems, and (e) individual model-specific predictions of features for each fitted model. The step of determining the value, the model-specific predicted value of the feature for each conforming model, includes the processor, including the step, based on the first and second accuracy scores of the conforming model. , Includes predictive modeling equipment.

Another embodiment of this aspect, when executed by the processor, is the step of (a) performing a plurality of predictive modeling procedures on the processor, each of which is associated with a predictive model. The steps that carry out the modeling procedure include a step that fits the associated prediction model to the initial data set that represents the initial prediction problem, and (b) each first individual accuracy score of the fit prediction model. The determination step, the first accuracy score of each conforming model, is included in the step and (c) the initial data set, which represents the accuracy with which the conforming model predicts the outcome of one or more of the initial prediction problems. A step of shuffling the value of the feature across the individual observations to generate a modified dataset that represents the modified prediction problem, and (d) a second individual of each of the fitted prediction models. The second accuracy score of each fit model represents the accuracy with which the fit model predicts the outcome of one or more of the modified prediction problems. ) A step of determining the model-specific predicted values of the features of each fitted model, and the model-specific predicted values of the features of each fitted model are based on the first and second accuracy scores of the fitted model. Includes manufactured products, including, having computer-readable instructions stored on it, performing operations.

Certain embodiments of this aspect can be implemented to achieve one or more of the following advantages: Some embodiments of this aspect may be usefully used to facilitate understanding of the prediction problem and to show how a concrete prediction model produces accurate prediction results. Some embodiments of this aspect can measure feature importance with respect to an arbitrary predictive model or with respect to various sets of predictive models. Some embodiments of this aspect guide the allocation of resources to the evaluation of predictive modeling procedures, to feature engineering tasks, and to the mixture of predictive models, thereby potentially predictive modeling for predictive problems. A cost-effective assessment of the space of the technique can be facilitated.

Secondary prediction modeling

Some modeling techniques tend to produce opaque and / or complex models that are difficult to understand and difficult to implement efficiently in software. Software that implements such a model uses enormous computational resources and can generate predictions that can be produced much more efficiently using software that implements other equally accurate models.

Predict the value of one or more output variables (“targets”) T based on the value of one or more input variables (“features”) F1 without significantly reducing the accuracy of the model. , There is a need for techniques to reduce the opacity and / or complexity of the primary prediction model M1. The inventors recognize and understand that these needs can be met by constructing a secondary model M2 of a primary model M1. The secondary model may predict the predictive value of the primary model for target T based on the same feature F1 (or a subset thereof) and / or one or more features not used by the primary model. Good.

In many cases, the secondary model produced by the embodiments of this aspect is as accurate as or even more accurate than the corresponding primary model, and the software that implements the secondary model has the corresponding primary model. It is substantially more efficient than the software it implements. When the embodiments of this aspect are used to generate a secondary model of a mixed primary model, the secondary model is often particularly accurate, and the software that implements the secondary model is often It is more efficient than software that implements the corresponding primary model (eg, it uses less computational resources). Secondary models generated according to some embodiments of this aspect can be useful for understanding complex primary models and / or simplify the task of generating software that implements accurate predictive models. be able to.

In general, another innovative aspect of the subject matter described herein is the step of obtaining a fitted primary prediction model, which is based on the value of one or more first input variables. A step that is configured to predict the value of an output variable that is one or more of the prediction problems, and a step that implements a secondary prediction modeling procedure on the fitted primary model, which is a secondary modeling procedure. Can be embodied in predictive modeling methods, including steps associated with secondary predictive models. The step of performing a secondary predictive modeling procedure on a fitted primary model is the step of generating secondary input data, including multiple secondary observations, with each secondary observation having one or more second observations. The step of generating the secondary input data, including the individual observations of the input variables and the predicted values of the output variables, is for each secondary observation, the individual observations of the second input variable and the first input. A step and a quadratic, including a step of obtaining the corresponding observations of the variables and a step of applying a primary prediction model to the corresponding observations of the first input variable to generate individual predictions of the output variables. A step to generate secondary training data and secondary test data from the input data, a step to generate a fitted secondary prediction model of the fitted primary model by fitting the secondary prediction model to the secondary training data, and a secondary It may include a step of testing a conforming secondary prediction model of the conforming primary model to the test data.

Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform a method action. One or more computer systems are identified by having software, firmware, hardware, or a combination thereof installed on the system that causes or causes the system to take action during operation. Can be configured to perform the actions of. One or more computer programs, when executed by a data processing device, can be configured to perform a particular action by including instructions that cause the device to perform the action.

The aforementioned and other embodiments may optionally include one or more of the following features, alone or in combination. In some embodiments, the step of obtaining a fitted primary prediction model includes the step of performing the primary prediction modeling procedure associated with the primary prediction model, and the step of performing the primary prediction modeling procedure is a plurality of primary observations. Each primary observation is from the step and the primary input data, including the individual observations of the first input variable and the corresponding observations of the output variable. It includes a step of generating primary training data and primary test data, a step of fitting the primary prediction model to the primary training data, and a step of testing a matching primary prediction model for the test data. In some embodiments, the step of obtaining a conforming primary model comprises mixing the two conforming predictive models.

In some embodiments, the step of obtaining a fitted primary model is, at least in part, a plurality of primary predictions for the prediction problem, based on the characteristics of the prediction problem and / or the attributes of the individual primary prediction modeling procedure. One or more predictive modeling from multiple primary predictive modeling procedures based on the steps to determine the suitability of the modeling procedure and the determined suitability of the selected modeling procedure for the predictive problem. It includes the steps of selecting a procedure and performing one or more predictive modeling procedures. In some embodiments, the step of implementing one or more predictive modeling procedures is the step of transmitting the instruction to multiple processing nodes, where the instruction is for executing the selected modeling procedure. Contains a resource allocation schedule that allocates resources for processing nodes to, and the resource allocation schedule is at least partially based on the suitability of the selected modeling procedure for the prediction problem, with multiple steps and according to the resource allocation schedule. A step that receives the result of executing the selected modeling procedure by the processing node of, and the result is a conforming primary from the step and the generated model, including the predictive model generated by the selected modeling procedure. Includes steps to select a model.

In some embodiments, the secondary prediction model is selected from the group consisting of the RuleFit model and the generalized addition model. In some embodiments, the actions of the method further include a step of performing cross-validation of the secondary model, the secondary input data includes at least one dataset, and the steps of generating secondary training data. , A step of obtaining a first subset of the dataset, and a step of generating secondary test data includes a step of obtaining a second subset of the dataset.

In some embodiments, the secondary training data is the first secondary training data, the secondary test data is the first secondary test data, and the fitted secondary model is the first fitted second. The next model, the step of performing the cross verification of the secondary model is (a) the step of generating the second secondary training data and the second secondary test data from the secondary input data, and the second The secondary training data of the second secondary test data contains a third subset of the dataset, the second secondary test data contains a fourth subset of the dataset, and (b) the secondary prediction model of the second. Includes a step of adapting to the secondary training data and obtaining a second fitted secondary predictive model, and (c) testing the second secondary test data with a second fitted secondary predictive model.

In some embodiments, the actions of the method further include dividing the dataset into multiple partitions, including at least a first partition and a second partition. In some embodiments, the step of dividing the dataset into multiple partitions includes the step of randomly assigning each observation in the dataset to a separate partition. In some embodiments, the first secondary training data includes the first partition of the dataset and the first secondary test data includes all partitions of the dataset except the first partition. The second secondary training data includes the second partition of the dataset, and the second secondary test data includes all partitions of the dataset except the second partition. In some embodiments, the first secondary training data comprises a subset of the first partition of the dataset and the first secondary test data is of all partitions of the dataset except the first partition. The second secondary training data contains a subset of the second partition of the dataset, the second secondary test data contains the individual of all partitions of the dataset except the second partition. Contains a subset of.

In some embodiments, the secondary input data includes a first partition and a second partition, the dataset contains a first partition of the secondary input data, and the actions of the method further include. The holdout data, including the second partition of the secondary input data, includes the step of testing the first and second conforming secondary models. In some embodiments, no predictive model fits the holdout data.

In some embodiments, the step of performing the quadratic predictive modeling procedure further comprises performing nested cross-validation of the quadratic predictive model. In some embodiments, the secondary input data contains at least one dataset, and the step of performing nested cross-validation of the secondary predictive model is to combine the dataset with at least the first partition of the dataset. The step of dividing the first plurality of partitions of the data set, including the second partition of the data set, and the first partition of the data set, at least the first partition of the first partition of the data set. And the step of dividing the first partition of the data set into a plurality of partitions, including the second partition of the first partition of the data set. In some embodiments, the secondary training data includes the first partition of the first partition of the dataset and the secondary test data is a dataset excluding the first partition of the first partition of the dataset. Includes all partitions of the first partition of.

In some embodiments, the secondary training data is the first secondary training data, the secondary test data is the first secondary test data, and the fitted secondary model is the first fitted secondary. The next model, the step of performing nested cross-validation of the secondary prediction model, further generates (a) second secondary training data and second secondary test data from the first partition of the dataset. The second secondary training data includes the second partition of the first partition of the dataset and the second secondary test data is the second of the first partition of the dataset. The step, which includes multiple partitions of the first partition of the non-partitioned dataset, and (b) fit the secondary prediction model to the second secondary training data to obtain the second secondary fit prediction model. Includes steps and (c) testing the second secondary conformance model in the second secondary test data.

In some embodiments, the steps of performing nested cross-validation are further a step of testing a first conforming secondary model and a second conforming secondary model on the second partition of the dataset, and of the dataset. Includes a step of comparing the first fitted secondary model with the second fitted secondary model based on the results of the step of testing the first and second fitted secondary models in the second partition.

In some embodiments, the actions of the method further include determining the accuracy score of each of the fit prediction models, where the accuracy score of each fit model is the result of a prediction problem with one or more fit models. Represents the accuracy of predicting. In some embodiments, the actions of the method further include determining the difference between the accuracy score of the fitted primary model and the accuracy score of the fitted secondary model. In some embodiments, the accuracy score of the fitted secondary model exceeds the accuracy score of the fitted primary model.

In some embodiments, the actions of the method further include determining the amount of computational resources used by each of the fitted prediction models to predict the outcome of one or more prediction problems. In some embodiments, the actions of the method further include determining the difference between the amount of computational resources used by the fitted primary model and the amount of computational resources used by the fitted secondary model. In some embodiments, the amount of computational resources used by the fitted secondary model is less than the amount of computational resources used by the fitted primary model.

In some embodiments, the actions of the method further include the step of developing a conforming secondary model. In some embodiments, the step of deploying the fitted quadratic model includes a step of generating multiple predictions by applying the fitted quadratic model to other data representing an instance of the prediction problem, and the quadratic input The data does not include other data. In some embodiments, the conforming secondary model comprises one or more sets of conditional rules, one or more sets of conditional rules is one or more machine-executable ifs. -Contains a set of ten statements.

In some embodiments, the secondary input data is the first secondary input data, and the step of developing the fitted secondary model is further fitted, at least in part, based on the second secondary input data. Includes steps to refresh the secondary model. In some embodiments, the conforming secondary model is the first conforming secondary model, and the step of refreshing the conforming secondary model, at least partially based on the second secondary input data, is the second. By the step of generating the second secondary training data and the second secondary test data from the secondary input data of, and by adapting the secondary prediction model to the second secondary training data, the conforming primary model A step to generate a second conforming secondary model, a step to test the second conforming secondary model of the primary model on the second secondary test data, a first conforming secondary model and a second conforming second. Includes steps to mix the following models and generate a refreshed secondary prediction model. In some embodiments, the conforming secondary model is the first conforming secondary model, and the step of refreshing the conforming secondary model, at least partially based on the second secondary input data, is the first step. From the step of generating the third secondary input data including at least a part of the secondary input data and at least a part of the second secondary input data, and the third secondary input data, the third Generate a second fitted secondary model of the fitted primary model by fitting the secondary prediction model to the third secondary training data and the steps to generate the secondary training data and the third secondary test data. The third secondary test data includes a step of testing a second conforming secondary model of the primary model.

In some embodiments, the first input variable is the second input variable. In some embodiments, both the first input variable and the second input variable include a particular input variable. In some embodiments, none of the first input variables is included in the second input variable.

In some embodiments, the quadratic modeling procedure is one of a plurality of quadratic modeling procedures, and the quadratic predictive model is one of a plurality of second predictive models. The action of the method involves performing multiple quadratic modeling procedures on the fitted primary model, thereby generating multiple fitted quadratic models of the fitted primary model. In some embodiments, the actions of the method further include determining the accuracy score of each of the fitted quadratic prediction models, where the accuracy score of each fitted quadratic model is one or more of the fitted quadratic models. Represents the accuracy of predicting the outcome of a prediction problem that exceeds. In some embodiments, the actions of the method further include determining which of the accuracy scores is the highest and developing a fitted quadratic model with the highest accuracy score.

Another embodiment of this aspect is a memory that is configured to store a machine-executable module that encodes a secondary predictive modeling procedure associated with a secondary predictive model, a secondary predictive modeling procedure. Is at least one processor configured to execute a memory and a machine executable module, including a plurality of tasks, including at least one preprocessing task and at least one model fitting task. The step of executing the machine executable module includes a predictive modeling device, including a processor, which causes the device to perform a secondary predictive modeling procedure on the fitted primary predictive model. The step of performing the secondary prediction modeling procedure is the step of performing a preprocessing task, including the step of obtaining a fitted primary prediction model, where the primary prediction model is one or more first input variables. May include steps configured to predict the value of an output variable that is one or more of the prediction problems based on the value of. The step of performing the secondary predictive modeling procedure is the step of generating secondary input data, including multiple secondary observations, where each secondary observation is for one or more second input variables. The step of generating the secondary input data, including the individual observations and the predictions of the output variables, corresponds to the individual observations of the second input variable and the corresponding input variables for each secondary observation. A step that includes a step of acquiring observations and a step of applying a primary prediction model to the corresponding observations of the first input variable to generate individual predictions of the output variables, and two from the secondary input data. Fits the secondary training data and the secondary test data, and fits the secondary prediction model to the secondary training data to generate the fitted secondary predictive model, and fits the secondary test data. Alignment of Primary Models Includes steps to perform model conformance tasks, including testing secondary predictive models.

Other aspects and advantages of the invention will become apparent from the following drawings, detailed description, and claims, all exemplifying the principles of the invention by way of example only.

The above summary, including description of some embodiments, motivations for them, and / or their advantages, is intended to assist the reader in understanding the present disclosure, and any of the claims. It is not limited in any way.
The present invention provides, for example,:
(Item 1)
It is a predictive modeling method, and the predictive modeling method is
Enforcing a predictive modeling procedure, including performing the predictive modeling procedure,
(A) Acquiring time series data containing one or more datasets, each dataset comprising a plurality of observations, each observation being (1) an indication of the time associated with said observation. And (2) include the individual values of one or more variables.
(B) Determining the time interval of the time series data and
(C) Identifying one or more of the variables as targets and identifying zero or more other variables as features.
(D) Determining the prediction range and skip range associated with the prediction problem represented by the time series data, the prediction range indicating the duration of the cycle in which the target value is predicted. The skip range indicates the time delay between the time associated with the oldest prediction within the forecast range and the time associated with the latest observation based on the prediction within the forecast range.
(E) To generate training data from the time series data, the training data includes a first subset of observations of at least one of the datasets, the first subset of observations. The time associated with the observation in the training input and training output set, including the training input and training output set of the observation, corresponds to the training input time range and the training output time range, respectively, and the skip range is the said. Separating the end of the training input time range from the start of the training output time range, the duration of the training output time range is at least as long as the expected range.
(F) To generate test data from the time series data, the test data includes a second subset of observations of at least one of the data sets, the second subset of observations. The time associated with the observation in the test input and test verification set, including the test input and test verification set of the observation, corresponds to the test input time range and the test verification time range, respectively, and the skip range is said to be said. Separating the end of the test input time range from the start of the test verification time range, the duration of the test verification time range is at least as long as the expected range.
(G) Fitting the prediction model to the training data
(H) Testing the conforming model with the test data
Predictive modeling methods, including.
(Item 2)
The method of item 1, wherein the time interval of the time series data is determined, at least in part, based on the time associated with at least a subset of the observations contained in at least one of the datasets.
(Item 3)
Determining the time interval of the time series data
For each of the datasets, determining the individual time intervals of the dataset
Determining that the time intervals of the data set are uniform
To set the time interval of the time series data to the time interval of the data set.
2. The method according to item 2.
(Item 4)
Determining the time interval of the dataset
Determining the individual time periods between the continuous observations for a pair or more of the continuous observations contained in the data set.
Determining that the time period between the observations of the continuous pair is uniform
Setting the time interval of the data set to the time cycle between the observations of the continuous pair
The method according to item 3, which comprises.
(Item 5)
Determining the time interval of the time series data
For each of the datasets, determining the individual time intervals of the dataset
Determining that at least two time intervals in the dataset are different
Including
The time intervals of the time series data are, at least in part, based on (1) the individual proportions of the observations contained in each of the datasets and / or (2) the individual time intervals of each of the datasets. The method according to item 2 to be determined.
(Item 6)
Determining the individual time intervals of the dataset
Determining the individual time period between each pair of continuous observations contained in the dataset.
Including
If the time cycle between multiple pairs of continuous observations exhibits multiple non-uniform durations, then the time interval of the data set is at least partially (1) each of the non-uniform durations. Determined based on the individual proportions of the multiple pairs of continuous observations presented and / or (2) the duration of the time cycle.
The method of item 5, wherein the time interval of the dataset is the respective duration of the time cycle, where the time cycle between the plurality of pairs of continuous observations is of uniform duration.
(Item 7)
5. The method of item 5, wherein the time interval of the time series data is the shortest time interval that is a multiple of an integer of each individual time interval of the dataset.
(Item 8)
For each dataset, if the time interval of the dataset is shorter than the time interval of the time series data, the observations of the dataset are downsampled, thereby setting the time interval of the dataset to the time of the time series data. 7. The method of item 7, further comprising converting to intervals.
(Item 9)
Downsampling the observations of the dataset means each instance of the time interval of the time series data within the time cycle corresponding to the dataset.
Identifying all observations in the dataset associated with the time corresponding to the individual instance of the time interval of the time series data.
Aggregating the identified observations to generate an aggregated observation,
Replacing the identified observations in the dataset with the aggregated observations
8. The method of item 8.
(Item 10)
9. The method of item 9, wherein the number of identified observations corresponding to an instance of the time interval of the time series data is equal to the ratio between the time interval of the time series data and the time interval of the dataset.
(Item 11)
Aggregating the identified observations includes (1) the corresponding variable values contained in the oldest of the identified observations, and (2) the latest of the identified observations. Corresponding variable values, (3) maximum value of the corresponding variable value included in the identified observation, (4) minimum value of the corresponding variable value included in the identified observation, (5) said identified. Including setting the value of each variable in the aggregated observation to the average of the corresponding variable values included in the observation, or (6) the value of the function of the corresponding variable value included in the identified observation. , Item 9.
(Item 12)
The method of item 5, wherein the time interval of the time series data is selected from the group consisting of the time intervals of the dataset.
(Item 13)
The data set includes a first data set exhibiting a first time interval and a second data set exhibiting a second time interval exceeding the first time interval, wherein the second time interval is , The method is further selected as the time interval of the time series data.
To downsample the observations of the first dataset, thereby converting the time interval of the first dataset to the time interval of the time series data.
The method of item 12, comprising.
(Item 14)
The method according to item 5, wherein the time interval of the time series data is different from each of the time intervals of the data set.
(Item 15)
At least the group of observations of the time series data includes the individual values of the first variable, the method further adapting the prediction model to the training data and testing the fit model against the test data. prior to,
Determining that the value of the first variable has a time value
For each observation in the group is to generate an individual value for the second variable, the value of the second variable being between the time value of the first variable and the reference time value. To have an offset
Adding the value of the second variable to the individual observations within the group
The method according to item 1.
(Item 16)
15. The method of item 15, further comprising removing the value of the first variable from the observations within the group.
(Item 17)
The method of item 15, wherein the reference time comprises the date of the event.
(Item 18)
17. The method of item 17, wherein the event comprises birth, marriage, graduation from school, initiation of employment for an employer, or initiation of work in a particular position.
(Item 19)
The variable includes a first variable and a second variable, and the method further comprises
Determined that the changes in the values of the first and second variables correlate with a time delay between the changes in the values of the first variable and the correlation changes in the values of the second variable. To do and
To display graphical content showing the duration of the temporal delay between the change in the value of the first variable and the change in the correlation of the value of the second variable through a graphical user interface.
The method according to item 1.
(Item 20)
The expected range is, at least in part, (1) the time interval of the time series data, (2) the number of observations in the time series data, (3) the time cycle corresponding to the time series data, and / or (4) Based on a natural time cycle selected from the group consisting of microseconds, milliseconds, seconds, minutes, hours, days, weeks, months, quarters, seasons, years, 10 years, centuries, and 1,000 years. The method according to item 1 to be determined.
(Item 21)
The method according to item 20, wherein the expected range is a multiple of an integer of the time interval of the time series data.
(Item 22)
21. The method of item 21, wherein the time period between the continuous predictions within the prediction range and the time associated with it is equal to the time interval of the time series data.
(Item 23)
The skip range includes, at least in part, a waiting time for collecting the time series data, a waiting time for communicating the time series data, a waiting time for analyzing the time series data, and a communication for analyzing the time series data. The method according to item 1, wherein the method is determined based on the waiting time in, and / or the waiting time for implementing an action based on the analysis of the time series data.
(Item 24)
At least in part, the total number of observations contained in the time-series data, the amount of variation in the value of at least one of the variables over time, the amount of seasonal variation in the value of at least one of the variables, and the plurality. Item 1 further comprises determining the duration of the training input time range based on the consistency of variation in the value of at least one of the variables over a time cycle and / or the duration of the expected range. The method described.
(Item 25)
Adapting the predictive model to the training data includes adapting the predictive model to a subset of the training data corresponding to a portion of the training input time range, which is part of the training input time range. 24. The method of item 24, wherein the method starts at a time following the start time of the training input time range and ends at the end time of the training input time range.
(Item 26)
25. The method of item 25, wherein the duration of a portion of the training input time range is a multiple of an integer of the duration of the expected range.
(Item 27)
The method of item 1, further comprising downsampling the training data prior to adapting the prediction model to the training data.
(Item 28)
27. The method of item 27, wherein downsampling the training data comprises removing from the training data all observations obtained from at least one of the datasets.
(Item 29)
Downsampling the training data
Setting the downsampled time interval of the training data to a multiple of an integer of the time interval of the time series data, and
For each instance of the downsampled time interval of the training data,
Identifying all observations in the training data that are associated with the time corresponding to the individual instances of the downsampled time interval of the training data.
Aggregating the identified observations to generate an aggregated observation,
Replacing the identified observations in the training data with the aggregated observations
27. The method of item 27.
(Item 30)
The method of item 1, further comprising downsampling the test data prior to testing the conforming model on the test data.
(Item 31)
The method of item 1, further comprising performing cross-validation of the prediction model.
(Item 32)
The training data is the first training data, the test data is the first test data, the conforming model is the first conforming model, and cross-validation of the prediction model can be performed. ,
(I) Generating a second training data and a second test data from the time series data, the second training data being a third subset of at least one observation in the dataset. The second test data comprises a fourth subset of observations of at least one of the datasets.
(J) To adapt the prediction model to the second training data and acquire the second conformance model.
(K) To test the second conforming model on the second test data.
31. The method of item 31.
(Item 33)
The first subset of the observations corresponds to a slide training window covering the first range of training time, and each observation contained in the first subset corresponds to a time within the first range of the training time. A third subset of the associated observations corresponds to the slide training window covering a second range of training time, and each observation contained in the third subset corresponds to a second range of training time. 32. The method of item 32, wherein the earliest time in the first range of the training time is earlier than the earliest time in the second range of the training time, which is associated with the time within.
(Item 34)
The second subset of the observations corresponds to a slide test window covering the first range of test times, and each observation contained in the second subset corresponds to a time within the first range of the test times. The fourth subset of the observations associated and corresponds to the slide test window covering the second range of test time, and each observation contained in the fourth subset corresponds to the second range of test time. 33. The method of item 33, wherein the earliest time within the first range of the test time is earlier than the earliest time within the second range of the test time, which is associated with the time within.
(Item 35)
34. The method of item 34, wherein the first test time range partially overlaps the second training time range.
(Item 36)
35. The method of item 35, wherein the second test time range does not overlap any part of the first training time range and does not overlap any part of the second training time range.
(Item 37)
32. The method of item 32, further comprising dividing the time series data into a plurality of partitions including at least a first partition and a second partition.
(Item 38)
37. The method of item 37, wherein dividing the time series data into a plurality of partitions comprises allocating each of the datasets to a corresponding partition.
(Item 39)
37. The method of item 37, wherein dividing the time series data into a plurality of partitions comprises dividing the time series data in time.
(Item 40)
Each of the partitions corresponds to a separate portion of the time cycle associated with the time series data, and each observation contained in the time series data corresponds to a portion of the time cycle that matches the time associated with the observation. 39. The method of item 39, which is assigned to the partition.
(Item 41)
The first training data comprises a subset of the observations contained in the first partition of the time series data.
The first test data comprises a separate subset of the observations contained in all partitions of the time series data except the first partition.
The second training data comprises a subset of the observations contained in the second partition of the time series data.
The second test data comprises a separate subset of the observations contained in all partitions of the time series data except the second partition.
The method according to item 37.
(Item 42)
The first partition of the time series data includes the first and second training data and the first and second test data, and the second partition of the time series data holds holdout data. Including, the method further comprises
Testing the first and second conforming models on the holdout data
32. The method of item 32.
(Item 43)
42. The method of item 42, wherein no predictive model fits the holdout data.
(Item 44)
The method of item 1, further comprising performing nested cross-validation of the prediction model.
(Item 45)
Performing nested cross-validation of the forecast model
Dividing the time series data into a plurality of first partitions including at least a first partition of the time series data and a second partition of the time series data.
The time series is divided into a plurality of partitions of the first partition of the time series data including at least the first partition of the first partition of the time series data and the second partition of the first partition of the time series data. To split the first partition of data
Including
The training data includes a first partition of the first partition of the time-series data, and the test data is at least the time-series data other than the first partition of the first partition of the time-series data. It has multiple partitions of the first partition,
The method according to item 44.
(Item 46)
The training data is the first training data, the test data is the first test data, the conforming model is the first conforming model, and nested cross-validation of the prediction model is performed. That is also
(I) The second training data and the second test data are generated from the first partition of the time series data, and the second training data is the first partition of the time series data. The second test data comprises at least a plurality of partitions of the first partition of the data set other than the second partition of the first partition of the time series data. When,
(J) To adapt the prediction model to the second training data and acquire the second conformance model.
(K) To test the second conforming model on the second test data.
45. The method of item 45.
(Item 47)
Performing the nested cross-validation further
Testing the first conforming model and the second conforming model in the second partition of the time series data,
Comparing the first conforming model with the second conforming model based on the results of testing the first and second conforming models on the second partition of the time series data.
46. The method of item 46.
(Item 48)
The method of item 1, further comprising determining model-specific predictions of one or more of the features of the time series data with respect to the fitted model.
(Item 49)
Removing features from the time series data, at least in part, based on model-specific predictions of the features, creating features derived from two or more features in the time series data, said. Resources during the process of adding derived features to the time series data, mixing the prediction model with another prediction model, and / or evaluating the suitability of the prediction modeling procedure for the prediction problem. 48. The method of item 48, further comprising performing at least one action selected from the group consisting of assigning.
(Item 50)
Determining the suitability of multiple predictive modeling procedures for the predictive problem, at least in part, based on the characteristics of the predictive problem and / or the attributes of the individual predictive modeling procedures.
To select one or more predictive modeling procedures from the plurality of predictive modeling procedures based on the determined suitability of the selected modeling procedure for the predictive problem.
Performing one or more of the predictive modeling procedures described above
The method according to item 1, further comprising.
(Item 51)
Performing the one or more predictive modeling procedures described above
The instruction is to be transmitted to a plurality of processing nodes, the instruction comprising a resource allocation schedule for allocating resources for the processing node for execution of the selected modeling procedure, the resource allocation schedule being at least. Partly based on the suitability of the selected modeling procedure for the prediction problem, and
Receiving the result of execution of the selected modeling procedure by the plurality of processing nodes according to the resource allocation schedule, the result of which is the prediction model generated by the selected modeling procedure. And / or include the score of the generated model for the time series data associated with the prediction problem.
From the generated model, selecting a predictive model for the predictive problem, at least in part, based on the score of the selected predictive model.
50. The method of item 50.
(Item 52)
The method of item 1, further comprising generating a mixed prediction model by mixing the fitted model with another fitted model.
(Item 53)
The method of item 1, wherein the method further comprises developing the conforming model.
(Item 54)
The time series data is the first time series data, and expanding the fit model applies the fit model to the second time series data representing one or more instances of the prediction problem. 53. The method of item 53, wherein the first time series data does not include the second time series data, comprising generating one or more predictions.
(Item 55)
The time series data is the first time series data, and expanding the conforming model includes refreshing the conforming model at least partially based on the second time series data. The method described in.
(Item 56)
The fit model is a first fit model, and refreshing the fit model, at least in part, based on the second time series data
Performing the predictive modeling procedure on the second time series data to generate a second fit model,
To generate a refreshed prediction model by mixing the first conforming model and the second conforming model.
55. The method of item 55.
(Item 57)
Refreshing the fitted model, at least partially based on the second time series data, provides at least a portion of the first time series data and at least a portion of the second time series data. 55. The method of item 55, comprising performing the predictive modeling procedure on a third time series data provided to generate a refreshed predictive model.
(Item 58)
The conforming model is deployed on one or more servers, and other conforming models are also deployed on the one or more servers, and the prediction requirements for the conforming model and the other conforming models are at least partially. Based on (1) an estimate of the amount of time used by each of the fitted models to generate a prediction, and / or (2) an estimate of how often a prediction request for each fit model is received. 53. The method of item 53, which is allocated between the servers.
(Item 59)
Each predictive request is assigned to a separate thread, each predictive request has an associated latency sensitivity value, and the number of threads running on a particular server is at least partially on that particular server. 58. The method of item 58, which is determined based on the latency sensitivity value of the thread to be executed.
(Item 60)
Determining the value of a metric that indicates the interaction strength of two or more of the features contained in the time series data.
When the value of the measurement reference exceeds the threshold value, a time series value of a new feature is generated based on the value of the two or more features, and the new feature is added to the time series data.
The method according to item 1, further comprising.
(Item 61)
The method of item 1, further comprising determining the time resolution of the time series data.
(Item 62)
The method of item 1, wherein the target is identified based on user input.
(Item 63)
Predictive modeling device
A memory configured to store a machine executable module that encodes a predictive modeling procedure, said predictive modeling procedure including at least one preprocessing task and at least one model fitting task. Memory, including multiple tasks,
Executing the machine-executable module with at least one processor configured to execute the machine-executable module causes the device to perform the predictive modeling procedure and said predictive modeling procedure. To carry out
To carry out the pre-processing task, and to carry out the pre-processing task
(A) Acquiring time series data containing one or more datasets, each dataset comprising a plurality of observations, each observation being (1) an indication of the time associated with said observation. And (2) include the individual values of one or more variables.
(B) Determining the time interval of the time series data and
(C) Identifying one or more of the variables as targets and identifying zero or more other variables as features.
(D) Determining the prediction range and skip range associated with the prediction problem represented by the time series data, the prediction range indicating the duration of the cycle in which the target value is predicted. The skip range indicates a time delay between the time associated with the oldest prediction within the forecast range and the time associated with the latest observation based on the prediction within the forecast range.
Including, and
Carrying out the model fitting task, and carrying out the model fitting task
(E) To generate training data from the time series data, the training data includes a first subset of observations of at least one of the datasets, the first subset of observations. The time associated with the observation in the training input and training output set, including the training input and training output set of the observation, corresponds to the training input time range and the training output time range, respectively, and the skip range is the said. Separating the end of the training input time range from the start of the training output time range, the duration of the training output time range is at least as long as the expected range.
(F) To generate test data from the time series data, the test data includes a second subset of observations of at least one of the data sets, the second subset of observations. The time associated with the observation in the test input and test verification set, including the test input and test verification set of the observation, corresponds to the test input time range and the test verification time range, respectively, and the skip range is said to be said. Separating the end of the test input time range from the start of the test verification time range, the duration of the test verification time range is at least as long as the expected range.
(G) Fitting the prediction model to the training data
(H) Testing the conforming model with the test data
Including, and
Including the processor and
A device that comprises.
(Item 64)
63. The method of item 63, wherein the machine executable module comprises a directed graph showing the dependencies between the tasks.
(Item 65)
It is a computer implementation prediction modeling method, and the computer implementation prediction modeling method is
(A) Performing a plurality of predictive modeling procedures, each of which is associated with a predictive model, and performing each of the modeling procedures initially predicts the associated predictive model. Including adapting to the initial dataset that represents the problem,
(B) The determination of each first individual accuracy score of the conformance prediction model, wherein the first accuracy score of each conformance model is such that the conformance model is one or more of the initial prediction problems. Represents the accuracy of predicting the outcome of
(C) Shuffling feature values across the individual observations contained in the initial dataset, thereby generating a modified dataset representing a modified predictive problem.
(D) The determination of each second individual accuracy score of the conformance prediction model, wherein the second accuracy score of each conformance model is one of the prediction problems that the conformance model has been modified. Or to represent the accuracy of predicting multiple outcomes,
(E) The prediction value peculiar to the individual model of the feature is determined for each conforming model, and the model-specific predicted value of the feature for each conforming model is the first and second prediction values of the conforming model. Based on the accuracy score of
Computer implementation predictive modeling methods, including.
(Item 66)
Prior to performing the plurality of prediction modeling procedures, the plurality of predictions for the prediction problem are based on the characteristics of the initial dataset, the characteristics of the initial prediction problem, and / or the characteristics of the characteristics. 65. The method of item 65, further comprising selecting a modeling procedure.
(Item 67)
The plurality of predictive modeling procedures include two or more modeling procedures selected from the group consisting of random forest modeling procedures, generalized addition modeling procedures, and support vector machine modeling procedures. The method of item 65.
(Item 68)
The plurality of predictive modeling procedures include an item including a first modeling procedure selected from the first modeling procedure family and a second modeling procedure selected from the second modeling procedure family. 65.
(Item 69)
65. The method of item 65, further comprising refitting the predictive model to the modified data set representing the modified predictive problem prior to determining a second accuracy score of the predictive model. ..
(Item 70)
The determined model-specific predictions of the features for a particular fit model increase as the difference between the first accuracy score and the second accuracy score of the particular fit model increases, item 65. The method described in.
(Item 71)
The determined model-specific predictions of the features for a particular fit model are the first accuracy score and the second accuracy score of the particular fit model relative to the first accuracy score of the particular fit model. 65. The method of item 65, comprising a proportion difference between.
(Item 72)
65. The method of item 65, further comprising determining a model-independent predicted value of the feature based on a model-specific predicted value of the feature.
(Item 73)
72. The method of item 72, wherein determining the model-independent predictor of the feature comprises calculating a statistical measure of the center and / or diffusion of the model-specific predictor of the feature.
(Item 74)
Determining the model-independent predictor of the feature involves calculating a statistical measure of the center of the model-specific predictor, which is the model-specific predictor. 72. The method of item 72, selected from the group consisting of mean, median, and mode of.
(Item 75)
Determining the model-independent predictor of the feature includes calculating a statistical measure of the spread of the model-specific predictor, the statistical measure of the spread being the model-specific predictor. 72. The method of item 72, selected from the group consisting of ranges, variances, and standard deviations of.
(Item 76)
72. The method of item 72, wherein determining the model-independent predicted values of the feature comprises calculating a model-specific combination of predicted values of the feature.
(Item 77)
76. The method of item 76, wherein calculating the model-specific prediction value combination comprises calculating a weighted combination of the model-specific prediction values.
(Item 78)
Calculating the weighted combination of the model-specific forecast values includes assigning individual weights to the model-specific forecast values, and is assigned to the specific model-specific forecast values corresponding to the specific fit prediction model. The method of item 77, wherein the weighting increases as the first accuracy score of the fit prediction model increases.
(Item 79)
The feature is the first feature, and the method further comprises
(C1) A second modified dataset that shuffles the values of the second feature across the individual observations contained in the initial dataset, thereby representing a second modified predictive problem. To generate and
(D1) The determination of each third individual accuracy score of the conformance prediction model, wherein the third accuracy score of each conformance model is a prediction problem in which the conformance model is the second modified prediction problem. Represents the accuracy of predicting one or more outcomes of
(E1) It is to determine the predicted value peculiar to the individual model of the second feature for each conforming model, and the predicted value peculiar to the model of the second feature for each conforming model is the conforming model. Based on the 1st and 3rd accuracy scores of
65. The method of item 65.
(Item 80)
The feature is a first feature, the initial dataset comprises the first feature and a plurality of second features, and the method further steps (c) for each of the second features. 65. The method of item 65, comprising determining model-specific predictions of the second feature of the initial dataset by performing (d), and (e).
(Item 81)
An item further comprising displaying graphical content that identifies model-specific predictions of the first and second features of the initial dataset, as well as the first and second features, via a graphical user interface. 80.
(Item 82)
The modeling procedure is a first modeling procedure including a specific modeling procedure associated with a specific prediction model, and model-specific prediction values of the first and second features are the particular prediction model. The method comprises a particular model-specific prediction of the first and second features that are unique to, and the method further comprises (a1) the particular modeling procedure associated with the particular prediction model. 80. The method of item 80, comprising performing the predictive modeling procedure of 2.
(Item 83)
Performing the particular predictive modeling procedure includes performing feature engineering on the initial dataset based on the particular model-specific predictions of the first feature and the second feature. The method of item 82.
(Item 84)
38. The method of item 83, wherein performing feature engineering involves removing a particular feature from the initial dataset based on said particular feature with low model-specific predictions.
(Item 85)
The specification is based on model-specific predictions of the particular feature below the threshold and / or within the defined percentile of the particular model-specific predictions of the first and second features of the initial dataset. 84. The method of item 84, further comprising determining that the model-specific predicted value of the particular feature is low based on the model-specific predicted value of the particular feature.
(Item 86)
Performing feature engineering is
Generating derived features based on two or more specific features in the initial dataset with high model-specific predictions.
Adding the derived features to the initial dataset, thereby generating a second initial dataset.
83. The method of item 83.
(Item 87)
The specification is based on model-specific predictions of the particular feature above the threshold and / or within the defined percentile of the particular model-specific predictions of the first and second features of the initial dataset. 86. The method of item 86, further comprising determining that the model-specific predicted value of the particular feature is high based on the model-specific predicted value of the particular feature.
(Item 88)
Performing the particular predictive modeling procedure further comprises adapting the particular predictive model to the second initial data set, the method further comprising:
The first accuracy score of the adapted specific model is to determine the first accuracy score of the adapted specific model, wherein the adapted specific model is of the initial prediction problem. Representing the accuracy of predicting one or more outcomes,
A second modified dataset that shuffles the values of the first feature across the individual observations contained in the second initial dataset, thereby representing a second modified predictive problem. To generate and
The second accuracy score of the fitted specific model is to determine the second accuracy score of the fitted specific model, wherein the fitted model is the second modified prediction problem. Represents the accuracy of predicting one or more outcomes of
Determining a second model-specific prediction of the first feature for the fitted specific model, the second of the first feature for the fitted particular model. The model-specific predictions for are based on the first and second accuracy scores of the particular model adapted.
86. The method of item 86.
(Item 89)
Prior to performing the plurality of second modeling procedures,
Choosing the second modeling procedure based on the suitability of the selected modeling procedure for the initial prediction problem.
Including
The suitability of the particular predictive modeling procedure for the initial predictive problem is, at least in part, one of the initial datasets with high model-specific predictive values for the particular predictive modeling procedure. Determined based on the characteristics of multiple specific features,
The method according to item 82.
(Item 90)
The instruction is to be transmitted to a plurality of processing nodes, the instruction comprising a resource allocation schedule for allocating resources for the processing node for execution of the second modeling procedure, the resource allocation schedule being at least. Partly based on the suitability of the second modeling procedure for the initial prediction problem.
The result of executing the second modeling procedure by the plurality of processing nodes according to the resource allocation schedule is received, and the result is the prediction model generated by the second modeling procedure. And / or include the score of the generated model for the data associated with the initial prediction problem.
From the generated model, selecting a prediction model for the initial prediction problem, at least in part, based on the score of the selected prediction model.
89. The method of item 89.
(Item 91)
Generating a mixed forecast model by combining two or more of the generated forecast models,
To evaluate the mixed prediction model
89. The method of item 89.
(Item 92)
65. The method of item 65, wherein the predicted value of the feature for the modified prediction problem is less than a threshold predicted value.
(Item 93)
Predictive modeling device
Memory configured to store processor executable instructions, and
A processor configured to execute the processor-executable instruction, and executing the processor-executable instruction, causes the device to execute the processor-executable instruction.
(A) Performing a plurality of predictive modeling procedures, each of which is associated with a predictive model, and performing each of the modeling procedures initially predicts the associated predictive model. Including adapting to the initial dataset that represents the problem,
(B) The determination of each first individual accuracy score of the conformance prediction model, wherein the first accuracy score of each conformance model is such that the conformance model is one or more of the initial prediction problems. Represents the accuracy of predicting the outcome of
(C) Shuffling feature values across the individual observations contained in the initial dataset, thereby generating a modified dataset representing the modified predictive problem.
(D) The determination of each second individual accuracy score of the conformance prediction model, wherein the second accuracy score of each conformance model is one of the prediction problems that the conformance model has been modified. Or to represent the accuracy of predicting multiple outcomes,
(E) The prediction value peculiar to the individual model of the feature is determined for each conforming model, and the model-specific predicted value of the feature for each conforming model is the first and second prediction values of the conforming model. Based on the accuracy score of
With the processor to perform steps including
A device that comprises.
(Item 94)
A manufactured product, the manufactured product having a computer-readable instruction stored on the product, and when the computer-readable instruction is executed by the processor, the processor receives the computer-readable instruction.
(A) Performing a plurality of predictive modeling procedures, each of which is associated with a predictive model, and performing each modeled procedure is an initial dataset that represents an initial predictive problem. Including fitting the associated predictive model and
(B) The determination of each first individual accuracy score of the conformance prediction model, wherein the first accuracy score of each conformance model is such that the conformance model is one or more of the initial prediction problems. Represents the accuracy of predicting the outcome of
(C) Shuffling feature values across the individual observations contained in the initial dataset, thereby generating a modified dataset representing a modified predictive problem.
(D) The determination of each second individual accuracy score of the conformance prediction model, wherein the second accuracy score of each conformance model is one of the prediction problems that the conformance model has been modified. Or to represent the accuracy of predicting multiple outcomes,
(E) The prediction value peculiar to the individual model of the feature is determined for each conforming model, and the model-specific predicted value of the feature for each conforming model is the first and second prediction values of the conforming model. Based on the accuracy score of
Manufactured products that carry out operations including.
(Item 95)
It is a predictive modeling method, and the predictive modeling method is
Acquiring a fitted primary prediction model, such that the primary prediction model predicts the values of one or more output variables of a prediction problem based on the values of one or more first input variables. It is composed of
Performing a secondary predictive modeling procedure on the fitted primary model, the secondary modeling procedure is associated with a secondary predictive model, and the secondary predictive modeling procedure is performed on the fitted primary model. That is
Generating secondary input data that includes multiple secondary observations, each secondary observation consisting of individual observations of one or more second input variables and predicted values of said output variables. To generate the secondary input data, to obtain the individual observation values of the second input variable and the corresponding observation values of the first input variable for each secondary observation. Includes applying the primary prediction model to the corresponding observations of the first input variable and generating individual predictions of the output variable.
To generate secondary training data and secondary test data from the secondary input data,
By adapting the secondary prediction model to the secondary training data, a conforming secondary prediction model of the conforming primary model can be generated.
To test the conforming secondary prediction model of the conforming primary model on the secondary test data.
Including, and
Predictive modeling methods, including.
(Item 96)
Acquiring the fitted primary model involves performing a primary predictive modeling procedure associated with the primary predictive model, and performing the primary predictive modeling procedure includes performing the primary predictive modeling procedure.
Acquiring primary input data including a plurality of primary observations, each primary observation comprising an individual observation value of the first input variable and a corresponding observation value of the output variable.
To generate primary training data and primary test data from the primary input data,
Adapting the primary prediction model to the primary training data
To test the conformance primary prediction model on the test data
95. The method of item 95.
(Item 97)
The method of item 95, wherein obtaining the conforming primary model comprises mixing the two conforming predictive models.
(Item 98)
Obtaining the conforming primary model
Determining the suitability of multiple primary predictive modeling procedures for the predictive problem, at least in part, based on the characteristics of the predictive problem and / or the attributes of the individual primary predictive modeling procedures.
To select one or more predictive modeling procedures from the plurality of primary predictive modeling procedures based on the determined suitability of the selected modeling procedure for the predictive problem.
Performing one or more of the predictive modeling procedures described above
95. The method of item 95.
(Item 99)
Performing the one or more predictive modeling procedures described above
The instruction is to be transmitted to a plurality of processing nodes, the instruction comprising a resource allocation schedule for allocating resources for the processing node for execution of the selected modeling procedure, the resource allocation schedule being at least. Partly based on the suitability of the selected modeling procedure for the prediction problem, and
Receiving the result of execution of the selected modeling procedure by the plurality of processing nodes according to the resource allocation schedule, the result of which is the prediction model generated by the selected modeling procedure. Including, and
To select the conforming primary model from the generated model
98. The method of item 98.
(Item 100)
95. The method of item 95, wherein the secondary prediction model is selected from the group consisting of a RuleFit model and a generalized addition model.
(Item 101)
Further including performing cross-validation of the secondary model, the secondary input data comprises at least one dataset, and generating the secondary training data comprises performing a first subset of the dataset. 95. The method of item 95, wherein generating the secondary test data, including obtaining, comprises obtaining a second subset of the dataset.
(Item 102)
The secondary training data is the first secondary training data, the secondary test data is the first secondary test data, and the conforming secondary model is the first conforming secondary model. , Performing cross-validation of the secondary model
(A) To generate a second secondary training data and a second secondary test data from the secondary input data, the second secondary training data is a third subset of the dataset. And that the second secondary test data includes a fourth subset of the dataset.
(B) Adapting the secondary prediction model to the second secondary training data and acquiring the second fitted secondary prediction model.
(C) To test the second conforming secondary prediction model on the second secondary test data.
101. The method of item 101.
(Item 103)
102. The method of item 102, further comprising dividing the data set into a plurality of partitions, including at least a first partition and a second partition.
(Item 104)
103. The method of item 103, wherein dividing the dataset into a plurality of partitions comprises randomly assigning each observation in the dataset to a separate partition.
(Item 105)
The first secondary training data comprises a first partition of the dataset.
The first secondary test data includes all partitions of the dataset except the first partition.
The second secondary training data comprises a second partition of the dataset.
The second secondary test data comprises all partitions of the dataset except the second partition.
The method of item 104.
(Item 106)
The first secondary training data comprises a subset of the first partitions of the dataset.
The first secondary test data comprises a separate subset of all partitions of the dataset except the first partition.
The second secondary training data comprises a subset of the second partition of the dataset.
The second secondary test data comprises a separate subset of all partitions of the dataset except the second partition.
The method of item 104.
(Item 107)
The secondary input data includes a first partition and a second partition.
The data set comprises a first partition of the secondary input data.
The method further comprises testing the first and second conforming secondary models on holdout data comprising a second partition of the secondary input data.
The method of item 102.
(Item 108)
The method of item 107, wherein no predictive model fits the holdout data.
(Item 109)
95. The method of item 95, wherein performing the secondary prediction modeling procedure further comprises performing nested cross-validation of the secondary prediction model.
(Item 110)
The secondary input data comprises at least one dataset.
Performing nested cross-validation of the secondary prediction model
Dividing the data set into a plurality of first partitions of the data set, including at least a first partition of the data set and a second partition of the data set.
The first partition of the data set has a plurality of partitions of the first partition of the data set including at least the first partition of the first partition of the data set and the second partition of the first partition of the data set. To partition
Including
The secondary training data comprises a first partition of the first partition of the dataset.
The secondary test data includes all partitions of the first partition of the data set except the first partition of the first partition of the data set.
The method of item 109.
(Item 111)
The secondary training data is the first secondary training data, the secondary test data is the first secondary test data, and the conforming secondary model is the first conforming secondary model. Further, performing nested cross-validation of the secondary prediction model
(A) The second secondary training data and the second secondary test data are generated from the first partition of the data set, and the second secondary training data is the data set. A second partition of the first partition is provided, and the second secondary test data includes a plurality of partitions of the first partition of the data set other than the second partition of the first partition of the data set. To prepare and to
(B) To acquire the second secondary conformity prediction model by adapting the secondary prediction model to the second secondary training data.
(C) To test the second secondary conformance model on the second secondary test data.
110. The method of item 110.
(Item 112)
Performing the nested cross-validation further
Testing the first conforming secondary model and the second conforming secondary model in the second partition of the dataset.
The first fitted secondary model is compared to the second fitted secondary model based on the results of testing the first and second fitted secondary models on the second partition of the dataset. With that
111. The method of item 111.
(Item 113)
The accuracy score of each conforming model further comprises determining the respective accuracy score of the conforming prediction model, wherein the accuracy score represents the accuracy with which the conforming model predicts the outcome of one or more prediction problems. the method of.
(Item 114)
The method of item 113, further comprising determining the difference between the accuracy score of the conforming primary model and the accuracy score of the conforming secondary model.
(Item 115)
The method of item 114, wherein the accuracy score of the conforming secondary model exceeds the accuracy score of the conforming primary model.
(Item 116)
95. The method of item 95, further comprising determining the amount of computational resources used by each of the fitted prediction models to predict the outcome of one or more prediction problems.
(Item 117)
The method of item 116, further comprising determining the difference between the amount of computational resources used by the conforming primary model and the amount of computational resources used by the conforming secondary model.
(Item 118)
117. The method of item 117, wherein the amount of computational resources used by the conforming secondary model is less than the amount of computational resources used by the conforming primary model.
(Item 119)
95. The method of item 95, further comprising developing the conforming secondary model.
(Item 120)
Expanding the fitted quadratic model involves generating multiple predictions by applying the fitted quadratic model to other data representing an instance of the prediction problem, the secondary input data. 119. The method of item 119, which does not include the other data.
(Item 121)
The conforming secondary model comprises one or more sets of conditional rules, the set of one or more conditional rules comprises one or more sets of machine-executable if-then statements. 119. The method of item 119.
(Item 122)
The quadratic input data is the first quadratic input data, and developing the fitted quadratic model further provides the fitted quadratic model, at least in part, based on the second quadratic input data. 119. The method of item 119, comprising refreshing.
(Item 123)
The fitted secondary model is a first fitted secondary model, and refreshing the fitted secondary model, at least in part, based on the second secondary input data
To generate the second secondary training data and the second secondary test data from the second secondary input data,
By adapting the secondary prediction model to the second secondary training data, a second fitted secondary model of the fitted primary model can be generated.
To test the second conforming secondary model of the primary model on the second secondary test data,
To generate a refreshed secondary prediction model by mixing the first fitted secondary model and the second fitted secondary model.
122. The method of item 122.
(Item 124)
The fitted secondary model is a first fitted secondary model, and refreshing the fitted secondary model, at least in part, based on the second secondary input data
Generating a third secondary input data comprising at least a portion of the first secondary input data and at least a portion of the second secondary input data.
To generate the third secondary training data and the third secondary test data from the third secondary input data,
By fitting the secondary prediction model to the third secondary training data, a second fitted secondary model of the fitted primary model can be generated.
To test the second conforming secondary model of the primary model on the third secondary test data.
122. The method of item 122.
(Item 125)
The method according to item 95, wherein the first input variable is the second input variable.
(Item 126)
95. The method of item 95, wherein both the first input variable and the second input variable include a particular input variable.
(Item 127)
The method according to item 95, wherein none of the first input variables are included in the second input variable.
(Item 128)
The quadratic modeling procedure is one of a plurality of quadratic modeling procedures, the quadratic prediction model is one of a plurality of second prediction models, and the method is described above. 95. The method of item 95, comprising performing the plurality of secondary modeling procedures on the fitted primary model, thereby generating a plurality of fitted secondary models of the fitted primary model.
(Item 129)
The accuracy score of each conforming secondary model further comprises determining the respective accuracy score of the conforming secondary prediction model, which indicates the accuracy with which the conforming secondary model predicts the outcome of one or more prediction problems. Represented, the method of item 128.
(Item 130)
Determining which of the accuracy scores is the highest
To develop the conforming secondary model with the highest accuracy score
129. The method of item 129.
(Item 131)
Predictive modeling device
A memory configured to store a machine-executable module that encodes a secondary predictive modeling procedure associated with a secondary predictive model, said secondary predictive modeling procedure with at least one preprocessing task. Memory, including multiple tasks, including at least one model conforming task.
Executing the machine-executable module with at least one processor configured to execute the machine-executable module implements the secondary predictive modeling procedure on the device and the fitted primary predictive model. And letting the fitted primary prediction model perform the secondary prediction modeling procedure.
Performing the preprocessing task, including acquiring the fitted primary prediction model, which primary prediction model is one of the prediction problems based on the values of one or more first input variables. Or it is configured to predict the value of multiple output variables,
Carrying out the model fitting task, and carrying out the model fitting task
Generating secondary input data that includes multiple secondary observations, where each secondary observation combines the individual observations of one or more second input variables with the predicted values of the output variables. Including, generating the secondary input data means obtaining the individual observation values of the second input variable and the corresponding observation values of the first input variable for each secondary observation, and the first. Including applying the primary prediction model to the corresponding observations of the input variable of 1 to generate individual predictions of the output variable.
To generate secondary training data and secondary test data from the secondary input data,
By adapting the secondary prediction model to the secondary training data, a conforming secondary prediction model of the conforming primary model can be generated.
To test the conforming secondary prediction model of the conforming primary model on the secondary test data.
Including, and
Including the processor and
A device that comprises.

Certain advantages of some embodiments may be understood by reference to the following description, which is interpreted in conjunction with the accompanying drawings. In drawings, similar reference characters generally refer to the same part throughout different drawings. It is also emphasized that the drawings are not necessarily at a constant scale and instead, in general, illustrate some principles of the invention.

FIG. 1 is a block diagram of a predictive modeling system according to some embodiments.

FIG. 2 is a block diagram of a modeling tool for building machine-executable templates that encode predictive modeling tasks, techniques, and methods, according to some embodiments.

FIG. 3 is a flowchart of a method for selecting a prediction model for a prediction problem according to some embodiments.

FIG. 4 shows another flowchart of a method for selecting a prediction model for a prediction problem, according to some embodiments.

FIG. 5 is a schematic diagram of a predictive modeling system according to some embodiments.

FIG. 6 is another block diagram of the predictive modeling system according to some embodiments.

FIG. 7 illustrates communication between components of a predictive modeling system according to some embodiments.

FIG. 8 is another schematic of the predictive modeling system according to some embodiments.

FIG. 9 is a flowchart of a method for time series prediction modeling according to some embodiments.

FIG. 10 is a flowchart of a method for determining a predicted value of a feature according to some embodiments.

FIG. 11A is a flow chart of a method for generating a secondary prediction model according to some embodiments.

FIG. 11B is a flow chart of a method for performing a secondary prediction modeling procedure according to some embodiments.

Overview of Predictive Modeling System With reference to FIG. 1, in some embodiments, the predictive modeling system 100 includes a predictive modeling search engine 110, a user interface 120, a library of predictive modeling techniques 130, and a predictive model. Includes deployment engine 140. The search engine 110 efficiently explores the predictive modeling search space (eg, potential pre-processing steps, modeling algorithms, and post-processing steps) to generate a predictive modeling solution suitable for the defined prediction problem. A search technique (or "modeling method") for (combination) may be implemented. The search technique may include an initial assessment of which predictive modeling technique is likely to provide a suitable solution for the predictive problem. In some embodiments, the search technique is a gradual evaluation of the search space (eg, using an increasing part of the dataset) and a different model for a predictive problem (eg, using consistent metrics). Includes a consistent comparison of the suitability of the chemical solutions. In some embodiments, the search technique can be adapted based on the results of previous searches to improve the effectiveness of the search technique over time.

The search engine 110 may use the library 130 of modeling techniques to evaluate potential modeling solutions within the search space. In some embodiments, the modeling technique library 130 includes a machine executable template that encodes the complete modeling technique. The machine executable template may include one or more predictive modeling algorithms. In some embodiments, the modeling algorithms included in the template may be related in some way. For example, the modeling algorithm may be a variant of the same modeling algorithm or a component of the modeling algorithm family. In some embodiments, the machine executable template further comprises one or more pre- and / or post-processing steps suitable for use with the template's algorithm. Algorithms, pre-processing steps, and / or post-processing steps may be parameterized. Machine executable templates may be applied to user datasets to generate potential predictive modeling solutions for the predictive problems represented by the dataset.

The search engine 110 may use the computational resources of a distributed computer system to search the search space or parts thereof. In some embodiments, the search engine 110 uses the resources of the distributed computer system to generate a search plan for efficiently performing the search, and the distributed computer system performs the search according to the search plan. To do. Decentralized computer systems, but not limited to, partition search plans and decentralize computer system resources to queue and monitor predictive modeling techniques, to virtualize computer system resources, to access databases. An interface that facilitates the execution of the predictive modeling solution according to the search plan may be provided, including an interface for allocating to the evaluation of the modeling technique, collecting and organizing the execution results, accepting user input, and the like.

The user interface 120 provides tools for monitoring and / or guiding searches in the predictive modeling space. These tools provide insights into the dataset of predictive problems (for example, by highlighting problematic variables in the dataset, identifying relationships between variables in the dataset, etc.), and / Or may provide insights into the results of the search. In some embodiments, the data analyst identifies the criteria for recognizing a suitable modeling solution, for example by defining the metrics used to evaluate and compare the modeling solutions. The interface may be used to guide the search, such as by. Therefore, the user interface may be used by the analyst to improve its own productivity and / or to improve the performance of the search engine 110. In some embodiments, the user interface 120 presents search results in real time and the user in real time (eg, adjusting the scope of the search, or the allocation of resources between evaluations of different modeling solutions). Allows you to navigate the search. In some embodiments, the user interface 120 provides tools for coordinating the efforts of multiple data analysts working on the same predictive and / or related predictive problems.

In some embodiments, the user interface 120 provides a tool for developing a machine executable template for the library 130 of modeling techniques. System users may use these tools to modify existing templates, create new templates, or remove templates from library 130. In this way, system users may update library 130 to reflect advances in predictive modeling research and / or to include dedicated predictive modeling techniques.

The model deployment engine 140 provides a tool for deploying a prediction model (eg, a prediction model generated by the search engine 110) in the operating environment. In some embodiments, the model deployment engine also provides tools for monitoring and / or updating predictive models. System users deploy predictive models generated by the search engine 110, monitor the performance of such predictive models, and (eg, based on new data or advances in predictive modeling techniques). Deployment engine 140 may be used to update the model. In some embodiments, the search engine 110 refits or adjusts the prediction model (eg, in response to changes in the underlying data set for the prediction problem) of the search space for the prediction problem. Data collected and / or generated by the deployment engine 140 may be used to guide the search (eg, based on the results of monitoring the performance of the deployed prediction model).

These and other aspects of the predictive modeling system 100 are described in more detail below.

Library of Modeling Techniques Library 130 of predictive modeling techniques includes a machine executable template that encodes a fully predictive modeling technique. In some embodiments, the machine executable template is suitable for use with one or more predictive modeling algorithms and zero or more preprocessing steps suitable for use with the algorithm. Includes zero or more post-processing steps. Algorithms, pre-processing steps, and / or post-processing steps may be parameterized. Machine executable templates may be applied to a dataset to generate a potential predictive modeling solution for the predictive problems represented by the dataset.

The template may encode suitable pre-processing steps, model fitting steps, and / or post-processing steps for use with the template's predictive modeling algorithm for machine execution. Examples of preprocessing steps include, but are not limited to, inputting missing values, feature engineering (eg, one-hot encoding, spline, text mining, etc.), feature selection (eg, rejecting features that give no information). , Rejecting highly correlated features, replacing the original features with the most important components, etc.). Examples of model fitting steps include, but are not limited to, algorithm selection, parameter estimation, hyperparameter tuning, scoring, diagnosis, and the like. Examples of post-processing steps include, but are not limited to, predictive calibration, censorship, mixing, and the like.

In some embodiments, the machine executable template contains metadata that represents the attributes of the predictive modeling technique encoded by the template. Metadata is one or more data processing techniques that a template can perform as part of a predictive modeling solution (eg, in a pre-processing step, in a post-processing step, or in a step of a predictive modeling algorithm). May be indicated. These data processing techniques may include, but are not limited to, text mining, feature normalization, dimension reduction, or other suitable data processing techniques. As an alternative, or in addition, the metadata is a template-encoded prediction model that includes, but is not limited to, constraints on the dimensions of the dataset, the characteristics of the target of the prediction problem, and / or the characteristics of the characteristics of the prediction problem. It may indicate one or more data processing constraints imposed by the template.

In some embodiments, the template metadata contains information related to estimating how well the corresponding modeling technique would be useful for a given dataset. For example, template metadata can be a wide range of data sets, tall data sets, sparse data sets, rich data sets, data sets with or without text, and various data types (eg, numbers,). Data sets containing variables of order, category, interpretation (eg, date, time, text, etc.), variables with various statistical properties (eg, statistical properties of missing values, radix, distribution, etc. of the variable). It may indicate how well the corresponding modeling technique is expected to work for datasets with specific characteristics, including datasets that include. As another example, template metadata may indicate how well the corresponding modeling technique is expected to work for prediction problems involving a particular type of target variable. In some embodiments, the template metadata may indicate the expected performance of the corresponding modeling technique with respect to one or more performance metrics (eg, objective functions).

In some embodiments, the template metadata characterizes the processing steps implemented by the corresponding modeling techniques, including, but not limited to, the allowed data types, structures, and / or dimensions of the processing steps. including.

In some embodiments, the template metadata is data that indicates the (actual or anticipatory) outcome of applying the predictive modeling technique represented by the template to one or more predictive problems and / or datasets. including. The results of applying a predictive modeling technique to a predictive problem or dataset are not limited, but the accuracy, predictive problem or data that the predictive model generated by the predictive modeling technique predicts the target of the predictive problem or dataset. Use predictive modeling techniques to generate predictive model accuracy ranks, predictive problems, or datasets of predictive models generated by predictive modeling techniques (as opposed to other predictive modeling techniques) for sets. It may also include a score (eg, a value generated by a predictive model for the objective function) that represents the usefulness of the thing.

Data showing the results of applying a predictive modeling technique to a predictive problem or dataset can be found in the search engine (for example, based on the results of trials before using the predictive modeling technique for the predictive problem or dataset). It may be provided by 110, provided by the user (eg, based on the user's expertise), and / or obtained from any other suitable source. In some embodiments, the search engine 110 is based, at least in part, on the relationship between the actual outcomes of an instance of a prediction problem and the outcomes predicted by the prediction model generated through predictive modeling techniques. And update such data.

In some embodiments, the template metadata is related to estimating how efficiently the modeling technique will be performed on the distributed computing infrastructure, of the corresponding modeling technique. Represents a characteristic. For example, template metadata is rowed by modeling techniques on a dataset of a given size, the impact on resource consumption of the number of cross-validation layers and the number of points retrieved in the hyperparameter space, and modeling techniques. It may indicate the processing resources required to train and / or test the unique parallelization of the processing steps.

In some embodiments, the modeling technique library 130 includes tools for assessing similarities (or differences) between predictive modeling techniques. Such tools include scores (eg, on a given scale), classifications (eg, "highly similar", "slightly similar", "slightly heterogeneous", "highly heterogeneous"), The similarity between the two predictive modeling techniques may be expressed as a dual determination (eg, "similar" or "dissimilar"). Such tools are based on processing steps common to modeling techniques, based on data showing the results of applying two predictive modeling techniques to the same or similar predictive problems, and so on. Similarities between object-modeling techniques may be determined. Considering, for example, two predictive modeling techniques that have a large number (or a high percentage) of their processing steps in common and / or produce similar results when applied to similar prediction problems, the tool models. You may assign a high similarity score to the technique, or classify the modeling technique as "highly similar".

In some embodiments, the modeling technique may be assigned to a group of modeling techniques. Group classification of modeling techniques is assigned by the user (eg, based on intuition and experience) and is the result of applying common processing steps to modeling techniques, different modeling techniques to the same or similar problem It may be assigned by a machine learning classifier (based on data indicating, etc.) or obtained from another suitable source. Tools for assessing similarities between predictive modeling techniques may rely on group classification to assess similarities between two modeling techniques. In some embodiments, the tool may treat all modeling techniques within the same group as "similar" and any modeling technique within different groups as "dissimilar". In some embodiments, the grouping of modeling techniques may be only one factor in assessing the tools of similarity between modeling techniques.

In some embodiments, the predictive modeling system 100 includes a library of predictive problems (not shown in FIG. 1). The library of prediction problems may contain data showing the characteristics of the prediction problem. In some embodiments, the data demonstrating the characteristics of the prediction problem includes data demonstrating the characteristics of the dataset representing the prediction problem. The characteristics of the dataset are, but not limited to, the width, height, sparseness, or density of the dataset, the number of targets and / or features in the dataset, and the data type of the dataset's variables (eg, numbers,). It may include the order, category, or interpretation (eg, date, time, text, etc.), the range of numeric variables in the dataset, the order of the dataset, the number of categories of categorical variables, and so on.

In some embodiments, the characteristics of the dataset are, but are not limited to, the number of total observations, the number of unique values for each variable across observations, the number of missing values for each variable across observations, outliers and normal values. Includes the statistical properties of variables in a dataset, including the existence and scope of, the nature of the distribution of values or category memberships of each variable, the radix of the variables, and so on. In some embodiments, the characteristics of the dataset are, but not limited to, the simultaneous distribution of groups of variables, the variable importance of one or more features to one or more targets (eg, features and). Relationships between variables in a dataset, including the degree of correlation between target variables), statistical relationships between two or more features (eg, the degree of multicollinearity between two features), and the like. Includes (eg, statistical relationships).

In some embodiments, the data demonstrating the characteristics of the forecasting problem is data demonstrating the subject of the forecasting problem (eg, finance, insurance, defense, e-commerce, retail, internet-based advertising, internet-based recommendation engine, etc.), Whether the origin of the variables (eg, each variable was obtained directly from automatic instrumentation, from human records of automatic instrumentation, from human measurements, from written human responses, from verbal human responses, etc.) Includes the existence and performance of known predictive modeling solutions for predictive problems, etc.

In some embodiments, the prediction modeling system 100 may support time series prediction problems (eg, one-dimensional or multidimensional time series prediction problems). For time series prediction problems, the purpose is generally to predict future values of the target as a function of pre-observation of all features, including the target itself. The data that characterizes the prediction problem is the time series prediction problem by indicating whether the prediction problem is a time series prediction problem and by identifying the time measurement variables in the dataset corresponding to the time series prediction problem. May be adapted to.

In some embodiments, the library of predictive questions includes tools for assessing similarities (or differences) between predictive questions. Such tools include scores (eg, on a given scale), classifications (eg, "highly similar", "slightly similar", "slightly heterogeneous", "highly heterogeneous"), The similarity between the two prediction problems may be expressed as a dual determination (eg, "similar" or "dissimilar"). Such tools have similarities between two predictive problems, such as based on data demonstrating the characteristics of a predictive problem and based on data showing the results of applying the same or similar predictive modeling technique to the predictive problem. May be determined. For example, considering two predictive problems represented by a dataset that have a large number (or high percentage) of characteristics in common and / or are susceptible to the same or similar predictive modeling techniques, the tool is a predictive problem. May be assigned a high similarity score, or the prediction problem may be classified as "highly similar".

FIG. 2 is a block of modeling tools 200 suitable for constructing machine executable templates that encode predictive modeling techniques, and for incorporating such templates into predictive modeling methods, according to some embodiments. The figure is illustrated. The user interface 120 may provide an interface to the modeling tool 200.

In the embodiment of FIG. 2, the modeling method builder 210 builds the modeling method library 212 on top of the modeling technique library 130. The modeling technique builder 220 builds the modeling technique library 130 on top of the modeling task library 232. The modeling method may correspond to one or more analysts' intuitions and experiences about what modeling techniques work in what situations, and / or modeling for predictive problems. The results of applying modeling techniques to previous prediction problems may be leveraged to guide the search of the search space. The modeling technique may correspond to a step-by-step recipe for applying a concrete modeling algorithm. The modeling task may correspond to a processing step within the modeling technique.

In some embodiments, the modeling technique may include a hierarchy of tasks. For example, the top-level "text mining" task may include a subtask for (a) creating a document term matrix, (b) ranking terms, and rejecting non-essential terms. In turn, the "term ranking and rejection" subtask may include a subtask for (b.1) building a ranking model and (b.2) using term ranks to reject columns from the document term matrix. Such a hierarchy may have an arbitrary depth.

In the embodiment of FIG. 2, the modeling tool 200 includes a modeling task builder 230, a modeling technique builder 220, and a modeling method builder 210. Each builder may include a tool or set of tools for encoding one of the modeling elements in a machine executable format. Each builder may allow the user to modify an existing modeling element or create a new modeling element. To build a complete library of modeling elements across the modeling layer illustrated in Figure 2, developers employ top-down, bottom-up, inside-out, outside-in, or compound strategies. May be good. However, since leaf-level tasks are the smallest modeling element in terms of logical dependencies, FIG. 2 depicts task creation as the first step in the process of building a machine executable template.

The user interface of each builder abstractly defines, but is not limited to, a set of special routines in a standard programming language, a formal grammar specifically designed for the purpose of encoding the elements of that builder, and the desired execution flow. It may be implemented using a rich user interface or the like. However, the logical structure of the operation allowed at each layer is independent of any particular interface.

When creating modeling tasks at the leaf level in the hierarchy, the modeling tool 200 may allow developers to incorporate software components from other sources. This ability utilizes the accumulated knowledge of the software installation infrastructure associated with statistical learning and how to develop such software. This installation platform includes scientific programming languages (eg Fortran), scientific routines written in general-purpose programming languages (eg C), scientific computational extensions to general-purpose programming languages (eg scikit-learn for Python), and commercial statistics. It covers environments (eg SAS / STAT) and open source statistical environments (eg R). When used to incorporate the capabilities of such a software component, the Modeling Task Builder 230 allows the software component to perform input and output specifications and / or any type of operation of the software component. It may be necessary to characterize whether it can be done. In some embodiments, the modeling task builder 230 checks the source code signature of the software component, reads the interface definition of the software component from the repository, and scrutinizes the software component with a set of requirements. This metadata is generated by doing this, or by performing some other form of automatic evaluation. In some embodiments, the developer manually supplies some or all of this metadata.

In some embodiments, the Modeling Task Builder 230 uses this metadata to create a "trumpet" that allows it to run the embedded software. Modeling Task Builder 230 expects, but is not limited to, compiling the component source code into an internal executable, linking the component object code into the internal executable, and the component's stand-alone executable. Accessing components through an emulator in a computer environment, accessing the functionality of components that operate as part of a software service on a local machine, and operating components that operate as part of a software service on a remote machine. Such a wrapper utilizes any mechanism for integrating software components, including accessing functions, accessing the functions of components operating through intermediate software services on a local or remote machine, and so on. May be implemented. No matter which built-in mechanism is used by the modeling task builder 230, after the wrapper is generated, the modeling tool 200 makes software calls to the components as it would in any other routine. Good.

In some embodiments, the developer may use Modeling Task Builder 230 to recursively assemble leaf-level modeling tasks into higher-level tasks. As previously shown, there are many different ways to implement a user interface to specify an array of task hierarchies. However, from a logical point of view, tasks that are not at the leaf level may include directed graphs of subtasks. At each of the top and middle levels of this hierarchy, there may be one starting subtask whose input is from the parent task in the hierarchy (or the parent modeling technique at the top level of the hierarchy). There may also be one ending subtask whose output goes to the parent task in the hierarchy (or the parent modeling technique at the top level of the hierarchy). Every other subtask at a given level may receive input from one or more previous subtasks and send output to one or more subsequent subtasks.

Propagating data according to a directed graph, combined with the ability to incorporate arbitrary code into leaf-level tasks, facilitates the implementation of arbitrary control flows within intermediate-level tasks. In some embodiments, the modeling tool 200 may provide additional built-in behavior. For example, while it would be straightforward to implement any particular conditional logic as a leaf-level task coded in an external programming language, Modeling Task Builder 230 is a built-in node that performs conditional evaluation in a general way. Alternatively, an arc may be provided and some or all of the data may be directed from one node to a different subsequent node based on the results of these evaluations. One before a similar alternative propagates it as input to a subsequent node, filtering the output from one node according to a rule or expression before propagating it as input to a subsequent node. Convert output from a node, split output from one node according to a rule or expression before propagating each partition to its subsequent node, multiple according to a rule or expression before accepting it as input It exists to iteratively apply subgraphs of node behavior, etc., using one or more loop variables that combine the outputs of the previous node.

In some embodiments, the developer may use the modeling technique builder 220 to assemble the tasks from the modeling task library 232 into the modeling technique. At least some of the modeling tasks in the modeling task library 232 may correspond to one or more pre-processing steps, model fitting steps, and / or post-processing steps of the modeling technique. The development of tasks and techniques may follow a linear pattern in which the techniques are assembled after the task library 232 is populated, or a more dynamic circular pattern in which the tasks and techniques are assembled simultaneously. Developers may be tempted to incorporate existing tasks into new techniques, recognize that the technique requires new tasks, and iteratively refine until the new technique is complete. As an alternative, developers will probably start with the concept of new techniques from scholarly publications and start building it with new tasks, but when providing suitable functionality, existing tasks from the modeling task library 232. May be pulled out. In all cases, the results from applying the modeling technique to the reference dataset or in field testing will allow the developer or analyst to evaluate the performance of the technique. This evaluation, in turn, can result in changes anywhere in the hierarchy from leaf-level modeling tasks to modeling techniques. By providing a common modeling task and modeling technique library (232, 130), as well as a highly productive builder interface (210, 220, and 230), Modeling Tool 200 allows developers to make rapid and accurate changes. At the same time, access to libraries (232, 130) may be used to allow such enhancements to be propagated to other developers and users.

The modeling technique may provide a focus for developers and analysts to conceptualize the entire predictive modeling procedure, using all the steps expected based on the best practices in the field. In some embodiments, the modeling technique encapsulates the best practice from statistical learning discipline. Modeling Tool 200 also provides, for example, a checklist of steps for developers to consider, such as detecting missing tasks, detecting additional steps, and / or anomalies between steps. The task graph for the new technique may be compared with that of the existing technique to provide guidance in the development of high quality techniques so as to detect the flow.

In some embodiments, the search engine 110 is used to build a predictive model for the dataset 240 using the techniques in the modeling technique library 130. The search engine 110 may prioritize the evaluation of the modeling technique in the modeling technique library 130 based on the prioritization scheme encoded by the modeling method selected from the modeling method library 212. Examples of preferred prioritization schemes for exploring the modeled space are described in the next section. In the embodiment of FIG. 2, the results of the modeled space search may be used to update the metadata associated with the modeling tasks and techniques.

In some embodiments, a unique identifier (ID) may be assigned to the modeling element (eg, technique, task, and subtask). The ID of the modeling element may be stored as metadata associated with the template of the modeling element. In some embodiments, these modeling element IDs may be used to efficiently perform modeling techniques that share one or more modeling tasks or subtasks. How to perform modeling techniques efficiently is described in more detail below.

In the embodiment of FIG. 2, the modeling result generated by the search engine 110 is fed back to the modeling task builder 230, the modeling technique builder 220, and the modeling method builder 210. The modeling builder may be adapted automatically (eg, using a statistical learning algorithm) or manually (eg, by the user) based on the modeling results. For example, the modeling method builder 210 may be adapted based on the patterns observed in the modeling results and / or based on the experience of the data analyst. Similarly, the results of performing concrete modeling techniques may signal automatic or manual adjustment of the default tuning parameter values for these techniques or tasks within them. In some embodiments, the fitting of the modeled builder can be semi-automatic. For example, the predictive modeling system 100 may flag potential improvements to methods, techniques, and / or tasks, and the user may decide whether to implement these potential improvements.

Modeling Spatial Search Engine FIG. 3 is a flowchart of Method 300 for selecting a prediction model for a prediction problem, according to some embodiments. In some embodiments, method 300 may correspond to the modeling method in the modeling method library 212.

In step 310 of method 300, the suitability of a plurality of predictive modeling procedures (eg, predictive modeling techniques) for a predictive problem is determined. The suitability of a predictive modeling procedure for a predictive problem may be determined based on the characteristics of the predictive problem, based on the attributes of the modeled procedure, and / or based on other suitable information.

The "suitability" of a predictive modeling procedure for a predictive problem may include data indicating the expected performance of the predictive model generated using the predictive modeling procedure for the predictive problem. In some embodiments, the expected performance of the prediction model for the prediction problem is one or more expected scores (eg, the expected value of one or more objective functions), and /. Alternatively, it may include one or more expected ranks (eg, relative to other predictive models generated using other predictive modeling techniques).

As an alternative, or in addition, the "suitability" of a predictive modeling procedure for a predictive problem is that the modeling procedure is expected to generate a predictive model that provides sufficient performance for the predictive problem. It may include data indicating the degree. In some embodiments, the "suitability" data for a predictive modeling procedure comprises a classification of suitability for the modeling procedure. Classification schemes are two categories (eg, "suitable" or "not suitable"), or two or more categories (eg, "highly suitable", "moderately suitable", "moderately inappropriate". , "Highly inappropriate").

In some embodiments, the search engine 110 predicts, at least in part, based on one or more of the properties of the prediction problem, including (but not limited to) the properties described herein. Determine the suitability of the predictive modeling procedure for the problem. As an example only, the suitability of a predictive modeling procedure for a predictive problem is the characteristics of the dataset corresponding to the predictive problem, the characteristics of the variables in the dataset corresponding to the predictive problem, the variables in the dataset. It may be determined based on the relationship between and / or the subject of the prediction problem. The search engine 110 may include a tool (eg, a statistical analysis tool) for analyzing the data set associated with the prediction problem so as to determine the characteristics of the prediction problem, the dataset, the dataset variables, and the like.

In some embodiments, the search engine 110 includes, but is not limited to, the attributes of the predictive modeling procedure described herein, at least in part, in one or more of the predictive modeling procedures. Determine the suitability of a predictive modeling procedure for a predictive problem based on the attributes that surpass. In only one embodiment, the suitability of a predictive modeling procedure for a predictive problem is determined based on the data processing techniques performed by the predictive modeling procedure and / or the data processing constraints imposed by the predictive modeling procedure. You may.

In some embodiments, the step of determining the suitability of a predictive modeling procedure for a predictive problem includes removing at least one predictive modeling procedure from consideration of the predictive problem. The decision to exclude a predictive modeling procedure from consideration may be referred to herein as "removing" and / or "removing the search space" of the excluded modeling procedure. In some embodiments, the user configures a search engine modeling procedure so that the pre-removed modeling procedure remains eligible for further execution and / or evaluation during the search space exploration. The decision to remove can be overridden.

The predictive modeling procedure may be excluded from consideration based on the result of applying one or more deductive rules to the attributes of the predictive modeling procedure and the characteristics of the predictive problem. The descriptive rules are not limited, but are as follows: (1) if the prediction problem contains categorical target variables, only the classification technique for execution is selected, (2) the numerical characteristics of the dataset are very diverse. Select or prioritize techniques that provide normalization if the scale range is large, (3) Select or prioritize techniques that provide text mining if the dataset has text features, (4) Observe the dataset Exclude all techniques that require the number of observations to exceed or equal to the number of features if they have more features, (5) Dimension reduction if the width of the dataset exceeds the threshold width. Select or prioritize the techniques provided: (6) If the dataset is large and sparse (eg, the size of the dataset exceeds the threshold size and the sparseness of the dataset exceeds the threshold sparseness), then the sparse data structure May include any rules for selecting, prioritizing, or excluding modeling techniques that select or prioritize techniques that are efficiently performed in, and / or rules can be expressed in the form of if-then statements. .. In some embodiments, the deductive rules are chained so that the execution of some rules in turn yields conclusions. In some embodiments, the deductive rules may be updated, refined, or refined based on historical performance.

In some embodiments, the search engine 110 determines the suitability of the predictive modeling procedure for the predictive problem based on the performance (expected or actual) of the similar predictive modeling procedure for the similar predictive problem. (As a special case, the search engine 110 may determine the suitability of a predictive modeling procedure for a predictive problem based on the performance (expected or actual) of the same predictive modeling procedure for a similar predictive problem. Good.)

As described above, the library of modeling techniques 130 may include tools for assessing similarities between predictive modeling techniques, and the library of predictive problems may include similarities between predictive problems. It may include tools for assessment. The search engine 110 may use these tools to identify predictive modeling procedures and predictive problems that are similar to the predictive modeling procedures and predictive problems in question. For the purpose of determining the suitability of the predictive modeling procedure for the predictive problem, the search engine 110 is in question, selecting the M modeling procedures that most closely resemble the modeling procedure in question. All modeling procedures that exceed the threshold similarity value for the modeling procedure may be selected and the like. Similarly, for the purpose of determining the suitability of a predictive modeling procedure for a predictive problem, the search engine 110 becomes a problem, selecting N predictive problems that are most similar to the predictive problem in question. All prediction problems that exceed the threshold similarity value for the prediction problem may be selected.

Given the set of predictive modeling procedures and the set of predictive problems that are similar to the modeling procedure and predictive problem in question, the search engine is the model in question for the predictive problem in question. The performance of the similarity modeling procedure for the similarity prediction problem may be combined to determine the expected suitability of the transformation procedure. As described above, a template for a modeling procedure may contain information related to estimating how well the corresponding modeling procedure would work for a given dataset. The search engine 110 may use model performance metadata to determine the performance value (expected or actual) of a similar modeling procedure for a similar prediction problem. These performance values can then be combined to generate an estimate of the suitability of the modeling procedure in question for the prediction problem in question. For example, the search engine 110 may calculate the suitability of the modeling procedure in question as a weighted sum of the performance values of the similar modeling procedure for the similarity prediction problem.

In some embodiments, the search engine 110 uses various modeling procedures (eg, the modeling in question) for other prediction problems (eg, prediction problems similar to the prediction problem in question). Based on the output of a "meta" machine learning model, which can be trained to determine the suitability of a modeling procedure for a prediction problem, based on the results of a modeling procedure similar to the procedure), at least in part. Determine the suitability of a predictive modeling procedure for a predictive problem. Machine learning models for estimating the suitability of predictive modeling procedures for predictive problems iterate to predict which technique is most likely to succeed for the predictive problem in question. Because it applies machine learning, it can be referred to as a "meta" machine learning model. Therefore, the search engine 110 produces a meta-prediction of the suitability of the modeling technique for the prediction problem by using a meta-machine learning algorithm directed at the results from solving other prediction problems. May be good.

In some embodiments, the search engine 110 is for a prediction problem, at least in part, based on user input (eg, user input that represents the intuition or experience of a data analyst regarding the suitability of a predictive modeling procedure). The suitability of the predictive modeling procedure may be determined.

Returning to FIG. 3, in step 320 of method 300, at least a subset of the predictive modeling procedures may be selected based on the suitability of the modeling procedure for the predictive problem. Modeling procedures are suitable categories (eg, "suitable" or "not suitable", "highly suitable", "moderately suitable", "moderately inappropriate", or "highly inappropriate") In an embodiment assigned to, the step of selecting a subset of modeling procedures is a modeling procedure assigned to one or more suitability categories (eg, all models assigned to a "suitable category"). It may include a step of selecting a modeling procedure, all modeling procedures not assigned to the "highly inappropriate" category, etc.).

In an embodiment in which the modeling procedure is assigned a preference value, the search engine 110 may select a subset of the modeling procedure based on the preference value. In some embodiments, the search engine 110 selects a modeling procedure with a suitability score above the threshold suitability score. The threshold suitability score may be provided by the user or determined by the search engine 110. In some embodiments, the search engine 110 increases or decreases the number of modeled procedures selected for execution, depending on the amount of processing resources available for executing the modeled procedure. , The threshold suitability score may be adjusted.

In some embodiments, the search engine 110 involves a modeling procedure with a suitability score within a defined range of the highest suitability score assigned to any of the modeling procedures for the prediction problem in question. Select. The range can be absolute (eg, score within S points of the highest score) or relative (eg, score within P% of the highest score). The range may be provided by the user or determined by the search engine 110. In some embodiments, the search engine 110 increases or decreases the number of modeled procedures selected for execution, depending on the amount of processing resources available for executing the modeled procedure. , The range may be adjusted.

In some embodiments, the search engine 110 selects part of a modeling procedure that has the highest suitability score for the prediction problem in question. Equally, the search engine 110 has the highest suitability (eg, when suitability scores for the modeling procedure are not available, but suitability ordering (ranking) for the modeling procedure is available). You may choose part of a modeling procedure that has a rank. Some may be provided by the user or determined by the search engine 110. In some embodiments, the search engine 110 increases or decreases the number of modeled procedures selected for execution, depending on the amount of processing resources available for executing the modeled procedure. , You may adjust a part.

In some embodiments, the user may choose one or more modeling procedures to be performed. User-selected procedures may be executed in addition to, or instead of, one or more modeling procedures selected by the search engine 110. By allowing the user to select a modeling procedure to perform, the intuition and experience of the data analyst, in particular, allows the modeling system 100 to accurately determine the suitability of the modeling procedure for predictive problems. The performance of the predictive modeling system 100 may be improved in scenarios that indicate no estimation.

In some embodiments, the search engine 110 uses the modeling procedure P0. .. .. Even if all PNs are determined to be suitable for the prediction problem in question, the modeling procedure P0. .. .. Rather than selecting a PN, one or more modeling procedures P1. .. .. The particle size of the search spatial evaluation may be controlled by selecting a modeling procedure P0 that represents PN (eg, similar). In addition, the search engine 110 uses the modeling procedure P1. .. .. The result of executing the selected modeling procedure P0 may be treated as representing the result of executing the PN. This approach to evaluating search space may save processing resources, especially if applied during the early stages of search space evaluation. If the search engine 110 later determines that the modeling procedure P0 is between the most suitable modeling procedures for the prediction problem, then a fine-grained evaluation of the relevant parts of the search space is performed by the similar modeling procedure P1. .. .. .. It can be done by running and evaluating the PN.

Returning to FIG. 3, in step 330 of method 300, a resource allocation schedule may be generated. The resource allocation schedule may allocate processing resources for the execution of the selected modeling procedure. In some embodiments, the resource allocation schedule allocates processing resources to the modeling procedure based on the determined suitability of the modeling procedure for the prediction problem in question. In some embodiments, the search engine 110 transmits the resource allocation schedule to one or more processing nodes with instructions for executing the modeling procedure selected according to the resource allocation schedule.

Allotted processing resources include time resources (eg, execution cycles of one or more processing nodes, execution time on one or more processing nodes, etc.), physical resources (eg, number of processing nodes, machines). It may include the amount of readable storage (eg, memory and / or secondary storage), and / or other assignable processing resources. In some embodiments, the allocated processing resources may be the processing resources of a distributed computer system and / or a cloud-based computer system. In some embodiments, costs may be incurred when processing resources are allocated and / or used (eg, data center operators in exchange for charges to use data center resources). May be collected by).

As shown above, the resource allocation schedule may allocate processing resources to the modeling procedure based on the suitability of the modeling procedure for the prediction problem in question. For example, resource allocation schedules provide more processing resources and higher predictive suitability for forecasting problems, so that more promising modeling procedures benefit from a greater proportion of limited processing resources. You may allocate less processing resources to the accompanying modeling procedure and to the modeling procedure with lower predictive suitability for the prediction problem. As another example, a resource allocation schedule is sufficient to allocate enough processing resources to process a larger dataset to a modeling procedure with higher predictive suitability and to process a smaller dataset. Processing resources may be allocated to modeling procedures with lower predictive suitability.

As another embodiment, the resource allocation schedule may schedule the execution of the modeling procedure with higher predictive suitability prior to the execution of the modeled procedure with lower predictive suitability. It can also have the effect of allocating more processing resources to more promising modeling procedures. In some embodiments, the result of executing the modeling procedure may be presented to the user via user interface 120 when the result becomes available. In such an embodiment, the search space is used early in the evaluation by scheduling the modeling procedure with the higher predictive suitability so that it is executed before the modeler procedure with the lower predictive suitability. It may provide the user with additional important information about the evaluation of the search plan, thereby facilitating rapid user-driven adjustment to the search plan. For example, based on preliminary results, the user determines that one or more modeling procedures that are expected to work very well are actually very incompletely working. May be good. The user may investigate the cause of the poor performance and determine, for example, that the bad performance is caused by an error in the preparation of the dataset. The user can then correct the error and resume execution of the modeling procedure affected by the error.

In some embodiments, the resource allocation schedule may allocate processing resources to the modeling procedure, at least in part, based on the resource utilization and / or parallelism characteristics of the modeling procedure. As explained above, the template corresponding to the modeling procedure provides metadata related to estimating how efficiently the modeling procedure will be executed on the distributed computing infrastructure. It may be included. In some embodiments, the metadata presents the resource utilization characteristics of the modeling procedure (eg, the processing resources required to train and / or test the modeling procedure on a dataset of a given size). Includes indicators for. In some embodiments, the metadata includes indicators of the concurrency characteristics of the modeled procedure (eg, the extent to which the modeled procedure can be executed in parallel on multiple processing nodes). Efficient allocation of processing resources to the modeling procedure may be facilitated by using the resource utilization and / or parallelism characteristics of the modeling procedure to determine the resource allocation schedule.

In some embodiments, the resource allocation schedule may allocate a defined amount of processing resources for the execution of the modeling procedure. The allocateable amount of processing resources may be specified in the processing resource budget, which may be provided by the user or obtained from another suitable source. The processing resource budget is borne by the processing resources used to execute the modeling procedure (eg, the amount of time used, the number of processing nodes used, the data center or cloud-based processing resources. You may impose restrictions on the costs to be paid, etc.). In some embodiments, the processing resource budget may impose a limit on all processing resources used in the process of generating a forecast model for a defined forecast problem.

Returning to FIG. 3, in step 340 of method 300, the result of executing the selected modeling procedure according to the resource allocation schedule may be received. These results may include one or more predictive models produced by the modeled procedure performed. In some embodiments, the prediction model received in step 340 is a prediction problem because the execution of the modeling procedure may include fitting the prediction model to one or more datasets associated with the prediction problem. Fits to the dataset associated with. The steps to fit the predictive model to the data set of the predictive problem are the steps to adjust hyperparameters of one or more predictive modeling procedures that generate the predictive model, one or more parameters of the generated predictive model. And / or other suitable model fitting steps may be included.

In some embodiments, the results received in step 340 include an assessment of the model's performance (eg, score) for the prediction problem. These assessments may be obtained by testing the prediction model on the test data set associated with the prediction problem. In some embodiments, the step of testing the predictive model involves cross-validating the model using different layers of the training dataset associated with the predictive problem. In some embodiments, execution of the modeling procedure involves testing the generated model. In some embodiments, testing of the generated model is done separately from the execution of the modeling procedure.

The model may be tested according to a suitable test technique and scored according to a suitable scoring metric (eg, objective function). Different scoring metrics, but not limited to, the accuracy of the model (eg, the rate at which the model correctly predicts the outcome of the prediction problem), the false positive rate (eg, the rate at which the model incorrectly predicts the "positive" outcome). , False negative rates (eg, the rate at which the model falsely predicts "negative" outcomes), positive predictive values, negative predictive values, susceptibility, specificity, etc., with different weights on different aspects of predictive model performance. May be good. The user can use standard scoring metrics (eg, goodness of fit, R-squares, etc.) from a set of options presented via user interface 120, or specific custom scoring metrics (eg, custom) via user interface 120. Objective function) may be selected. The search engine 110 may use user-selected or user-defined scoring metrics to score the performance of the predictive model.

Returning to FIG. 3, in step 350 of method 300, the prediction model may be selected for the prediction problem based on the evaluation (eg, score) of the generated prediction model. Spatial search engine 110 may use any suitable criteria to select a prediction model for the prediction problem. In some embodiments, the spatial search engine 110 may select a model with the highest score, or any model with a score above the threshold score, or any model with a score within the defined range of the highest score. Good. In some embodiments, the score of the predictive model may be only one factor considered by the spatial search engine 110 when selecting a predictive model for the predictive problem. Other factors considered by the spatial search engine may include, but are not limited to, the complexity of the prediction model, the computational demand of the prediction model, and the like.

In some embodiments, the step of selecting a predictive model for a predictive problem is the step of iteratively selecting a subset of the predictive model and the predictive model selected on a larger or different part of the dataset. It may include a step to train. The iterative process may continue until the predictive model is selected for the predictive problem or until the processing resources budgeted to generate the predictive model are exhausted.

The steps to select a subset of predictive models are: to select a portion of the predictive model with the highest score, to select all models with a score above the threshold score, to within the specified range of scores for the highest scoring model It may include the step of selecting all models having a score of, or the step of selecting any other suitable group of models. In some embodiments, the step of selecting a subset of the predictive model can be similar to the step of selecting a subset of the predictive modeling procedure, as described above with reference to step 320 of method 300. Therefore, the details of the steps of selecting a subset of the prediction model are not repeated here.

The step of training the selected predictive model may include the step of generating a resource allocation schedule that allocates the processing resources of the processing node for training the selected model. The allocation of processing resources is, at least in part, based on the suitability of the modeling technique used to generate the selected model and / or the score of the selected model for other samples in the dataset. It may be determined. The steps to train the selected predictive model further include the step of transmitting instructions to the processing nodes to fit the selected predictive model to the specified part of the dataset, and the fitted model and / or fitted model score. , A step of receiving the results of the training process and may be included. In some embodiments, the step of training the selected predictive model is similar to the step of executing the selected predictive modeling procedure, as described above with reference to steps 320-330 of Method 300. Can be done. Therefore, the details of the steps to train the selected predictive model are not repeated here.

In some embodiments, steps 330 and 340 are repeated until the prediction model is selected for the prediction problem or until the processing resources budgeted to generate the prediction model are exhausted. You may be broken. At the end of each iteration, the suitability of the predictive modeling procedure for the predictive problem may be re-determined, at least in part, based on the results of running the modeled procedure. The set may be selected for execution during the next iteration.

In some embodiments, the number of modeling procedures performed in the iterations of steps 330 and 340 may tend to decrease as the number of iterations increases, training and / or testing the generated model. The amount of data used for this may tend to increase as the number of iterations increases. Therefore, the initial iterations may "look for a wide range" by executing a relatively large number of modeling procedures on a relatively small dataset, and subsequent iterations will be identified during the initial iterations. More rigorous testing of the most promising modeling procedures may be performed. Alternatively, or in addition, the initial iterations may implement a coarser granularity assessment of the search space, and subsequent iterations may implement a finer granularity assessment of the portion of the search space that is determined to be the most promising. May be implemented.

In some embodiments, method 300 comprises one or more steps (not shown in FIG. 3). The additional steps of Method 300 include, but are not limited to, processing the dataset associated with the prediction problem, mixing two or more prediction models to form a mixed prediction model, and / or prediction. It may include the step of adjusting the prediction model selected for the problem. Some embodiments of these steps are described in more detail below.

Method 300 may include a step in which the dataset associated with the prediction problem is processed. In some embodiments, the steps of processing the data set of the prediction problem include the steps of characterizing the data set. The steps to characterize the dataset include data leaks (eg, the dataset contains features that strongly correlate with the target, but the values of the features are available as inputs to the prediction model under the conditions imposed by the prediction problem. Steps to identify (scenarios) that will not be, steps to detect missing observations, steps to detect missing variable values, steps to detect anomalous variable values, and / or significant predicted values ("important variables") It may include, but is not limited to, a step of identifying potential problems associated with the dataset, including, but not limited to, identifying variables that are likely to have.

In some embodiments, the steps of processing a predictor problem dataset include applying feature engineering to the dataset. The step of applying feature engineering to a dataset is the step of combining two or more features and replacing the constituent features with the combined features, different aspects of date / time variables (eg, time and seasonal information). It may include a step of extracting into a separate variable, a step of normalizing the variable value, a step of filling in the missing variable value, and the like.

Method 300 may include steps in which two or more prediction models are mixed to form a mixed prediction model. The mixing steps may be iterative in connection with the steps of performing predictive modeling techniques and evaluating the generated predictive model. In some embodiments, the mixing step may be performed in some of the execution / evaluation iterations (eg, when multiple promising predictive models are generated, in subsequent iterations).

Two or more models may be mixed by combining the outputs of the constitutive models. In some embodiments, the mixed model may include a weighted linear combination of the outputs of the constitutive model. The mixed prediction model may work better than the composition prediction model, especially if the different construction models are complementary. For example, a mixed model generates a model when the constitutive model tends to work well for different parts of the forecast problem dataset, and when the model mix works well for other (eg, similar) predictive problems. It may be expected to work well when the modeling techniques used to do this are heterogeneous (eg, one model is a linear model and the other is a tree model). In some embodiments, the configuration models that are mixed together are identified by the user (eg, based on the user's intuition and experience).

Method 300 may include a step in which the prediction model selected for the prediction problem is adjusted. In some cases, the deployment engine 140 provides the user with source code that implements the prediction model, thereby allowing the user to adjust the prediction model. However, disclosing the source code of a predictive model may be undesirable in some cases (eg, when the predictive modeling technique or predictive model contains specialized capabilities or information). To allow the user to adjust the predictive model without exposing the source code of the model, the deployment engine 140 adjusts the parameters of the model based on the representation of the predictive model (eg, mathematical representation). Human-readable rules may be constructed and human-readable rules may be provided to the user. The user can then use human-readable rules to adjust the model's parameters without accessing the model's source code. Therefore, the predictive modeling system 100 may support the evaluation and adjustment of the dedicated predictive modeling technique without exposing the source code for the dedicated modeling technique to the end user.

In some embodiments, the machine executable template corresponding to the predictive modeling procedure may include efficiency-enhancing features to reduce redundant computations. These efficiency-enhancing features can be especially valuable when a relatively small amount of processing resources are budgeted to explore the search space and generate predictive models. As described above, the machine executable template may store the unique ID of the corresponding modeling element (eg, technique, task, or subtask). In addition, the predictive modeling system 100 may assign a unique ID to the dataset sample S. In some embodiments, when the machine executable template T is run on the dataset sample S, the template is running the template on its modeling element ID, dataset / sample ID, and data sample. Store the results in a storage structure (eg, table, cache, hash, etc.) accessible to other templates. When the template T is called on the dataset sample S, the template checks the storage structure to determine if the result of running the template on the dataset sample is already stored. If so, rather than reprocessing the dataset sample to get the same result, the template simply reads the corresponding results from the storage structure, returns these results, and exits. The storage structure may persist within individual iterations of the loop in which the modeling procedure is executed, across multiple iterations of the procedure execution loop, or across multiple search space searches. Computational savings achieved through this efficiency-enhancing feature because many tasks and subtasks are shared by different modeling techniques, and method 300 often involves performing different modeling techniques on the same dataset. Can be a considerable amount.

FIG. 4 shows a flowchart of method 400 for selecting a prediction model for a prediction problem, according to some embodiments. The method 300 may be embodied by an embodiment of the method 400.

In the embodiment of FIG. 4, the spatial search engine 110 uses the modeling method library 212, the modeling technique library 130, and modeling to search the space of modeling techniques available for the solution of the predictive modeling problem. Use task library 232. First, the user may select a modeling method from library 212, or the spatial search engine 110 may automatically select a default modeling method. Available modeling methods are, but are not limited to, selection of modeling techniques based on the application of descriptive rules, selection of modeling techniques based on the performance of similar modeling techniques for similarity prediction problems, meta-machine learning models. It may include the selection of output-based modeling techniques, any combination of the modeling techniques described above, or other suitable modeling techniques.

In step 402 of method 400, the search engine 110 prompts the user to select a dataset to solve the predictive modeling problem. The user can create a new dataset from either a preloaded dataset or an instruction to read data from a file or other information system. In the case of files, the search engine 110 may include, but is not limited to, comma-separated values, tab-separated extended markup language (XML), JavaScript® object notation, native database files, etc. More formats may be supported. For instructions, the user specifies the types of information systems, their network addresses, access certificates, references to subsets of the data in each system, and rules for mapping the target data schema into the desired data schema. You may. Such information systems may include, but are not limited to, databases, data warehouses, data integration services, distributed applications, web services and the like.

In step 404 of method 400, the search engine 110 loads the data (eg, by reading a default file or accessing a default information system). Internally, the search engine 110 may construct a two-dimensional matrix using features on one axis and observations on the other axis. Conceptually, each column of the matrix may correspond to a variable, and each row of the matrix may correspond to an observation. The discovery engine 110 retrieves metadata from the original source (eg, an explicitly defined data type) and / or metadata generated during the loading process (eg, explicit data type of a variable, variable). Related metadata may be attached to the variable, including whether is considered to be a numeric, ordered, basic, or interpretation type).

In step 406 of method 400, the search engine 110 prompts the user to identify which of the variables is the target and / or which is the feature. In some embodiments, the search engine 110 is also referred to as a statistical optimization technique with a measure of model performance used to score the model (eg, a statistical learning algorithm implemented by the search engine 110). In a sense, it encourages the user to identify the optimized model performance metrics).

In step 408 of method 400, the search engine 110 evaluates the dataset. The evaluation may include steps to calculate the characteristics of the dataset. In some embodiments, the evaluation comprises performing a data set analysis that may help the user better understand the prediction problem. Such an analysis involves applying one or more algorithms to identify problematic variables (eg, those with outliers or normals), detecting variable importance, and variable. It may include a step of determining the impact and a step of identifying the impact hotspot.

Analysis of the dataset may be performed using any suitable technique. Variable importance, which measures the degree of significance each feature has in predicting a target, is "gradient boost tree", Breiman and Cutler's "random forest", "alternate conditional expectation", and / or It may be analyzed using other suitable techniques. Variable effects, which measure the direction and size of the effect of features on the target, may be analyzed using "normalized regression", "logistic regression", and / or other suitable techniques. Impact hotspots, which identify the areas where features provide the most information in predicting a target, may be analyzed using the "RuleFit" algorithm and / or other suitable techniques.

In some embodiments, in addition to the step of assessing the importance of the features contained in the original dataset, the evaluation performed in step 408 of Method 400 comprises feature generation. The feature generation technique may include the step of generating additional features by interpreting the logical type of the variables in the dataset and applying various transformations to the variables. Examples of transformations include, but are not limited to, polynomial and logarithmic transformations of numerical features. For interpretive variables (eg, date, time, currency, units of measure, percentages, and location coordinates), the examples of the transformation are, but not limited to, date strings to test each aspect of the date for predictive power. Includes steps to analyze continuous time variables, days of the week, months, and seasons.

Systematic transformations of numerical and / or interpretive variables, followed by those systematic tests using potential predictive modeling techniques, allow the predictive modeling system 100 to search more of the potential model space and more. It may be possible to achieve precise predictions. For example, in the case of "date / time", separating time and seasonal information into separate features is very important because these distinct features often exhibit very different relationships with the target variable. Can be beneficial.

Creating derived features by interpreting and transforming the original features can increase the dimensions of the original dataset. The predictive modeling system 100 may apply a dimension reduction technique that can counteract the increasing dimension of the dataset. However, some modeling techniques are more sensitive to dimensions than others. Also, different dimension reduction techniques tend to work better with some modeling techniques than others. In some embodiments, the predictive modeling system 100 maintains metadata representing these interactions. System 100 may systematically evaluate the various combinations of dimension reduction and modeling techniques and prioritize the combinations that the metadata indicates are most likely to succeed. System 100 may further update the metadata over time based on the empirical performance of the combination and incorporate new dimension reduction techniques as they are discovered.

In step 410 of method 400, the predictive modeling system 100 presents to the user the results of the dataset evaluation (eg, the results of the dataset analysis, the characteristics of the dataset, and / or the results of the dataset transformation). In some embodiments, the results of the dataset evaluation may be presented via the user interface 120 (eg, using graphs and / or tables).

In step 412 of method 400, the user may refine the dataset (eg, based on the results of the dataset evaluation). Such refinements deal with missing or outliers of one or more features, change the type of interpretive variable, change the transformation under consideration, exclude features from consideration, specific values. It may include steps to select methods for directly editing features, transforming features using functions, combining feature values using expressions, adding completely new features to a dataset, and so on.

Steps 402-412 of Method 400 may represent one embodiment of the step of processing a dataset of predictive problems, as described above in connection with some embodiments of Method 300.

In step 414 of method 400, the search space engine 100 may load the modeling techniques available from the modeling technique library 130. The determination of which modeling technique is available may depend on the modeling method chosen. In some embodiments, loading of the modeling technique may occur in parallel with one or more of steps 402-412 of Method 400.

In step 416 of method 400, the user instructs the search engine 110 to start searching for the modeling solution in either manual or automatic mode. In automatic mode, the search engine 110 uses a default sampling algorithm to partition the dataset (step 418) and a default prioritization algorithm to prioritize modeling techniques (step 420). The step of prioritizing modeling techniques is to determine the suitability of the modeling technique for the prediction problem and to select at least a subset of the modeling technique for execution based on those determined suitability. It may include steps and.

In manual mode, the search engine 110 proposes a data partition (step 422) and proposes prioritization of modeling techniques (step 424). The user may accept the proposed data partition or specify a custom partition (step 426). Similarly, the user may accept the proposed prioritization of the modeling technique or specify a custom prioritization of the modeling technique (step 428). In some embodiments, the user exceeds one or more (eg, using the modeling technique builder 220 and / or the modeling task builder 230) before the search engine 110 begins performing the modeling technique. The modeling technique can be modified (step 430).

To facilitate cross-validation, the predictive modeling system 100 may divide the dataset into K "layers" (or suggest splitting the dataset). Cross-validation includes the step of fitting the predictive model to a divided dataset K times during each fit so that different layers act as test sets and the remaining layers act as training sets. .. Cross-validation can generate useful information about how the accuracy of predictive models fluctuates with different training data. In steps 418 and 422, the predictive modeling system divides the dataset into K layers, the number of layers K being the default parameter. At step 426, the user may change the number of layers K or stop using cross-validation altogether.

To facilitate rigorous testing of the predictive model, the predictive modeling system 100 may divide the dataset into training sets and "holdout" test sets (or suggest splitting the dataset). In some embodiments, the training set is further divided into K layers for cross-validation. The training set may then be used to train and evaluate the predictive model, while the holdout test set may be strictly reserved for testing the predictive model. In some embodiments, the predictive modeling system 100 makes the holdout test set inaccessible (not for training) until the user with the specified authority and / or certificate releases it. ) The use of holdout test sets for testing can be strongly enforced. In steps 418 and 422, the predictive modeling system 100 may partition the dataset so that the default percentage of the dataset is reserved for the holdout set. At step 426, the user may change the percentage of dataset reserved for the holdout set or stop using the holdout set altogether.

In some embodiments, the predictive modeling system 100 divides the dataset during evaluation of the modeled search space to facilitate efficient use of computational resources. For example, the predictive modeling system 100 may divide the cross-validation layer of the dataset into smaller samples. Reducing the size of the data sample to which the predictive model is fitted may reduce the amount of computational resources required to evaluate the relative performance of different modeling techniques. In some embodiments, smaller samples may be generated by taking a random sample of layer data. Similarly, reducing the size of the data sample to which the prediction model is fitted may reduce the amount of computational resources required to adjust the parameters of the prediction model or the hyperparameters of the modeling technique. Hyperparameters include variable settings for modeling techniques that can affect the speed, efficiency, and / or accuracy of the model fitting process. Examples of hyperparameters include, but are not limited to, penalties parameters for elastic net models, the number of trees in the gradient boost tree model, the number of neighbors in the nearest neighbor model, and the like.

In step 432-458 of Method 400, the selected modeling technique may be performed using the partitioned data to evaluate the search space. These steps are described in more detail below. For convenience, some aspects of the search space evaluation for data partitioning are described in the following paragraphs.

Adjusting hyperparameters using sample data containing a cross-validation layer test set can lead to model overfitting, which can make performance comparisons of different models unreliable. Using a "prescriptive approach" can help avoid this problem and can provide some other important benefits. Therefore, some embodiments of the search engine 110 implement "nested cross-validation," a technique to which two loops of k-layer cross-validation are applied. The outer loop provides a test set for comparing a given model with other models and calibrating the predictions for each model on future samples. The inner loop provides both a test set for adjusting the hyperparameters of a given model and a training set for the derived features.

The cross-validation predictions generated in the inner loop may also facilitate a mixing technique that combines multiple different models. In some embodiments, the input into the blender is a prediction from an out-of-sample model. Using predictions from in-sample models can result in overfitting when used with several mixed algorithms. Without a well-defined process for consistently applying nested cross-validation, even the most experienced users can skip steps or implement them incorrectly. Therefore, the application of the double loop of k-validation is that the predictive modeling system 100 simultaneously adjusts the complex model with five important goals: (1) many hyperparameters (2). ) Create informative derived features, (3) adjust the mix of two or more models, (4) calibrate the predictions of single and / or mixed models, and (5) ) It may be possible to achieve maintaining a pure pristine test set that allows accurate comparison of different models.

In step 432 of method 400, the search engine 110 generates a resource allocation schedule for execution of the initial set of selected modeling techniques. The resource allocation represented by the resource allocation schedule may be determined based on modeling technique prioritization, partitioned data samples, and available computational resources. In some embodiments, the search engine 110 greedily allocates resources to the selected modeling technique (eg, assigns computational resources to, in turn, the highest priority modeling technique that has not yet been executed).

In step 434 of method 400, the search engine 110 initiates execution of the modeling technique according to a resource allocation schedule. In some embodiments, performing a set of modeling techniques may include training one or more models on the same data sample extracted from the dataset.

In step 436 of method 400, the search engine 110 monitors the state of execution of the modeling technique. When the modeling technique is finished running, the search engine 110 collects the results, which may include model fit metrics for the fit model and / or the corresponding data sample (step 438). Such metrics include, but are not limited to, any metrics that can be extracted from the underlying software components that make the fit, including Gini coefficients, r-squares, residual mean square errors, and any variations thereof. It may be included.

In step 440 of Method 400, the search engine 110 excludes the poorest performing modeling technique from consideration (eg, based on the performance of the model generated according to the model fit metrics). The search engine 110, but not limited to, steps to eliminate those that do not generate a model that meets the minimum threshold of the model fit metric, that produces the current model among the top-level parts of all generated models. Except, using any suitable technique, including the step of eliminating all modeling techniques, or the step of eliminating any modeling technique that has not generated a model within a range of the top-level model. You may decide whether to exclude the modeling technique. In some embodiments, different procedures may be used to eliminate modeling techniques at different stages of evaluation. In some embodiments, the user may be allowed to specify different exclusion techniques for different modeling problems. In some embodiments, the user may be allowed to build and use custom exclusion techniques. In some embodiments, metastatistical learning techniques may be used to select between exclusion techniques and / or to adjust the parameters of these techniques.

When the search engine 110 calculates the model performance and excludes the modeling technique from consideration, the predictive modeling system 100 may present the user with the progress of the search space evaluation through the user interface 120 (step 442). In some embodiments, in step 444, the search engine 110 modifies the process by which the user evaluates the search space based on the progress of the search space evaluation, the user's expertise, and / or other suitable information. Make it possible. If the user specifies modifications to the search spatial evaluation process, the spatial evaluation engine 110 reallocates processing resources accordingly (eg, determines which jobs are affected and moves them within the scheduling queue. Either let them or remove them from the queue). Other jobs continue processing as before.

The user may modify the search spatial evaluation process in many different ways. For example, a user may reduce the priority of some modeling techniques, or take into account some modeling techniques altogether, even if the model generated with the selected metric performs well. It may be excluded. As another example, the user may increase the priority of some modeling techniques or select some modeling techniques from consideration, even if the generated model is poorly performing. Good. As another example, the user may prioritize the evaluation of a defined model or the execution of a defined modeling technique for additional data samples. As another embodiment, the user may modify one or more modeling techniques and select the modified technique from consideration. As another embodiment, the user is used to train modeling techniques (eg, by adding features, removing features, or selecting different features), or to fit a model. The features may be changed. Such changes can be useful if the feature scale requires normalization, or if the results indicate that some of the features are "data leaks".

In some embodiments, steps 432-444 may be repeated. Modeling techniques that are not excluded (eg, by the system in step 440 or by the user in step 444) survive another iteration. Based on the performance of the model generated in the previous iteration (or multiple iterations), the search engine 110 adjusts the priority of the corresponding modeling technique and allocates processing resources to the modeling technique accordingly. When computational resources become available, the engine uses the resources available to launch model technique execution jobs based on updated priorities.

In some embodiments, in step 432, the search engine 110 uses a plurality of different mathematical combinations (eg, using a stepwise selection of models to include in the blender) to create a new model. Models may be "mixed". In some embodiments, the predictive modeling system 100 provides a modular framework that allows the user to incorporate their own automatic mixing techniques. In some embodiments, the predictive modeling system 100 allows the user to manually specify different model blenders.

In some embodiments, the predictive modeling system 100 may provide one or more advantages in developing a mixed predictive model. First, mixing can work better when a wide variety of candidate models are available for mixing. Also, mixing does not mean that the differences between the candidate models simply correspond to minor variations in the algorithm, but rather the differences between linear models, tree-based models, support vector machines, and nearestest classification approaches. It can work better when dealing with major differences. The predictive modeling system 100 may achieve a substantial favorable start by automatically generating a wide variety of models and maintaining metadata representing how the candidate models differ. The predictive modeling system 100 may also provide a framework that allows any model to be incorporated into a mixed model, for example by automatically normalizing the magnitude of variables across candidate models. The framework may allow users to easily add their own customized or independently generated models to the automatically generated models to further increase diversity. ..

In addition to increasing the variety of candidate models available for mixing, the predictive modeling system 100 also provides some user interface and analytical features that can result in good mixing. First, the user interface 120 includes candidate model fits such as double lift charts and several different alternative measures of graphics so that the user can easily identify the exact and complementary models to mix. , Bidirectional model comparison may be provided. Second, the modeling system 100 selects specific candidate models and mixing techniques, or uses some or all of the candidate models to some of the mixing techniques in the modeling technique library. Or give the user the option to adapt everything automatically. The nested cross-validation framework then states that the data used to rank each mixed model is not used when adjusting the blender itself or when adjusting the hyperparameters of its component model. To enforce. This discipline may provide the user with a more accurate comparison of alternative blender performance. In some embodiments, the modeling system 100 implements the processing of the mixed model in parallel so that the computational time for the mixed model approaches the computational time of its slowest component model.

Returning to FIG. 4, in step 446 of method 400, the user interface 120 presents the final result to the user. Based on this presentation, the user refines the dataset (eg, by returning to step 412) and adjusts the allocation of resources to the modeling technique to be performed (eg, by returning to step 444). Modify one or more of the modeling techniques to improve accuracy (eg by returning to step 430), modify the dataset (eg by returning to step 402), etc. You may.

In step 448 of Method 400, the user may select one or more top-level predictive model candidates, rather than restarting the search spatial evaluation or part thereof. In step 450, the predictive modeling system 100 may present the results of a holdout test for the selected predictive model candidate. Holdout test results may provide a final measure of how to compare these candidates. In some embodiments, only a fully privileged user may publish the holdout test results. Preventing the publication of holdout test results until a candidate prediction model is selected may facilitate a fair assessment of performance. However, the search engine 110 may actually calculate the holdout test results during the modeling job execution process (steps 432-444) as long as the results remain hidden until after the candidate prediction model has been selected. ..

User Interface Returning to FIG. 1, user interface 120 may provide tools for monitoring and / or guiding searches in the predictive modeling space. These tools provide insights into the dataset of predictive problems (for example, by highlighting problematic variables in the dataset, identifying relationships between variables in the dataset, etc.), and / Or may provide insights into the results of the search. In some embodiments, the data analyst identifies the criteria for recognizing a suitable modeling solution, for example by defining the metrics used to evaluate and compare the modeling solutions. The interface may be used to guide the search, such as by. Therefore, the user interface may be used by the analyst to improve its own productivity and / or to improve the performance of the search engine 110. In some embodiments, the user interface 120 presents search results in real time and the user in real time (eg, adjusting the scope of the search, or the allocation of resources between evaluations of different modeling solutions). Allows you to navigate the search. In some embodiments, the user interface 120 provides tools for coordinating the efforts of multiple data analysts working on the same predictive and / or related predictive problems.

In some embodiments, the user interface 120 provides a tool for developing a machine executable template for the library 130 of modeling techniques. System users may use these tools to modify existing templates, create new templates, or remove templates from library 130. In this way, system users may update library 130 to reflect advances in predictive modeling research and / or to include dedicated predictive modeling techniques.

User interface 120 allows users to manage multiple modeling projects within an organization, create and modify elements of the modeling method hierarchy, perform a comprehensive search of accurate predictive models, and gain insights into datasets and model results. It may include various interface components that allow it to acquire and / or develop the finished model to generate predictions for new data.

In some embodiments, the user interface 120 distinguishes between four types of users: an administrator, a technique developer, a model builder, and an observer. The administrator may control the allocation of human and computational resources to the project. Technique developers may create and modify modeling techniques and their component tasks. The model builder focuses primarily on searching for good models, but may also make minor adjustments to techniques and tasks. Observers may view certain aspects of project progress and modeling results, but may be prohibited from making any changes to the data or initiating any model building. Individuals may play more than one role in a concrete project or across multiple projects.

A user acting as an administrator may access project management components of user interface 120 to set project parameters, assign project responsibilities to users, and allocate computational resources to projects. In some embodiments, the administrator may use project management components to organize multiple projects into groups or hierarchies. All projects in the group may inherit the group settings. In the hierarchy, all children of the project may inherit the project settings. In some embodiments, a user with sufficient permission may override the inherited settings. In some embodiments, a user with sufficient permission may further divide them into different sections so that only the user with the corresponding permission can change the settings. In some cases, the administrator may access resources that are orthogonal to the project body. For example, certain techniques and tasks may be made available to all projects unless explicitly prohibited. Others may be banned on all projects unless expressly permitted. Also, if a user with these rights is assigned to that particular project, some resources may be allocated on a user basis so that the project can only access the resources.

In managing users, the administrator may control all groups of users allowed to the system, their allowed roles, and system-level permissions. In some embodiments, the administrator may add the user to the system by adding the user to the corresponding group and issuing some form of access certificate to the user. In some embodiments, the user interface 120 is a different type of certificate, including, but not limited to, a username and password, a unified authorization framework (eg, OAuth), a hardware token (eg, a smart card), and the like. May be supported.

Once approved, the administrator may specify that a user has a default role to assume for any project. For example, a particular user may be designated as an observer unless specifically approved by the administrator for another role for a particular project. While another user may be supplied as a technique developer for all projects unless specifically excluded by the administrator, another user may only branch to a particular group or project hierarchy of the project. May be supplied as a technique developer for. In addition to the default role, the administrator may also assign more specific permissions to users at the system level. For example, some administrators may be able to grant access to certain types of computational resources, and some technique developers and model builders may be able to access certain features within the builder. It is possible that some model builders may be authorized to start a new project, consume more than a given level of computational resources, or invite new users to a project they do not own.

In some embodiments, the administrator may assign access, permissions, and responsibilities at the project level. Access may include the ability to access any information within a particular project. Permits may include the ability to perform specific actions for a project. Access and permissions may disable system-level permissions or provide finer-grained control. As an example of the former, a user, usually with a full builder permit, may be constrained to a partial builder permit for a particular project. As an example of the latter, a user may be restricted from loading new data into an existing project. Responsibility may include action items that the user is expected to complete for the project.

A user acting as a developer may access the builder area of the interface to create and modify modeling methods, techniques, and tasks. As previously discussed, each builder may present one or more tools with different types of user interfaces that perform the corresponding logical actions. In some embodiments, the user interface 120 allows the developer to use the "property" sheet to edit the metadata attached to the technique. The technique may also have tuning parameters that correspond to the variables of a particular task. Developers may publish these tuning parameters in technique-level property sheets to specify default values and whether the model builder can override these defaults.

In some embodiments, the user interface 120 is tasked with conditional logic, filtering the output, transforming the output, splitting the output, combining inputs, iterating over subgraphs, and any other built-in actions. A graphical flow diagram tool for defining a hierarchical directed graph of may be provided. In some embodiments, the user interface 120 provides equipment for creating wrappers around existing software to implement leaf-level tasks, including properties that can be set for each task. May be provided.

In some embodiments, the user interface 120 may provide senior developers with built-in access to an integrated development environment (IDE) for implementing leaf-level tasks. As an alternative, the developer may code the components within an external environment and wrap the code as leaf-level tasks, but it can be more convenient if these environments are directly accessible. In such an embodiment, the IDE itself may be wrapped in an interface and logically incorporated into the task builder. From the user's point of view, the IDE may operate within the same interface framework as the task builder and on the same computing infrastructure. This ability may allow advanced developers to more quickly iterate to develop and modify techniques. Some embodiments may further provide code coordination features that facilitate coordination among multiple developers programming the same leaf-level task at the same time.

The model builder may utilize techniques generated by the developer to build predictive models for concrete datasets. Different model builders have different levels of experience and can therefore request different support from the user interface. For relatively new users, the user interface 120 presents a process that is as automatic as possible, but may still give the user the ability to explore options and thereby learn more about predictive modeling. For intermediate users, the user interface 120 rapidly assesses how easy it will be to solve a particular problem, and those existing predictive models are automatically generated by the predictive modeling system 100. Presenting information to facilitate comparing how much it is comparable to what can be done, and making accelerated launches on complex projects that will ultimately benefit from substantive practice entrainment. You may. For advanced users, user interface 120 faces some surplus decimal precision extractions for existing predictive models, fast assessment of the applicability of new techniques to the problems users have been working on, and organizations. You may facilitate the development of techniques for the entire category of problems you get. By capturing the knowledge of advanced users, some embodiments facilitate the dissemination of that knowledge throughout the rest of the body.

To support the full range of user requirements, some embodiments of user interface 120 provide a set of interface tools that reflect the model building process. Each tool may also provide a set of features from basic to advanced. The first step in the model building process may involve loading and preparing the dataset. As previously discussed, users may specify how to upload files or access data from online systems. In the context of modeling project groups or hierarchies, users may also specify which parts of the parent dataset will be used for the current project and which parts will be added. Good.

For the basic user, the predictive modeling system 100 may continue to build the model immediately after the dataset is defined, and the user interface 120 is not limited, but there are few unanalyzable data and observations. To flag annoying problems, including too many observations to expect good results, too many observations to perform in a reasonable amount of time, too many missing data, or variables whose distribution can lead to abnormal results. Only pause. For intermediate users, user interface 120 may facilitate a better understanding of the data by presenting a table of dataset characteristics and a graph of variable importance, variable impact, and impact hotspots. The user interface 120 is also by providing visualization tools, including, but not limited to, the results of unsupervised machine learning algorithms such as correlation matrices, partially dependent plots, and / or k-means and hierarchical clustering. , May facilitate understanding and visualization of relationships between variables. In some embodiments, the user interface 120 allows advanced users to create completely new dataset features by defining expressions that transform existing features or combinations thereof.

Once the dataset is loaded, the user may specify model fit metrics that are optimized. For basic users, the predictive modeling system 100 may select model fit metrics and the user interface 120 may provide a description of the selection. For intermediate users, user interface 120 may provide information to help the user understand the trade-offs of choosing different metrics for a particular dataset. For advanced users, the user interface 120 is based on the low-level performance data collected by the search engine 110, by writing an equation (eg, an objective function), or by uploading a custom metric calculation code. May be allowed to specify custom metrics.

Once the dataset is loaded and the model fit metrics are selected, the user may launch the search engine. For the base user, the search engine 110 may use the default prioritization settings for the modeling technique, and the user interface 120 may include model performance, how far the dataset has been executed, and general computational resources. High-level information about consumption may be provided. For intermediate users, the user interface 120 may allow the user to define a subset of techniques that take into account some of the initial priorities and make slight adjustments. In some embodiments, the user interface 120 provides finer-grained performance and progress data, allowing intermediate users to make in-flight adjustments as previously described. In some embodiments, the user interface 120 provides the intermediate user with additional insight and control over computational resource consumption. In some embodiments, the user interface 120 provides significant (eg, complete) control of the techniques considered and their priorities, all available performance data, and significant (eg, complete) resource consumption. N) Control may be provided to advanced users. Some embodiments of user interface 120 include either providing distinctly different interfaces to different levels of users, or "folding" more advanced features for less advanced users by default. It can support users at their corresponding levels.

During or after exploring the search space, the user interface may provide information about the performance of one or more modeling techniques. Some performance information may be displayed in a tabular format, and other performance information may be displayed in a graph format. For example, the information presented in tabular form may include, but is not limited to, a comparison of model performance by technique, a portion of the data to be evaluated, the nature of the technique, or the current consumption of computational resources. The information presented in graph format is, but is not limited to, directed graphs of tasks in modeling procedures, comparison of model performance across different partitions of a dataset, representation of model performance such as receiver operating characteristics and lift charts, predicted value pairs. It may include actual values as well as consumption of computational resources over time. The user interface 120 may include a modular user interface framework that allows easy inclusion of any type of new performance information. Also, some embodiments may allow the display of several types of information for each data partition and / or for each technique.

As previously discussed, some embodiments of user interface 120 support the coordination of multiple users on multiple projects. Throughout the project, the user interface 120 may allow users to share data, modeling tasks, and modeling techniques. Within the project, user interface 120 may allow users to share data, models, and results. In some embodiments, the user interface 120 may allow the user to modify the nature of the project and use the resources allocated to the project. In some embodiments, the user interface 120 may allow multiple users to modify project data, add models to the project, and then compare these contributions. In some embodiments, the user interface 120 may identify which users made specific changes to the project, when the changes were made, and what project resources the users used.

Model Deployment Engine The model deployment engine 140 provides tools for deploying predictive models in the operating environment. In some embodiments, the model deployment engine 140 monitors the performance of the deployed predictive model and generates a model that generates the deployed model so that the performance data accurately reflects the performance of the deployed model. Update performance metadata associated with the technique.

The user may develop a conformance prediction model when he believes that the conformance model can guarantee field testing or add values. In some embodiments, the user and the external system access the prediction module (eg, within the interface service layer of the prediction modeling system 100) to define one or more prediction models to be used, which is new. Observations may be supplied. The prediction module may then return the predictions provided by these models. In some embodiments, the administrator controls which users and external systems have access to the prediction module, and / or sets usage limits such as the number of predictions allowed per unit time. May be good.

For each model, the search engine 110 may store a record of the modeling techniques used to generate the model, as well as the state of the fitted model, including coefficients and hyperparameter values. These values may be sufficient for the execution engine to generate predictions for new observational data, as each technique is already machine executable. In some embodiments, model predictions may be generated by applying the pre-processing and modeling steps described in the modeling technique to each instance of new input data. However, in some cases it may be possible to increase the speed of future forecast calculations. For example, the fit model may perform some independent checks on the values of a particular variable. You may reduce the total amount of calculations used to generate the predictions by combining some or all of these checks and then simply referencing them when it is convenient. Similarly, some component models of the mixed model may perform the same data transformation. Therefore, some embodiments may reduce the calculation time by identifying duplicate calculations, performing them only once, and referencing the results of the calculations in the component model that uses them.

In some embodiments, the deployment engine 140 identifies an opportunity for parallel processing, thereby responding to each prediction when the underlying hardware is able to execute multiple instructions in parallel. Improve the performance of predictive models by reducing time. Some modeling techniques may describe a series of steps in succession, and in fact some of the steps can be logically independent. By investigating the data flow between each step, the deployment engine 140 can identify the status of logical independence and then restructure the execution of the forecast model so that the independent steps run in parallel. Will be done. The mixed model may offer a special class of parallelization, because once any common data conversion is complete, the configuration prediction model can be executed in parallel.

In some embodiments, the deployment engine 140 may cache the state of the prediction model in memory. Using this approach, continuous prediction requests for the same model do not have to bear the time to load the model state. Caching may work particularly well when there are many demands for predictions for a relatively small number of observations, and thus this loading time is potentially the majority of the total run time.

In some embodiments, the deployment engine 140 may provide at least two implementations of the predictive model, namely service-based and code-based implementations. For service-based forecasting, computations work within a decentralized computing infrastructure as described below. The final forecast model may be stored in the data service layer of the distributed computing infrastructure. When a user or an external system requests a prediction, it may indicate which model is used and provides at least one new observation. The prediction module then loads the model from the data service layer or from the module's in-memory cache, verifies that the submitted observations match the structure of the original dataset, and calculates the predictions for each observation. You may. In some implementations, the prediction model may be run on a dedicated pool of cloudworkers, thereby facilitating the generation of predictions with low dispersion response times.

Service-based forecasts may occur either interactively or via APIs. For bidirectional prediction, the user may enter a value for each new observation feature or upload a file containing data for one or more observations. The user may then receive the predictions directly through the user interface 120 or download them as a file. For API predictions, the external system may access the prediction module via local or remote APIs, submit one or more observations, and receive the corresponding calculated predictions in exchange.

Some implementations of the deployment engine 140 may allow organizations to create small instances of one or more of the distributed computing infrastructures for the purpose of making service-based forecasts. At the interface layer of a distributed computing infrastructure, each such instance may use parts of a monitoring and prediction module accessible by an external system without accessing user-related features. The analysis service layer may not use the technique IDE module, and the remaining modules in this layer may be stripped of extra equipment and optimized to meet predictive demands. The data service layer does not have to use user or model building data management. Such stand-alone predictive instances are deployed on a parallel pool of cloud resources, distributed to other physical locations, or even downloaded to one or more dedicated machines that act as "prediction appliances." May be good.

To create a dedicated predictive instance, the user may define a targeted computing infrastructure, for example, whether it is a set of cloud instances or a set of dedicated hardware. The corresponding modules may then be supplied and either installed on the target computing infrastructure or packaged for installation. The user may either configure the instance with an initial set of predictive models or create a "blank" instance. After the initial installation, the user may manage the available prediction models by installing new ones or updating existing ones from the primary installation.

With respect to code-based prediction, the deployment engine 140 may generate source code for calculating predictions based on a particular model, and the user may incorporate the source code into software. When the model is based on a technique in which all leaf-level tasks are implemented in the same programming language as required by the user, the deployment engine 140 predicts the model by matching the code for the leaf-level tasks. You may generate the source code for. The deployment engine 140 may use a more elaborate approach when the model incorporates code from a different language, or the language is different from what is desired by the user.

One approach is to use a source-to-source compiler to translate the source code of leaf-level tasks into the target language. Another approach is to then generate a functional stub in the target language that calls the link-in object code in the original language or accesses an emulator that executes such object code. The former approach may involve the use of a cross-compiler to specifically generate object code for the user's target computing platform. The latter approach may involve the use of an emulator that will run on the user's target platform.

Another approach is to generate a summary description of a particular model and then compile the description into the target language. To generate a summary description, some embodiments of Deployment Engine 140 may use metadata to account for a number of potential pre-processing, model fitting, and post-processing steps. The deployment engine may then extract specific behaviors for the complete model and use metamodels to encode them. In such embodiments, a compiler for the target programming language may be used to translate the metamodel into the target language. Therefore, if the user wants the predictive code in a supported language, the compiler may generate it. For example, in a decision tree model, decisions in a tree may be abstracted into logical if / there / else statements that can be implemented directly in a wide variety of programming languages. Similarly, a set of mathematical operations supported by a common programming language may be used to implement a linear regression model.

However, disclosing the source code of a predictive model in any language may be undesirable in some cases (eg, when the predictive modeling technique or predictive model contains specialized capabilities or information). Therefore, the deployment engine 140 may transform the predictive model into a set of rules that reserve the predictive power of the predictive model without disclosing the procedure details. One approach is to apply an algorithm that generates such rules from a set of hypothetical predictions that the prediction model will generate in response to hypothetical observations. Some such algorithms may generate a set of if-then rules (eg, RuleFit) for making predictions. For these algorithms, the deployment engine 140 may then translate the resulting if-then rules into the target language instead of transforming the original prediction model. An additional benefit of transforming a predictive model into a set of if-then rules is that, in general, the basic model of conditional logic is more similar across programming languages than a predictive model with arbitrary control and data flow. , It is easier to translate a set of if-then rules into a target programming language.

Once the model begins to make predictions for new observations, the deployment engine 140 may track these predictions, measure their accuracy, and use these results to improve the prediction modeling system 100. .. For service-based forecasts, each observation and forecast may be stored through the data service layer, as the forecasts occur within the same distributed computing environment as the rest of the system. By providing an identifier for each prediction, some embodiments may allow the user or an external software system to submit the actual values if they are recorded. For code-based predictions, some embodiments may include code that stores observations and predictions in a local system or back to an instance of the data service layer. Again, providing identifiers for each prediction may facilitate the collection of model performance data for actual target values as they become available.

For the accuracy of predictions and / or observations obtained through other channels, the information collected directly by the deployment engine 140 is to improve the model for prediction problems (eg, to "refresh" existing models). , Or to generate a model by re-searching the modeled search space partially or completely). New data is added to improve the model in the same way that the data was originally added to create the model, or by submitting target values for the data previously used in the forecast. Can be done.

Some models may be refreshed (eg, fit) by applying the corresponding modeling techniques to the new data and combining the resulting new model with the existing model, while others , May be refreshed by applying the corresponding modeling technique to the original and new data combinations. In some embodiments, when refreshing the model, some of the model parameters (eg, to refresh the model more quickly, or because the new data provides information that is particularly relevant to a particular parameter). Only may be recalculated.

Alternatively, or in addition, a new model may be generated by exploring the modeled search space partially or completely, with the new data contained in the dataset. Re-searching the search space may be limited to a portion of the search space (eg, limited to modeling techniques that worked well in the original search), or may cover the entire search space. In either case, the initial suitability score for the modeling technique that generated the expanded model may be recalculated to reflect the performance of the expanded model for the prediction problem. The user may choose to exclude some of the previous data for recalculation. Some embodiments of the deployment engine 140 may track different versions of the same logical model, including which subset of data was used to train which version.

In some embodiments, the prediction data is used to perform post-request analysis of trends in input parameters or the prediction itself over time, and to warn the user of potential problems with the quality of the input or model predictions. May be done. For example, if the model performance aggregation scale begins to deteriorate over time, the system warns the user to consider refreshing the model or investigating whether the input itself is skewed. May be good. Such shifts can be caused by temporal changes in certain variables, or by drifting the entire population. In some embodiments, most of the analysis is performed after the prediction request has been completed so as to avoid slowing down the prediction response. However, the system makes particularly bad predictions (eg, if the input values are outside the range of values calculated as valid given characteristics of the original training data, modeling techniques, and final model fit state). Verification at the estimated time may be performed to avoid it.

Post-hoc analysis can be important if the model is developed so that the user extrapolates well beyond the population used in the training. For example, the model is trained with data from one geographic region, but is used to make predictions for populations within completely different geographic regions. At some point, such extrapolation to a new population can result in substantially worse model performance than expected. In these cases, the deployment engine 140 warns the user and / or uses new values to extend the original training data by refitting one or more modeling techniques. The model may be refreshed automatically.

Benefits of Some Embodiments Predictive modeling system 100 significantly improves analyst productivity at any skill level and / or provides achievable predictive model accuracy with a given amount of resources. It may be significantly increased. By automating procedures, you can reduce the workload, and by systematizing processes, you can enhance consistency and allow analysts to spend more time generating unique insights. Can be. Predicting outcomes, predicting properties, and inferring measurements exemplify these benefits in three common scenarios.

Predict results

If the organization can accurately predict outcomes, its behavior can be planned and promoted more effectively. Therefore, a common use of machine learning is to develop algorithms that generate predictions. For example, many industries face the problem of predicting costs for large, time-consuming projects.

In some embodiments, the techniques described herein can be used to anticipate cost overruns (eg, software overcost or build overcost). For example, the techniques described herein may be applied to the problem of anticipating cost overruns, as follows:

1. 1. Select the appropriate model fit metric for the response variable type (eg, numeric or binary, nearly Gaussian, or strongly non-Gaussian). The predictive modeling system 100 may recommend data characteristic-based metrics that require less skill and effort by the user, but allow the user to make final choices.

2. 2. Preprocess data to address outliers and missing data values. The predictive modeling system 100 may provide a detailed overview of the data characteristics that allows the user to become more aware of the situation of the modeling problem and more effectively assess the potential modeling issues. .. The predictive modeling system 100 may include automated procedures for outlier detection and replacement, missing value completion, and detection and processing of other data anomalies that require less skill and effort by the user. Predictive modeling system procedures for addressing these challenges are systematic and can lead to more consistent modeling results than ad hoc data editing procedures over methods, datasets, and time.

3. 3. Split data for modeling and evaluation. The predictive modeling system 100 may automatically divide the data into training, validation, and holdout sets. This division is more flexible than the training and trial divisions used by some data analysts and may be consistent with widely accepted recommendations from the machine learning community. The use of methods, datasets, and a consistent partitioning approach over time can make the results more equal and enable more effective allocation of deployment resources in commercial situations.

4. Select the model structure, generate the derived features, select the model tuning parameters, fit and evaluate the model. In some embodiments, the predictive modeling system 100 includes many different, but not limited to, decision trees, neural networks, support vector machine models, regression models, boost trees, random forests, deep learning neural networks, and the like. The model type can be adapted. The predictive modeling system 100 may offer the option of automatically constructing an ensemble from these component models that exhibit the best individual performance. Accuracy can be improved by exploring the larger space of the potential model. The predictive modeling system may automatically generate various derived features suitable for different data types (eg, box-cox transformations, text preprocessing, key components, etc.). Accuracy can be improved by exploring the larger space of the potential transformation. Predictive modeling system 100 may use cross-validation to select the best values for these tuning parameters as part of the model building process, thereby improving the selection of tuning parameters and the parameters. Create an audit trail of how your choices affect your results. The predictive modeling system 100 may fit and evaluate different model structures that are considered part of this automated process and rank the results in terms of validation set performance.

5. Select the final model. The selection of the final model can be made by the predictive modeling system 100 or by the user. In the latter case, the predictive modeling system allows the user to, for example, have a ranked validation set performance assessment of the model, compare performance, and rank by a quality scale other than that used in the conformance process, and / Or support may be provided to assist in making this decision, including the opportunity to build an ensemble model from these component models that exhibit the best individual performance.

One important practical aspect of the model development process for predictive modeling systems is that once the dataset is assembled, all subsequent calculations can occur within the same software environment. This aspect represents a significant difference from traditional model building efforts, often with a combination of different software environments. An important practical disadvantage of such a multi-platform analytical approach is the need to transform the results into a common data format that can be shared between different software environments. Often, this conversion is done either manually or with a custom "single" reformatting script. Errors in this process can lead to extremely serious data distortion. The predictive modeling system 100 may avoid such reformatting and data transfer errors by performing all calculations within one software environment. More generally, because it is highly automatic, many different model structures are adapted and optimized, and the predictive modeling system 100 is substantially faster and more systematic to the final model and therefore easier. It is possible to provide an explainable and repeatable route. Also, as a result of the predictive modeling system 100 exploring more different modeling methods and including more possible predictors, the resulting model is more accurate than that obtained by conventional methods. Can be.

Predict the nature

In many areas, organizations face uncertainty in the outcome of the production process and want to predict how a given set of conditions will affect the final nature of the output. Therefore, a common use of machine learning is to develop algorithms that predict these properties. For example, concrete is a common building material whose final structural properties can vary dramatically from context to context. Neither the model developed from the first principle nor the conventional regression model provides sufficient prediction accuracy due to the significant variation in concrete properties over time and their dependence on their highly variable composition.

In some embodiments, the techniques described herein can be used to predict the outcome properties of the production process (eg, the properties of concrete). For example, the techniques described herein may be applied to the problem of predicting the properties of concrete, such as:

1. 1. Divide the dataset into training, validation, and test subsets.

2. 2. Clean up the modeled dataset. The predictive modeling system 100 may automatically check for missing data, outliers, and other critical data anomalies, recommend processing strategies, and provide users with the option of accepting or rejecting them. The approach may require less skill and effort by the user and / or may provide more consistent results over methods, datasets, and time.

3. 3. Select the response variable and select the primary conformance metric. The user may select the expected response variables from those available in the modeled dataset. Once the response variables have been selected, the predictive modeling system 100 may recommend compatible conformance metrics that the user can accept or disable. This approach does not require much user skill and effort. Based on the response variable type and the fitted metrics selected, the predictive modeling system includes traditional regression models, neural networks, and other machine learning models (eg, random forests, boost trees, support vector machines). A set of predictive models may be provided. The predictive modeling system 100 may increase the expected accuracy of the final model by automatically searching between the spaces of possible modeling approaches. The default set of model selection excludes certain model types from consideration, adds other model types supported by the predictive modeling system rather than part of the default list, or (eg, R or Python). It may be disabled to add your own custom model type (implemented in).

4. Generate input features, fit the model, optimize model-specific tuning parameters, and evaluate performance. In some embodiments, feature generation may include scaling for numerical covariates, box-cox transformations, key components, and the like. The tuning parameters of the model may be optimized via cross-validation. The validation set performance scale may be calculated and presented for each model, along with other summary characteristics (eg, regression model model parameters, boost tree or random forest variable importance scale).

5. Select the final model. The selection of the final model can be made by the predictive modeling system 100 or by the user. In the latter case, the predictive modeling system allows the user to, for example, have a ranked validation set performance assessment of the model, compare performance, and rank by a quality scale other than that used in the conformance process, and / Or support may be provided to assist in making this decision, including the opportunity to build an ensemble model from these component models that exhibit the best individual performance.

Infer the measurement

Organizations may wish to use cheaper metrics instead of more expensive ones, as some measurements are much more expensive to make than others. Therefore, a common use of machine learning is to infer the possible output of expensive measurements from the known outputs of cheaper measurements. For example, "curl" is a property that captures how a paper product tends to deviate from its flat shape, but can typically only be determined after the product is complete. Therefore, being able to infer paper curls from mechanical properties that are easily measured during manufacturing can result in enormous cost savings in achieving a given level of quality. With respect to typical end-use properties, the relationship between these properties and manufacturing process conditions is not well understood.

In some embodiments, the techniques described herein can be used to infer measurements. For example, the techniques described herein may be applied to the problem of inferring measurements, such as:

1. 1. Characterize the modeled dataset. The Predictive Modeling System 100 may provide key overview characteristics and recommendations for handling critical data anomalies that allow the user to freely accept, reject, or request further information about it. For example, the key characteristics of a variable may be calculated and displayed, the spread of missing data may be displayed, processing strategies may be recommended, and outliers in numerical variables may be detected. If found, processing strategies may be recommended, and / or other data anomalies may be detected automatically (eg, normal values, variables that never change, variables that give no information) and are recommended. The processing may be made available to the user.

2. 2. Divide the dataset into training / validation / holdout subsets.

3. 3. Feature generation / model structure selection / model matching. The predictive modeling system 100 may combine these steps to automate and allow extensive internal iterations. Multiple features may be automatically generated and evaluated using both traditional techniques such as key components and newer methods such as boost trees. Many different model types may be fitted and compared, including regression models, neural networks, support vector machines, random forests, boost trees, and more. In addition, the user may have the option of including other model structures that are not part of this default collection. Model substructure selection (eg, selection of the number of hidden units in the neural network, specifications of other model-specific tuning parameters, etc.) is automated by extensive cross-validation as part of this model fitting and evaluation process. It may be done as a target.

4. Select the final model. The selection of the final model can be made by the predictive modeling system 100 or by the user. In the latter case, the predictive modeling system allows the user to, for example, have a ranked validation set performance assessment of the model, compare performance, and rank by a quality scale other than that used in the conformance process, and / Or support may be provided to assist in making this decision, including the opportunity to build an ensemble model from these component models that exhibit the best individual performance.

In some embodiments, the predictive modeling system 100 implements the model in order to automate and efficiently implement data preprocessing (eg, anomaly detection), data partitioning, multiple feature generation, model fitting, and model evaluation. The time required to develop can be much shorter than the time in a traditional development cycle. In addition, in some embodiments, the predictive modeling system automatically observes well-known data anomalies such as missing data and outliers, as well as normal values (repeated observations that match the data distribution but are not false). The resulting model is more accurate because it includes data preprocessing procedures that deal with both less widely recognized anomalies such as post-determinants (ie, highly predictive covariates resulting from information leakage). , Can be more useful. In some embodiments, the predictive modeling system 100 is able to explore a much wider range of model types and many more specific models of each type than is conventionally feasible. This model diversity can greatly reduce the likelihood of inadequate results even when applied to poor quality datasets.

Implementation of Predictive Modeling System With reference to FIG. 5, in some embodiments, the predictive modeling system 500 (eg, the embodiment of predictive modeling system 100) has at least one client computer 510 and at least one server. Includes 550 and one or more processing nodes 570. Illustrative configurations are for illustrative purposes only, and it is intended that there can be any number of clients 510 and / or servers 550.

In some embodiments, the predictive modeling system 500 may perform one or more (eg, all) steps of method 300. In some embodiments, the client 510 may implement the user interface 120, and the predictive modeling module 552 of the server 550 may include other components of the predictive modeling system 100 (eg, the modeled space search engine 110). , A library of modeling techniques 130, a library of prediction problems, and / or a modeling deployment engine 140). In some embodiments, the computational resources allocated by the search engine 110 for exploring the modeled search space may be one or more resources of processing node 570, one or more. Node 570 may perform modeling techniques according to a resource allocation schedule. However, the embodiment is not limited by the mode in which the components of the predictive modeling system 100 or the predictive modeling method 300 are distributed among the client 510, the server 550, and one or more processing nodes 570. Further, in some embodiments, all components of the predictive modeling system 100 are implemented on a single computer (instead of being distributed among clients 510, servers 550, and processing nodes 570). Alternatively, it may be implemented on two computers (eg, client 510 and server 550).

One or more communication networks 530 connect the client 510 to the server 550, and one or more communication networks 580 connect the server 550 to the processing node 570. Communication is by stand-alone telephone line, LAN or WAN link (eg, T1, T3, 56kb, X.25), wideband connection (ISDN, Frame Relay, ATM), and / or wireless link (IEEE 802.11, Bluetooth (eg, IEEE 802.11, Bluetooth). It may be done via any medium such as registered trademark)). Preferably, the network 530/580 can carry TCP / IP protocol communications, and the data transmitted by the client 510, server 550, and processing node 570 (eg, HTTP / HTTPS requests, etc.) is such. It can be transmitted via a TCP / IP network. The type of network is not limited, but any suitable network may be used. Non-limiting examples of networks that serve or may be part of the communication network 530/580 are wireless or wired Ethernet® based, which can be adapted to many different communication media and protocols. Includes intranets, local or wide area networks (LAN or WAN), and / or global communications networks known as the Internet.

The client 510 is preferably implemented with software 512 running on hardware. In some embodiments, the hardware is a MICROSOFT WINDOWS® operating system from Microsoft Corporation (Redmond, Washington), a MACINTOSH operating system from Apple Computer (Cupertino, California), and MACINTOSH operating systems. Various types of Unix®, such as SUN SOLARIS from RED HAT, INC. It may also include a personal computer (eg, a PC with an INTEL processor or APPLE MACINTOSH) capable of running an operating system such as GNU / Linux® from (Durham, North Carolina). Client 510 may also be a smart or dumb terminal, network computer, wireless device, wireless phone, information appliance, workstation, minicomputer, mainframe computer, mobile information terminal, tablet, smartphone, or other computer operated as a general purpose computer. It may be implemented on hardware such as a device or a special purpose hardware device that is used only to perform its function as a client 510.

In general, in some embodiments, the client 510 sends and receives emails and / or instant messages, requests and browses content available via the World Wide Web, and participates in chat rooms. , Or can be manipulated and used for a variety of activities, including performing other commonly performed tasks using a computer, handheld device, or mobile phone. The client 510 can also be operated by the user on behalf of others, such as employees, who provide the client 510 to the user as part of employment.

In various embodiments, software 512 of client computer 510 includes client software 514 and / or web browser 516. The web browser 514 allows the client 510 to request a web page or other downloadable program, applet, or document using a web page request (eg, from server 550). An embodiment of a web page can be displayed, run, played, processed, streamed, and / or stored, and can contain links or pointers to other web pages, computer executable or A data file that contains interpretable information, graphics, audio, text, and / or video. Examples of commercial web browser software 516 are provided by INTERNET EXPLORER provided by Microsoft Corporation, NETSCAPE NAVIGATOR provided by AOL / Time Warner, FIREFOX provided by Mozilla Foundation, or CHROM provided by Mozilla.

In some embodiments, software 512 includes client software 514. Client software 514 provides client 510 with functionality that allows users to send and receive, for example, email, instant messaging, phone calls, video messages, streaming audio or video, or other content. Examples of client software 514 include, but are not limited to, OUTLOOK and OUTLOOK EXPRESS provided by Microsoft Corporation, THUNDERBIRD provided by Mozilla Foundation, and INSTANT MESSENGER provided by AOL / Time Warner. Standard components associated with client computers, including central processing units, volatile and non-volatile storage devices, input / output devices, and displays, are not shown.

In some embodiments, the web browser software 516 and / or the client software 514 may allow the client to access the user interface 120 for the predictive modeling system 100.

The server 550 interacts with the client 510. The server 550 preferably has sufficient memory, data storage, and processing power and is a server-class operating system (eg, SUN Solaris, GNU / Linux®, and MICROSOFT WINDOWS® operating systems. Implemented on one or more server-class computers running the system). Depending on the capacity of the device and the size of the user base, system hardware and software other than those specifically described herein may also be used. For example, the server 550 may be a logical group of one or more servers, such as a server farm or server network, or part of it. As another embodiment, there may be a plurality of servers 550 associated with or connected to each other, or a plurality of servers may operate independently using shared data. In a further embodiment, the application software is implemented in the components with different components running on different server computers, on the same server, or in some combination, as is typical in large systems. Can be done.

In some embodiments, the server 550 includes a predictive modeling module 552, a communication module 556, and / or a data storage module 554. In some embodiments, the predictive modeling module 552 may implement a modeled space exploration engine 110, a library of modeling techniques 130, a library of predictive problems, and / or a modeled deployment engine 140. In some embodiments, server 550 uses communication module 556 to propagate the output of predictive modeling module 552 to client 510 and / or to supervise the execution of modeling techniques on processing node 570. You may. The modules described throughout this specification may be in whole or using any suitable programming language (C ++, C #, Java®, LISP, BASIC, Perl, etc.). It can be implemented in part as a software program and / or as a hardware device (eg, ASIC, FPGA, processor, memory, storage, and equivalent).

The data storage module 554 may store, for example, the prediction modeling library 130 and / or the library of prediction problems. The data storage module 554 may be, for example, a MySQL database server by MySQL AB (Uppsala, Sweden), a PostgreSQL database server by PostgreSQL Global Development Group (Berkeley, CA), or an ORACLE Corp. It may be implemented using the Oracle database server provided by (Redwood Shores, CA).

FIG. 6-8 illustrates one possible implementation of the predictive modeling system 100. The discussion in Figure 6-8 is given as an example of some embodiments and is by no means limiting.

To execute the procedure described above, the predictive modeling system 100 may use a distributed software architecture 600 running on various client and server computers. The goal of Software Architecture 600 is to achieve a wealth of user experience and computationally intensive processing at the same time. The software architecture 600 may implement a variant of the basic four-layer Internet architecture. As illustrated in FIG. 6, it extends the infrastructure to take advantage of cloud-based computations coordinated through applications and data layers.

Similarities and differences between Architecture 600 and the basic four-tier Internet architecture may include:

(1) Client 610. Architecture 600 makes essentially the same assumptions about the client 610 as any other Internet application. Key use cases include frequent access over long time cycles to perform complex tasks. Therefore, target platforms include a wealth of web clients running on laptops or desktops. However, the user may access the architecture via a mobile device. Therefore, the architecture is designed to adapt to native client 612, which has direct access to the interface service API, using a relatively thin client-side library. Of course, any cross-platform GUI layer such as Java® and Flash can access these APIs as well.

(2) Interface service 620. This layer of architecture is an extended version of the basic Internet presentation layer. Due to the elaborate user interaction that can be used to direct machine learning, alternative implementations are via this layer, static HTML, dynamic HTML, SVG visualization, executable Javascript® code, and even the built-in IDE. May support a wide variety of content, including. Also, as new Internet technologies evolve, implementations need to adapt to new forms of clients or change the division of work between client, presentation, and application layers to perform user interaction logic. obtain. Therefore, their interface service layer 620 provides a flexible framework for integrating multiple content distribution mechanisms of varying abundance, with common support equipment such as authentication, access control, and input validation. May be good.

(3) Analysis service 630. The application layer is focused on delivering analytical services, as the architecture may be used to generate predictive analytical solutions. The computational strength of machine learning drives a major enhancement to the standard application layer, namely the dynamic allocation of machine learning tasks to a large number of virtual "workers" running in a cloud environment. For all kinds of logical operation requests generated by the execution engine, the analysis service layer 630 accepts the requests, divides the requests into jobs, assigns the jobs to workers, provides the data necessary for job execution, and provides the data necessary for job execution. Cooperate with other layers so that the execution results are collated. There are also relevant differences from the standard application layer. The predictive modeling system 100 may allow users to develop their own machine learning techniques, so some implementations have their capabilities divided across the client, interface services, and analytics service layers. One or more complete IDEs may be provided. The execution engine then incorporates new and improved techniques created through these IDEs into future machine learning calculations.

(4) Worker cloud 640. To efficiently perform the modeling calculations, the predictive modeling system 100 may divide them into smaller jobs and allocate them to virtual worker instances running in a cloud environment. Architecture 600 enables different types of workers and different types of clouds. Each worker type corresponds to a specific virtual machine configuration. For example, the default worker type provides general machine learning capabilities for trusted modeling code. However, another type enforces additional security "sandboxing" for user-developed code. Alternative types may provide configurations that are optimized for specific machine learning techniques. Jobs can be allocated appropriately as long as the analysis service layer 630 understands the purpose of each worker type. Similarly, the analytics service layer 630 can manage workers in different types of clouds. Organizations may have the option to maintain a pool of instances in their private cloud and run instances in their public cloud. It can also have different pools of instances running on different types of commercial cloud services or even dedicated internal services. Jobs can be assigned appropriately as long as the analysis service layer 630 understands the trade-off between capacity and cost.

(5) Data service 650. Architecture 600 assumes that different services operating within different layers can benefit from different corresponding storage options. Therefore, it is a framework for delivering a rich array of data services 650, such as file storage for any type of permanent data, temporary databases for caching and other purposes, and long-term records. Provide a permanent database for management. Such services may even be specialized for certain types of content, such as virtual machine image files used for cloud workers and IDE servers. In some cases, the implementation of data service layer 650 may implement a particular access idiom on a particular type of data so that other layers can work together smoothly. For example, standardizing the format for datasets and model results means that when assigning jobs to workers, analytics service layer 630 can simply pass references to the user's dataset. The worker can then access the dataset from the data service layer 650 and, in turn, return a reference to the model results stored via the data service 650.

(6) External system 660. Like any other internet application, the use of APIs may allow external systems to integrate with the predictive modeling system 100 at any layer of architecture 600. For example, a business dashboard application can access graphic visualization and modeling results through the interface service layer 620. External data warehouses or even live business applications can provide modeled datasets to analytics service layer 630 through a data integration platform. The reporting application can access all modeling results from a particular time cycle through the data service layer 650. However, under most circumstances, external systems will not have direct access to the worker cloud 640 and will utilize them via the analytics service layer 630.

As with multitier architectures, the layers of architecture 600 are logical. Physically, services from different tiers can run on the same machine, different modules in the same tier can run on different machines, and multiple instances of the same module can run. It can work across several machines. Similarly, services within one layer can operate across multiple network segments, and services from different layers may or may not operate on different network segments. However, the logical structure helps to align developer and operator expectations about how different modules will interact, while balancing service-level implementations such as scalability, reliability, and security. Give the operator the flexibility needed to do this.

The high-level layer is considered reasonably similar to that of a typical Internet application, but the addition of cloud-based computations can substantially change how information flows through the system. Good.

Internet applications typically offer two distinct types: synchronous or asynchronous user interaction. In conceptually synchronous operations such as finding and booking flights on an air route, the user makes a request and waits for a response before making the next request. In conceptually asynchronous behavior, such as setting alerts for online transactions that meet certain criteria, the user expects to make a request and notify the user of the result at a later time in the system. (Typically, the system provides the user with an initial request "ticket" and provides notifications through a designated communication channel.)

In contrast, building and refining machine learning models may involve interaction patterns somewhere in the middle. Setting up a modeling problem may involve an initial set of conceptually synchronous steps. However, when a user instructs a system to start computing an alternative solution, a user who understands the scale of the corresponding calculation is unlikely to expect an immediate response. Specifically, this expectation of delayed results makes this stage of interaction appear asynchronously.

However, the predictive modeling system 100 does not force the user to "fire and forget", i.e., stops its own engagement with the problem until it receives a notification. In fact, this may prompt the user to continue exploring the dataset and scrutinize them as soon as preliminary results arrive. Such additional exploration or initial insight motivates the user to change model building parameters "in flight". The system may then process the requested changes and reassign processing tasks. The predictive modeling system 100 may enable this request / revision dynamic continuously throughout the user's session.

Therefore, the analysis service and data service layer may mediate between the request / response loop from one user and the request / response loop to the worker cloud on the other. FIG. 7 illustrates this viewpoint.

FIG. 7 emphasizes that the predictive modeling system 100 does not always fit neatly into a layered model, assuming that each layer relies mostly solely on the layer immediately below it. Rather, the analysis service 630 and the data service 650 emphasize user and computation. Considering this point of view, there are three "columns" of information flow.

(1) Interface <-> Analytic + Data. The leftmost column of flow 710 first transforms the user's raw datasets and modeling requirements into a refined dataset and list of computational jobs, then fuses the results for easy grasping. Deliver to users in a format that allows. Thus, goals and constraints flow from interface service 620 to analysis service 630, while progress and expectations flow backwards. In parallel, raw datasets and user annotations flow from interface service 620 to data service 650, while trained models and their performance metrics flow backwards. At any time, the user can initiate the change and enforce the adjustments by the analysis service 630 and the data service 650 tier. Note that in addition to this dynamic circular flow, there are also more conventional linear interactions (eg, when interface service 620 reads system state from analysis service 640 or static content from data service 650). I want to.

(2) Analytic + Data <-> Worker. The rightmost column of flow 730 supplies workers, allocates computational jobs, and provides data for these jobs. Thus, job assignments, their parameters, and data references flow from analysis service 630 to worker cloud 640, while progress and expectations flow backwards. The refined dataset flows from the data service 650 to the worker cloud 640, while the modeling results flow backwards. The updated instructions from the user force the analytics service layer 630 to suspend the worker during flight, assign the updated modeling job, and force the dataset to be refreshed from the data service layer 650. be able to. In turn, the updated allocations and datasets change the flow of results returned from the worker.

(3) Analytic <-> Data. The two layers in the middle can be interlocked between them so as to mediate between the left and right columns. Most of this traffic 720 is involved in tracking the execution progress and intermediate calculations of the cloud worker. However, the flow can be particularly complex when responding to the above-mentioned in-flight changes to model building instructions. Analysis and data services assess the current state of the calculation, determine which intermediate calculations are still valid, and build the new calculation job correctly. Of course, there are also more conventional linear interactions (eg, when the analytics service reads the rules and configurations for the cloudworker from the data service).

This conceptual model of information flow provides a context for an array of functional modules within a layer. They are not stateless blocks that simply provide an application programming interface (API) to higher level blocks and consume APIs from lower level blocks. Rather, they are dynamic participants in coordination and computation between users. FIG. 8 presents an array of these functional modules. From the user's point of view, the interface services layer provides several distinct areas of functionality.

(1) User / project management 802. Each machine learning project has at least one assigned administrator who can use the project management components of the interface to manage project level parameters, responsibilities, and resources. This functional component also supports system level management functions.

(2) Monitoring 810. This module provides diagnostics on the computing infrastructure. It works with the corresponding module 818 within the analytics services layer to track compute resource usage, both in real time for each worker instance and overall for each compute job.

(3) Technique designer 804. This module supports the methods, techniques, and graphical interfaces for using the task builder described above. An example of how this graphical interface can be implemented is that Javascript® operates within the client 610, communicates with the technique designer 804 via an AJAX request, and graphically illustrates the graph for the user. Render to and push changes back to the server.

(4) Technique IDE812. As previously described, some implementations of the predictive modeling system 100 may provide technique developers with built-in access to the IDE to implement leaf-level tasks. Such an IDE can support a general-purpose programming language used for machine learning such as Python, or a specialized scientific computing environment such as R. This functionality may be performed across the client 610, interface service 620, and analysis service 630 layers. Client component 610 can download and execute a Javascript® container for the IDE environment that first registers a session with the interface service component via AJAX. After authenticating the registration request and verifying its validity, the interface service component downloads the user's project data to the client 610 and hands over the session to a dedicated IDE server instance running within the analytics service layer. The server instance then communicates directly with the client 610 via websocket.

(5) Data tool 806. This module allows the model builder to define a dataset, understand it, and prepare it for model building.

(6) Modeled dashboard 814. Each project has its own modeling dashboard. An instance of this module launches a modeling process for the project, measures them as the results arrive, and provides the model builder with control and measurement equipment to make in-flight adjustments. It calculates which modeling technique works for which dataset and passes these requirements to the analytics services layer. Once the execution engine begins to build the model, this module provides execution state and control.

(7) Insight 808. Once the machine learning process begins to produce substantive results, this module provides the model builder with deeper insights. Examples include a text mining overview, predictor importance, and a one-way relationship between each predictor and the target. Most of these insights are easy to understand and do not require a deep knowledge of statistics.

(8) Prediction 816. Once the execution engine builds at least one model, this module provides an interface for making predictions based on new data.

Activities within the interface services layer trigger activities within the analytics services layer. As discussed above, the technique IDE and monitoring module are divided to be partially performed within the analysis service layer (see monitoring module 818 and technique IDE module 820). Other modules in this layer include:

(1) Job queue 822. Each project may have its own job queue instance that fulfills model calculation requests from the corresponding modeling dashboard instance. The job includes a reference to the partition of the project's dataset, modeling techniques, and priorities within the project. The module then builds and maintains a job priority list. When computational resources are available, broker 824 requests the next job from the job queue. A user with sufficient permission can add, remove, or prioritize modeling jobs in the queue at any time. The queue is sustained via the temporary DB module 826, where the backend storage provides extremely fast response times.

(2) Broker 824. These modules instantiate workers, assign jobs to them, and monitor their health. One broker may operate for each worker cloud. The broker adds a secure buffer to dynamically supply and terminate workers to meet the current level of demand from the open job queue. Upon launch, each worker automatically registers with a broker for its cloud environment and provides information about its computing power. Brokers and workers send hardbeat messages to each other every few seconds. The worker will automatically restart and re-register in the event of a crash or loss of contact with its broker. The broker will drop the worker from its pool of available resources and record a warning if it misses too many heartbeat messages. When a new job arrives from the job queue and the worker completes an existing job, the broker continually recalculates the number of workers and the allocation of jobs to these workers.

(3) Worker Cloud 640. These modules include a pool of workers. Each worker is a running virtual machine instance, or another unit of built-in compute resources in its cloud environment, that receives jobs from the corresponding brokers. From a worker's point of view, a job includes project references, project dataset partitions, and modeling techniques. For each task in the assigned modeling technique, the worker first queries the file storage module 830, which has a special directory subtree for the modeling results, by any other worker projecting. You may check to see if you have completed it for that dataset partition in. When the first worker processes the step, it makes a calculation and stores it in file storage 830 so that other workers can reuse it. Since the modeling technique is constructed from the tasks in the common modeling task library, there may be a substantial level of commonality in task execution across the modeling techniques. Caching the results of task execution can enable implementations that significantly reduce the amount of computational resources consumed.

The data service layer 650 provides a variety of different storage mechanisms to support modules within other layers.

(1) Temporary DB826. This module provides an interface to a storage mechanism for data that benefits from extremely fast access and / or is transient and maintains the storage mechanism. In some implementations, this uses an in-memory DBMS deployed in a master-slave configuration with automatic failover. This module provides an interface for storing objects as key-value pairs. The key is linked to the specific user and project, but is still very small. The value can be a string, a list, or a set.

(2) Sustainable DB828. This module provides an interface to a storage mechanism for persistent data and maintains the storage mechanism. In some implementations, the main types of data handled by this module include JSON objects and are highly scalable non-SQL deployed in clusters with automatic failover for both high availability and high performance. You may use a database. Objects stored through this module typically range in size up to a few megabytes.

(3) File storage device 830. This module provides an interface to the storage mechanism for files and maintains the storage mechanism. The types of data stored through this module include uploaded datasets, derived data, model calculations, and forecasts. The module may layer file directories and naming conventions on cloud storage. In addition, when cloud workers access the module, they may also temporarily cache the stored files in their local storage.

(4) VM image storage device 832. This module provides an interface to storage for VM images used to run IDE and worker instances and maintains storage. It stores images in a self-sufficient VM container format. For IDE instances, this reserves the user's state for the duration of the session, while loading a new worker instance as a blank copy from that worker type template.

Together, these services manage a wide variety of information, including:

(1) UI session 834: Data for rendering the current state of the active user session and performing simple request authentication and access control.

(2) UI object 836: Content displayed by the UI.

(3) Cache 838: Application content stored in the cache.

(4) System configuration 840: Configuration parameters for launching the computing infrastructure and executing the model search service.

(5) System health 842: Real-time data collected from modules in system 600.

(6) User / Project Management 844: Settings and user privileges for each project, as well as individual user settings.

(7) Dataset 846: A data file uploaded by the user for the project.

(8) Modeling calculation 848: Intermediate modeling result, final fit model, and calculated prediction.

(9) VM image 850: An image used to start a new IDE server.

Again, the specific modules 802-850 described above are logical constructs. Each module may contain executable code from many different source files, and a given source file may provide functionality to many different modules.

Overtime Prediction In some embodiments, the prediction modeling system 100 takes into account the observation of X at the time before t, and optionally the observation of other predictor variables P at the time before t, at time t. Includes a time series model that can optionally predict the value of target X at t + 1, ..., T + i. In some embodiments, the predictive modeling system 100 divides past observations to train a supervised learning model, measure its performance, and improve accuracy. In some embodiments, the time series model provides useful time-related predictive features, eg, predicting the previous value of the target at different delays. In some embodiments, the predictive modeling system 100 takes into account the amount of new information in such observations and the cost of refitting the model as time advances and new observations arrive. Refresh the time series model.

Here are examples that illustrate the useful use of time series models. In this example, we want the supermarket chain to forecast daily sales for the next 6 weeks for each supermarket location. The data available is other variables (eg, historical and planned dates for population and economic growth around each location, holidays and major social events, and historical and planned dates for chain promotions). Includes daily sales data from 10,000 locations up to 3 years ago. In some embodiments, a time series model trained with historical data can accurately predict daily sales for the next 6 weeks for each supermarket location.

Here, some embodiments of techniques for generating and using time series models are described. When a data scientist develops a predictive modeling technique using the Modeling Technique Builder 220, the data scientist may show that the modeling technique is specific to a time series prediction problem. The modeling technique builder 220 then encodes this property in the metadata of the modeling technique. The dataset itself may also have time series specific metadata such as the date range in which the data occurred, the time resolution of the observation, any downsampling that has already occurred, and so on.

When the search engine 110 loads the dataset (eg, in step 404 of method 400 illustrated in FIG. 4), it is whether the dataset is considered to contain time series data and, if so, a time index. It may automatically detect what is considered to be. The time index may include time resolution and time steps (“time intervals”). Time resolution is a unit in which time is retained (eg, seconds or days). For example, if dates (eg, all dates) are encoded in standard day format (eg, months / days / years), engine 110 may use days as the resolution. As another embodiment, if the dates (eg, all dates) include hours, minutes, and seconds, engine 110 may use seconds as the time resolution. Similarly, the engine 110 may include any suitable time metric (eg, 1,000 years, centuries, 10 years, years, quarters, seasons, months, weeks, days, hours, minutes, seconds, milliseconds, microseconds). , Nanoseconds, etc.) can be used as the common time resolution. A time step is a time cycle (eg, minimum time cycle, most typical time cycle, user-defined time cycle, etc.) between continuous observations (eg, daily, monthly, or yearly data).

Where the dataset contains time series data with a mixture of time resolutions, the engine 110 uses the most common resolutions as part of the time index or after converting all time data to a common resolution. The "minimum common resolution" may be used. In some embodiments, the engine 110 uses an internal objective function that weights the frequency and difference of potential indexes to select an index (eg, the optimal index). For example, if 90% of the date variables are in day resolution and 10% are in second resolution, the objective function may determine that day resolution is the best choice. The reverse mixture (90% second resolution, 10% day resolution) may give rise to the determination that second resolution is the best choice. At 50% mixing, the objective function may determine that it is best to compromise on the resolution between days and seconds (eg, hours). The choice of objective function may also be determined by meta-machine learning about the space of the previously used objective function and their accuracy with respect to prediction problems with various properties.

In some embodiments, the user either blocks or enforces the treatment of the dataset as a time series dataset (eg, in step 406 of method 400), and the dataset is a time series dataset. Alternatively, the search engine's determination that it is not a time series dataset may be overridden. If the user chooses to force the treatment of the dataset as a time series dataset, the user may specify variables that are time-based, how to interpret their data format, and / or time index. .. In some cases, the user may provide information on how to transform variables to use a given time index.

For automatically detected time series datasets, the user accepts the proposed time index, chooses from any automatically detected alternatives, or transforms variables to use a custom time index. A custom time index may be specified, which may also require that the method be specified. Once the time index is accepted, selected, or specified, other time variables may be converted to offsets from each other and used as predictive features. For example, if the dataset uses month / day / year as the time index and one of the variables is "date of birth", the engine 110 will have other events (eg "marriage", "children". The offset from the "date of birth" (also known as the "age") may be calculated with respect to the "date of birth", etc.

After the user reviews or presents the time series dataset and checks, selects, or defines the time index, engine 110 evaluates the dataset (eg, in step 408 of method 400) and (eg, method). This evaluation is presented to the user (in step 410 of 400). The presentation may include schematics showing target and predictor variable values over time, either individually or in groups. In some embodiments, the presentation illustrates the dependence of the current value of each time series variable on past values at one or more time delays. Where the dataset is a panel dataset containing a mixture of time series and cross-sectional variables, the presentation of the cross-sectional variables, either in the same period or at different delays from each other. Dependencies between them may be illustrated.

The user is used to exit the training window (eg, time range of data used for training) and the validation window (eg, time range of data used for validation) or holdout window (eg, holdout test). It may indicate a "skip range" in the data, which is the gap between the start of the data time range). In some cases, there are operational and logistical reasons for the delay between the last historical observation and the first expectation. In a supermarket example, the data from the previous week arrives at headquarters on Sunday and can be completely clean. However, operationally, even after the forecast model is complete, it can take several days both to communicate the forecast to all store locations and for the store to make adjustments to its operation in response to the forecast. Similarly, there may be a delay in receiving or clearing historical data. In either case, the practical goal may be to start the prediction after some gap from the last available historical observation. The skip range allows the engine 110 to evaluate the candidate prediction model with this gap in place.

The user may indicate the desired range of expectations (eg, the number of future time cycles predicted by the model, or the number of distinctly different future events predicted by the model). In some cases, this range may be only one observation, eg, sales for the next quarter. In other cases, as in the supermarket chain embodiment, the range may include some future cycles, which in that embodiment is 42 days.

In some cases, engine 110 may propose an expected range based on the frequency of the data and / or the total number of cycles in the data. For example, using daily data and a relatively small number of time cycles, the engine 110 may propose a 7-day forecast, while using a relatively large number of time cycles, the engine 110 may propose a 30-day forecast. Good. Using monthly data, engine 110 may propose a 3, 6, or 12 month forecast, depending on the number of time cycles. Using quarterly data, Engine 110 may propose 4, 8, or 12 quarterly forecasts. For annual data, engine 110 may propose 5, 10, or 20 year forecasts.

In some cases, the engine 110 uses a verification range equal to the expected range or a multiple of the expected range (eg, a logical multiple) (eg, if the user specifies an unusually short expected range).

For time series data, engine 110 cross-validates and with a set of training ranges (eg, in steps 418 and 422 of method 400), a set of corresponding validation ranges offset by the skip range, and a holdout range. Holdouts may be implemented. Engine 110 may have a default target number training and validation range that can be adjusted according to the amount of data available. A dataset with a relatively small time cycle may be split into fewer training and validation ranges, while a dataset with a relatively large time cycle may be split into more training and validation ranges. Good.

Depending on the size of the data, the frequency of the data, the length of the skip range, and / or the expected range, the engine 110 may choose the length of the training range. For example, if daily observations and skips and forecast ranges are represented by weeks, months, or years (or multiples of corresponding days such as 7, 30, and 365), the engine will be an integer week, month, or You may choose the training range for the year. The length of these training windows is the total number of observations, the total amount of data, the total amount of variation in target and predictor variables over time, the amount of seasonal variation exhibited by the variables, and the variation of these variables over different time windows. It can depend on consistency and / or target length of expected cycle. The engine 110 may use an internal objective function that weights these factors to select the length of the training window (eg, the optimum length). For example, the default may be five training and validation ranges that are evenly divided across the entire dataset. However, if the amount of variation over a longer time cycle is low, the engine 110 may shorten the time window. Alternatively, if the data is seasonal, the engine 110 uses only three windows, thereby placing several years of data within each range. Alternatively, if there are several specific cycles in the dataset that exhibit high variability, the engine 110 may divide the data so that each window contains one of these cycles. The choice of objective function may be determined by meta-machine learning about the space of the previously used objective function and their accuracy with respect to prediction problems with various properties.

With the desired number of training and verification ranges, plus the length of these ranges plus skip ranges, the engine 110 can divide the dataset into a consistent set of training and verification ranges. For panel data, each training and validation pair can be further subdivided into layers (eg, by randomly assigning compartmentalized observations to the layers). In the supermarket embodiment, the training range may be 30 weeks, the skip range may be 1 week, and the verification range may be 6 weeks, about 4 training sets over 3 years. Causes. However, since there are 10,000 stores, these stores can be "downsampled" as described below to further improve performance. The engine may also reverse the subwindows in the training and validation windows used to tune the model hyperparameters as well as the mixed model.

In some embodiments, the holdout data is only within the last time window. However, if the dataset is panel data and is downsampled, the holdout data comes from the same time period as the other data, but may come from different non-overlapping samples.

Similar to the cross-sectional model (see, eg, the description in step 350 of method 300 above), the engine 110 iterates through the dataset, training each model in a small portion of the training window and its performance for that portion. May then be evaluated and then determined based on its performance whether to continue testing the model on additional data. For time series data, each part may end on the last observation in the training window, and the initial part may start so that the partial training window is a logical multiple of the validation window. Larger parts may use larger multiples. For example, a weekly verification window may include a first part that starts 4 weeks before the end of the training window, a second part that starts 8 weeks before, a 3rd part that starts 12 weeks before, and so on. You may use it. The verification window measured by the month is the first part starting 3 months before the end of the training window, the second part starting 6 months ago, the third part starting 9 months ago, 12 months ago. A fourth part or the like starting with may be used. The verification window measured by the year uses the first part, which starts 4 years before the end of the training window, the second part, which starts 8 years ago, and the 3rd part, which starts 12 years ago. May be good. Alternatively, the partial cycle may be 5, 10, and 15 years before the end of the training window. The parts are linear (eg, 3, 6, 9, 12 cycles or 4, 8, 12, 16 cycles) or geometrically (eg, 3, 6, 12, 24 cycles or 4, 8, 16, 32 cycles) may be increased. Exponential growth of parts is also possible, such as specific schedules based on problem domain and / or data analysis (eg, 3, 6, 24, 192 cycles or 4, 8, 32, 256 cycles). ..

To improve execution speed and reduce resource consumption, engine 110 may suggest default downsampling. There are two types of downsampling, namely temporal and cross-sectional. For downsampling over time, the total number of observations within each partition may be reduced by a fixed percentage or aggregated to longer time interval resolutions. For example, a time window containing 10 billion observations would be reduced to 10 million observations. Alternatively, a window of data with observations every millisecond may aggregate these observations every minute. In this case, all of the observation values in the downsampling window may be contained in a single value for aggregate observation. The aggregated value may be, for example, a start value, an end value, the most extreme value, a mean value, or a value calculated through a weighting function to take into account some characteristics of the variation. Engine 110 may use an objective function that weighs on the potential reduction in observations due to variability of targets and predictors within the corresponding time window. For example, a 1,000-fold reduction in observation can justify a 5% reduction in variability. The choice of objective function may also be determined by meta-machine learning about the space of the previously used objective function and their accuracy with respect to prediction problems with various properties.

Downsampling over time may be used, for example, for very high frequency data where the variation of the data is lower than the frequency of the data. Cross-sectional downsampling is generally more common because there can be a large number of compartments whose behavior does not change much over time. Regarding the embodiment of the supermarket chain, there are 10,000 divisions, one for each store location. Most of the variability can be due to factors over time, with relatively low cross-sectional variability across stores. For example, similar stores may actually be classified into a much smaller number of representative groups. Therefore, engine 110 may simply build a random subsample of divisions, in this case store locations, and train the prediction model with one or more subsamples. For example, instead of using all 10,000 stores, the engine 110 can be trained with a random selection of 1,000 stores. The engine 110 may also apply this downsampling when cross-validation is used along cross-validation dimensions. For example, in the case of 5-fold cross-validation, the engine may use 5 layers of 400 stores instead of 5 layers of 2,000 stores. The rate of downsampling may be determined by a function of the number of compartments, by an objective function that balances multiple properties of the sample, or by the user. In some cases, the choice of downsampling ratio may be refined by meta-machine learning for many different prediction problems.

One challenge in generating accurate predictions from time series models is that they can be sensitive to the choice of training and validation time windows. In some embodiments, the engine automatically evaluates this sensitivity. For example, engine 110 may evaluate its sensitivity to time window selection for all modeling techniques as it performs. As another embodiment, the engine 110 may evaluate the sensitivity after the model exceeds a threshold with predictive accuracy. The third option is to evaluate the sensitivity of the upper model based on their relative prediction accuracy. The fourth option is to activate the sensitivity analysis on demand when the user requests it. Sensitivity analysis may include fitting the model with a slide training and validation window and measuring how the model accuracy varies with the points contained in the window. For example, the graph may plot the model accuracy on the vertical axis and the start observation of the training window on the horizontal.

Once the engine has completed fitting the time series models (eg, in step 446 of method 400) and presented them to the user, the user may further explore the effect of different training data windows on model performance. Good. The user interface may adjust the training and validation windows so that the user selects a particular fit model or set of fit models and includes any observations that are not in the holdout window. If the engine does not use subwindows for adjusting hyperparameters, the optimal hyperparameters calculated during the original fit of the model may be used when refitting the model with a new window. When the nesting training and validation window is available, the engine automatically recalculates the optimal hyperparameters on the inverted window and / or recalculates the optimal hyperparameters, or first. You may allow the user the choice of whether to use the calculated one.

After the user has scrutinized the holdout results (eg, in step 450 of method 400), the user recaptures any subset of the model using any combination of training, validation, and data from the holdout window. May be compatible. Since the data from the holdout window is typically the most recent data, including it in the fit of the model can improve the accuracy of the model in predicting future values. Therefore, before deploying a model for making new predictions, the user may refit the model onto the training data, whose last observation is the last observation in the holdout window. The choice of the first observation used in this refit may depend on the size of the training window calculated during the sensitivity analysis (eg, the optimal size). The user may override the calculated starting point based on an analysis of the sensitivity and / or domain knowledge of the prediction problem.
Some embodiments of time series modeling techniques

With reference to FIG. 9, method 900 for time series predictive modeling may include steps 910-980. In step 910, time series data is acquired. Time series data may include one or more datasets. Each dataset may contain multiple observations. Each observation may include (1) an indication of the time associated with the observation and (2) the value of one or more variables. In step 920, the time interval of the time series data is determined. In step 930, one or more variables of the time series data are identified as targets. Optionally, one or more variables of the time series data may also be identified as features. In step 940, the "expected range" and "skip range" associated with the prediction problem represented by the time series data are determined. The prediction range may indicate the duration of the time cycle in which the value of the target is predicted. The skip range may indicate a time delay between the time associated with the oldest prediction within the forecast range and the time associated with the latest observation based on the prediction within the forecast range.

In step 950, training data is generated from the time series data. The training data includes a first subset of observations of at least one of the datasets. The first subset of observations includes training inputs and output sets of observations. The times associated with the observations in the training input and training output sets correspond to the training input time range and the training output time range, respectively. The skip range separates the start of the training output time range from the end of the training input time range. The duration of the training output time range is at least as long as the expected range. In step 960, test data is generated from the time series data. The test data includes a second subset of observations of at least one of the datasets. The second subset of observations includes the test inputs and test validation sets of the observations. The time associated with the test input and the observation in the test verification set corresponds to the test input time range and the test verification time range, respectively. The skip range separates the start of the test verification time range from the end of the test input time range. The duration of the test verification time range is at least as long as the expected range. At step 970, the prediction model is fitted to the training data. In step 980, the conforming model is tested with test data.

In step 910, time series data is acquired. Time series data may be obtained from any suitable source using any suitable technique (measured using sensors, received over a communication network, loaded from a computer-readable medium). Etc.). The time series data may include one or more datasets, each of which may contain one or more observations. The dataset may correspond to individual "classifications" of data, as described above. Each observation in the dataset may include an indication of the time associated with the observation (eg, a time stamp). The time associated with the observation may be the time when the value of the observation is measured, reported, received, etc. The "time" associated with the observation may include date and / or time, or any other suitable time data.

In step 920, the time interval of the time series data is determined. In some embodiments, the time interval of the data is explicitly indicated by the metadata associated with the data. In some embodiments, the time interval of the data analyzes the data (eg, based on the time resolution of the time indicator associated with the observation and / or the interval between the times associated with the continuous observation). By doing so, it is determined.

The time interval between observations in the dataset can be uniform or non-uniform. If the time interval between observations within the dataset is uniform, the uniform interval may be the time interval of the dataset. If the time intervals between observations within the dataset are non-uniform, the dataset may be modified so that the time intervals between observations within the modified dataset are uniform. The time interval of the modified dataset may be determined, for example, by applying an objective function (eg, a weighted objective function) to the original dataset. The objective function is modified based on, for example, (1) the individual proportions of multiple pairs of continuous observations exhibiting each non-uniform duration, and / or (2) the duration of the non-uniform time interval. The time interval may be determined. In some embodiments, the modified time interval is the shortest common time cycle (eg, the shortest time cycle that is a multiple of each integer of the non-uniform time cycle).

The original dataset with a non-uniform time interval is the modified data with a modified uniform time interval by sampling (eg, downsampling) and / or aggregating the observations of the original dataset. It may be converted into a set. The step of downsampling the observations of the original dataset is the time corresponding to the time interval instances of the time series data, for each modified time interval instance within the time cycle covered by the observations in the original dataset. A step that identifies all observations in the original dataset, a step that aggregates the identified observations to generate an aggregated observation, and a step that inserts the aggregated observations into a modified dataset. May be accompanied by. The step of aggregating the identified observations sets the value of each variable in the aggregated observation to (1) the corresponding variable value contained in the oldest of the identified observations, (2) identification. Corresponding variable value included in the latest of the observed observations, (3) Maximum value of the corresponding variable value included in the identified observation, (4) Corresponding included in the identified observation By setting the minimum value of the variable value, (5) the average of the corresponding variable values contained in the identified observation, or (6) the value of the function of the corresponding variable value contained in the identified observation. It may be carried out.

The time interval between datasets in time series data can be uniform or non-uniform. If the time intervals between the datasets are uniform, the uniform intervals may be the time intervals of the time series data. If the time intervals between the datasets are non-uniform, then one or more of the datasets may be modified so that the time intervals between the modified datasets are uniform. The modified time interval of the time series data may be selected from the interval of the dataset or determined using any other suitable technique. For example, the modified time intervals of time series data are (1) individual percentages of observations contained in the dataset exhibiting each of the non-uniform time intervals, and / or (2) non-uniformity of the dataset. It may be calculated using an objective function (eg, a weighted objective function) based on the duration of the time interval. In some embodiments, the modified time interval of the time series data is the shortest common time period (eg, the shortest time period that is a multiple of each integer of the non-uniform time intervals of the dataset). Some embodiments of the technique for modifying the time interval of a dataset are described above.

In some embodiments of Method 900, feature engineering may be performed on time series data. Such feature engineering may be performed, for example, before or after the time interval of the time series data is determined. In some embodiments, the steps of performing feature engineering on the time-series data generate a step of identifying a first variable in the time-series data that has a value representing time and a value of the second variable. A step in which each value of the second variable is an offset between the time value and the reference time value of the first variable, and a step of adding the second variable to the time series data (eg,). , Each value of the second variable may be added to the observation, including the value of the first variable from which the value of the second variable was derived). In some embodiments, the first variable may be removed from the time series data. In some embodiments, the reference time is the date of the event (eg, birth, marriage, graduation from school, start of employment for an employer, start of work in a particular position, etc.). This feature engineering technique may be used to convert absolute time values to relative time values, which relate to patterns in data from different datasets spanning different time periods (eg, related to the age of the entity). Patterns) can be greatly facilitated, and thus accurate prediction of values associated with such patterns.

In some embodiments, the step of performing feature engineering includes the step of downsampling time series data in time. Several techniques for downsampling time series data over time are described above. The step of downsampling time series data over time may increase the efficiency of time series modeling techniques based on the time series data (eg, reduce computational resource usage).

In some embodiments of Method 900, graphic information (eg, graphs or charts) may be presented (eg, displayed) via a user interface. The graphic information may indicate a time delay between a change in the value of one variable and a change in the correlation of the value of another variable. Such correlations may be detected using any suitable technique.

In step 930, one or more variables of the time series data are identified as targets. Targets may be identified, for example, based on metadata associated with time series data, based on a description of the prediction problem, and / or based on user input. Optionally, one or more variables of the time series data may also be identified as features. Features may be identified, for example, based on metadata associated with time series data, based on a description of the prediction problem, and / or based on user input.

In step 940, the prediction range associated with the prediction problem represented by the time series data is determined. The prediction range may indicate the duration of the time cycle in which the value of the target is predicted. The expected range is (1) the time interval of the time series data, (2) the number of observations contained in the time series data, (3) the time cycle covered by the observations contained in the time series data, (4) microseconds, Natural time cycle selected from the group consisting of millisecond, second, minute, hour, day, week, month, quarter, season, year, 10 years, century, and 1,000 years, (5) for user input, etc. It may be determined based on. In some embodiments, the expected range is an integer multiple of the time interval of the time series data. In general, the expected range may increase as the number of observations in the time series data increases.

In step 940, the skip range associated with the prediction problem represented by the time series data is determined. The skip range may indicate a time delay between the time associated with the oldest prediction within the forecast range and the time associated with the latest observation based on the prediction within the forecast range. The skip range is, at least in part, the waiting time for collecting time series data, the waiting time for communication of time series data, the waiting time for analyzing time series data, the waiting time for communication for analyzing time series data, and / Or may be determined based on the latency of implementing the action based on the analysis of time series data. Such latency may be determined, for example, based on user input and / or metadata associated with time series data. In some embodiments, the skip range is determined or user-defined based on the metadata associated with the time series data.

In step 950, training data is generated from the time series data. The training data includes a first subset of observations of at least one of the datasets. The first subset of observations includes training inputs and output sets of observations. The times associated with the observations in the training input and training output sets correspond to the training input time range and the training output time range, respectively. The skip range separates the start of the training output time range from the end of the training input time range. The duration of the training output time range is at least as long as the expected range.

In some embodiments, the duration of the training input time range is the total number of observations in the time series data, the amount of variation over time of at least one value of the variable, of at least one value of the variable. It is determined based on the amount of seasonal variation, the consistency of variation in the value of at least one of the variables over multiple time periods, and / or the duration of the expected range.

In some embodiments, a subset of training data is identified for the purpose of training a predictive model on the subset rather than training the model on all training data. The step of training a model based on a subset of training data may use less computational resources (and time) than training a model based on all training data. In some embodiments, each subset of training data ends at the end time of the training input time range. The training data subset may start at different times within the training input time range and / or sample observations within the training input time range at different sample rates. The duration of the training data subset may be a multiple of an integer of the expected duration.

In some embodiments, the training data may be downsampled (eg, downsampled in time or cross-sampling). The training data is selected by selecting a downsampled time interval and downsampling each of the datasets in the training data according to the downsampled time interval (eg, using the techniques described above). It may be downsampled in time. The training data may be cross-sectionally downsampled by removing one or more of the datasets from the training data. In some embodiments, the training data may be both temporally downsampled and cross-sectional downsampled.

In step 960, test data is generated from the time series data. The test data includes a second subset of observations of at least one of the datasets. The second subset of observations includes the test inputs and test validation sets of the observations. The time associated with the test input and the observation in the test verification set corresponds to the test input time range and the test verification time range, respectively. The skip range separates the start of the test verification time range from the end of the test input time range. The duration of the test verification time range is at least as long as the expected range.

In some embodiments, the duration of the test input time range is the total number of observations in the time series data, the amount of variation over time in at least one of the variables, the duration of at least one of the variables. It is determined based on the amount of seasonal variation, the consistency of variation in the value of at least one of the variables over multiple time periods, and / or the duration of the expected range.

In some embodiments, a subset of the test data is identified for the purpose of testing the predictive model on the subset rather than training the model on all training data. The step of training a model based on a subset of test data may use less computational resources (and time) than training a model based on all training data. In some embodiments, each subset of training data ends at the end time of the training input time range. The test data subset may start at different times within the training input time range and / or sample observations within the test input time range at different sample rates. The duration of the test data subset may be a multiple of an integer of the expected duration.

In some embodiments, the test data may be downsampled (eg, downsampled in time or cross-sampling). Several techniques for temporal and cross-sectional downsampling are described above.

At step 970, the prediction model is fitted to the training data. In step 980, the conforming model is tested with test data. Cross-validation (including, but not limited to, nested cross-validation) and holdout techniques may be used to fit and / or test the predictive model. For cross-validation purposes, time series data may be cross-validated and / or temporally divided.

Several embodiments of method 300 for selecting a prediction model for a prediction problem may be used to select a time series prediction model for a time series prediction problem. In some embodiments, model-specific predictions of one or more of the features of the time series data are determined (eg, using the embodiment of Method 1000 described below). May be good. In some embodiments, the time series prediction models may be mixed (eg, using the techniques described herein). In some embodiments, the time series prediction model may be expanded and / or refreshed (eg, using the techniques and / or performance enhancements described herein). In some embodiments, the intensity of interaction between two or more variables (eg, features) in the time series data is determined (eg, using the techniques described herein). You may.
Universal feature importance

When tackling supervised machine learning problems, the main challenge is often to measure features that have the best predictors of the target. Measuring the importance of such features is two separate steps in predictive modeling: (1) understanding prediction problems in general, and (2) how a concrete fit model produces prediction results. Can be useful in understanding.

For purpose (1), the metric of feature importance is the evaluation of the dataset (eg, step 408 of method 400), the presentation of the rating to the user (eg, in step 410 of method 400), (eg, eg, step 410 of method 400). The user (in step 412 of method 400) may be informed of how to refine the dataset and / or modeling techniques to try or propose trials (eg, in steps 422 and 424). For purpose (2), feature importance metrics signal the automatic development of the mixed model (eg, in step 432 of method 400) and the relative performance of the alternative model (eg, in step 446 of method 400). You may assist the user in understanding.

In practice, the step of determining feature importance for a primary purpose generally occurs within the context of a particular model or model family (eg, Random Forest). Of course, each model can have advantages and disadvantages as a device for measuring feature importance. Therefore, the step of calculating feature importance using several different types of models can achieve a better understanding. For example, random forests, generalized addition models, and support vector machines are radically different types of models for machine learning, producing different measurements of the importance of the same features for the same dataset. You may. Such differences may provide deeper insight into the structure of the prediction problem and suggest means of future exploration. For purpose (2), feature importance is generally specific to the model under consideration.

Some embodiments of techniques for calculating feature importance for arbitrary models are described herein.

Given the dataset (or any sample thereof) and modeling techniques, the search engine 110 may use universal partial dependencies to calculate the importance of any feature. First, the engine 110 uses modeling techniques to obtain accuracy metrics for the predicted model fitted to the sample. Engine 110 can either implement this conformance from the beginning or use a previous fit. The engine 110 then takes all its values across all observations, shuffles them, and reassigns them to the observations (eg, randomly reassigns them) for a given feature. This random shuffle can reduce (eg, destroy) any predicted value of its features. The engine may then re-score the model on a dataset with shuffled feature values to generate new values for accuracy metrics. (Optionally, the engine may refit the model to the shuffled dataset before rescoring the fitted model on the shuffled dataset.) The reduced accuracy of the model is predicted. Shows the amount lost, and therefore the importance of features to the model and / or within the scope of the modeling technique.

Using the techniques described above for calculating the importance of one feature for one model and / or modeling technique, the engine iterates across the features and within the model and / or modeling technique. The relative importance of a feature can be determined, iterative across the model and / or modeling technique, and the relative importance of the feature can be determined across the model and / or modeling technique, or both.

To meet objective (1), engine 110 may generally maintain a list of modeling techniques that produce exemplary results of feature importance. When evaluating the dataset, engine 110 may automatically start all or some of these modeling techniques. The engine may choose the modeling technique to start based on the priority of the dataset. The user interface may display feature importance values individually for each modeling technique used or across all modeling techniques used. The engine 110 may also allow the user to select an arbitrary modeling technique from the modeling technique library 130 used to measure feature importance.

In addition to assisting the user in understanding the prediction problem, the engine 110 may use the results from this application of feature importance to guide additional automated analysis. For example, for features that score high in importance across the board, the engine may allocate more resources to exploring the interactions of these features. For features that score low in importance across the board, the engine may completely drop these features from the dataset. When some features have high importance for some model or modeling technique, and low importance for other model or modeling technique, the engine will use more modeling technique. You may try and direct a deeper search of the predictive model, using more data earlier in the model space exploration process.

To meet objective (2), the engine may calculate the feature importance values of the conforming model automatically or on demand. If automatic, the engine 110 may calculate feature importance values for any of all models or parts thereof. Some may be calculated by any suitable method. For example, some are N highest performance models, all models that meet at least a certain level of performance threshold for a performance metric, or that performance metric is within a given portion of the performance of the top model. All models may be included. In some embodiments, the engine may take into account the available computational resources and make some adjustments to be larger or smaller in proportion to these resources.

In some embodiments, the user may request feature importance calculations on demand. In some cases, the user may wish to confirm the importance of some or all features for a given model. In other cases, the user may wish to confirm the importance of some or all features across different models.

Using the techniques described above to provide feature importance values for either predictive problems or specific models in general can have a number of benefits. For example

(1) If a feature does not provide information in general, or at least in all accurate models, the collection of data corresponding to that feature may be stopped. In some cases, there is an actual cost to extract features from their source location or to make available features such as labor, which pays the vendor for the data.

(2) User expectations of feature importance and differences in measured feature importance can be justified for further investigation. It can be found that there are errors in the datasets that make up the difference, or that the difference is genuine and provides new insights into the prediction problem.

(3) In some cases, it may be desirable to generate a model that makes predictions using as few features as possible. In these cases, the user searches between concrete modeling techniques or all modeling techniques using only the N features of maximum importance, or only features with importance values above a specified threshold. Can be restarted.

(4) Understanding the important features can help the user improve the predictive model by experimenting with different methods of transforming and combining the most important features.
Some embodiments of the technique for determining the predicted value of one or more features of a dataset.

FIG. 10 shows method 1000 for determining predicted values (eg, "importance") of features that are one or more of the initial datasets that represent the initial prediction problem. In some embodiments, the predictive modeling system 100 is during evaluation of the dataset (eg, in step 408 of method 400) and / or as described above (eg, with reference to method 300). ) Method 1000 (or a portion thereof) may be performed when processing the dataset. In some embodiments, Method 1000 can be used to determine the predicted value of any dataset feature to any predictive model or predictive modeling technique.

In step 1010, system 100 implements a plurality of predictive modeling procedures. Each predictive modeling procedure is associated with a predictive model. The steps of performing each modeling procedure include adapting the associated prediction model to at least a portion of the initial data set that represents the initial prediction problem. The initial dataset contains previous observations, and each observation generally contains at least some of the features of the initial dataset.

In step 1020, system 100 determines each first accuracy score of the fit prediction model. The first accuracy score of the fitted model represents the accuracy with which the fitted model predicts the outcome of one or more of the initial prediction problems. Any suitable metric and / or technique for determining the accuracy of the model may be used, including, but not limited to, the step of testing the model in the holdout portion of the initial dataset.

At step 1030, system 100 "shuffles" the value of a particular feature F across the observations in the initial dataset, thereby producing a modified dataset that represents the modified prediction problem. In some embodiments, shuffling is performed by reassigning (eg, randomly) the values of feature F from those original observations to different observations. This shuffle operation can reduce (eg, destroy) any predicted value of feature F. Reducing (eg, destroying) the predicted value of feature F in the dataset, including, but not limited to, removing feature F from the dataset or reassigning the same value of feature F to each observation. Other techniques for this are possible. In some embodiments, any technique may be used that reduces the predicted value of feature F below the threshold.

In step 1040, system 100 determines a second accuracy score for the fitted prediction model for the modified prediction problem. The second accuracy score represents the accuracy with which the fitted model predicts the outcome of one or more of the modified prediction problems. Any suitable metric and / or technique for determining the accuracy of the model may be used, including, but not limited to, the step of testing the model on the holdout portion of the modified dataset. In some embodiments, the fit model is refitted to the modified dataset prior to determining the second accuracy score.

In step 1050, for each predictive modeling procedure (or fit model), system 100 calculates the predictive value of feature F. In some embodiments, the predicted value of feature F for the modeling procedure or model is based on changes in accuracy (eg, between a first accuracy score and a second accuracy score for the model). Calculated (based on the difference). In some embodiments, the model is based on the first and second accuracy scores so that the predicted values generally increase as the difference between the first and second accuracy scores increases. A function is used to determine the predicted value of a feature for a transformation procedure or model. The predicted values determined in step 1050 may be referred to herein as "model-specific predicted values" because the individual predicted values may be specific to a particular modeling procedure or model.

In step 1060, the system 100 determines whether to analyze the predicted value of another feature. In some embodiments, the determination is based on user input (eg, the system continues to analyze predicted features until all features specified by the user have been analyzed). In some embodiments, the system analyzes all features in the dataset. In some embodiments, the system analyzes only a subset of the features in the dataset. Such features can be selected based on any suitable criteria. If the system determines in step 1060 that another feature to analyze exists, the system then repeats steps 1030-1050 for that feature.

In some embodiments, method 1000 includes an additional step (not shown) of selecting a predictive modeling procedure performed in step 1010. System 100 may select modeling procedures from, for example, library 130 of predictive modeling techniques. In some embodiments, the system 100 selects two or more modeling procedures from two or more different predictive modeling families. Examples of the predictive modeling family can include linear regression techniques (eg, generalized addition models), tree-based techniques (eg, random forests), support vector machines, neural networks (eg, multilayer perceptron), and the like. .. For example, system 100 includes a modeling procedure from the tree family (eg, a random forest modeling procedure), another modeling procedure from a linear regression family (eg, a generalized addition model), and a support vector machine modeling procedure. You may choose.

The system 100 includes, but is not limited to, (1) a step of processing the data set based on the model-specific predicted value, (2) a step of presenting the evaluation of the data set to the user, and (3) a model-specific predicted value. Based on the steps of selecting predictive models for mixing based on, (4) presenting the user with the evaluated models and their associated model-specific predictive values, and (5) model-specific predictive values. And, based on the steps of allocating resources during the process of assessing the space of the potential predictive modeling solution for the predictive problem, and (6) the model-specific predictive values of the features, the model-independent predictive values of the features ( Any suitable using one or more of the model-specific predicted values (feature importance values) determined through Method 1000, including the step of calculating the feature importance values). Tasks (eg, predictive modeling tasks) may be performed. In some embodiments, after calculating the model-independent feature importance value, the system 100 uses the model-independent feature importance value to describe any suitable task (eg, described in the previous sentence). Tasks (1)-(5)) may be carried out.

In some embodiments, as part of a predictive modeling procedure, during evaluation of a dataset (eg, in step 408 of method 400) and / or described above (eg, with reference to method 300). As such), the system 100 performs feature generation and / or feature engineering based on model-specific predictions when processing the dataset. For example, system 100 may remove "less important" features from the dataset. In this context, a feature is a feature in a dataset if the predicted value of the feature is less than a threshold and the feature has one of the M lowest predicted values among the features in the dataset. It may be classified as "less important" if it does not have one of the N highest predicted values among them. As another embodiment, the system may create features derived from "more important" features in the dataset. In this context, a feature is a feature of a feature in a dataset if the predicted value of the feature exceeds a threshold and the feature has one of the N highest predicted values among the features in the dataset. It may be classified as "more important" if it does not have one of the M minimum predicted values in between. In some embodiments, the system 100 may use method 1000 to calculate the predicted values of the derived features.

In some embodiments, the system 100 may present (eg, display) a rating of the dataset to the user (eg, in step 410 of method 400), and the presented rating is a feature of the dataset. And / or information derived from it may be included. For example, for one or more modeling procedures or models, the system 100 identifies (1) "more important" and / or "less important features" and (2) displays predicted values of the features. And (3) rank features by their predictors, and / or (4) stop collecting less important features and / or remove less important features from the dataset. May be recommended. In response to presenting the user with a rating of the dataset, the system 100 may receive user input (eg, in step 412 of method 400) that specifies the refinement of the dataset.

In some embodiments, the system 100 selects a prediction model for mixing based on model-specific predictions and mixes the selected models (eg, in step 432 of method 400). System 100 may use any suitable technique to select a predictive model for mixing. For example, system 100 may select a "complementary top-level model" for mixing. In this context, "complementary top-level models" may include accurate models that achieve their high accuracy through different mechanisms. System 100 has M minimum accuracy values among the conforming models when the model has one of the N highest accuracy values among the conforming models when the accuracy of the model exceeds the threshold accuracy. Models may be classified as "top-level" models, such as when they do not have one of them. The system has (1) different most important features for the model (eg, features with the highest predicted values of the model), or (2) features that are important for the first model are important for the second model. Instead, the two models may be classified as "complementary models" if features that are not important to the first model are important to the second model. In this context, a feature is "important" to the model if the feature has a high predictive value for the model (eg, the highest predictor, one of the N highest predictors, a predictor above a threshold, etc.). Can be. In this context, a feature is "important" to the model if the feature has a low predictive value for the model (eg, the lowest predictive value, one of the N lowest predictive values, a predictor value below the threshold, etc.). Not ". In some embodiments, the system 100 may use the classification techniques described above to select two or more complementary top-level models for mixing.

In some cases, the step of mixing complementary top-level models may give rise to a mixed model with very high accuracy for the component model. In contrast, the step of mixing non-complementary models may not result in a mixed model with significantly higher accuracy than the component model.

In some embodiments, the system 100 may present to the user the predicted prediction models evaluated (eg, in step 446 of method 400) and their associated model-specific prediction values. In some embodiments, the system 100 calculates and / or displays feature importance values for only a subset of predictive models (eg, top-level models). The step of presenting the feature importance value to the user may assist the user in understanding the relative performance of the evaluated model. For example, based on the feature importance values presented, the user (or system 100) is the top model M, which is superior to the other top models, and the other top models, which are important to the model M. One or more features F may be identified that are not important to. The user may conclude that the model M makes better use of the information represented by the feature F with respect to the other top-level models (or the system 100 may indicate that). .. Based on this finding, stakeholders interested in the outcome expected by Model M may invest in a system for measuring or controlling feature F, which improves the outcome expected by the model. Can be done.

In some embodiments, during the process of evaluating the space of a potential predictive modeling solution for a predictive problem (eg, while performing method 300), the system 100 is a dataset representing the predictive problem. Resources for evaluation of the modeling procedure may be allocated based on the predicted values of the features in. For example, the system 100 may select or propose a predictive modeling procedure to evaluate (eg, in step 310 of method 300 or in step 424 of method 400). As described above, the system 100 may select or propose a predictive modeling procedure that is predicted to be suitable or highly suitable for the dataset. Several techniques for determining the suitability of a modeling procedure for a dataset with specified characteristics are described above. When determining the suitability of a predictive modeling procedure for a predictive problem based on the characteristics of the predictive problem, the system 100 may treat the characteristics of the more important features of the dataset as the characteristics of the predictive problem. In this way, the suitability score generated by System 100 may be matched to the more important features of the dataset. Because System 100 allocates resources based on aptitude scores, the step of matching aptitude scores to the more important features of the dataset is to allocate resources to the evaluation of the predictive modeling procedure, in part based on feature importance. You may let me.

In addition, or as an alternative, the system 100 (eg, during evaluation of the dataset and / or when processing the dataset as described above with reference to step 408 of method 300 or method 400). ) Allocate more resources to feature generation tasks with more important features of the dataset, and / or mix more resources (eg, in step 432 of method 400) to the resources of the complementary top-level model. It may be assigned.

In some embodiments, the system 100 may calculate model-independent predictions for feature F based on model-specific predictions for features. The system 100 includes, but is not limited to, (1) a step of calculating statistical measurements of model-specific predicted values (eg, mean, median, standard deviation, etc.), or (2) model-specific predicted values. Model-independent predictions may be calculated using any suitable technique, including the step of determining the combination. In the latter case, the combination may be a weighted combination. In a weighted combination, the model-specific feature values of a more accurate model may be weighted more heavily than the model-specific feature values of a less accurate model. In some embodiments, the model-specific feature values of the least accurate modeling procedure may be excluded from the calculations and / or combinations described in this paragraph.
Secondary model

As discussed above (eg, in the section entitled "Model Deployment Engine"), a model of the model ("Secondary Model") using a suitable quadratic modeling technique (eg, RuleFit). It may be desirable to simplify the task of creating and generating predictive code, and / or to prevent disclosure of dedicated details of the modeling techniques used to create the model. It may be possible to interpret the quadratic model more easily than the model in the quadratic model, which may provide insight into the "black box" model in another way.

However, there are concerns that the secondary model may not be systematically less accurate for prediction than the source model. To reduce (eg, minimize) any loss of accuracy associated with the step of transitioning from the source model to the secondary model (in some cases, a secondary model with higher accuracy than those source models). Some embodiments of the technique (for generation) are described below.

In some embodiments, after the user presents the results of a holdout test on the conforming model (eg, in step 446 of method 400) and / or the top model (eg, in step 450 of method 400), the user , (A) interpret the model, understand how it uses features and generate accurate predictions, and / or (b) may desire to deploy the model using the deployment engine 140. .. Some modeling techniques use opaque and / or complex methods to build a model, thereby making the model difficult to understand and, at least in some cases, increasing the difficulty of generating predictive code. obtain. The steps to prevent disclosure of dedicated model building techniques can further complicate the task of making the model understandable and generating predictive code for the model.

In some embodiments, the predictive modeling system 100 addresses these issues by building a secondary model of the source model. In some embodiments, the predictive modeling system 100 generally produces a model that is relatively easy to interpret, and one or more modeling techniques in which the predictive code can be generated relatively easily. (For example, RuleFit, generalized addition model, etc.) are used to build a secondary model. Such techniques are referred to herein as "secondary modeling techniques." Engine 110 produces a fitted "primary" model (eg, a primary model that the user wants to better understand, a primary model that the user wants to generate a prediction code, and / or a primary model with dedicated features). After that, the system 100 can use the quadratic modeling technique to create a quadratic model of the primary model.

For each feature in the primary model, there is a corresponding set of feature values derived from or derived from the original dataset. The secondary modeling technique may use the same features as the primary model, and therefore the original values of such features or a subset thereof may be used for training and test data of the secondary modeling technique. Good. In some embodiments, instead of using the actual values of the target from the dataset, the secondary modeling technique uses the predicted values of the target from the primary model.

In some cases, secondary modeling techniques may use alternative or ancillary training and / or test data. Such alternatives are capable of a wider range of other real-world data from either the same or different data sources (eg, via interpolation and extrapolation) (eg, through real-world samples). It may include real-world data combined with machine-generated data (for sex-extrapolating purposes), or data entirely generated by machine-based stochastic models. In some embodiments, the value of the target variable used to train the secondary model is a predicted value from the primary model.

One concern is that any error in the primary model can be complicated or expanded when building the secondary model, thereby systematically reducing the accuracy of the secondary model. First, we recognize and understand that there is a question as to whether this is empirically true. Second, in some cases, if this is true, we recognize and understand that using a more accurate primary model is likely to reduce the loss of accuracy. .. For example, a mixed model (eg, as explained towards the end of the section entitled "Modeled Spatial Search Engine") is sometimes more accurate than any single model, so two. The step of adapting the next model to the mixture of the primary models can reduce any loss of accuracy associated with the secondary modeling.

We have empirically determined that concerns about the accuracy of secondary models are largely misguided. Testing of the system was performed on 381 datasets, resulting in 1195 classifications and 1849 primary regression models. For the classification models tested, 43% of the secondary models were not as accurate as the corresponding primary models, but only 10% were worse, according to accurate log loss measurements. Thirty percent of the secondary models were actually more accurate than the primary model. In 27% of cases alone, secondary models above 10% were not as accurate as primary models. About one-third of these cases (nearly 9% of the total population) occurred when the dataset was very small. An additional third of these cases (again, nearly 9% of the total population) are very accurate in log loss where the primary model is less than 0.1, and the secondary model is still less than 0.1. Occurred when it was very accurate in log loss of. Thus, in more than 90% of cases with sufficiently large datasets, the secondary model was either within 10% of the primary model, or the secondary model was very accurate by absolute standard. .. In 41% of cases, the best secondary model was derived from a mixture of primary models.

For the regression models tested, 39% of the secondary models were not as accurate as the corresponding primary models, but only 10% were worse, according to the residual mean square error measurements of accuracy. Forty-seven percent of the secondary models were actually more accurate than the primary model. In 14% of cases alone, secondary models above 10% were not as accurate as primary models. About 10% of these cases (nearly 1.5% of the total population) occurred when the dataset was very small. In 35% of all cases, the best secondary model was derived from a mixture of primary models.

Based on this empirical data, we recognize that the quadratic model is generally either relatively accurate or absolutely accurate when the original dataset is large enough. I understand. In many cases, the secondary model is actually more accurate than the primary model. Finally, for more than one-third of all classification and regression problems tested, the most accurate secondary model was derived from a mixture of primary models.

We find that the secondary model is (a) to understand a complex primary model, (b) to simplify the task of generating predictive code for a predictive model, and (c) to build a dedicated model. Recognize and understand that it can be beneficial to protect the technique. Mixing primary models often improves on these benefits. Mixed models produce the best predictions at most times, but they are also generally more complex because they combine the complexity of all the components of all the models involved in the mixture. The step of generating a prediction code for a mixed model also generally combines the task of generating a prediction code for all component models included in the mixture. In addition to these component tasks, mixed models are generally slower to generate predictions (and / or generate predictions) because all of the mixed models are generally calculated to generate each prediction. Requires more computing resources to do). Secondary models generally reduce this time (and / or the use of computational resources) to the time it takes to calculate a single model. The mixed model also contains the exclusive features of each component model. Therefore, the secondary and mixed models operate in a highly complementary fashion.

In addition, the system 100 can automatically separate the dataset into training, testing, and holdout partitions, so that any particular secondary model works well compared to the primary model. It can be easily determined whether or not to do so.
Some embodiments of secondary modeling techniques

With reference to FIG. 11A, method 1100 for generating a quadratic prediction model may include steps 1110 and 1120. In step 1110, a fitted primary prediction model is acquired. The primary prediction model is configured to predict the value of one or more output variables in the prediction problem based on the value of one or more first input variables. The primary model may include any suitable technique (eg, a step of executing a machine executable module that implements a predictive modeling procedure, a step of implementing an embodiment of predictive modeling method 300, two or more predictive models. It may be obtained using mixing steps, etc.) or may be any suitable type of prediction model (eg, time series model, etc.). In step 1120, a quadratic predictive modeling procedure is performed on the fitted primary model. A quadratic modeling procedure is a quadratic predictive model (eg, a RuleFit model, a generalized addition model, any model organized as a set of conditional rules, two or more of the above types of models. (Mixed etc.) and associated.

Referring to FIG. 11B, method 1120 for performing a quadratic predictive modeling procedure may include steps 1122, 1124, 1126, and 1128. In step 1122, secondary input data is generated. The secondary input data includes a plurality of secondary observations. Each secondary observation is an output variable predicted by the primary model based on the observed value of one or more second input variables and the value of the first input variable corresponding to the value of the second input variable. Including the value of. The steps to generate the secondary input data are the step of acquiring the observation value of the second input variable and the corresponding observation value of the first input variable and the corresponding observation of the first input variable for each secondary observation. It may include a step of applying a first-order prediction model to the values and generating the predicted values of the output variables.

The primary and secondary models may use the same set of input variables. Alternatively, the quadratic model may use one or more of the input variables of the primary model (eg, all), and may use one or more of the other input variables. You may. Alternatively, the quadratic model may not use any of the input variables of the primary model and may use one or more other input variables.

In step 1124, secondary training data and secondary test data are generated from the secondary input data. Some embodiments of techniques for generating training and test data from input datasets are described herein.

In step 1126, a fitted secondary prediction model of the fitted primary model is generated by fitting the secondary prediction model to the secondary training data. In step 1128, the conforming secondary prediction model of the conforming primary model is tested with the secondary test data. Cross-validation (including, but not limited to, nested cross-validation) and holdout techniques may be used to fit and / or test the predictive model.

In some embodiments, the primary and secondary conforming models may be compared. For example, the accuracy score for each fitted model may be determined. The accuracy score of each fitted model may represent the accuracy with which the fitted model predicts the outcome of one or more prediction problems. (In some embodiments, the accuracy score of the secondary model represents the accuracy with which the fitted model predicts the observation of the target variable, rather than the accuracy with which the fitted model predicts the target variable value predicted by the primary model. In the embodiment, the accuracy score of the quadratic model represents the accuracy with which the fitted model predicts the target variable value predicted by the primary model.) If there is, the accuracy score of the fitted quadratic model is the fitted primary model. Exceeds the accuracy score of. As another example, the amount of computational resources is used by each of the fitted prediction models to predict the outcome of one or more prediction problems. In some embodiments, the amount of computational resources used by the fitted secondary model is less than the amount of computational resources used by the fitted primary model.

In some embodiments, multiple quadratic modeling procedures are performed and multiple quadratic models of the primary model are generated. In some embodiments, the accuracy score and / or computational resource usage of the secondary model is determined and compared. One or more of the secondary models may be selected for deployment, at least in part, based on the model's accuracy score and / or computational resource usage.

Several embodiments of method 300 for selecting a predictive model for a predictive problem may be used to select a secondary predictive model for the predictive problem. In some embodiments, model-specific predictions of one or more of the features of the input data are relative to the secondary model (eg, using the method 1000 embodiment described below). It may be determined. In some embodiments, the secondary prediction models may be mixed (eg, using the techniques described herein). In some embodiments, the secondary prediction model may be deployed and / or refreshed (eg, using the techniques and / or performance enhancements described herein).

Several embodiments of method 300 for selecting a predictive model for a predictive problem may be used to select a secondary predictive model for the predictive problem.
Text language conditioning

Some features of the dataset may include unstructured text blocks. Because different languages have different characteristics, the best predictive modeling step can depend on one or more languages present in the unstructured text features.

In some embodiments, the system 100 detects a language in a text string, or more generally, calculates the likelihood that the string is in a given language. System 100 may incorporate knowledge of the text language into the predictive modeling process through conditional text processing. For example, modeling techniques may include preprocessing steps, including steps to extract structured features from text. The feature extraction procedure may vary based on language. As another example, some modeling techniques may either be specific to a particular language or behave differently with respect to different languages.

For example, the character ngram works well with Chinese text. They often produce good results in English text, but are not very consistent, so word ngrams are usually preferred for processing English text. There are also "synthetic" languages that form a single word from many morphemes such as German and Finnish. These processed texts in these languages generally require additional special tokenization to extract morphemes from words, as word ngrams can lose much of the information in this structure. In addition, processing texts from different languages can benefit from different actions for the removal of stopwords, stem interpretations, and lemma recognition.

In some embodiments, the individual modeling techniques may have conditional logic for text processing that depends on the detected language. For example, modeling techniques may use word ngrams for European languages and letter ngrams for Chinese. Engine 110 may pass one or more detected languages to the modeling technique when shipped for execution.

In some embodiments, engine 110 excludes modeling techniques or modeling technique families from the evaluation based on language, or modifies their resource allocation (eg, processing priority) based on language. You may. In some embodiments, the modeling technique may have associated metadata that defines one or more languages for which the technique is suitable. In some embodiments, the modeling technique may have associated metadata that indicates the degree to which the technique is likely to work with respect to a set of languages. The performance estimates may be calculated via meta-machine learning based on previous information about the modeling technique or the extent to which different modeling techniques work in different languages.
Interaction strength

In a predictive model, two or more predictor variables taken together (eg, features) can affect the target variables in addition to or in place of these variables taken separately. In some embodiments, the system determines a metric value or set of metric that exhibits such an interaction across a wide variety of predictive problems. In some embodiments, engine 110 incorporates metrics into the process of automatically applying appropriate interaction modeling techniques. In some embodiments, the engine 110 allows the user to choose between alternative pre-packaged interaction modeling techniques or apply their own custom techniques.

For example, in the context of car insurance, there may be interactions between "age" and "gender" characteristics. At some ages, women are safer drivers than men. At other ages, men and women are equally safe. And at yet other ages, men are safer. As another embodiment, there may be an interaction between latitude and longitude in the context of geographic analysis. Using only latitude and longitude features without any interaction, the model can predict that Singapore will have low GDP per capita instead of its very high GDP per capita. Using interaction information, the model can correctly predict that Singapore will have high GDP per capita.

In general, the step of detecting an interaction involves implementing a method for detecting a statistically significant interaction between two or more features. In some embodiments, the engine 110 may detect the interaction depending on the evaluation of the dataset (eg, in step 408 of method 400). In some embodiments, individual modeling techniques may be constructed to be individually tested for interactions (eg, by modeling technique builder 220). The former approach may provide a general solution and the latter approach may provide additional flexibility to deal with specific situations.

In the general case, engine 110 determines a combination of (a) a subset of features that are candidates for an interaction, (b) the sequence of interactions to be tested, and (c) a specific combination of features to be tested. You may. For each decision, engine 100 may use either a discovery problem solution or the result of meta-machine learning for past prediction problems. For example, some studies show that testing for three-way or higher-order interactions leads to little improved prediction accuracy. Therefore, in some embodiments, the engine 110 will not be specifically overridden by the user, or a meta-machine learning algorithm that indicates a particular problem will improve accuracy by modeling higher-order interactions. Unless you have a reasonable probability that it is an exceptional case, you may test for only two-way interactions, or by default only for two-way interactions.

After determining the set of potential interactions, the engine 110 may select an interaction detection method. In some cases, the engine 110 may use the same interaction detection method for all possible interactions. In other cases, the engine 110 may select different interaction detection methods for different combinations of features based on the characteristics of the features. Some examples of the interaction method are ANOVA, Partial Dependence Functions (JH Friedman and BE Popescu, Prective learning learning via ruleLes evience, Liberation, Defense and interaction detection, Statistica Sinica 12 (2), 2002), Grove (D. Sorokina et al., Detecting statistical interactions with additive groves of trees, in ICML, 2008), and FAST (Yin Lou et al, Accurate intelligible models with Includes pairwise interactions, in KDD, 2013).

In some embodiments, the system 100 implements such techniques for detecting interactions using the same mechanism described above for performing other machine learning techniques. After the engine 110 obtains the interaction detection results, the engine builds new features based on these interactions that are significant (1) and adds them to the dataset and / or (2). If the modeling technique has the ability to handle interactions directly, a list of significant parameters as parameters may be passed to the modeling technique. Execution of the modeling technique can then be continued in the manner described above.
Predictive performance improvement

The user may potentially use the deployment engine (eg, using the model deployment engine 140) to adapt the model to the independent prediction service. In some cases, the step of using a particular model and calculating predictive targets from feature observations can require significant computational resources. In some embodiments, engine 110 limits resource consumption (eg, minimal), especially when the prediction service handles demands on many models and / or frequent demands on one or more models. It is configured to provide a timely response.

Examples are described herein. In online games, a game provider may support many different types of games, with many instances of each type of game, and many users playing in each instance. In order to increase (eg, optimize) user satisfaction and revenue from games, such providers may wish to predict user behavior based on the outcome of the game the user plays. Such providers may use such predictions to provide suggestions to players or adjust their future gaming experience. Thus, such providers may rely on dozens to hundreds of different models, each of which can be used, for example, to make thousands to millions of forecasts per day. Also, the relative prediction requirements across the model can fluctuate significantly over time due to demographic effects, changing popularity, differences in average game completion time, and the like.

To make behavior predictions, users may want an independent prediction service that is completely separate from the model-building computing infrastructure. Independent forecasting services may be launched within different computing environments or managed as distinctly different components within a shared computing environment. Once instantiated, service execution, security, and monitoring can be completely separated from the model building environment, allowing users to deploy and manage it independently.

After instantiating the service, the deployment engine may allow the user to install the conformance model on the service. Implementations of modeling techniques suitable for fitting a model to enhance (eg, optimize) performance can be suboptimal for making predictions. For example, the step of fitting a model requires that the same algorithm be invoked repeatedly, so it is often worth investing significant overhead in enabling fast parallel execution of the algorithm. However, if the expected rate of the forecast request is not very high, the same overhead may be worthless due to the independent forecast service. In some cases, the modeling technique developer may even provide a special version of one or more of its component execution tasks that provides better performance characteristics within the predictive environment. In particular, implementations designed for highly parallel execution or execution on specialized processors can be advantageous for predictive performance. Similarly, when a modeling technique involves a task specified in a programming language, the step of precompiling the task at the time of service instantiation, rather than waiting for the initial request for service startup or prediction from that model, is , Can provide performance improvements.

Also, the model fitting task generally uses a different computing infrastructure than the forecasting service. To protect the cloud infrastructure from errors during the execution of the modeling technique, and to prevent access to the modeling technique by other users in the cloud, the modeling technique provides secure computing during model fitting. It may be executed in the container. However, the prediction service may often be started on a dedicated machine or cluster. Removing the secure container layer can therefore reduce overhead costs without any practical disadvantages.

Therefore, based on the specific tasks performed by the model modeling technique, the expected load, and the characteristics of the target computing environment for prediction, the deployment engine sets a set of rules for packaging and deploying the model. You may use it. These rules may optimize execution.

Since a given prediction service can run multiple models, the service may allocate computational resources across prediction requests for each model. There are two basic cases: deployment to one or more server machines, and deployment to computing clusters.

In the case of server deployment, the challenge is how to allocate requests among multiple servers. Prediction services may have several types of a priori information. Such information includes (a) an estimate of the time required to perform a prediction for each configured model, (b) the expected frequency of requests for each configured model at different times, and (c) model execution. May include the desired priority of. Estimates of execution time may be calculated based on measuring the actual execution speed of the prediction code for each model under one or more conditions. The desired priority of model execution may be specified by the service manager. The expected frequency of requests can be calculated from the historical data of the model, expected based on a meta-machine learning model, or provided by the administrator.

The service may include an objective function that combines some or all of these factors to calculate some of the aggregate computing power of all available servers that can be initially allocated to each model. When a service receives and executes a request, it naturally gets updated information about execution times and estimates of the expected frequency of requests. Therefore, the service may recalculate these parts and reallocate the model to the server accordingly.

The deployed forecasting service may have two different types of server processes: routers and workers. One or more routers may form a routing service that accepts requests for predictions and assigns them to workers. An incoming request has a model identifier that indicates the predictive model to use, a user or client identifier that indicates the user or software system making the request, and a vector that is one or more predictor variables for that model. You may.

When a request enters a dedicated prediction service, the routing service may check a number of combinations of model identifiers, user or client identifiers, and vectors of predictor variables. The routing service then allocates the request to the worker and (1) runs the given model and / or (2) the server cache for instructions and data used by the given user or client. Hits may be increased (eg, maximized). The routing service may also consider the number of vectors of predictor variables and achieve a batch size mixture submitted to each worker that balances latency and throughput.

Examples of algorithms for allocating requirements for models across workers include round robin, weighted round robin based on model computational strength and / or worker's computational power, and dynamic allocation based on reported loads. Good. To facilitate the rapid routing of requests to a designated server, the routing service uses a hash function to select the same server given the same set of observed characteristics (eg, model identifiers). May be good. The hash function may be a simple hash function or a consistent hash function. A consistent hash function requires less overhead as the number of nodes (in this case, corresponding to the worker) changes. Therefore, if a worker goes down or a new worker is added, a consistent hash function can reduce the number of hash keys that must be recalculated.

In addition to improving performance (eg, optimizing) by intelligently distributing forecast requests among available services, predictive services intelligently know how each worker executes each model. The performance of individual models may be enhanced (eg, optimized) by configuring in. For example, if a given server receives a mixture of requests for several different models, the step of loading and unloading the model for each request can incur significant overhead. However, the step of aggregating requests for batch processing can incur significant latency. In some embodiments, the service can intelligently make this trade-off if the administrator specifies a latency tolerance for the model. For example, an emergency request may have a latency tolerance of only 100 milliseconds, in which case the server may process only one or at most several requests. In contrast, a 2 second latency tolerance can allow batch size in hundreds. Due to the overhead, the step of doubling the latency tolerance can increase the throughput by 10 to 100 times.

Similarly, steps that use operating system threads can improve throughput while increasing latency due to thread configuration and initialization overhead. In some cases, the predictions can be extremely latency sensitive. If all requests to a given model are likely to be latency sensitive, the service may configure the server to handle these requests to operate in single-threaded mode. Also, if only a subset of requests are likely to be latency sensitive, the service may allow the requester to flag a given request as sensitive. In this case, the server may operate only in single-threaded mode while fulfilling specific requirements.

In some cases, a group of users may have a batch of predictions that the group wants to use distributed computing clusters and calculate as quickly as possible. A decentralized computing framework (eg, Apache Spark) generally allows organizations to set up clusters to launch the framework, and any program designed to work with the flakework then data. And can submit jobs with executable instructions.

Predictions are stateless behavior in the context of cluster computing, because the execution of one prediction on a model does not affect the results of another prediction on that model, or the results of any other model. Therefore, in general, it is very easy to make them in parallel. Therefore, given the batch of data and executable instructions, the normal behavior of the framework's partitioning and allocation algorithms can result in linear scaling.

In some cases, the steps to make predictions may be part of a large workflow in which data is generated and consumed in many steps. In such cases, the predictive job may be integrated with other operations through a publish-subscribe mechanism such as Apache Kafka. Prediction services subscribe to channels that generate new observations that require predictions. After the service makes predictions, publish them to one or more channels that other programs can consume.
Performance improvement

Searching between fit modeling techniques and / or a number of alternative techniques can be computationally intensive. Computational resources can be expensive. Some embodiments of System 100 for generating predictive models identify opportunities to reduce resource consumption.

Based on user preference, the engine 110 may adjust its search so that the model reduces execution time and consumption of computational resources. In some cases, the prediction problem may contain a lot of training data. In such cases, the benefit of cross-validation is usually lower in terms of reducing model bias. Therefore, the calculation time for a single launch on a 5-10x amount of data is typically much greater than the 5-10 launches on a 1/5 to 1/10 amount of data. Users may prefer to fit the model on all training data at once, rather than on each cross-validation layer.

Even if the user does not have a relatively large training set, the user may still wish to save time and resources. In such cases, engine 110 may offer a "more greedy" option that uses some more aggressive search approaches. First, the engine 110 can try a smaller subset of possible modeling techniques (eg, only those with relatively high expected performance). Second, the engine 110 may more aggressively remove the low performance model in each training and evaluation. Third, the engine 110 may take larger steps when searching for optimal hyperparameters for each model.

In general, the steps to find better (eg, optimal) hyperparameters can be expensive. Thus, even if the user wants the engine 110 to evaluate a wide range of potential models and not actively remove them, the engine still limits (eg, optimizes) hyperparameter searches. Allows you to save resources. The cost of this search is generally proportional to the size of the dataset. One strategy is to adjust the hyperparameters on a small portion of the dataset and then extrapolate these parameters throughout the dataset. In some cases, adjustments are made to take into account large amounts of data. In some embodiments, the engine 110 can use one of two strategies. First, the engine 110 can make adjustments based on discoverable problem solving for its modeling technique. Second, the engine 110 engages in meta-machine learning, tracking how the hyperparameters of each modeling technique fluctuate with the dataset size, building a meta-predictive model of these hyperparameters, and then the user The metamodel can be applied where it is desired to make a trade-off.

When working with the category prediction problem, there can be minority and majority classes. The minority class can be much smaller but relatively more important, as in the case of fraud detection. In some embodiments, the engine 110 "downsamples" the majority class so that the number of training observations for the majority class is more similar to that for the minority class. In some cases, modeling techniques may automatically adapt to such weights directly during model fitting. If the modeling technique does not adapt to such weights, the engine 110 can make post-adaptation adjustments proportional to the amount of downsampling. This approach can sacrifice some accuracy due to much shorter execution times and lower resource consumption.

Some modeling techniques may be performed more efficiently than others. For example, some modeling techniques may be optimized to start on parallel computing clusters or servers with specialized processors. The metadata for each modeling technique may show any such performance advantage. When engine 110 is allocating computing jobs, it may detect jobs for modeling techniques whose benefits apply within the currently available computing environment. The engine 110 may then use a larger dataset for these jobs during each search. These modeling techniques may then be completed faster. Also, if their accuracy is high enough, it may not even be necessary to test other modeling techniques that are functioning relatively poorly.
User interface (UI) enhancement

The engine 110 can help the user generate a better predictive model by extracting more information from the user prior to model building and can provide the user with a better understanding of model performance after model fitting.

In some cases, the user may have additional information about the dataset, which is suitable for better directing the search for an accurate predictive model. For example, the user may want to know that an observation has a special meaning and show that meaning. The engine 110 may allow the user to easily create new variables for this purpose. For example, one composite variable is that the engine should use a particular observation as part of a training, validation, or holdout data partition instead of randomly assigning that particular observation to such a partition. May be indicated. This ability can be useful in situations where certain values occur infrequently and the corresponding observations should be carefully assigned to different partitions. This ability can be useful in situations where the user is training a model using different machine learning systems and wants to perform a comparison where the training, validation, and holdout partitions are identical. ..

Similarly, some observations may represent particularly important events in which the user desires to assign additional weights. Therefore, the additional variables inserted into the dataset may indicate the relative weight of each observation. The engine 110 may then use this weighting when training the models and calculating their accuracy, the goal being to generate more accurate predictions under higher weighted conditions.

In other cases, the user may have a priori information about how a feature should behave in the model. For example, the user may know that a feature should have a monotonous effect on a predictive target over a range. In car insurance, the likelihood of an accident is generally considered to increase monotonically with age after the age of 30. Another embodiment is to create a bandwidth for another continuous variable. Personal income is continuous, but there are analytical conventions for assigning values to the band from $ 10K increment to $ 100K, then from the $ 25K band to $ 250K, and to any income above $ 250K. There are then cases where restrictions on the dataset require constraints on specific features. At one point, a categorical variable can have a large number of values relative to the size of the dataset. The user should either ignore the category features where the engine 110 has categories that are likely to exceed a certain number, or limit the number of categories to the most frequent X and all other values "other". It may be desirable to indicate which one should be assigned to the category. In all these situations, the user interface may offer the user the option of defining this information for each feature detected (eg, in step 412 of method 400).

The user interface may provide guidance assistance in transforming features. For example, a user may wish to convert a continuous variable to a categorical variable, but there may be no standard convention for that variable. By analyzing the shape of the distribution, the engine 110 may choose to place an optimal number of categorical bands and "nodes" demarcating the boundaries between each band in the distribution. Optionally, the user may override these defaults within the user interface by adding or removing nodes and moving the location of the nodes.

Similarly, with respect to features that are already categorized, engine 110 may simplify their representation by combining one or more categories into a single category. Based on the relative frequency of each observed category and the frequency with which they appear relative to the values of other features, the engine 110 may calculate the optimal way to combine the categories. Optionally, the user may override these calculations by removing the original category from the combined category and / or putting an existing category inside the combined category.

In some cases, the prediction problem may include events that occur at irregular intervals. In such cases, it may be useful to automatically create new features that capture the number of these events that have occurred within a particular time frame. For example, in an insurance forecasting problem, the dataset may have a record of each time the policyholder makes a claim. However, it may be more useful to consider the number of claims made by policyholders in the last X years when building a model for predicting future risks. Such a situation when the engine evaluates a dataset by detecting the data structure relationship between the record corresponding to the entity and the other records corresponding to the event (eg, step 408 of method 400). May be detected. When presenting a dataset to the user (eg, in step 410), the user interface may propose to automatically create or create such features. It is also calculated to maximize the statistical dependency between this variable and the occurrence of future events, based on how often the event occurs, or using some other discovery problem solving. You may also propose a time frame threshold. The user interface may also allow the user to override the creation of such features, force the creation of such features, and override the proposed time frame thresholds.

When the system makes model-based predictions, the user may wish to scrutinize these predictions and search for anomalies. For example, the user interface provides a list of all or a subset of predictions for the model, one that was extreme in terms of either the magnitude of the predictor's value or its low probability of having that value. May be shown. It is also possible to provide insight into the reasons for extrema. For example, in a car insurance risk model, a particular high value may have the reason "age <25 and marriage history = single".

Further Description of Some Embodiments The examples provided herein have described operations as being resident on a separate computer, or as being performed by a separate computer. It should be understood that the functionality of the components can be implemented on a single computer or on any number of computers in a distributed fashion.

The above embodiment may be implemented by any of a number of methods. For example, embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, software code is provided on any suitable processor or set of processors, whether provided within a single computer or distributed among multiple computers. Can be run on. Further, it should be understood that a computer can be embodied in any of several forms such as a rack mount computer, a desktop computer, a laptop computer, or a tablet computer. In addition, a computer may be embodied in a device, including a personal digital assistant (PDA), a smartphone, or any suitable portable or fixed electronic device, which is generally not considered a computer but has suitable processing power. Good.

Such computers may be interconnected by one or more networks of any preferred form, including as a local area network or wide area network, such as as a corporate network or the Internet. Such networks may be based on any suitable technique, may operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.

Also, the various methods and processes outlined herein are coded as software that can be run on one or more processors that employ any one of the various operating systems or platforms. It may be converted. In addition, such software may be written using any of several preferred programming languages and / or programming or scripting tools and may also be run on a framework or virtual machine. It may be compiled as executable machine language code or intermediate code.

In this aspect, some embodiments, when run on one or more computers or other processors, make a method of implementing the various embodiments discussed above, one or the other. Computer-readable media (or multiple computer-readable media) (eg, computer memory, one or more floppy® discs, compact discs, optical discs, magnetic tapes, flash memory) encoded by more than one program. , A circuit configuration within a field programmable gate array or other semiconductor device, or other tangible computer storage medium). One or more computer-readable media can be transient. One or more computer-readable media differ by one or more such that one or more programs stored on it implement various aspects of predictive modeling as discussed above. It can be portable so that it can be loaded on a computer or other processor. The term "program" or "software" is used in any type of computer code or computer executable instructions that may be employed to program a computer or other processor to implement the various aspects described in this disclosure. As used herein in a general sense to refer to a set. In addition, according to one aspect of the disclosure, one or more computer programs that perform a predictive modeling method when executed need not reside on a single computer or processor, but predictive modeling. It should be understood that it can be distributed in a modular fashion among several different computers or processors to implement various aspects of.

Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. In general, a program module includes routines, programs, objects, components, data structures, etc. that perform a particular task or implement a particular abstract data type. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

The data structure may also be stored on a computer-readable medium in any suitable form. For simplicity of illustration, a data structure may be shown to have fields that are associated through locations within the data structure. Such relationships may also be achieved by allocating storage for fields with locations in computer-readable media that convey the relationships between fields. However, any suitable mechanism, including through the use of pointers, tags, or other mechanisms that establish relationships between data elements, is used to establish relationships between information within fields of a data structure. May be done.

The predictive modeling technique may also be embodied as a method for which examples are provided. The actions performed as part of the method may be ordered in any suitable method. Thus, although shown as a continuous action in an exemplary embodiment, embodiments may be constructed in which the actions are performed in a different order than shown, which may include performing several actions simultaneously.

In some embodiments, the method may be implemented as computer instructions stored in a portion of the computer's random access memory to provide control logic that affects the processes described above. In such an embodiment, the program is one of several high-level languages such as FORTRAN, PASCAL, C, C ++, C #, Java®, Javascript®, Tcl, or BASIC. It may be written in one. In addition, the program may be written with scripts, macros, or functionality built into commercial software such as EXCEL or VISUAL BASIC. In addition, the software may be implemented in assembly language for microprocessors residing on the computer. For example, the software may be implemented in Intel 80x86 assembly language if configured to run on an IBM PC or PC clone. Software includes, but is not limited to, "computer-readable programming means" such as floppy (registered trademark) disks, hard disks, optical disks, magnetic tapes, PROMs, EPROMs, or CD-ROMs. May be good.

The various aspects of the present disclosure may be used alone, in combination, or in various sequences not specifically described above, and thus the present invention describes its application as described above. Or not limited to the details and arrangement of components illustrated in the drawings. For example, the aspects described in one embodiment may be combined with the aspects described in another embodiment in any manner.

Terminology The terminology and terminology used herein is for explanatory purposes only and should not be considered limiting.

It should be understood that the indefinite articles "a" and "an" as used in the specification and claims mean "at least one" unless explicitly stated to the contrary. The phrase "and / or" as used in the description and claims is "one or both" of the elements so combined, i.e., in some cases conjunctive, and in the other. In some cases it should be understood to mean elements that exist distantly. Multiple elements described by "and / or" should be construed in the same manner, i.e., "one or more" of such combined elements. In addition to the elements specifically identified by the "and / or" clause, other elements may optionally be present, whether related to or unrelated to the specifically identified elements. Therefore, as a non-limiting example, the reference to "A and / or B", when used in conjunction with an unconstrained term such as "with", is only A in one embodiment (optionally B). (Including elements other than), in another embodiment only B (optionally including elements other than A), and in yet another embodiment both A and B (optionally including other elements), etc. it can.

As used in the specification and claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" is inclusive, i.e. at least one inclusive, but also in some element or list of elements. It shall be construed to include more than one and optionally additional unlisted items. In contrast, clearly indicated "only" terms such as "only one of" or "exactly one of", or when used in a claim, "consisting of". "Will refer to the inclusion of exactly one element of several elements or a list of elements. Generally, the term "or" as used is "any one", "one of", "only one of", or "exactly one of", etc. When the term of exclusivity of is preceded, it shall be construed only as indicating an exclusive alternative (ie, "one or the other, not both"). By "essentially consisting of", when used in the claims, shall have its usual meaning as used in the field of patent law.

As used in the specification and claims, the phrase "at least one" refers to a list of one or more elements and any one or more of the elements in the list of elements. It means at least one element selected from more than that, but does not necessarily include at least one of all the elements specifically described in the list of elements, any of the elements in the list of elements. It should be understood that the combination of is not excluded. This definition is also optional, in addition to the specifically identified elements in the list of elements, which the phrase "at least one" refers to, whether related to or unrelated to the specifically identified elements. Allow to be present in. Thus, as a non-limiting example, "at least one of A and B" (or equally "at least one of A or B", or equally "at least one of A and / or B". In one embodiment, none of B is present, optionally more than one, at least one A (and optionally including elements other than B), in another embodiment any A also does not exist, optionally contains more than one, at least one B (and optionally contains elements other than A), and in yet another embodiment, at least one A optionally contains more than one. , And at least one B (and optionally other elements), etc., including more than one at will.

The use of "contains", "contains", "has", "contains", "with", and variants thereof is described in subsequent items and additional items. Is intended to include.

The use of ordering terms such as "first,""second,""third," etc. in a claim to modify a claim element alone is one claim compared to another element. It does not imply any priority, precedence, or order of the elements, or the time order in which the act of the method takes place. The term of order is used as a marker to distinguish one claim element having a certain name from another element having the same name (due to the use of the term of order) to distinguish the claim elements. Only used.
Equal

Although some aspects of at least one embodiment of the invention have been described in this way, it should be appreciated that various changes, modifications and improvements will be readily recalled to those skilled in the art. Such changes, modifications, and improvements are intended to be part of this disclosure and are intended to be within the spirit and scope of the invention. Therefore, the above description and drawings are merely examples.

Claims (1)

The invention described herein.