KR102448694B1 - Systems and related methods and devices for predictive data analysis - Google Patents

Abstract

Predictive data analysis techniques are improved techniques for generating predictive models for time-series prediction problems, improved techniques for determining predicted values of input variables (“features”), and improved techniques for generating secondary models of primary predictive models. technology may be included.

Description

Systems and related methods and devices for predictive data analysis

<Cross-Reference to Related Applications>

This application is entitled "Systems and Techniques for Determining the Predictive Value of a Feature" and was filed on October 21, 2016 and has Attorney's Docket No. DRB-001C1CP. U.S. Patent Application Serial No. 15/331,797 and U.S. Provisional Patent Application No. 62/411,526 entitled "Systems and Techniques for Predictive Data Analysis" filed on October 21, 2016 and having Attorney Docket No. DRB-002PR to the maximum extent permitted by applicable law, each of which is incorporated herein by reference.

<Field of Invention>

FIELD OF THE INVENTION The present invention relates generally to systems and techniques for data analysis. Some embodiments specifically relate to systems and techniques for using statistical learning methods to develop, select, and/or understand predictive models for predictive problems.

Many organizations and individuals use electronic data to improve their operations or aid their decision-making. For example, many business enterprises use data management technologies to improve the efficiency of various business processes, such as executing transactions, tracking inputs and outputs, or marketing products. As another example, many businesses use operational data to evaluate the performance of business processes, to measure the effectiveness of efforts to improve processes, or to determine how processes should be adjusted.

In some cases, electronic data may be used to predict problems or opportunities. To build predictive models, some organizations combine operational data that describes what happened in the past with evaluation data that describes the successor values of performance statistics. Based on the results predicted by the predictive model, the organization can make decisions, adjust processes, or take other actions. For example, an insurance company may attempt to build a predictive model that more accurately predicts future claims, or when a policyholder considers switching to a competing insurer. Automakers may try to build predictive models that more accurately predict demand for new car models. Fire departments may try to build predictive models to predict high fire risk days or to predict which structures will be at risk from a fire.

A machine-learning technique (eg, a supervised statistical-learning technique) may be used to generate a predictive model from a data set comprising previously recorded observations of at least two variables. The variable(s) to be predicted may be referred to as “target(s)”, “response(s)” or “dependent variable(s)”. The remaining variable(s) that can be used to make predictions may be referred to as “feature(s)”, “predictor(s)” or “independent variable(s)”. Observations are generally partitioned into at least one "training" data set and at least one "test" data set. The data analyst then selects a statistic-learning procedure and runs the procedure on the training data set to create a predictive model. The analyst then tests the generated model against the testing data set to determine how well the model predicts the value(s) of the target(s) relative to the actual observation of the target(s).

Motivation for some embodiments

Data analysts can use analytics techniques and computational infrastructure to build predictive models from electronic data, including computational and evaluation data. Data analysts typically use one of two approaches to building predictive models. In the first approach, organizations dealing with predictive problems simply use packaged predictive modeling solutions already developed for the same or similar predictive problems. This "cookie cutter" approach is inexpensive, but generally only works for a handful of predictive problems common to a relatively large number of organizations (eg, fraud detection, bad management, marketing response, etc.). Using the second approach, data analytics teams build customized predictive modeling solutions to predictive problems. This "artisanal" approach tends to be used for a small number of high-value forecasting problems, as it is generally expensive and time consuming.

The space of potential predictive modeling solutions to predictive problems is usually large and complex. Statistical learning techniques (e.g., mathematics, statistics, physics, engineering, economics, sociology, biology, medicine, artificial intelligence, data mining, etc.) and many academic traditions (e.g., finance, insurance, retail, manufacturing, Many commercial sectors (such as health care) are affected by this application. As a result, there are many other predictive modeling algorithms that may have various transformations and/or tuning parameters, as well as different pre-processing and post-processing steps with their own transformations and/or parameters. The amount of potential predictive modeling solutions (eg, a combination of pre-processing steps, modeling algorithms and post-processing steps) is already quite large and is growing rapidly as researchers develop new techniques.

Given this vast space of predictive modeling techniques, craftsmanship approaches to creating predictive models tend to be time consuming and not explore large portions of the modeling search space. Analysts tend to explore the modeling space in an ad hoc fashion, based on intuition or previous experience and extensive trial and error testing. They may not pursue a potentially useful path of search or may not properly tailor the search in response to the results of the initial effort. Also, the scope of trial and error tends to be limited by the analyst's time constraints, so craftsmanship approaches typically only explore a fraction of the modeling search space.

The craftsmanship approach can also be very expensive. Developing predictive models through a craftsmanship approach often entails significant investments in computing resources and well-paid data analysts. Given these significant costs, organizations are expected to have a low cost but only a small fraction of the vast predictive modeling space (e.g., the portion of the modeling space a priori expected to contain acceptable solutions to some particular predictive problem). It avoids the craftsmanship approach in favor of the cookie-cutter approach, which tends to explore the bay. The cookie cutter approach can produce predictive models that perform poorly compared to the unexplored option.

A need exists for tools to systematically and cost-effectively evaluate the space of potential predictive modeling techniques for predictive problems. In many ways, the traditional approach to creating predictive models is similar to the exploration of valuable resources (eg, oil, gold, minerals, gems, etc.). Exploration can lead to valuable discoveries, but it is not much more efficient than well-planned exploratory excavation or geological exploration in combination with drilling based on a vast library of previous results. The inventors recognize and understand that statistical learning techniques can be used to systematically and cost-effectively evaluate the space of potential predictive modeling solutions to predictive problems.

Time Series Predictive Modeling

Many prediction problems pose the problem of predicting the value of one or more output variables (“targets”) at one or more future times based on the values of one or more input variables (“features”) at one or more past times. Such a prediction problem may be referred to as a "time-series prediction problem", and a prediction model that models such a problem may be referred to as a "time-series prediction model" or a "time-series model".

Techniques are needed to rigorously and efficiently explore the modeling search space for time series models. We describe certain aspects of the time series modeling procedure, e.g., the amount of training data used to train the model, the time interval between observations of the input variable, the length of time period covered by the training data, and the amount of time the training data covers. The recency of the period, the period between the time associated with the feature value provided to the model and the associated time with the target value predicted by the model ("skip range"), and the period over which the model predicts the value of the target ("expected range") .

In general, innovative aspects of the subject matter described herein may be implemented in a predictive modeling method comprising performing a predictive modeling procedure, the predictive modeling procedure comprising: (a) generating time series data comprising one or more data sets; obtaining, each data set comprising a plurality of observations, each observation comprising (1) an indication of a time associated with the observation and (2) respective values of one or more variables; (b) determining a time interval of the time series data; (c) identifying the one or more variables as targets and zero or more other variables as features; (d) determining an expected range and a skip range associated with a prediction problem represented by the time series data, wherein the expected range represents a duration of time for which values of the targets are to be predicted, and wherein the skip range is indicating a time lag between a time associated with the earliest prediction in the expected range and a time associated with the most recent observation on which the prediction in the expected range is based; (e) generating training data from the time series data, the training data comprising a first subset of observations in at least one of the data sets, the first subset of observations comprising the observations training-input and training-output collections of a range separating an end of the training-input time range from a beginning of the training-output time range, wherein a duration of the training-output time range is at least as long as the expected duration; (f) generating testing data from the time series data, the testing data comprising a second subset of observations in at least one of the data sets, the second subset of observations comprising the observations and test-input and test-validation collections of , wherein the skip range separates the end of the test-input time range from the start of the test-validation time range, and the duration of the test-validation time range is at least as long as the expected range; (g) fitting a predictive model to the training data; and (h) testing the customized model against the testing data.

Other embodiments of this aspect include corresponding computer systems, apparatus and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method. A system of one or more computers may be configured to perform a particular action by having software, firmware, hardware, or a combination thereof installed on the system in operation, causing the system to perform the action. One or more computer programs may be configured to perform a particular action by including instructions that, when executed by the data processing apparatus, cause the apparatus to perform the action.

The foregoing and other embodiments may optionally include one or more of the following features, each alone or in combination. In some embodiments, the time interval of the time series data is determined based at least in part on times associated with at least a subset of observations included in at least one of the data sets. In some embodiments, determining a time interval of time series data comprises: for each of the data sets, determining a respective time interval of the data set; determining that the time intervals of data sets are non-uniform; and setting a time interval of time series data to a time interval of the data sets. In some embodiments, determining the time interval of the data set comprises: for one or more pairs of successive observations included in the data set, determining each time period between the successive observations; determining that the period of time between the pair of consecutive observations is uniform; and setting a time interval of the data set to a time interval between the pair of consecutive observations.

In some embodiments, determining the time interval of the time series data comprises: for each of the data sets, determining each time interval of the data set; and determining that the time intervals of at least two of the data sets are different, wherein the time intervals of the time series data include (1) respective portions of the observations included in each of the data sets, and/ or (2) based at least in part on each of the respective time intervals of the data sets. In some embodiments, determining each time interval of the data set comprises: determining each time period between each pair of consecutive observations included in the data set; If the period between the pair of consecutive observations represents a plurality of non-uniform durations, then the time interval of the data set is (1) respective portions of the pair of consecutive observations representing each of the non-uniform durations, and/ or (2) determining based at least in part on durations of the period; and if the period of time between the pair of consecutive observations is a non-uniform duration, then the time interval of the data set is the respective duration of the time period. In some embodiments, the time interval of the time series data is the shortest time interval that is an integer multiple of each of the time intervals of each of the data sets.

In some embodiments, the actions of the present invention include: for each data set, if the time interval of the data set is shorter than the time interval of the time series data, by down-sampling the observations of the data set, The method further includes converting a time interval into a time interval of the time series data. In some embodiments, down-sampling the observations of the data set comprises: for each instance of the time interval of the time series data in a time period corresponding to the data set: each of the time intervals of the time series data identifying all observations in the data set associated with times corresponding to an instance; aggregating the identified observations to produce an aggregate observation; and replacing the identified observations in the data set with the aggregate observation. In some embodiments, the number of identified observations corresponding to the instance of the time interval of the time series data is equal to a ratio between the time interval of the time series data and the time interval of the data set. In some embodiments, aggregating the identified observations comprises: (1) a corresponding variable value initially included in the identified observations; (2) the value of each variable in the aggregated observation; the most recently included corresponding variable value of the observations, (3) the maximum value of the corresponding variable value included in the identified observations, (4) the minimum value of the corresponding variable value included in the identified observations, ( 5) setting to the average of the corresponding variable values included in the identified observations, or (6) the value of a function of the corresponding variable values included in the identified observations.

In some embodiments, the time interval of the time series data is selected from the group consisting of time intervals of the data sets. In some embodiments, the data sets include a first data set representing a first time interval and a second data set representing a second time interval greater than the first time interval, wherein the second time interval comprises the time series data , the method further comprising: converting a time interval of the first data set to a time interval of the time series data by down-sampling the observations of the first data set. In some embodiments, the time interval of the time series data is different from each of the time intervals of the data set.

In some embodiments, the at least group of observations in the time series data comprises respective values of a first variable, the method comprising: fitting the predictive model to the training data; and applying the customized model to the testing data. before testing: determining that the values of the first variable include time values; generating, for each observation in the group, a respective value of a second variable, the value of the second variable comprising an offset between the time value of the first variable and a reference time value; and adding the values of the second variable to each observation in the group. In some embodiments, the action of the method further comprises removing said values of said first variable from observations within said group. In some embodiments, the reference time includes a date of the event. In some embodiments, the event comprises a birth, marriage, school graduation, commencement of employment with an employer, or commencement of work in a particular position.

In some embodiments, the variables include a first variable and a second variable, and the method comprises: changes in the value of the first variable and correlated changes in the value of the second variable, wherein the determining that changes in values of the first variable and the second variable are correlated; and displaying, via a graphical user interface, graphical content representing a duration of the time lag between changes in the value of the first variable and the correlated changes in the value of the second variable.

In some embodiments, the expected range includes (1) a time interval of the time series data, (2) the number of observations included in the time series data, (3) a period corresponding to the time series data, and/or (4) micro is determined based at least in part on a natural period selected from the group consisting of seconds, milliseconds, seconds, minutes, hours, days, weeks, months, quarters, seasons, years, decades, hundred years and millennia. In some embodiments, the expected range is an integer multiple of a time interval of the time series data. In some embodiments, a period of time between times associated with successive predictions of the expected range is equal to a time interval of the time series data.

In some embodiments, the skip range includes a latency in the collection of the time series data, a latency in communication of the time series data, a latency in the analysis of the time series data, and a communication of the analysis of the time series data. is determined based, at least in part, on a wait time at , and/or a wait time at implementation of actions based on the analysis of the time series data.

In some embodiments, an action of the method comprises determining a duration of the training-input time range, the total number of observations included in the time series data, an amount of variation in at least one value of the variables over time, determining based, at least in part, on an amount of periodic variation in at least one values of the variables, a consistency of variation in at least one values of the variables over a plurality of time periods, and/or a duration of the expected range. further comprising the step of In some embodiments, fitting the predictive model to the training data comprises fitting the predictive model to a subset of the training data corresponding to a portion of the training-input time range, wherein the training- The portion of the input time range starts at a time subsequent to the start time of the training-input time range and ends at the end time of the training-input time range. In some embodiments, the duration of said portion of said training-input time range is an integer multiple of said duration of said expected range.

In some embodiments, the actions of the method further comprise down-sampling the training data prior to fitting the predictive model to the training data. In some embodiments, down-sampling the training data comprises: removing from the training data all observations obtained from at least one of the data sets. In some embodiments, the down-sampling of the training data comprises: setting the down-sampled time interval of the training data to an integer multiple of the time interval of the time series data; and for each instance of the down-sampled time interval of the training data: identifying all observations of the training data associated with times corresponding to each instance of the down-sampled time interval of the training data; aggregating the identified observations to aggregate an aggregate observation, and replacing the identified observations in the training data with the aggregate observation. In some embodiments, the actions of the method further comprise down-sampling the testing data prior to testing the customized model against the testing data.

In some embodiments, the action of the method further comprises performing cross-validation of the predictive model. In some embodiments, the training data is first training data, the testing data is first testing data, the customized model is a first customized model, and performing mutual validation of the predictive model comprises: (i) generating second training data and second testing data from the time series data, wherein the second training data comprises a third subset of at least one observation of the data sets, the second testing data comprises a fourth subset of the at least one observations in the data sets; (j) fitting the predictive model to the second training data to obtain a second customized model; and (k) testing the second customized model against the second testing data.

In some embodiments, the first subset of observations corresponds to a sliding training window covering a first range of training times, and each observation included in the first subset is a time within the first range of training times. wherein the third subset of observations corresponds to a sliding training window covering a second range of training times, and each observation included in the third subset corresponds to a time within the second range of training times. associated, wherein the earliest time of the first range of training times is earlier than the earliest time of the second range of training times. In some embodiments, the second subset of observations corresponds to a sliding test window covering a first range of testing times, and each observation included in the second subset is a time within the first range of testing times. wherein the fourth subset of observations corresponds to a sliding test window covering a second range of testing times, and each observation included in the fourth subset corresponds to a time within the second range of testing times. associated, the earliest time of the first range of testing times is earlier than the earliest time of the second range of testing times. In some embodiments, the first testing time range partially overlaps the second training time range. In some embodiments, the second testing time range does not overlap any portion of the first training time range and does not overlap any portion of the second training time range.

In some embodiments, the action of the method further comprises partitioning the time series data into a plurality of partitions including at least a first partition and a second partition. In some embodiments, partitioning the time series data into a plurality of partitions comprises assigning each of the data sets to a corresponding partition. In some embodiments, partitioning the time series data into a plurality of partitions comprises temporally partitioning the time series data, each of the partitions corresponding to a respective portion of a time period associated with the time series data; Each observation included in the time series data is assigned a partition corresponding to a portion of the time period that matches the time associated with the observation.

In some embodiments, the first training data comprises a subset of observations included in the first partition of the time series data; the first testing data includes respective subsets of the observations included in all partitions of the time series data except the first partition; the second training data includes a subset of the observations included in the second partition of the time series data; and the second testing data includes respective subsets of the observations included in all partitions of the time series data except the second partition. In some embodiments, the first partition of the time series data comprises the first training data and the second training data and the first testing data and the second testing data, and the second partition of the time series data is a holdout holdout data, and wherein the action of the method further comprises testing the first customized model and the second customized model against the holdout data. In some embodiments, no predictive model is customized to the holdout data.

In some embodiments, the actions of the method further comprise performing nested mutual validation of the predictive model. In some embodiments, performing nested mutual validation of the predictive model comprises: the time series data into a first plurality of partitions comprising at least a first partition of the time series data and a second partition of the time series data. partitioning; and a plurality of divisions of the first division of the time series data including at least a first division of the first division of the time series data and a second division of the first division of the time series data. dividing one partition, wherein the training data comprises the first partition of the first partition of the time series data, and the testing data is at least other than the first partition of the first partition of the time series data. a plurality of partitions of the first partition of the time series data of

In some embodiments, the training data is first training data, the testing data is first testing data, and the customized model is a first customized model, performing nested mutual validation of the predictive model. The steps include: (i) generating, from the first partition of the time series data, second training data and second testing data, wherein the second training data is the second partition of the first partition of the time series data. wherein the second testing data comprises at least a plurality of partitions of the first partition of the data set other than the second partition of the first partition of the time series data; (j) fitting the predictive model to the second training data to obtain a second customized model; and (k) testing the second customized model against the second testing data.

In some embodiments, performing the nested mutual validation comprises: testing the first customized model and the second customized model for the second partition of the time series data; and comparing the first customized model and the second customized model based on a result of testing the first customized model and the second customized model on the second partition of the time series data. include more

In some embodiments, the action of the method further comprises determining, for the customized model, model-specific predictive values of the one or more features of the time series data. In some embodiments, the action of the method is to remove a feature from the time series data, based at least in part on the model-specific predicted values of the features, and a feature derived from two or more features in the time series data. generating and adding the derived features to the time series data, mixing the predictive model with other predictive models, and/or allocating resources during the process of evaluating the adequacy of predictive modeling procedures for the predictive problem. The method further includes performing at least one action selected from the group.

In some embodiments, the action of the method comprises: determining, based at least in part on characteristics of the prediction problem and/or properties of respective predictive modeling procedures, the suitability of a plurality of predictive modeling procedures for the prediction problem. step; selecting one or more predictive modeling procedures from the plurality of predictive modeling procedures based on the determined suitability of the selected modeling procedures for the predictive problem; and performing the one or more predictive modeling procedures.

In some embodiments, performing the one or more predictive modeling procedures comprises: sending an instruction to a plurality of processing nodes, wherein the instruction is a resource that allocates resources of the processing nodes for execution of the selected modeling procedures. the method comprising: an allocation schedule, wherein the resource allocation schedule is based at least in part on adequacy of the selected modeling procedures for the prediction problem; receiving results of execution of the selected modeling procedures by the plurality of processing nodes according to the resource allocation schedule, wherein the results are predictive models generated by the selected modeling procedures, and/or the prediction problem comprising scores of the generated models for time series data associated with ; and selecting, from the generated models, a predictive model for the predictive problem based at least in part on a score of the selected predictive model.

In some embodiments, the actions of the method further comprise mixing the customized model with other customized models to produce a blended predictive model.

In some embodiments, the action of the method further comprises deploying the customized model. In some embodiments, the time series data is first time series data, and placing the customized model comprises applying the customized model to second time series data representing one or more instances of the prediction problem to generate one or more predictions. and generating, wherein the first time series data does not include the second time series data. In some embodiments, the time series data is first time series data, and deploying the customized model comprises refreshing the customized model based at least in part on second time series data. In some embodiments, the customized model is a first customized model, and refreshing the customized model based at least in part on the second time series data comprises: the predictive modeling to generate a second customized model performing a procedure on the second time series data; and mixing the first customized model and the second customized model to generate a refreshed predictive model. In some embodiments, refreshing the customized model based at least in part on the second time series data comprises: at least a portion of the first time series data and at least a portion of the second time series data to generate a refreshed predictive model. and performing the predictive modeling procedure on third time series data including a portion.

In some embodiments, the customized model is deployed on one or more servers, other customized models are also deployed on the one or more servers, and a prediction request for the customized model and the other customized model includes: based at least in part on 1) an estimate of the amount of time used by each of the customized models to generate a prediction, and/or (2) an estimate of the frequency at which prediction requests for the customized models are received. assigned among the servers. In some embodiments, each prediction request is assigned to a respective thread, each prediction request has an associated latency-sensitivity value, and the number of threads executing on a particular server is the number of threads executing on the particular server. determined based at least in part on latency-sensitivity values of the threads.

In some embodiments, the actions of the method include: determining a value of a metric representing an interaction strength of two or more features included in the time series data; and if the value of the metric exceeds a threshold, generating time-series values of a new feature based on the values of the two or more features, and adding the new feature to the time-series data.

In some embodiments, the action of the method further comprises determining a temporal resolution of the time series data. In some embodiments, the target is identified based on user input.

Another embodiment of this aspect includes a predictive modeling apparatus, the apparatus comprising: a memory configured to store a machine-executable module encoding a predictive modeling procedure, the predictive modeling procedure performing at least one pre-processing task and a plurality of tasks including at least one model-fitting task; and at least one processor configured to execute the machine-executable module, wherein the execution of the machine-executable module causes the apparatus to perform the predictive modeling procedure. The performance of the pre-process operation comprises: (a) obtaining time series data comprising one or more data sets, each data set comprising a plurality of observations, each observation comprising (1) a time series associated with the observation. an indication and (2) respective values of one or more variables, (b) determining a time interval of the time series data, (c) identifying the one or more variables as targets, and zero or more other variables identifying them as features; and (d) determining an expected range and a skip range associated with a prediction problem represented by the time series data, wherein the expected range indicates a duration of time for which values of the targets are to be predicted, and wherein the skip range is the It represents the time lag between the time associated with the earliest prediction in the expected range and the time associated with the most recent observation on which the predictions in the expected range are based. Performing the predictive modeling procedure comprises performing a model-fitting task, wherein performing the model-fitting task: (e) generating training data from the time series data, the training data comprising at least one of the data sets a first subset of observations in a data set of The times associated with observations correspond to a training-input time range and a training-output time range, respectively, wherein the skip range separates the end of the training-input time range from the beginning of the training-output time range, and wherein the duration of an output time range is at least as long as the expected duration; two subsets, wherein the second subset of observations includes test-input and test-validation collections of the observations, the time associated with the observations in the test-input and test-validation collections are respectively corresponding to a test-input time range and a test-validation time range, the skip range separating the end of the test-input time range from the beginning of the test-validation time range, and the test-validation time range wherein the duration of the range is at least as long as the expected range; (g) fitting a predictive model to the training data; and (h) testing the customized model against the testing data. A machine-executable module may include a directed graph representing dependencies between tasks.

Determining the predicted value of a feature

Even when accurate predictive models for predictive problems are available, it can be difficult to understand (1) the predictive problem itself and (2) how a particular predictive model produces accurate predictive results. A metric of “feature importance” may facilitate this understanding. In general, a feature importance metric represents a predicted value for a feature in a data set in order to predict the outcome of the prediction problem that the data set represents. Prior art techniques for measuring feature importance are generally only applicable to certain types of predictive models, and are generally not suitable for use with other types of predictive models. Therefore, there is a need for a tool that can measure feature importance for any predictive model or set of various predictive models. In addition, there is a need for tools that can use feature importance metrics to guide the evaluation of predictive modeling procedures, engineering work functions, and resource allocation for predictive model mixes, thereby cost-effectively reducing the space of potential predictive modeling techniques for predicting problems. facilitate evaluation.

In general, another innovative aspect of the subject matter described herein may be embodied in a method comprising: (a) performing a plurality of predictive modeling procedures, each of which is associated with a predictive model wherein performing each modeling procedure comprises fitting the associated predictive model to an initial data set representing an initial predictive problem; (b) determining a first respective accuracy score of each of the customized predictive models, wherein the first accuracy score of each customized model determines that the customized model determines one or more results of the initial prediction problem. indicating accuracy of prediction; (c) shuffling the value of a feature over each of the observations included in the initial data set, thereby generating a modified data set representing a modified prediction problem; (d) determining a respective second respective accuracy score of each of the customized predictive models, wherein the second accuracy score of each customized model determines that the customized model determines one or more results of the modified prediction problem. indicating accuracy of prediction; and (e) determining a respective model-specific predictive value of the feature for each of the customized models, wherein the model-specific predictive value of the feature for each customized model is determined by the customized model. based on the first accuracy score and the second accuracy score of

Another embodiment of this aspect includes a corresponding computer system, apparatus, and computer program recorded on one or more computer storage devices, each configured to perform the action of the method. A system of one or more computers may be configured to perform a particular operation by installing software, firmware, hardware, or a combination thereof on the system in operation, causing the system to perform the operation. One or more computer programs may be configured to perform particular operations by including instructions that, when executed by the data processing apparatus, cause the apparatus to perform the actions.

The foregoing and other embodiments may each optionally include one or more of the following features, alone or in combination. In some embodiments, the action of the method comprises: prior to performing the plurality of predictive modeling procedures, the prediction based on characteristics of the initial data set, characteristics of the initial prediction problem, and/or characteristics of the feature. The method further comprises selecting the plurality of predictive modeling procedures for a problem. In some embodiments, the plurality of predictive modeling procedures comprises two or more modeling procedures selected from the group consisting of a random forest modeling procedure, a generalized additive modeling procedure, and a support vector machine modeling procedure. In some embodiments, the plurality of predictive modeling procedures comprises a first modeling procedure selected from a first family of modeling procedures and a second modeling procedure selected from a second family of modeling procedures.

In some embodiments, the action of the method further comprises, prior to determining the second accuracy score of the predictive model, refitting the predictive models to the modified data set representative of the modified prediction problem. In some embodiments, the determined model-specific predictive value of the feature for a particular customized model increases as the difference between the first accuracy score and the second accuracy score of the particular customized model increases. In some embodiments, the determined model-specific predictive value of the feature for a particular customized model is a first accuracy score and a second accuracy score of the particular customized model relative to the first accuracy score of the particular customized model. Include the percentage difference between

In some embodiments, the action of the method further comprises determining a model-independent predictive value of the feature based on model-specific predictive values of the feature. In some embodiments, determining the model-independent predictive value of the feature comprises calculating a statistical measure of the centroid and/or spread of model-specific predictive values of the feature. In some embodiments, determining the model-independent predictive value of the feature comprises calculating a statistical measure of centroids of the model-specific predictive values, wherein the systematic measure of centroids is a mean, a median. (median), and a mode of model-specific prediction values. In some embodiments, determining the model-independent predictive value of the feature comprises calculating a statistical measure of the spread of the model-specific predictive values, wherein the statistical measure of the spread is a range, a variance, and the model. - selected from the group consisting of the standard deviation of the unique predictive values. In some embodiments, determining the model-independent predictive value of the feature comprises calculating a combination of model-specific predictive values of the feature. In some embodiments, calculating the combination of model-specific prediction values comprises calculating a weighted combination of the model-specific prediction values. In some embodiments, calculating the weighted combination of model-specific prediction values comprises assigning respective weights to the model-specific prediction values, wherein a specific model corresponding to a specific customized model- The weight assigned to a unique predictive value increases as the first accuracy score of the customized predictive model increases.

In some embodiments, the feature is a first feature, and the action of the method comprises (c1) shuffling the values of the second feature over respective observations included in the initial data set, thereby resulting in a second modified prediction problem. generating a second modified data set representing (d1) determining a third accuracy score of each of the customized models, wherein the third accuracy score of each customized model determines whether the customized model determines one or more results of the second modified prediction problem. indicating accuracy of prediction; and (e1) determining a respective model-specific prediction value of the second feature for each of the customized models, wherein the model-specific prediction value of the second feature for each of the customized models is based on a first accuracy score and a third accuracy score of the customized model.

In some embodiments, the feature is a first feature, and the initial data set comprises the first feature and a plurality of second features, the method comprising: step (c) for each of the second features; and determining model-specific prediction values of second features of the initial data set by performing (d) and (e).

In some embodiments, the action of the method comprises, via a graphical user interface, identifying first and second features of the initial data set and model-specific prediction values of the first and second features. The method further includes displaying graphic content. In some embodiments, the modeling procedures are first modeling procedures comprising a specific modeling procedure associated with a specific predictive model, wherein the model-specific prediction values of the first feature and the second features are specific to the specific predictive model. comprising specific model-specific prediction values of the first feature and the second feature, wherein the action of the method comprises (a1) a plurality of second predictive modeling procedures comprising the specific modeling procedure associated with the specific predictive model. It further comprises the step of performing. In some embodiments, performing the specific predictive modeling procedures comprises performing feature engineering on the initial data set based on specific model-specific prediction values of the first feature and the second features. .

In some embodiments, performing the feature engineering comprises removing particular features from the initial data set based on the features having low model-specific predictive values. In some embodiments, the action of the method is based on the model-specific prediction value of the particular feature being lower than a threshold and/or the model-specific predictive value of the particular feature being the first feature of the initial data set and determining that the model-specific predictive value of the particular feature is low based on being a specified percentile value of the particular model-specific predictive values for a second feature.

In some embodiments, performing the feature engineering comprises: generating a derived feature based on which two or more specific features of the initial data set have high model-specific prediction values; and adding the derived features to the initial data set, thereby generating a second initial data set. In some embodiments, the action of the method is that the model-specific prediction values of the specific features are based on being higher than a threshold value and/or that the model-specific prediction values of the specific features are based on a first value of the initial data set. determining that the model-specific prediction values of the particular features are high based on being a specified percentile value of the particular model-specific prediction values for a feature and the second feature.

In some embodiments, performing the specific predictive modeling procedure further comprises: customizing the specific predictive model to the second initial data set, wherein the actions of the method include: determining a 1 accuracy score, wherein the first accuracy score of the customized specific model is indicative of an accuracy with which the customized specific model predicts one or more outcomes of the initial prediction problem; generating a second modified data set representing a second modified prediction problem by shuffling the values of the first feature over respective observations included in the second initial data set; determining a second accuracy score of the customized specific predictive model, wherein the second accuracy score of the customized specific model is an accuracy with which the customized model predicts one or more outcomes of the second modified prediction problem. representing the above steps; and determining a second model-specific predictive value of the first feature for the particular customized model, wherein a second model-specific predictive value of the first feature for the particular customized model is determined by the customization. based on the first accuracy score and the second accuracy score of the specified model.

In some embodiments, the action of the method further comprises, before performing the plurality of second modeling procedures: selecting the second modeling procedures based on suitability of the selected modeling procedures for the initial prediction problem. and the suitability of the particular predictive modeling procedure for the initial prediction problem is based, at least in part, on characteristics of one or more particular features of the initial data set having high model-specific predictive values for the particular predictive modeling procedure. it is decided

In some embodiments, the action of the method comprises sending an instruction to a plurality of processing nodes, the instruction comprising a resource allocation schedule allocating resources of the processing nodes for execution of the second modeling procedures; , wherein the resource allocation schedule is based, at least in part, on adequacy of the second modeling procedures for the initial prediction problem; receiving results of execution of the second modeling procedures by the plurality of processing nodes according to the resource allocation schedule, the results including predictive models generated by the second modeling procedures, and/or the comprising a score of models generated for data associated with an initial prediction problem; and selecting, from the generated models, a predictive model for the initial prediction problem based at least in part on a score of the selected predictive model. In some embodiments, the actions of the method include: combining two or more of the generated predictive models to generate a blended predictive model; and evaluating the blended predictive model.

In some embodiments, the predicted value of the model-independent feature for the modified prediction problem is less than the threshold prediction value.

In some embodiments, the initial data set is an initial time series data set, the initial prediction problem is an initial time series prediction problem, the modified data set is a modified time series data set, and the modified prediction problem is a modified time series prediction problem. In some embodiments, the customized model includes one or more time series prediction models.

Another embodiment of this aspect includes a predictive modeling apparatus comprising: a memory configured to store processor-executable instructions; and a processor configured to execute the processor-executable instructions, wherein the execution of the processor-executable instructions causes the apparatus to perform the steps of: (a) performing a plurality of predictive modeling procedures; performing each of the predictive modeling procedures associated with a predictive model, and performing each modeling procedure comprises fitting the associated predictive model to an initial data set representing an initial predictive problem. ; (b) determining a respective first accuracy score of each of the customized predictive models, wherein the first accuracy score of each customized model determines whether the customized model determines one or more results of the initial prediction problem. indicating the accuracy of the prediction; (c) shuffling the values of the feature over each of the observations included in the initial data set, thereby generating a modified data set representing the modified prediction problem; (d) determining a respective second accuracy score of each of the customized predictive models, wherein the second accuracy score of each customized model is one or more outcomes of the prediction problem for which the customized model is modified. indicating the accuracy of predicting them; and (e) determining each model-specific predictive value of the feature for each of the customized models, wherein the model-specific predictive value of the feature for each customized model is based on the first accuracy score and the second accuracy score of the model.

Another embodiment of this aspect includes an article of manufacture having stored thereon computer-readable instructions, wherein when executed by a processor, cause the processor to perform the following operations: (a) a plurality of predictive modeling procedures wherein each of the predictive modeling procedures is associated with a predictive model, and wherein performing each modeling procedure comprises fitting the associated predictive model to an initial data set representative of an initial prediction problem. movement; (b) determining a respective first accuracy score of each of the customized predictive models, wherein the first accuracy score of each customized model comprises: indicative of predictive accuracy; (c) generating a modified data set representing a modified prediction problem by shuffling the values of the feature over respective observations included in the initial data set; (d) determining a respective second accuracy score of each of the customized predictive models, wherein the second accuracy score of each customized model is one or more results of the prediction problem for which the customized model is modified. indicating the accuracy of predicting them; and (e) determining each model-specific predictive value of the feature for each of the customized models, wherein the model-specific predictive value of the feature for each customized model is determined by the customized model. based on the first accuracy score of and the accuracy score.

Certain embodiments of this aspect may be realized to implement one or more of the following advantages. Some embodiments of this aspect can be advantageously used to facilitate understanding of prediction problems and to show how a particular predictive model accurately predicts an outcome. Some embodiments of this aspect may measure feature importance for any predictive model or various sets of predictive models. Some embodiments of this aspect may guide the allocation of resources to evaluate predictive modeling procedures, feature engineering tasks, and blend predictive models, thereby cost-effectively evaluating the space of potential predictive modeling techniques for predictive problems. to facilitate

Secondary Predictive Modeling

Certain modeling techniques tend to produce opaque and/or complex models that are difficult to understand and efficiently implement in software. Software implementing these models can use significant computing resources to produce predictions that can be generated much more efficiently using software implementing other identically accurate models.

A first-order predictive model (M1) that predicts the values of one or more output variables (“targets”) (T) based on the values of one or more input variables (“features”) (F1) without significantly reducing the accuracy of the model. Techniques are needed to reduce opacity and/or complexity. The inventors recognize and understand that this need can be met by building a second-order model M2 of the first-order model M1. The second-order model may predict the predicted value of the first-order model for targets T based on the same feature F1 (or a subset thereof) and/or one or more features not used by the first-order model. have.

In many cases, the quadratic model generated by embodiments of this aspect is exactly the same as or more accurate than the corresponding first-order model, and the software implementing the second-order model is substantially more accurate than the software implementing the corresponding first-order model. Efficient. When embodiments of this aspect are used to generate a second-order model of a mixed first-order model, the second-order model is in many cases particularly accurate and in many cases the software implementing the second-order model is to implement the corresponding first-order model. More efficient than software (eg, using less computational resources). Second order models generated in accordance with some embodiments of this aspect may be beneficial in understanding complex first order models and/or may simplify the task of creating software implementing accurate predictive models.

In general, another innovative aspect of the subject matter described herein may be embodied in a predictive modeling method, the method comprising: obtaining a customized first-order predictive model, the first-order predictive model comprising one or more first input variables configured to predict values of one or more output variables of a prediction problem based on values of ; and performing a quadratic predictive modeling procedure on the customized first-order model, wherein the second-order modeling procedure is associated with a second-order predictive model, wherein the second-order prediction is performed on the customized first-order model. Performing the modeling procedure may include: generating secondary input data comprising a plurality of secondary observations, each secondary observation having respective observed values of one or more second input variables and a prediction of the output variables. and generating the secondary input data for each secondary observation comprising: obtaining respective observed values of the second input variables and corresponding observed values of the first input variables; and applying the first-order predictive model to the corresponding observed values of the input variables to generate respective predicted values of the output variables. generating training data and secondary testing data, customizing the secondary predictive model to the secondary training data, thereby generating a customized secondary predictive model of the customized primary model, and the secondary testing testing the customized secondary predictive model of the customized primary model against data.

Another embodiment of this aspect includes a corresponding computer system, apparatus, and computer program recorded on one or more computer storage devices, each configured to perform the action of the method. A system of one or more computers may be configured to perform a particular operation by installing software, firmware, hardware, or a combination thereof on the system in operation, causing the system to perform the operation. One or more computer programs may be configured to perform particular operations by including instructions that, when executed by the data processing apparatus, cause the apparatus to perform the actions.

The foregoing and other embodiments may each optionally include one or more of the following features, alone or in combination. In some embodiments, obtaining the customized first-order model includes performing a first-order predictive model procedure associated with the first-order predictive model, and performing the first-order predictive modeling procedure includes: a plurality of obtaining primary input data comprising primary observations, each primary observation including respective observed values of the first input variables and corresponding observed values of the output variables; from the primary input data, generating primary training data and primary testing data; fitting the primary prediction model to the primary training data; and applying the customized primary prediction model to the testing data. including testing for In some embodiments, obtaining the customized first-order model comprises mixing two customized predictive models.

In some embodiments, obtaining the customized first-order model comprises: based at least in part on characteristics of the prediction problem and/or properties of the respective first-order predictive modeling procedures, a plurality of models for the prediction problem. determining the adequacy of the first-order predictive modeling procedures; selecting one or more predictive modeling procedures from the plurality of primary predictive modeling procedures based on the determined relevance of the selected modeling procedures for the predictive problem; and performing the one or more predictive modeling procedures. In some embodiments, performing the one or more predictive modeling procedures comprises: sending an instruction to a plurality of processing nodes, wherein the instruction is a resource that allocates resources of the processing nodes for execution of the selected modeling procedures. an allocation schedule, wherein the resource allocation schedule is based at least in part on suitability of the selected modeling procedures for the prediction problem; receiving, according to the resource allocation schedule, results of execution of the selected modeling procedures by the plurality of processing nodes, the results comprising predictive models generated by the selected modeling procedures; and selecting the customized first-order model from the generated models.

In some embodiments, the quadratic predictive model is selected from the group consisting of a RuleFit model and a generalized additive model. In some embodiments, the action of the present invention further comprises performing mutual validation of the secondary model, wherein the secondary input data includes at least one data set, and generating the secondary training data. performing includes obtaining a first subset of the data set, and generating the secondary testing data includes obtaining a second subset of the data set.

In some embodiments, the secondary training data is first secondary training data, the secondary testing data is primary secondary testing data, the customized secondary model is a first customized secondary model, and The performing mutual validation of the secondary model includes: (a) generating second secondary training data and second secondary testing data from the secondary input data, wherein the second secondary training data is comprising a third subset of the data set, and wherein the second secondary testing data comprises a fourth subset of the data set; (b) fitting the secondary prediction model to the second secondary training data to obtain a second customized secondary prediction model; and (c) testing the second secondary predictive model against the second secondary testing data.

In some embodiments, the action of the method further comprises partitioning the data set into a plurality of partitions comprising at least a first partition and a second partition. In some embodiments, partitioning the data set into a plurality of partitions comprises randomly assigning each observation of the data set to each partition. In some embodiments, the first secondary training data comprises a first partition of the data set; the first secondary testing data includes all partitions of the data set except for the first partition; the second secondary training data comprises a second partition of the data set; and the second secondary testing data includes all partitions of the data set except for the second partition. In some embodiments, the first secondary training data comprises a subset of the first partition of the data set; the first secondary testing data includes respective subsets of all partitions of the data set except for the first partition; the second secondary training data comprises a subset of the second partition of the data set; and the second secondary testing data includes respective subsets of all of the partitions of the data set except the second partition.

In some embodiments, the secondary input data includes a first partition and a second partition, the data set includes a first partition of the secondary input data, and the action of the method comprises: and testing the first and second customized quadratic models against holdout data comprising a second partition of . In some embodiments, no predictive model is customized to the holdout data.

In some embodiments, performing the secondary predictive modeling procedure further includes performing nested mutual validation of the secondary predictive model. In some embodiments, the secondary input data comprises at least one data set; The performing nested mutual validation of the quadratic predictive model comprises: the data into a first plurality of partitions of the data set comprising at least a first partition of the data set and a second partition of the data set. partitioning a set into a plurality of partitions of the first partition of the data set comprising at least a first partition of the first partition of the data set and a second partition of the first partition of the data set partitioning the first partition of the data set. According to some embodiments, the secondary training data comprises a first partition of the first partition of the data set; and the secondary testing data includes all partitions of the first partition of the data set except for the first partition of the first partition of the data set.

In some embodiments, the secondary training data is first secondary training data, the secondary testing data is primary secondary testing data, the customized secondary model is a first customized model, and the secondary Performing nested mutual validation of the predictive model comprises: (a) generating, from a first partition of the data set, second secondary training data and second secondary testing data, wherein the second the differential training data comprises a second partition of the first partition of the data set, and the second secondary testing data includes the first partition of the data set other than a second partition of the first partition of the data set comprising a plurality of partitions of ; (b) fitting the quadratic predictive model to the second quadratic training data to obtain a second quadratic customized predictive model; and (c) testing the second secondary customized model against the second secondary testing data.

In some embodiments, performing the nested mutual validation comprises: testing the first customized quadratic model and the second customized quadratic model against a second partition of the data set; and based on a result of testing the first customized quadratic model and the second customized quadratic model on a second partition of the data set, the first customized quadratic model and the second customized quadratic model The method further includes comparing the car models.

In some embodiments, the actions of the method further comprise determining an accuracy score of each of the customized predictive models, wherein the accuracy score of each customized model is determined by determining that the customized model has one or more predictive problems. It represents the accuracy of predicting the results of In some embodiments, the actions of the method further comprise determining a discrepancy between an accuracy score of the customized first-order model and an accuracy score of the customized second-order model. In some embodiments, the accuracy score of the customized second-order model exceeds the accuracy score of the customized first-order model.

In some embodiments, the actions of the method further comprise determining an amount of computational resource used by each of the customized predictive models to predict outcomes of one or more predictive problems. According to some embodiments, the action of the method further comprises determining a discrepancy between the amount of computational resources used by the customized first-order model and the amount of computational resources used by the customized second-order model. do. In some embodiments, the amount of computational resources used by the customized second-order model is less than the amount of computational resources used by the customized first-order model.

In some embodiments, the action of the method further comprises deploying the customized quadratic model. In some embodiments, placing the customized quadratic model comprises generating a plurality of predictions by applying the customized quadratic model to other data representative of instances of the prediction problem, wherein the quadratic The input data does not include other data. In some embodiments, the customized quadratic model comprises a set of one or more conditional rules, wherein the set of one or more conditional rules comprises a set of one or more machine-executable if-then statements.

In some embodiments, the secondary input data is first secondary input data, and wherein disposing the customized secondary model refreshes the customized secondary model based at least in part on the second secondary input data. further comprising the step of In some embodiments, the customized quadratic model is a first customized quadratic model, and refreshing the customized quadratic model based at least in part on the second secondary input data comprises: generating, from the differential input data, second secondary training data and second secondary testing data; generating a second customized secondary model of the customized primary model by customizing the secondary predictive model to the second secondary training data; testing the second customized secondary model of the primary model against the second secondary testing data; and mixing the first customized quadratic model and the second customized quadratic model to generate a refreshed quadratic predictive model. In some embodiments, the customized quadratic model is a first customized quadratic model, and refreshing the customized quadratic model based at least in part on the second quadratic input data comprises: the first 2 generating third secondary input data including at least a portion of the difference input data and at least a portion of the second secondary input data; generating third secondary training data and third secondary testing data from the third secondary input data; generating a second customized secondary model of the customized primary model by customizing the secondary predictive model to the third secondary training data; and testing the second customized secondary model of the primary model on the third secondary testing data.

In some embodiments, the first input variables are the second input variables. In some embodiments, both the first input variables and the second input variables include a particular input variable. In some embodiments, none of the first input variables are included in the second input variables.

In some embodiments, the secondary modeling procedure is one of a plurality of secondary modeling procedures, the secondary prediction model is one of a plurality of second prediction models, and the method comprises: By performing procedures on the customized first-order model, a plurality of customized second-order models of the customized first-order model are generated. In some embodiments, the action of the method further comprises determining an accuracy score of each of the customized quadratic predictive models, wherein the accuracy score of each customized quadratic model is: It represents the accuracy of predicting the outcomes of one or more prediction problems. In some embodiments, the actions of the method include determining which accuracy score has the highest; and placing the customized quadratic model with the highest accuracy score.

Another embodiment of this aspect includes a predictive modeling apparatus, the apparatus comprising: a memory configured to store a machine-executable module encoding a second-order predictive modeling procedure associated with a second-order predictive model, the second-order predictive modeling the memory comprising a plurality of tasks including at least one pre-processing task and at least one model-fitting task; and at least one processor configured to execute the machine-executable module, wherein the execution of the machine-executable module causes the apparatus to perform the second-order predictive modeling procedure on the customized first-order predictive model. do. Performing the secondary predictive modeling procedure includes performing the pre-processing operation comprising obtaining a customized primary predictive model, wherein the primary predictive model is based on values of one or more first input variables. and predict values of one or more output variables of the prediction problem. Performing the secondary predictive modeling procedure includes: performing the model-fitting operation, the performing comprising: generating secondary data comprising a plurality of secondary observations, each secondary observation comprising: includes respective observed values of one or more second input variables and predicted values of the output variables, wherein generating the secondary input data for each secondary observation comprises: each of the second input variables obtaining observed values of and corresponding observed values of the first input variables, and applying the first-order predictive model to the corresponding observed values of the input variables to generate respective predicted values of the output variables. The customized primary model by: generating secondary training data and secondary testing data from the secondary input data; fitting the secondary predictive model to the secondary training data; generating a customized secondary predictive model of , and testing the customized, secondary predictive model of the customized primary model against the secondary testing data.

Other aspects and advantages of the present invention will become apparent from the following drawings, detailed description and claims, all of which illustrate the principles of the invention by way of example only.

The above summary, including a description of several embodiments, their motivations, and/or their advantages, is intended to aid the reader in understanding the present invention and does not limit the scope of the claims in any way.

Certain advantages of some embodiments may be understood by reference to the following description taken in conjunction with the accompanying drawings. In the drawings, like reference numbers generally refer to like parts throughout different drawings. Further, the drawings are not necessarily to scale, emphasis instead generally being placed on illustrating the principles of some embodiments of the invention.
1 is a block diagram of a predictive modeling system in accordance with some embodiments.
2 is a block diagram of a modeling tool for building machine-executable templates encoding predictive modeling tasks, techniques, and methods in accordance with some embodiments.
3 is a flowchart of a method for selecting a predictive model for a predictive problem in accordance with some embodiments.
4 shows another flowchart of a method for selecting a predictive model for a predictive problem in accordance with some embodiments.
5 is a schematic diagram of a predictive modeling system in accordance with some embodiments.
6 is another block diagram of a predictive modeling system according to some embodiments.
7 illustrates communication between components of a predictive modeling system in accordance with some embodiments.
8 is another schematic diagram of a predictive modeling system in accordance with some embodiments.
9 is a flowchart of a method for time-series predictive modeling in accordance with some embodiments.
10 is a flow diagram of a method for determining a predicted value of a feature in accordance with some embodiments.
11A is a flowchart of a method for generating a quadratic predictive model in accordance with some embodiments.
11B is a flowchart of a method for performing a secondary predictive modeling procedure in accordance with some embodiments.

Overview of Predictive Modeling Systems

1 , in some embodiments, the predictive modeling system 100 includes a predictive modeling search engine 110 , a user interface 120 , a library of predictive modeling techniques 130 , and a predictive model placement engine 140 . do. A search engine is a search technique (or, for example, a potential combination of pre-processing steps, modeling algorithms, and post-processing steps) to efficiently search the predictive modeling search space to generate a predictive modeling solution appropriate to a particular predictive problem. "Modeling Method") can be implemented. The search technique may include an initial assessment of which predictive modeling techniques are likely to provide an appropriate solution to the prediction problem. In some embodiments, a search technique is an incremental evaluation of the search space (e.g., using an increasing portion of a data set) and the relevance of different modeling solutions to a prediction problem (e.g., using a consistent metric). consistent comparison of In some embodiments, the search technique may adapt based on the results of previous searches to improve the effectiveness of the search technique over time.

The search engine 110 may use the library 130 of modeling techniques to evaluate potential modeling solutions in the search space. In some embodiments, modeling technique library 130 includes machine-executable templates that encode complete modeling techniques. The machine-executable template may include one or more predictive modeling algorithms. In some embodiments, modeling algorithms included in templates may be related in some way. For example, a modeling algorithm may be a variant of the same modeling algorithm or a member of a family of modeling algorithms. In some embodiments, the machine-executable template further comprises one or more pre-processing and/or post-processing steps suitable for use with the algorithm(s) of the template. The algorithm(s), pre-processing steps and/or post-processing steps may be parameterized. A machine-executable template can be applied to a user data set to create a potential predictive modeling solution to the predictive problem represented by the data set.

The search engine 110 may use the computational resources of the distributed computing system to search the search space or a portion thereof. In some embodiments, the search engine 110 generates a search plan for efficiently performing a search using the resources of the distributed computing system, and the distributed computing system executes the search according to the search plan. A distributed computing system is a queuing and monitoring of predictive modeling techniques, virtualization of computing system resources, access to databases, allocation of computing system resources for segmentation of search plans and evaluation of modeling techniques, collection and organization of execution results , an interface for facilitating evaluation of a predictive modeling solution according to a search plan including, but not limited to, an interface for acceptance of user input, etc. may be provided.

The user interface 120 provides tools for monitoring and/or guiding the search of the predictive modeling space. These tools provide insight into data sets of prediction problems and/or insights into search results (e.g., by highlighting problematic variables in data sets, identifying relationships between variables in data sets, etc.) can do. In some embodiments, the data analyst may use the interface to guide the search, for example, by specifying metrics used to evaluate and compare modeling solutions, specifying criteria for recognizing suitable modeling solutions, etc. . Accordingly, the user interface may be used by the analyst to improve their own productivity and/or to improve the performance of the search engine 110 . In some embodiments, the user interface 120 presents the results of the search in real time and allows the user to guide the search in real time (eg, adjust the scope of the search or the allocation of resources during evaluation of different modeling solutions) make it possible). In some embodiments, user interface 120 provides tools for coordinating the efforts of multiple data analysts working on the same prediction problem and/or related prediction problems.

In some embodiments, user interface 120 provides tools for developing machine-executable templates for library 130 of modeling techniques. System users may use these tools to modify existing templates, create new templates, or remove templates from library 130 . In this manner, system users may update library 130 to reflect advances in predictive modeling research and/or to include proprietary predictive modeling techniques.

The model deployment engine 140 provides tools for deploying a predictive model (eg, a predictive model generated by the search engine 110 ) in an operating environment. In some embodiments, the model deployment engine also provides tools for monitoring and/or updating predictive models. System users deploy predictive models generated by search engine 110, monitor the performance of such predictive models, and update such models (eg, based on new data or advances in predictive modeling techniques). For this purpose, the batch engine 140 may be used. In some embodiments, the search engine 110 may be used to guide the search of the search space for a prediction problem (eg, to refit or tune a predictive model in response to changes in the underlying data set for the prediction problem). ) may use the data collected and/or generated by the deployment engine 140 (eg, based on the results of monitoring the performance of the deployed predictive model).

These and other aspects of the predictive modeling system 100 are described in greater detail below.

A library of modeling techniques

The library of predictive modeling techniques 130 includes machine-executable templates encoding complete predictive modeling techniques. In some embodiments, the machine-executable template includes one or more predictive modeling algorithms, zero or more pre-processing steps suitable for use with the algorithm(s), and zero or more predictive modeling algorithms suitable for use with the algorithm(s). Post-processing steps included. The algorithm(s), pre-processing steps and/or post-processing steps may be parameterized. A machine-executable template can be applied to a data set to create a potential predictive modeling solution to the prediction problem represented by the data set.

The template may encode pre-processing steps, model-fitting steps and/or post-processing steps suitable for use with the template's predictive modeling algorithm(s) for machine execution. Examples of pre-processing steps include missing-value input, feature engineering (e.g., one-hot encoding, splines, text mining, etc.), feature selection (e.g., missing features without information, unrelated high feature dropouts, replacement of original features with top major components, etc.). Examples of model-fitting steps include, but are not limited to, algorithm selection, parameter estimation, hyper-parameter tuning, scoring, diagnosis, and the like. Examples of post-processing steps include, but are not limited to, proofreading, censoring, blending, and the like.

In some embodiments, the machine-executable template includes metadata that describes properties of the predictive modeling technique encoded by the template. The metadata may represent one or more data processing techniques that the template may perform as part of a predictive modeling solution (eg, in a pre-processing step, post-processing step, or step of a predictive modeling algorithm). Such data processing techniques may include, but are not limited to, text mining, feature normalization, dimension reduction, or other suitable data processing techniques. Alternatively or additionally, the metadata may be contained in a template including, but not limited to, constraints on the dimensionality of the data set, characteristics of the target(s) of the prediction problem, and/or characteristics of the feature(s) of the prediction problem. may indicate one or more data processing constraints imposed by a predictive modeling technique encoded by

In some embodiments, the metadata of the template includes information related to estimating how effective the corresponding modeling technique is for a given data set. For example, the metadata of a template can be large data sets, tall data sets, sparse data sets, dense data sets, data sets with or without text, and different data types (e.g. , data sets containing variables of numbers, ordinal numbers, categories, interpreting (e.g., date, time, text), etc.), various statistical characteristics (e.g., missing values of variables, cardinality, variance) It may indicate how well a corresponding modeling technique is predicted to perform on a data set having a particular characteristic including, but not limited to, a data set including a variable having a statistical characteristic associated with the same. In another example, the metadata of the template represents predicted performance of the corresponding modeling technique in terms of one or more performance metrics (eg, objective functions).

In some embodiments, the metadata of the template includes a characterization of the processing step implemented by a corresponding modeling technique including but not limited to allowed data type(s), structure and/or dimensions of the processing step.

In some embodiments, the metadata of the template includes data representing the results (actual or expected) of applying the predictive modeling techniques represented by the template to one or more predictive problems and/or data sets. The result of applying a predictive modeling technique to a predictive problem or data set is the accuracy with which the predictive model generated by the predictive modeling technique predicts the target(s) of the predictive problem or data set, for the predictive problem or data set (other predictive modeling A score indicating the degree of accuracy of the predictive model generated by the predictive modeling technique (related to the technique), the utility of using the predictive modeling technique to generate a predictive model for a predictive problem or data set (e.g., for an objective function). values generated by the predictive model), and the like.

Data representing the results of applying the predictive modeling technique to the predictive problem or data set is provided by the search engine 110 (eg, based on the results of previous attempts to use the predictive modeling technique to the predictive problem or data set). provided by the user (eg, based on the user's expertise), and/or may be obtained from any other suitable source. In some embodiments, the search engine 110 updates such data based, at least in part, on the relationship between the actual results of instances of the prediction problem and the results predicted by the predictive model generated through predictive modeling techniques.

In some embodiments, the metadata of the template describes characteristics of the corresponding modeling technique related to estimating how efficiently the modeling technique will execute on a distributed computing infrastructure. For example, the template's metadata may include the processing resources needed to train and/or test modeling techniques on a data set of a given size, the impact on resource consumption of the number of mutual validation folds, and the points retrieved in the hyper-parameter space. , the intrinsic parallelism of processing steps performed by the modeling technique, etc.

In some embodiments, the library 130 of modeling techniques includes tools for evaluating similarities (or differences) between predictive modeling techniques. These tools can score similarities (eg, on a predetermined scale) between two predictive modeling techniques, classify them (eg, "very similar", "somewhat similar", "somewhat dissimilar", "very dissimilar"). "), binary crystals (eg, "like" or "dislike"), and the like. Such a tool can determine the similarity between two predictive modeling techniques based on processing steps common to the modeling techniques, etc., based on data representing the results of applying the two predictive modeling techniques to the same or similar predictive problem. For example, given two predictive modeling techniques that, when applied to similar predictive problems, have many (or a high percentage) of their processing steps in common and/or produce similar results, the tool has a high degree of similarity to the modeling techniques. You can assign a score or classify the modeling technique as "very similar".

In some embodiments, modeling techniques may be assigned to families of modeling techniques. A family classification of modeling techniques can be assigned by the user (eg, based on intuition and experience), and (eg, processing steps common to modeling techniques, as a result of applying different modeling techniques to the same or similar problem) may be assigned by a machine-learning classifier, etc.), or may be obtained from other suitable sources. A tool for evaluating the similarity between predictive modeling techniques may rely on family classification to evaluate the similarity between two modeling techniques. In some embodiments, the tool may treat all modeling techniques in the same family as “similar” and any modeling techniques in different families as “dissimilar”. In some embodiments, the classification of a family of modeling techniques may be only one factor in the instrumental assessment of similarities between modeling techniques.

In some embodiments, predictive modeling system 100 includes a library of predictive problems (not shown in FIG. 1 ). The library of prediction problems may contain data representative of characteristics of the prediction problem. In some embodiments, the data representative of the characteristics of the prediction problem comprises data representative of the characteristics of the data set representative of the prediction problem. A characteristic of a data set may be the width, height, sparseness, or density of the data set; the number of targets and/or features in the data set, the data type of the data set variable (e.g., numeric, ordinal, category, or interpreting (e.g., date, time, text, etc.); range; the number of classes for ordinal and categorical variables in the data set; and the like.

In some embodiments, a characteristic of a data set is the total number of observations; the number of eigenvalues for each variable across the observations; the number of missing values for each variable across the observations; the presence and extent of outliers and inliers; an attribute of the variance of the value of each variable or class membership; cardinality of variables; including, but not limited to, statistical properties of variables in data sets including them. In some embodiments, a characteristic of a data set is a joint variance of a group of variables; variable importance of one or more features to one or more targets (eg, correlations between features and target variables); a statistical relationship between two or more features (eg, the degree of multicollinearity between the two features); including, but not limited to, relationships (eg, statistical relationships) between variables in a data set comprising the same.

In some embodiments, data representative of the nature of the prediction problem may include data representative of the subject of the prediction problem (eg, finance, insurance, defense, e-commerce, retail, internet-based advertising, internet-based recommendation engine, etc.); origin of the variable (eg, whether each variable was obtained directly from an automated instrument, from a human record of an automated instrument, from a human measurement, from a recorded human response, an oral human response, etc.); the existence and performance of known predictive modeling solutions to predictive problems; etc.

In some embodiments, the predictive modeling system 100 may support time series prediction problems (eg, one-dimensional or multi-dimensional time series prediction problems). For time-series prediction problems, the goal is usually to predict the future value of a target as a function of previous observed values of all features, including the target itself. Data representative of the nature of the prediction problem can accommodate the time-series prediction problem by indicating whether the prediction problem is a time-series prediction problem and identifying temporal variables in the data set corresponding to the time-series prediction problem.

In some embodiments, the library of prediction problems includes tools for evaluating similarities (or differences) between prediction problems. These tools score similarities (eg, on a predetermined scale) between two prediction problems, classify them (eg, "very similar", "somewhat similar", "somewhat dissimilar", "very dissimilar"). , can be expressed as a binary crystal (eg, “like” or “dislike”). Such a tool may determine the similarity between two predictive problems based on data representative of the characteristics of the predictive problem, etc., based on data representative of the results of applying the same or similar predictive modeling techniques to the predictive problem. For example, for two prediction problems represented by data sets that have many (or a high percentage of) characteristics in common and/or are susceptible to the same or similar predictive modeling techniques, the tool is highly sensitive to the prediction problem. You can assign a similarity score or classify the prediction problem as "very similar".

2 shows a block diagram of a modeling tool 200 suitable for building machine-executable templates encoding predictive modeling techniques and incorporating such templates into predictive modeling methods in accordance with some embodiments. The user interface 120 may provide an interface to the modeling tool 200 .

In the example of FIG. 2 , in FIG. 2 , the modeling method builder 210 builds a library 212 of modeling methods on top of the library 130 of modeling techniques. The modeling technique builder 220 builds the library 130 of modeling techniques on top of the library 232 of modeling tasks. Modeling methods may correspond to one or more analyst's intuition and experience as to which modeling techniques are effective in which circumstances, and/or use modeling techniques for prior prediction problems to guide exploration of the modeling search space for prediction problems. The results of application can be used. A modeling technique may correspond to a step-by-step recipe for applying a specific modeling algorithm. A modeling task may correspond to a processing step within a modeling technique.

In some embodiments, modeling techniques may include hierarchies of tasks. For example, a top-level “text mining” task may include sub-tasks that (a) generate a document-term matrix and (b) rank terms and drop unimportant terms. Then, the “grading and dropping terms” sub-task is a sub-task for (b.1) building a rating model and (b.2) using term ratings to drop columns from the document-term matrix. may include These layers can have any depth.

In the example of FIG. 2 , in FIG. 2 , the modeling tool 200 includes a modeling task builder 230 , a modeling technique builder 220 and a modeling method builder 210 . Each builder may include a tool or set of tools for encoding one of the modeling elements in a machine-executable format. Each builder may allow users to modify existing modeling elements or create new modeling elements. To construct a complete library of modeling elements across the modeling layers shown in Figure 2, developers can employ top-down, bottom-up, extroverted, introverted, or combined strategies. However, in terms of logical dependencies, since leaf-level tasks are the smallest modeling elements, Figure 2 shows task creation as the first step in the process of constructing a machine-executable template.

Each builder's user interface is not limited to, but is not limited to, a collection of routines specialized in a standard programming language, a formal syntax specifically designed for the purpose of encoding the builder's elements, and a rich user interface to abstractly specify the desired flow of execution. However, the logical structure of the operations allowed in each layer is independent of any particular interface.

When creating modeling tasks at the leaf-level of a hierarchy, modeling tool 200 may allow developers to integrate software components from other sources. This function utilizes the installed base of software related to statistical learning and accumulated knowledge of how to develop such software. These installed bases include scientific programming languages (eg Fortran), scientific routines written in general-purpose programming languages (eg C), and general-purpose programming languages (eg scikit-learn for Python). of scientific computing extensions, commercial statistical environments (eg, SAS/STAT), and open source statistical environments (eg, R). When used to incorporate the functionality of such software components, modeling task builder 230 may require a specification of the input and output of the software component and/or characterization of the types of operations that the software component may perform. In some embodiments, modeling task builder 230 checks a software component's source code signature, retrieves the software component's interface definition from a repository, probes the software component with a sequence of requests, or performs other forms of automation. You create this metadata by performing an evaluation. In some embodiments, the developer manually supplies some or all of this metadata.

In some embodiments, modeling job builder 230 uses this metadata to create a “wrapper” that enables the integrated software to run. The modeling job builder 230 compiles the component's source code into an internal executable, links the component's object code to the internal executor, accesses the component through an emulator of the computing environment expected by the component's standalone executor, Accessing the functionality of a component running as part of a software service on a local machine, accessing the functionality of a component running as part of a software service on a remote machine, and accessing the functionality of a component through an intermediate software service running on a local or remote machine. Such wrappers may be implemented using any mechanism to incorporate software components, including but not limited to accessing, and the like. Whatever integration mechanism modeling task builder 230 uses after the wrapper is created, modeling tool 200 may call the software into the component like any other routine.

In some embodiments, a developer may use the modeling task builder 230 to recursively assemble leaf-level modeling tasks into higher-level tasks. As mentioned above, there are many different ways of implementing a user interface for specifying the arrangement of a task hierarchy. However, from a logical point of view, non-leaf-level tasks can contain directed graphs of sub-tasks. At each of the top and middle levels of this hierarchy, there may be one starting sub-task whose input is from the parent task of the hierarchy (or the parent modeling technique at the top level of the hierarchy). There may also be one end sub-task in the hierarchy whose output is directed to the parent task (or the parent modeling technique at the top level of the hierarchy). All other sub-tasks at a given level receive input from one or more previous sub-tasks and send output to one or more subsequent sub-tasks.

Propagating data according to directed graphs, in combination with the ability to incorporate arbitrary code into leaf-level operations, facilitates the implementation of arbitrary control flows within mid-level operations. In some embodiments, the modeling tool 200 may provide additional built-in operations. For example, while it is straightforward to implement any particular conditional logic as a leaf-level task coded in an external programming language, the modeling task builder 230 provides built-in nodes or arcs that perform conditional evaluation in the usual way. may provide, directing some or all of the data from a node to a different subsequent node based on the results of this evaluation. Filter the output from one node according to a rule or expression before propagating as input to subsequent nodes, transform the output from one node before propagating as input to subsequent nodes, and propagate each split to each subsequent node Split the output from one node according to a rule or expression before, combine the outputs of a plurality of previous nodes according to a rule or formula before accepting it as an input, and use one or more loop variables to sub-graph the node's behavior Similar alternatives exist for iterative application of

In some embodiments, a developer may use modeling skills builder 220 to assemble tasks from modeling tasks library 232 into modeling techniques. At least some modeling tasks in the modeling tasks library 232 may correspond to pre-processing steps, model-fitting steps, and/or post-processing steps of one or more modeling techniques. The development of tasks and skills may follow a linear pattern in which the skills are assembled after the work library 232 is populated, or a more dynamic circular pattern in which tasks and skills are assembled simultaneously. Developers are encouraged to combine existing work into new skills, recognizing that these skills require new work, and can iteratively refine new skills until they are complete. Alternatively, a developer may start with a concept of a new technology, perhaps from a journal publication, and build it from a new task, but draw existing work from the modeling task library 232 when providing the appropriate functionality. In all cases, the results of applying a modeling technique to a reference data set or field test will enable the developer or analyst to evaluate the performance of the technique. This evaluation, in turn, can result in changes at any point in the hierarchy, from leaf-level modeling tasks to modeling techniques. The modeling tool 200 provides a high-productivity builder interface 210, 220 and 230, as well as a common modeling task and modeling technology library 232, 130, so that developers can quickly and accurately make changes, as well as libraries 232, 130 ) to propagate these enhancements to other developers and users who have access to

Modeling techniques can provide a focus for developers and analysts to conceptualize the entire predictive modeling process, with every step expected based on best practice in the field. In some embodiments, modeling techniques encapsulate best practices from statistical learning training. In addition, the modeling tool 200 may generate a working graph for a new technique relative to a working graph of an existing technique, for example, detecting missing jobs, detecting additional jobs, and/or detecting abnormal flow during steps. It can provide guiding guidance in the development of high-quality technologies by, for example, providing a checklist of steps for developers to consider and compare.

In some embodiments, the search engine 110 is used to build a predictive model for the data set 240 using techniques from the modeling techniques library 130 . The search engine 110 may prioritize the evaluation of the modeling technique in the modeling technique library 130 based on the priority scheme encoded by the modeling method selected from the modeling method library 212 . An example of a suitable prioritization scheme for exploration of the modeling space is described in the next section. In the example of FIG. 2 , the search results of the modeling space may be used to update metadata associated with modeling tasks and techniques.

In some embodiments, unique identifiers (IDs) may be assigned to modeling elements (eg, descriptions, tasks, and sub-tasks). The ID of the modeling element may be stored as metadata associated with the template of the modeling element. In some embodiments, these modeling element IDs may be used to efficiently execute modeling techniques that share one or more modeling tasks or sub-tasks. Methods of efficiently implementing modeling techniques are described in more detail below.

In the example of FIG. 2 , the modeling result generated by the search engine 110 is fed back to the modeling job builder 230 , the modeling technique builder 220 , and the modeling method builder 210 . The model builder may be adapted automatically (eg, using a statistical learning algorithm) or manually (eg, by a user) based on modeling results. For example, the modeling method builder 210 may be adapted based on observed patterns in the modeling results and/or based on the experience of the data analyst. Similarly, the results of executing certain modeling techniques may inform automatic or manual adjustment of default tuning parameter values for techniques or tasks within these models. In some embodiments, the adaptation of the modeling builder may be semi-automatic. For example, predictive modeling system 100 may flag potential improvements to methods, techniques, and/or operations, and a user may determine whether to implement such potential improvements.

modeling space exploration engine

3 is a flow diagram of a method 300 for selecting a predictive model for a predictive problem in accordance with some embodiments. In some embodiments, method 300 may correspond to a modeling method in modeling method library 212 .

At step 310 of method 300, the relevance of a plurality of predictive modeling procedures (eg, predictive modeling techniques) for a predictive problem is determined. The suitability of a predictive modeling procedure for a predictive problem may be determined based on the nature of the predictive problem, the nature of the modeling procedure, and/or other suitable information.

The "relevance" of a predictive modeling procedure for a predictive problem may include data indicative of the expected performance of a predictive model generated using the predictive modeling procedure for a predictive problem. In some embodiments, the expected performance of a predictive model on a predictive problem is generated using one or more expected scores (eg, predicted values of one or more objective functions) and/or (eg, using other predictive modeling techniques). It contains one or more predicted grades) for other predictive models that have been

Alternatively or additionally, the “relevance” of a predictive modeling procedure for a predictive problem may include data indicating the degree to which the modeling procedure is expected to produce a predictive model that provides adequate performance for the predictive problem. In some embodiments, the “relevance” data of the predictive modeling procedure includes a classification of the relevance of the modeling procedure. A classification scheme can be divided into two classes (e.g., "adequate" or "inappropriate") or more than two classes (e.g., "very appropriate", "moderately appropriate", "moderately inadequate", "very inappropriate").

In some embodiments, the search engine 110 determines the suitability of a predictive modeling procedure for a prediction problem based at least in part on one or more characteristics of the prediction problem, including but not limited to the characteristics described herein. decide As an example, the adequacy of a predictive modeling procedure for a prediction problem depends on the characteristics of the data set corresponding to the prediction problem, the characteristics of the variables in the data set corresponding to the prediction problem, the relationships between the variables in the data set, and/or the subject of the prediction problem. can be determined based on Search engine 110 may include tools (eg, statistical analysis tools) for analyzing data sets associated with the prediction problem to determine characteristics of the prediction problem, data set, data set variables, and the like.

In some embodiments, the search engine 110 provides predictive modeling for a predictive problem based, at least in part, on one or more attributes of the predictive modeling procedure, including, but not limited to, attributes of the predictive modeling procedure described herein. determine the adequacy of the procedure; As an example, the adequacy of a predictive modeling procedure for a predictive problem may be determined based on data processing techniques performed by the predictive modeling procedure and/or data processing constraints imposed by the predictive modeling procedure.

In some embodiments, determining the appropriateness of a predictive modeling procedure for the predictive problem comprises removing at least one predictive modeling procedure from consideration for the predictive problem. A decision to remove a predictive modeling procedure from consideration may be referred to herein as "pruning" and/or "search space theorem" the modeling procedure removed. In some embodiments, the user may override the search engine's decision to refine the modeling procedure so that the previously compiled modeling procedure remains eligible for further execution and/or evaluation during exploration of the search space.

A predictive modeling procedure may be excluded from consideration based on the properties of the predictive modeling procedure and the results of applying one or more deductive rules to the nature of the prediction problem. Deductive rules may include, but are not limited to: (1) if the prediction problem includes a categorical target variable, select only classification techniques for execution; (2) select or prioritize techniques that provide normalization when the numeric features of the data set span widely different size ranges; (3) select or prioritize techniques that provide text mining if the data set has text features; (4) if the data set has more features than observations, remove all descriptions that require a number of observations greater than or equal to the number of features; (5) select or prioritize techniques that provide size reduction if the width of the data set exceeds a threshold width; (6) If the data set is large and sparse (e.g., the size of the data set exceeds the threshold size and the sparsity of the data set exceeds the critical sparsity), find a technique that runs efficiently on sparse data structures. select or prioritize; and/or any rules for selecting, prioritizing, or eliminating modeling techniques in which the rules may be expressed in the form of if-then statements. In some embodiments, deductive rules are chained, such that sequential execution of several rules produces a conclusion. In some embodiments, a priori rules may be updated, refined, or improved based on historical performance.

In some embodiments, the search engine 110 determines the adequacy of a predictive modeling procedure for a predictive problem based on performance (expected or actual) of similar predictive modeling procedures for a similar predictive problem (as a special case, the search engine ( 110) may determine the adequacy of a predictive modeling procedure for a predictive problem based on the performance (expected or actual) of the same predictive modeling procedure for a similar predictive problem).

As described above, the library of modeling techniques 130 may include tools for evaluating similarities between predictive modeling techniques, and the library of prediction problems may include tools for evaluating similarities between prediction problems. . Search engine 110 may use this tool to identify predictive modeling procedures and predictive modeling procedures in question and predictive problems that are similar to prediction problems. For the purpose of determining the adequacy of the predictive modeling procedure for the predictive problem, the search engine 110 selects the M modeling procedure most similar to the modeling procedure in question, and any modeling that exceeds a threshold similarity value for the modeling procedure in question. Selecting a procedure, etc. may be performed. Similarly, for the purpose of determining the adequacy of the predictive modeling procedure for the prediction problem in question, the search engine 110 selects the N prediction problems most similar to the prediction problem in question, and selects the N prediction problems that exceed the threshold similarity value for the prediction problem in question. Selecting all prediction problems, and so on.

For a set of predictive modeling procedures and a set of modeling procedures and prediction problems that are similar to the prediction problem in question, the search engine is used to evaluate the prediction problem in question to determine the expected adequacy of the modeling procedure in question for the prediction problem in question. It is possible to combine the performance of similar modeling procedures for As mentioned above, a template of a modeling procedure may include information related to evaluating how well a corresponding modeling procedure will perform on a given data set. The search engine 110 may use the model performance metadata to determine performance values (expected or actual) of similar modeling procedures for similar prediction problems. These performance values can then be combined to produce an estimate of the adequacy of the modeling procedure for the prediction problem in question. For example, the search engine 110 may calculate the adequacy of the modeling procedure in question as a weighted sum of performance values of the similar modeling procedure for a similar prediction problem.

In some embodiments, search engine 110 provides various modeling procedures (eg, modeling procedures similar to the modeling procedure in question) for other prediction problems (eg, prediction problems similar to the prediction problem in question). Determine the adequacy of a predictive modeling procedure for a predictive problem based at least in part on the output of a “meta” machine-learning model that can be trained to determine the adequacy of a modeling procedure for a predictive problem based on the results of Machine-learning models for estimating the adequacy of predictive modeling procedures for predictive problems are “meta” because they recursively apply machine learning to predict which techniques are most likely to succeed for the predictive problem in question. It may be referred to as a machine-learning model. Accordingly, the search engine 110 may generate meta-predictions of the adequacy of the modeling technique for the prediction problem by using a meta-machine-learning algorithm trained on the results of solving other prediction problems.

In some embodiments, the search engine 110 determines the predictive modeling procedure for a predictive problem based at least in part on user input (eg, user input indicative of a data analyst's intuition or experience regarding the appropriateness of the predictive modeling procedure). appropriateness can be determined.

Referring to FIG. 3 , at step 320 of method 300 , at least a subset of predictive modeling procedures may be selected based on the relevance of the modeling procedures to the predictive problem. In embodiments where the modeling procedure is assigned a relevance category (e.g., "adequate" or "inappropriate", "very appropriate", "moderately appropriate", "moderately inadequate" or "very inadequate", etc.), the modeling procedure Choosing a subset of may include

In embodiments in which relevance values are assigned to modeling procedures, search engine 110 may select a subset of modeling procedures based on relevance values. In some embodiments, the search engine 110 selects a modeling procedure that has a relevance score that exceeds a threshold relevance score. The threshold relevance score may be provided by the user or determined by the search engine 110 . In some embodiments, the search engine 110 may adjust the threshold relevance score to increase or decrease the number of modeling procedures selected for execution, depending on the amount of processing resources available for execution of the modeling procedures.

In some embodiments, the search engine 110 selects a modeling procedure that has a relevance score within a certain range of the highest relevance score assigned to any modeling procedure for the prediction problem in question. A range can be absolute (eg, a score within the S point of the highest score) or relative (eg, a score within P% of the highest score). The range may be provided by the user or may be determined by the search engine 110 . In some embodiments, the search engine 110 may adjust the scope to increase or decrease the number of modeling procedures selected for execution according to the amount of processing resources available for execution of the modeling procedures.

In some embodiments, the search engine 110 selects the portion of the modeling procedure with the highest relevance score for the prediction problem in question. Equally, the search engine 110 determines the rating of the modeling procedure with the highest relevance rating (eg, if a relevance score for the modeling procedure is not available, but an order of relevance (grading) of the modeling procedure is available). part can be selected. The portion may be provided by the user or determined by the search engine 110 . In some embodiments, the search engine 110 may adjust portions to increase or decrease the number of modeling procedures selected for execution, depending on the amount of processing resources available for execution of the modeling procedures.

In some embodiments, a user may select one or more modeling procedures to be executed. The user-selected procedure may be executed in addition to or in lieu of one or more modeling procedures selected by the search engine 110 . In particular, in scenarios where the intuition and experience of the data analyst indicates that the modeling system 100 has not accurately estimated the appropriateness of the modeling procedure for the predictive problem, allowing the user to select a modeling procedure for execution, the predictive modeling system ( 100) can be improved.

In some embodiments, the search engine 110 does not select one or more other modeling procedures P0...PN, even if all of the modeling procedures P0...PN are determined to be appropriate for the prediction problem in question. The granularity of the search space evaluation can be controlled by selecting a modeling procedure P0 that represents (eg, similar) procedure P1...PN. Also, the search engine 110 may treat the result of executing the selected modeling procedure P0 as representing the result of executing the modeling procedure P1...PN. This coarse-grained approach to evaluating the search space can save processing resources, especially when applied during the initial stage of the evaluation of the search space. If the search engine 110 later determines that the modeling procedure P0 is in the middle of the most appropriate modeling procedure for the prediction problem, a refined evaluation of the relevant part of the search space is achieved by executing and evaluating the similar modeling procedure P1...PN. can be performed.

Referring to FIG. 3 , at step 330 of method 300 , a resource allocation schedule may be generated. The resource allocation schedule may allocate processing resources for the execution of the selected modeling procedure. In some embodiments, the resource allocation schedule allocates processing resources to the modeling procedure based on the determined suitability of the modeling procedure for the prediction problem in question. In some embodiments, the discovery engine 110 sends the resource allocation schedule to one or more processing nodes with instructions for executing the selected modeling procedure according to the resource allocation schedule.

The allocated processing resources include time resources (eg, execution cycles of one or more processing nodes, execution time of one or more processing nodes, etc.), physical resources (eg, multiple processing nodes, a significant amount of machine-readable storage). (eg, memory and/or secondary storage), and/or other allocable processing resources. In some embodiments, the allocated processing resources may be processing resources of distributed computing systems and/or cloud-based computing systems. In some embodiments, costs may arise when processing resources are allocated and/or used (eg, a fee may be collected by the operator of the data center in exchange for using the resources of the data center).

As described above, the resource allocation schedule may allocate processing resources to the modeling procedure based on the adequacy of the modeling procedure for the prediction problem in question. For example, a resource allocation schedule may allocate more processing resources to modeling procedures with higher predictive relevance for prediction problems and less processing resources to modeling procedures with lower predictive relevance for prediction problems. Therefore, more promising modeling procedures benefit from a larger fraction of the limited processing resources. As another example, a resource allocation schedule allocates processing resources sufficient to process a larger data set to a modeling procedure with a higher predicted relevance, and allocates processing resources sufficient to process a smaller data set to a lower predicted relevance. can be assigned to a modeling procedure with

As another example, the resource allocation schedule may schedule the execution of a modeling procedure with a higher predicted relevance prior to the execution of a modeling procedure with a lower predicted relevance, which also dedicates more processing resources to a more promising modeling procedure. It can have the effect of allocating. In some embodiments, the results of executing the modeling procedure may be presented to the user via the user interface 120 when the results become available. In such an embodiment, scheduling a modeling procedure with a higher predicted relevance to be run before a modeling procedure with a lower predicted relevance may provide the user with additional important information about the evaluation of the search space at an early stage of evaluation. and thereby facilitates coordinating rapid user-driven adjustments to the search plan. For example, based on preliminary results, a user may determine that one or more modeling procedures that were expected to perform very well actually perform very poorly. The user may investigate the cause of the poor performance and determine, for example, that the poor performance is due to an error in the preparation of the data set. Thereafter, the user can correct the error and resume execution of the modeling procedure affected by the error.

In some embodiments, the resource allocation schedule may allocate processing resources to the modeling procedure based at least in part on resource utilization characteristics and/or parallelism characteristics of the modeling procedure. As described above, a template corresponding to a modeling procedure may include metadata related to estimating how efficiently the modeling procedure will be executed on a distributed computing infrastructure. In some embodiments, this metadata includes an indication of the resource usage characteristics of the modeling procedure (eg, processing resources needed to train and/or test the modeling procedure for a data set of a given size). In some embodiments, this metadata includes an indication of the parallel nature of the modeling procedure (eg, the degree to which the modeling procedure may be executed in parallel on a plurality of processing nodes). Using resource utilization characteristics and/or parallelism characteristics of a modeling procedure to determine a resource allocation schedule may facilitate efficiently allocating processing resources to a modeling procedure.

In some embodiments, the resource allocation schedule may allocate a specified amount of processing resources for the execution of the modeling procedure. The amount of allocable processing resources may be specified in the processing resource budget, which may be provided by the user or obtained from other suitable sources. The processing resource budget depends on the processing resources used to execute the modeling procedure (eg, the amount of time used, the number of processing nodes used, the cost of using the data center or cloud-based processing resources, etc.). restrictions may be imposed on In some embodiments, the processing resource budget may impose limits on the total processing resources used in the process of generating a predictive model for a particular predictive problem.

3 , in step 340 of method 300 , a result of executing the selected modeling procedure according to the resource allocation schedule may be received. These results may include one or more predictive models generated by the modeling procedures executed. In some embodiments, the predictive model received at step 340 is applied to the data set(s) associated with the predictive problem, as execution of the modeling procedure may include fitting the predictive model to one or more data sets associated with the predictive problem. customized Fitting the predictive model to the data set(s) of the predictive problem includes tuning one or more high-parameters of the predictive modeling procedure that generates the predictive model, tuning one or more parameters of the generated predictive model, and/or other Appropriate model-fitting steps may be included.

In some embodiments, the results received at step 340 include an evaluation (eg, a score) of the model's performance on the prediction problem. This assessment may be obtained by testing the predictive model against the data set(s) associated with the predictive problem. In some embodiments, testing the predictive model includes mutually validating the model using different folds of the training data set associated with the predictive problem. In some embodiments, executing the modeling procedure comprises testing the generated model. In some embodiments, testing of the generated model is performed independently of execution of the modeling procedure.

The model may be tested according to an appropriate testing technique and scored according to an appropriate scoring metric (eg, an objective function). Different scoring metrics measure modeling accuracy (e.g., the rate at which the model correctly predicts the outcome of a prediction problem), the false positive rate (e.g., the rate at which the model incorrectly predicts a "positive" outcome), and the false negative rate. Different weights can be given to different aspects of the performance of a predictive model, including but not limited to (e.g., the rate at which the model incorrectly predicts a "negative" outcome), positive predictive values, negative predictive values, sensitivity, specificity, etc. have. A user may select a standard scoring metric (eg, goodness-of-fitness) from a set of options presented via user interface 120 , or a specific custom scoring metric (eg, a custom objective function) via user interface 120 . fit), R-square, etc.). The search engine 110 may score the performance of the predictive model using user-selected or user-specific scoring metrics.

3 , at step 350 of method 300 , a predictive model may be selected for a predictive problem based on an evaluation (eg, a score) of the generated predictive model. The spatial search engine 110 may use any suitable criteria to select a predictive model for a predictive problem. In some embodiments, spatial search engine 110 may select the model with the highest score, or any model with a score above a threshold score, or any model with a score within a certain range of the highest score. In some embodiments, the score of a predictive model may be the only factor considered by the spatial search engine 110 when selecting a predictive model for a predictive problem. Other factors considered by the spatial search engine may include, but are not limited to, the complexity of the predictive model, computational requirements of the predictive model, and the like.

In some embodiments, selecting a predictive model for a predictive problem may include iteratively selecting a subset of the predictive models and training the selected predictive model on a larger or different portion of the data set. This iterative process may continue until a predictive model is selected for the predictive problem or until the processing resources budgeted for generating the predictive model are exhausted.

Selecting a subset of predictive models includes selecting the portion of the predictive model with the highest score, selecting all models with scores above a threshold score, and scoring within a certain range of scores of the highest-scoring models. selecting all models with, or selecting any other suitable group of models. In some embodiments, selecting a subset of predictive models may be similar to selecting a subset of a predictive modeling procedure, as described above with reference to step 320 of method 300 . Accordingly, the details of selecting a subset of predictive models are not discussed herein.

Training the selected predictive model may include generating a resource allocation schedule that allocates processing resources of the processing node for training of the selected model. The allocation of processing resources may be determined based, at least in part, on the adequacy of modeling techniques used to generate the selected model and/or the score of the selected model relative to other samples of the data set. Training the selected predictive model includes sending instructions to a processing node to fit the selected predictive model to a particular portion of a data set, and receiving the results of a training process comprising the customized model and/or scores of the customized model. may include more. In some embodiments, training the selected predictive model may be similar to executing the selected predictive modeling procedure, as described above with reference to steps 320 - 330 of method 300 . Accordingly, the details of training the selected predictive model are not discussed herein.

In some embodiments, steps 330 and 340 may be performed iteratively until a predictive model is selected for the predictive problem or until the processing resources budgeted for generating the predictive model are exhausted. At the end of each iteration, the adequacy of the predictive modeling procedure for the predictive problem may be re-determined based at least in part on the results of executing the modeling procedure, and a new set of predictive modeling procedures may be selected for execution during the next iteration. .

In some embodiments, the number of modeling procedures executed in iterations of steps 330 and 340 tends to decrease as the number of iterations increases, and the amount of data used to train and/or test the generated model decreases as the number of iterations increases. It tends to increase as the number of times increases. Thus, initial iterations can “cast a wide net” by running a relatively large number of modeling procedures on a relatively small data set, with subsequent iterations of the most promising modeling procedures identified during the initial iterations. More stringent tests can be performed. Alternatively or additionally, an initial iteration may implement a coarse-grained evaluation of the search space, and subsequent iterations may implement a fine-grained evaluation of the portion of the search space determined to be the most promising. .

In some embodiments, method 300 includes one or more steps not shown in FIG. 3 . Additional steps of method 300 include processing the data set associated with the predictive problem, blending two or more predictive models to form a blended predictive model, and/or tuning the selected predictive model for the predictive problem. may include Some embodiments of these steps are described in more detail below.

Method 300 may include processing a data set associated with a prediction problem. In some embodiments, processing the data set of the prediction problem comprises characterizing the data set. Characterization of the data set is to identify missing data (e.g., the data set contains features that are strongly correlated with the target, but the values of the features are not available as inputs to the prediction problem under conditions imposed by the prediction problem. scenarios), detecting missing observations, detecting missing variable values, identifying outlining variable values, and/or identifying variables likely to have significant predictive values (“significant variables”). However, this may include identifying potential problems with data sets that contain them.

In some embodiments, processing the data set of the prediction problem comprises applying feature engineering to the data set. Applying feature engineering to a data set involves combining two or more features, replacing constituent features with the combined features, extracting different aspects of date/time variables (e.g., time and seasonal information) as separate variables, and , normalizing variable values, and filling in missing variable values.

Method 300 may include mixing two or more predictive models to form a blended predictive model. The blending step may be performed iteratively in connection with implementing the predictive modeling technique and evaluating the generated predictive model. In some embodiments, the mixing step may be performed only as part of the run/evaluation iteration (eg, in iterations after a plurality of promising predictive models have been generated).

Two or more models can be mixed by combining the outputs of the constituent models. In some embodiments, the blended model may include a weighted linear combination of the outputs of the constituent models. A mixed predictive model may perform better than a constituent predictive model, especially when the other constituent models are complementary. For example, when a constitutive model tends to perform well on different parts of the data set of a prediction problem, when a mixture of models performs well on different (e.g., similar) prediction problems, creating a model Mixed models can be expected to perform well, such as when the model techniques used are dissimilar (eg, one model is a linear model and the other is a tree model). In some embodiments, the constituent models to be blended are identified by the user (eg, based on the user's intuition and experience).

Method 300 may include tuning a selected predictive model for a predictive problem. In some cases, the placement engine 140 provides the user with source code implementing the predictive model, thereby allowing the user to tune the predictive model. However, disclosing the source code of predictive models may not be desirable in some cases (eg, where predictive modeling techniques or predictive models contain proprietary functions or information). To allow a user to tune a predictive model without exposing the model's source code, the placement engine 140 provides a human readable method for tuning parameters of the model based on a representation (eg, a mathematical representation) of the predictive model. Rules can be configured and human readable rules can be provided to users. The user can then tune the parameters of the model using human readable rules without access to the model's source code. Accordingly, the predictive modeling system 100 can support the evaluation and tuning of proprietary predictive modeling techniques without exposing the source code for the proprietary modeling techniques to end users.

In some embodiments, a machine-executable template corresponding to a predictive modeling procedure may include efficiency-enhancing features to reduce redundant operations. This efficiency-enhancing feature can be particularly valuable when relatively small amounts of processing resources are budgeted for exploring the search space and generating predictive models. As described above, a machine-executable template may store a unique ID for a corresponding modeling element (eg, a skill, task, or sub-task). In addition, the predictive modeling system 100 may assign a unique ID to the data set sample S. In some embodiments, when the machine-executable template T is executed on data set sample S, the template stores its modeling element ID, data set/sample ID, and results of executing the template on data samples accessible to other templates. Store them in structures (eg tables, caches, hashes, etc.). When template T is called on data set sample S, the template checks the storage structure to determine whether the result of executing the template on that data set sample is already stored. If stored, rather than re-processing the data set samples to get the same result, the template simply retrieves the corresponding result from the storage structure, returns the result, and exits. The storage structure may persist within individual iterations of the loop in which the modeling procedure is executed, across multiple iterations of the procedure-execution loop, or across multiple search space searches. Since many tasks and sub-tasks are shared by different modeling techniques, the computational savings achieved through these efficiency-enhancing features can be significant, and the method 300 often avoids running different modeling techniques on the same data set. include

4 illustrates a flow diagram of a method 400 for selecting a predictive model for a predictive problem in accordance with some embodiments. Method 300 may be implemented by example of method 400 .

In the example of FIG. 4 , the spatial search engine 110 uses the modeling method library 212 , the modeling technique library 130 , and the modeling task library 232 to provide a spatial list of modeling techniques available for solutions to the predictive modeling problem. explore Initially, the user may select a modeling method from the library 212 , or the spatial search engine 110 may automatically select a default modeling method. Available modeling methods include the selection of modeling techniques based on the application of deductive rules, the selection of modeling techniques based on the performance of similar modeling techniques for similar predictive problems, the selection of modeling techniques based on the output of meta-machine-learning models, and the modeling described above. any combination of techniques, or other suitable modeling techniques.

At step 402 of method 400, search engine 110 prompts the user to select a data set for the predictive modeling problem to be solved. Users can select from previously loaded data sets or create new data sets from files or commands for retrieving data from other information systems. In the case of files, the search engine 110 may include one or more, including but not limited to, comma-separated values, tab-delimited, cXtensible Markup Language (XML), JavaScript Object Notation, native database files, and the like. format can be supported. In the case of instructions, the user can specify the type of information system, its network address, access credentials, a reference to a subset of data within each system, and rules for mapping the target data schema to the desired data set schema. . Such information systems may include, but are not limited to, databases, data warehouses, data integration services, distributed applications, web services, and the like.

At step 404 of method 400, search engine 110 loads data (eg, by reading a specified file or accessing a specified information system). Internally, the search engine 110 may construct a two-dimensional matrix with features on one axis and observations on the other axis. Conceptually, each column of the matrix may correspond to a variable, and each row of the matrix may correspond to an observation. Search engine 110 may include metadata obtained from the original source (eg, explicitly specified data types) and/or metadata generated during the loading process (eg, explicit data types of variables; ordinal, variables); You can attach relevant metadata to the variable, including whether it appears to be a radix or interpreted type, etc.

At step 406 of method 400, search engine 110 prompts the user to identify which of the variables are targets and/or which are features. In some embodiments, search engine 110 also provides metrics of model performance used to score models to users (eg, by statistical learning algorithms implemented by search engine 110 , in terms of statistical optimization techniques). We urge you to identify the metrics of model performance that are optimized in

At step 408 of method 400, search engine 110 evaluates the data set. This evaluation may include calculating characteristics of the data set. In some embodiments, this assessment includes performing an analysis of the data set, which may help the user better understand the prediction problem. Such analysis involves applying one or more algorithms to identify the variable in question (e.g., variables with outliers or inliers), determining variable importance, determining variable effects, and effect hotspots ( hotspots).

Analysis of the data set may be performed using any suitable technique. Variable importance, which measures the degree of importance each feature has in predicting its target, is a "gradient boosted tree", Breiman and Cutler's "Random Forest", "alternating condition expectation" and and/or analyzed using other suitable techniques. Variable effects that measure the direction and magnitude of the effect of a feature on a target may be analyzed using "normalized regression", "logistic regression" and/or other suitable techniques. Effect hotspots that identify the extent to which a feature provides the most information when predicting a target may be analyzed using the "RuleFit" algorithm and/or other suitable techniques.

In some embodiments, in addition to evaluating the importance of features included in the original data set, the evaluation performed at step 408 of method 400 includes feature creation. Feature creation techniques may include interpreting logical types of variables in a data set and applying various transformations to the variables to create additional features. Examples of transformations include, but are not limited to, polynomial and logarithmic transformations for numeric features. For interpreted variables (e.g., date, time, currency, unit of measure, percentage, and location coordinates), an example of a transformation is to convert a date string to successive times to test each aspect of the date for predictive power. including but not limited to parsing into variables, day, month, and season.

Systematic transformation of numerical and/or interpreted variables that precede systematic testing of potential predictive modeling techniques enables predictive modeling system 100 to further search the potential model space and achieve more precise predictions. For "date/time" for example, separating temporal and seasonal information into separate features can be very useful as these individual features often exhibit very different relationships to the target variable.

Generating derived features by interpreting and transforming the original features can increase the dimensionality of the original data set. The predictive modeling system 100 may apply a dimensionality reduction technique capable of offsetting an increase in the dimension of the data set. However, some modeling techniques are more dimensionally sensitive than others. Also, other dimensionality reduction techniques tend to work better with some modeling techniques than others. In some embodiments, the predictive modeling system 100 maintains metadata describing these interactions. The system 100 may systematically evaluate various combinations of dimensionality reduction techniques and modeling techniques that prioritize the combinations with the highest probability of success indicated by the metadata. System 100 may further update this metadata based on the empirical performance of combinations over time and may incorporate new dimensionality reduction techniques as they are discovered.

At step 410 of method 400, predictive modeling system 100 provides the user with the results of data set evaluation (eg, results of data set analysis, characteristics of data sets, and/or results of data set transformations). present. In some embodiments, the results of the data set evaluation are presented via user interface 120 (eg, using graphs and/or tables).

At step 412 of method 400 , the user may refine the data set (eg, based on the results of the data set evaluation). This refinement handles missing values or outliers for one or more features, changes the type of interpreted variables, changes the transformation under consideration, removes features from consideration, edits specific values directly, functions It can include selection methods for transforming features using , combining the values of features using formulas, adding entirely new features to the data set, and so on.

Steps 402 - 412 of method 400 may represent one embodiment of processing a data set of a prediction problem, as described above with respect to some embodiments of method 300 .

At step 414 of method 400 , search space engine 100 may load modeling techniques available from modeling technique library 130 . The determination of which modeling techniques are available may depend on the chosen modeling method. In some embodiments, the loading of modeling techniques may occur in parallel with one or more steps 402 - 412 of method 400 .

At step 416 of method 400, the user instructs the search engine 110 to initiate a search for modeling solutions in either a manual mode or an automatic mode. In automatic mode, search engine 110 partitions the data set using a default sampling algorithm (step 418) and prioritizes modeling techniques using a default priority algorithm (step 420). Prioritizing the modeling techniques may include determining the appropriateness of the modeling technique for the prediction problem, and selecting at least a subset of the modeling techniques for implementation based on the determined appropriateness.

In the manual mode, the search engine 110 presents data partitioning (step 422) and prioritization of modeling techniques (step 424). The user may accept the presented data partition or specify a custom partition (step 426). Likewise, the user may accept the suggested priorities of modeling techniques or specify a custom priority for modeling techniques (step 428). In some embodiments, a user may select one or more modeling techniques (eg, using modeling technique builder 220 and/or modeling job builder 230 ) before search engine 110 starts executing the modeling technique. can be modified (step 430).

To facilitate cross-validation, predictive modeling system 100 may partition (or suggest partitioning of) the data set into K “folds”. Mutual validation involves fitting the predictive model K times to a partitioned data set such that during each fit a different fold serves as the test set and the remaining folds serve as the training set. Mutual validation can generate useful information about how the accuracy of a predictive model changes with different training data. At steps 418 and 422, the predictive modeling system may partition the data set into K folds, where the number of folds K is the default parameter. In step 426, the user may change the number of folds K or cancel the use of mutual validation altogether.

To facilitate rigorous testing of predictive models, predictive modeling system 100 may partition (or suggest partitioning of) a data set into a training set and a "holdout" test set. In some embodiments, the training set is further divided into K folds for mutual validation. The training set can be used to train and evaluate the predictive model, while the holdout test set can be strictly preserved to test the predictive model. In some embodiments, predictive modeling system 100 provides holdout tests for testing (not training) by making the holdout tests inaccessible until a user with specified privileges and/or credentials releases them. The use of the set can be performed strongly. At steps 418 and 422 , the predictive modeling system 100 may partition the data set such that a default percentage of the data set is reserved for the holdout set. In step 426, the user may change the percentage of data sets reserved for the holdout set, or cancel use of the holdout set altogether.

In some embodiments, the predictive modeling system 100 partitions the data set to facilitate efficient use of computing resources during evaluation of the modeling search space. For example, the predictive modeling system 100 may split the mutually validating folds of a data set into smaller samples. Reducing the size of the data samples to which the predictive model is tailored can reduce the amount of computing resources required to evaluate the relative performance of different different modeling techniques. In some embodiments, smaller samples may be generated by taking random samples of the fold's data. Likewise, reducing the size of the data samples to which the predictive model is customized may reduce the amount of computing resources required to tune the parameters of the predictive model or hyper-parameters of the modeling technique. Hyper-parameters include setting parameters in modeling techniques that can affect the speed, efficiency and/or accuracy of the model fitting process. Examples of hyper-parameters include, but are not limited to, the penalty parameter of the elastic-net model, the number of trees in the gradient boosted tree model, and the number of neighbors in the nearest neighbor model.

At steps 432-458 of method 400, the selected modeling technique may be implemented using the segmented data to evaluate the search space. This step is described in more detail below. For convenience, some aspects of the evaluation of the search space related to data partitioning are described in the following paragraphs.

Tuning hyper-parameters using sample data comprising a test set of mutual validation folds can lead to model over-fitting, thereby making comparisons of the performance of different models unreliable. Using a “specific approach” can help avoid this problem, and can provide several other important advantages. Accordingly, some embodiments of search engine 110 implement a “nested mutual validation” technique, whereby two loops of k-fold cross validation are applied. The outer loop provides a set of tests for both comparing a given model to other models and correcting each model's predictions on future samples. The inner loop provides both a test set for tuning the hyper-parameters of a given model and a training set for the derived features.

Additionally, mutually validated predictions generated in inner loops can facilitate mixing techniques that combine multiple different models. In some embodiments, the input to the mixer is a prediction from a model outside the sample. Using predictions from in-sample models can result in over-fitting when used with some blending algorithms. Without a well-defined process for consistently applying nested mutual validation, even the most experienced users can omit steps or implement them incorrectly. Thus, application of the double loop of k-fold cross validation enables the predictive modeling system 100 to simultaneously achieve the following five important objectives: (1) tuning a complex model with multiple hyper-parameters (2) ) develop informed derived features, (3) tune a mixture of two or more models, (4) calibrate the predictions of single and/or mixed models, and (5) pure non-contact capable of accurately comparing different models. Keep the test set.

At step 432 of method 400, search engine 110 generates a resource allocation schedule for execution of an initial set of selected modeling techniques. The allocation of resources indicated by the resource allocation schedule may be determined based on priorities of modeling techniques, divided data samples, and available computational resources. In some embodiments, search engine 110 greedily allocates resources to selected modeling techniques (eg, in turn assigns computational resources to highest-priority modeling techniques that have not yet been implemented).

At step 434 of method 400, search engine 110 initiates execution of the modeling technique according to the resource allocation schedule. In some embodiments, executing the set of modeling techniques may include training one or more models on the same data samples extracted from the data set.

At step 436 of method 400, search engine 110 monitors the execution status of the modeling technique. When the modeling technique has finished running, the search engine 110 collects results, which may include metrics of model fit and/or customized models to corresponding data samples (step 438). These metrics include, but are not limited to, Gini coefficients, r-squared, residual mean squared error, any variations thereof, and the like, any metric that can be extracted from the underlying software component performing customization. may include

At step 440 of method 400, search engine 110 removes the worst-performing modeling technique from consideration (eg, based on the performance of the model generated according to model fit metrics). The search engine 110 removes those that do not generate a model that satisfies the minimum threshold of the model fitting metric, removes all modeling techniques except for those that have generated the current model from the top part of all generated models, or removes the specific model of the top model. Appropriate techniques can be used to determine which modeling techniques to remove, including, but not limited to, removing any modeling techniques that do not create models that are within scope. In some embodiments, different procedures may be used to eliminate modeling techniques at different stages of evaluation. In some embodiments, a user may be allowed to specify different removal techniques for different modeling problems. In some embodiments, users may be allowed to build and use custom removal techniques. In some embodiments, meta-statistical-learning techniques may be used to select among and/or adjust parameters of removal-techniques.

As the search engine 110 calculates model performance and removes modeling techniques from consideration, the predictive modeling system 100 may present to the user the progress of the search space evaluation via the user interface 120 (step (step ) 442)). In some embodiments, at step 444, the search engine 110 allows the user to modify the process of evaluating the search space based on the progress of the search space evaluation, the user's expertise and/or other suitable information. . If the user specifies modifications to the search space evaluation process, the spatial evaluation engine 110 reallocates processing resources accordingly (eg, determines which jobs are affected, and puts them into the scheduling queue ( (moved in or removed from queue). Other jobs continue processing as before.

Users can modify the search space evaluation process in many different ways. For example, a user may reduce the priority of some modeling techniques or remove some modeling techniques from consideration, even if the performance of the model generated for the selected metric was good. As another example, even when the performance of the generated modeling was inferior, the user may increase the priority of some modeling techniques or select some modeling techniques as considerations. As another example, as another example, a user may prioritize evaluation of a particular model or execution of a specified modeling technique over additional data samples. As another example, a user may modify one or more modeling techniques and select the modified techniques for consideration. As another example, a user may change features used to customize a model or train modeling techniques (eg, by adding features, removing features, or selecting different features). This change can be useful if the results indicate that the size of the features needs normalization or that some of the features are "missing data".

In some embodiments, steps 432-444 may be performed iteratively. Modeling techniques that are not removed (eg, by the system at step 440 or by the user at step 444) survive another iteration. Based on the performance of the model generated in the previous iteration (or iterations), the search engine 110 adjusts the priority of the corresponding modeling technique and allocates processing resources to the modeling technique accordingly. When computational resources become available, the engine uses the available resources to launch model-description-execution tasks based on the updated priorities.

In some embodiments, at step 432 the search engine 110 “blends” the plurality of models using different mathematical combinations (eg, using stepwise model selection for inclusion in the mixer). to create a new model. In some embodiments, the predictive modeling system 100 provides a modular framework that allows users to plug in their own automatic blending techniques. In some embodiments, the predictive modeling system 100 allows a user to manually specify different model mixes.

In some embodiments, predictive modeling system 100 may provide one or more advantages when developing blended predictive models. First, blending can be more effective when a wide variety of candidate models are available for blending. Also, blending can be more effective when differences between candidate models correspond to major differences in approaches, such as among linear models, tree-based models, support vector machines, and nearest neighbor classification, rather than simply minor changes in the algorithm. The predictive modeling system 100 can deliver a substantial lead start by automatically generating a wide variety of models and maintaining metadata that describes how candidate models differ. The predictive modeling system 100 may also provide a framework that allows any model to be incorporated into a mixed model, for example, by automatically normalizing the scale of a variable across candidate models. This framework can further increase versatility by allowing users to easily add their own customized or independently generated models to automatically generated models.

In addition to increasing the variety of candidate models available for blending, predictive modeling system 100 also provides a number of user interface features and analysis features that can result in good blending. First, the user interface 120 can provide interactive model comparisons, including candidate model fit and some other alternative measures of graphics, such as dual lift charts, so that the user can provide accurate and complementary measures for blending. model can be easily identified. Second, the modeling system 100 provides the user with the option to select specific candidate models and blending techniques, or to automatically customize some or all blending techniques from a library of modeling techniques using some or all of the candidate models. The nested mutual validation framework then enforces the condition that the data used to rank each blend model is not used when tuning the mixer itself or the hyper-parameters of its component models. This discipline can provide users with a more accurate comparison of the performance of alternative mixers. In some embodiments, the modeling system 100 implements the processing of the mixed model in parallel, such that the computation time for the mixed model approaches the computation time of its slowest component model.

Returning to FIG. 4 , at step 446 of method 400 , user interface 120 presents the final result to the user. Based on this representation, the user refines the data set (eg, by returning to step 412 ) and adjusts the allocation of resources to the execution modeling technique (eg, by returning to step 444 ). and modify one or more modeling techniques to improve accuracy (e.g., by returning to step 430), modifying the data set (e.g., by returning to step 402), etc. can do.

At step 448 of method 400 , rather than resuming search space evaluation or a portion thereof, the user may select one or more top predictive model candidates. In step 450 , the predictive modeling system 100 may present the results of a holdout test for the selected predictive model candidate(s). The holdout test results can provide a final gauge on how these candidates are compared. In some embodiments, only users with appropriate privileges can release holdout test results. Preventing release of holdout test results until a candidate predictive model is selected may facilitate an unbiased evaluation of performance. However, the search engine 110 may actually compute the holdout test results during the modeling task execution process (eg, steps 432-444) as long as the results remain hidden until after a candidate predictive model is selected.

user interface

Returning to FIG. 1 , the user interface 120 may provide tools for monitoring and/or guiding the search of the predictive modeling space. These tools provide insight into data sets of prediction problems and/or insights into search results (e.g., by highlighting problematic variables in data sets, identifying relationships between variables in data sets, etc.) can do. In some embodiments, the data analyst may use the interface to guide the search, for example, by specifying metrics used to evaluate and compare modeling solutions, specifying criteria for recognizing appropriate modeling solutions, etc. have. Accordingly, the user interface may be used by the analyst to improve their own productivity and/or to improve the performance of the search engine 110 . In some embodiments, the user interface 120 presents the results of the search in real time and allows the user to guide the search in real time (eg, to adjust the scope of the search or allocation of resources during evaluation of different modeling solutions). make it possible In some embodiments, user interface 120 provides tools for coordinating the efforts of multiple data analysts to be effective on the same prediction problem and/or related prediction problems.

In some embodiments, user interface 120 provides tools for developing machine-executable templates for library 130 of modeling techniques. A system user can use these tools to modify an existing template, create a new template, or remove a template from the library 130 . In this way, system users may update library 130 to reflect advances in predictive modeling research and/or to include proprietary predictive modeling techniques.

User interface 120 allows users to manage multiple modeling projects within an organization, create and modify elements of a modeling methodology hierarchy, perform comprehensive searches for accurate predictive models, and access data sets and model results. It may include various interface components that allow deploying the finished model to gain insights and/or make predictions on new data.

In some embodiments, user interface 120 distinguishes four types of users: administrators, technical developers, model builders, and observers. Administrators can control the allocation of people and computing resources to the project. Technology developers can create and modify modeling techniques and component tasks. The model builder focuses on finding good models, although you can make some adjustments to your skills and work. An observer may see certain aspects of the project progress and modeling results, but may be prevented from making any changes to the data or from initiating any model-building. An individual may perform more than one role in a particular project or multiple projects.

A user acting as an administrator can access the project management component of the user interface 120 to set project parameters, assign project responsibilities to the user, and allocate computing resources to the project. In some embodiments, an administrator may use a project management component to organize multiple projects into groups or hierarchies. All projects within a group can inherit the group's settings. In a hierarchy, all children of a project can inherit the settings of the project. In some embodiments, users with sufficient privileges may override inherited settings. In some embodiments, users with sufficient privileges may further divide the settings into different sections, so that only users with corresponding privileges can change them. In some cases, an administrator may grant access to specific resources orthogonal to the project's organization. For example, certain skills and tasks may be made available to any project, unless expressly prohibited. Others may be prohibited unless expressly permitted. In addition, some resources can be assigned on a per-user basis, so that a project can access resources only when the user who owns those rights is assigned to a specific project.

In user management, an administrator can control the groups, permitted roles, and system-level privileges of all users permitted to the system. In some embodiments, administrators can add users to the system by adding users to corresponding groups and issuing them some form of access credentials. In some embodiments, the user interface 120 provides different kinds of credentials including, but not limited to, a username and password, a unified authentication framework (eg, OAuth), a hardware token (eg, a smart card), and the like. can support

Once granted, the administrator can specify that a particular user has a assumed default role for any project. For example, certain users can be designated as observers, unless specifically authorized for other roles by the administrator for a particular project. Other users may serve as technical developers for all projects unless specifically excluded by the administrator, and other users may serve only as technical developers for a specific group of projects or branches of the project hierarchy. In addition to the default roles, administrators can assign additional, more specific privileges to users at the system level. For example, an administrator may grant access to certain types of computing resources, some technical developers and model builders may access certain features within a builder; Some model builders may be authorized to start a new project, consume more computational resources than a given level, or invite new users to a project they do not own.

In some embodiments, administrators may assign access, permissions, and responsibilities at the project level. Access may include the ability to access any information within a particular project. Privileges may include the ability to perform specific actions on a project. Access and privileges can override system-level privileges or provide more granular control. As an example of the former, a user who typically has full builder rights may be limited to partial builder rights for a specific project. As an example of the latter, certain users may be restricted from loading new data into existing projects. Responsibilities may include action items a user is expected to complete for a project.

Users acting as developers can access the Builder area of the interface to create and modify modeling methods, techniques, and tasks. As noted above, each builder may present one or more tools with different types of user interfaces that perform corresponding logical operations. In some embodiments, user interface 120 may enable a developer to use a “properties” sheet to edit metadata attached to a description. Techniques may also have tuning parameters that correspond to variables for a particular task. Developers can publish these tuning parameters to a skill-level property sheet, specifying default values and whether model builders can override these defaults.

In some embodiments, user interface 120 specifies a hierarchically directed graph of tasks, filters outputs, transforms outputs, splits outputs, combines inputs, and , can provide graphical flowchart tools for iterating over sub-graphs, and the like. In some embodiments, user interface 120 may provide facilities for creating a wrapper around existing software for implementing leaf-level tasks, including properties that may be set for each task. .

In some embodiments, user interface 120 may provide advanced developers with built-in access to an interactive development environment (IDE) for implementing leaf-level tasks. Developers can alternatively code components in an external environment and wrap the code as leaf-level operations, but it may be more convenient to have direct access to these environments. In such an embodiment, the IDE itself can be wrapped in an interface and logically integrated into the task builder. From the user's point of view, the IDE can run on the same computational infrastructure as the Job Builder within the same interface framework. These capabilities allow advanced developers to iterate their development and revision techniques more quickly. Some embodiments may further provide a code collaboration feature to facilitate coordination among multiple developers concurrently programming the same leaf-level task.

Model builders can leverage the skills generated by developers to build predictive models for their specific data sets. Different model builders may have different levels of experience and thus may require different support than the user interface. For relatively new users, the user interface 120 may present as automated processes as possible, but may still provide the user with the ability to explore options, thereby learning more about predictive modeling. For intermediate users, the user interface 120 quickly evaluates how easily a particular problem will be solved, compares how existing predictive models stack up against what the predictive modeling system 100 can automatically generate, and , can present information to facilitate obtaining accelerated start-up for complex projects that will ultimately benefit from substantial manual tuning. For advanced users, the user interface 120 provides for the extraction of a few extra decimal places of accuracy for existing predictive models, a rapid evaluation of the applicability of the new technique to the problem being worked on, and the Can facilitate the development of skills for an entire class of possible problems. By capturing knowledge about advanced users, some embodiments facilitate the dissemination of that knowledge across the rest of the organization.

To support this breadth of user needs, some embodiments of user interface 120 provide a sequence of interface tools that reflect the model building process. Additionally, each tool can provide a variety of features, from basic to advanced. A first step in the model building process may include loading and preparing a data set. As mentioned above, a user can specify how to upload a file or access data from an online system. In terms of modeling project groups or hierarchies, users can also specify which parts of the parent data set to use for the current project and which parts to add.

For a basic user, the predictive modeling system 100 can immediately proceed to building a model after the data set has been specified, with unparsable data, too few observations to predict good results, too few observations to run in a reasonable amount of time. It stops only when the user interface 120 flags an embarrassing problem including, but not limited to, many observations, too many missing values, or variables whose variance could lead to an anomalous result. For intermediate users, user interface 120 may facilitate a deeper understanding of data by presenting tables of data set characteristics and graphs of variable importance, variable effects, and effect hotspots. Additionally, the user interface 120 provides visualization tools including, but not limited to, correlation matrices, partial dependence plots, and/or results of unsupervised machine-learning algorithms such as k-means and hierarchical clustering, thereby providing visualization tools for relationships between variables. can facilitate the understanding and visualization of In some embodiments, user interface 120 allows advanced users to create entirely new data set features by specifying formulas to transform existing features or combinations thereof.

Once the data set is loaded, the user can specify the model-fit metrics to be optimized. For a primary user, the predictive modeling system 100 may select a model-fit metric, and the user interface 120 may present a description of the selection. For an intermediate user, user interface 120 may present information to help the user understand trade-offs in selecting different metrics for a particular data set. For advanced users, the user interface 120 allows the user to write formulas (eg, objective functions) based on low-level performance data collected by the search engine 110 , or even custom metric calculation code. You can specify custom metrics by uploading

Once the data set is loaded and model-fit metrics are selected, the user can launch the search engine. For a basic user, the search engine 110 may use default priority settings for modeling techniques, and the user interface 120 provides information about model performance, how far the run has progressed into the data set, and the general consumption of computational resources. It can provide high-level information about For an intermediate user, the user interface 120 may allow the user to specify a subset of techniques to consider and slightly adjust some of the initial priorities. In some embodiments, user interface 120 provides more detailed performance and progress data, allowing intermediate users to make in-flight adjustments as described above. In some embodiments, the user interface 120 provides the intermediate user more intuition and control of computational resource consumption. In some embodiments, user interface 120 provides advanced users with significant (eg, complete) control of the considered technologies and their priorities, all available performance data, and control of critical (eg, complete) resource consumption. can be provided to Some embodiments of user interface 120 may assist users at their corresponding levels, either by providing separate interfaces to different levels of users, or by default "collapsing" more advanced features for less advanced users.

During and after navigation of the search space, the user interface may present information regarding the performance of one or more modeling techniques. Some performance information may be displayed in a table format while other performance information may be displayed in a graphical format. For example, information presented in a table format may include, but is not limited to, comparison of model performance by technology, a portion of evaluated data, technology attributes, or current consumption of computational resources. Information presented in graphical format includes directed graphs of the tasks of the modeling procedure, comparison of model performance across different partitions of data sets, representation of model performance such as receiver operating characteristics and lift charts, predicted versus actual values, and calculations over time. may include, but is not limited to, consumption of resources. User interface 120 may include a modular user interface framework that allows for the easy inclusion of any type of new capability information. Additionally, some embodiments may allow for the display of some type of information for each data segmentation and/or each technology.

As noted above, some embodiments of user interface 120 support the collaboration of multiple users on multiple projects. Across projects, user interface 120 may allow users to share data, modeling tasks, and modeling techniques. Within a project, user interface 120 may allow users to share data, models, and results. In some embodiments, user interface 120 may allow a user to modify properties of a project and use resources allocated to the project. In some embodiments, user interface 120 may allow multiple users to modify project data, add models to the project, and then compare their contributions. In some embodiments, user interface 120 may identify which users made particular changes to the project, when the changes were made, and which project resources the user used.

model placement engine

The model deployment engine 140 provides tools for deploying predictive models in an operational environment. In some embodiments, the model deployment engine 140 monitors the performance of the deployed predictive model and updates performance metadata associated with the modeling technique that generated the deployed model, so that the performance data accurately reflects the performance of the deployed model. do.

Users can deploy customized predictive models when they believe the customized model can warrant field testing or add value. In some embodiments, users and external systems may access the prediction module (eg, at the interface service layer of the predictive modeling system 100 ), specify one or more predictive models to be used, and provide new observations. The prediction module may then return the predictions provided by the model. In some embodiments, administrators may control which users and external systems have access to this prediction module and/or may set usage restrictions, such as the number of predictions allowed per unit time.

For each model, the search engine 110 may store the state of the model after customization and a record of the modeling techniques used to create the model, including coefficients and hyper-parameter values. As each technique is already machine-executable, this value may be sufficient for the execution engine to generate predictions for new observational data. In some embodiments, predictions of the model may be generated by applying the pre-processing and modeling steps described in modeling techniques to each instance of new input data. However, in some cases, it may be possible to speed up future prediction calculations. For example, a customized model can make several independent checks of the values of certain variables. Combining some or all of these checks and then simply referencing them when convenient can reduce the total amount of computation used to generate the prediction. Similarly, several component models of a mixed model may perform the same data transformation. Accordingly, some embodiments can reduce computation time by identifying redundant computations, performing them only once, and referencing the computation results in a component model that uses them.

In some embodiments, the placement engine 140 improves the performance of predictive models by identifying opportunities for parallel processing, whereby the response time to perform each prediction when the underlying hardware can execute multiple instructions in parallel. reduces the While some modeling techniques may describe a sequence of steps sequentially, in practice some steps may be logically independent. By examining the data flow during each step, the placement engine 140 can identify situations of logical independence and then reconstruct the execution of the predictive model so that the independent steps are executed in parallel. Mixed models can present a special class of parallelism, since the construct prediction models can be run in parallel once any common data transformations are complete.

In some embodiments, placement engine 140 may cache the state of the predictive model in memory. Using this approach, successive prediction requests of the same model may not spend time loading the model state. This loading time is a potentially large fraction of the total execution time, as caching can work particularly well when there are many requests for predictions for a relatively small number of observations.

In some embodiments, placement engine 140 may provide implementations of at least two predictive models, service-based and code-based. For service-based prediction, the computation is performed within a distributed computing infrastructure, as described below. The final predictive model may be stored in the data service layer of the distributed computing infrastructure. When a user or external system requests a prediction, it may indicate which model will be used and provide at least one new observation. The prediction module can then load the model from the data service layer or the module's in-memory cache, verify that the submitted observations match the structure of the original data set, and compute a prediction value for each observation. In some implementations, the predictive model may be run on a dedicated pool of cloud workers, thereby facilitating the creation of predictions with low variance response times.

Service-based prediction can occur interactively or through APIs. For interactive prediction, a user can enter a value for a feature for each new observation or upload a file containing data for one or more observations. The user can then receive the prediction directly via user interface 120 or download it as a file. For API prediction, an external system may access the prediction module through a local or remote API, submit one or more observations, and receive a corresponding computed prediction in return.

Some implementations of deployment engine 140 may allow an organization to create one or more miniaturized instances of a distributed computing infrastructure for the purpose of performing service-based prediction. At the interface layer of the distributed computing infrastructure, each such instance may use portions of the monitoring and prediction modules that are accessible by external systems without access to user-related functions. The analytics service layer may not use the technical IDE module and the remaining modules in this layer may be removed and optimized to service prediction requests. The data service layer cannot use user or model-building data management. These standalone prediction instances can be deployed in a parallel pool of cloud resources, distributed to different physical locations, or even downloaded to one or more dedicated machines acting as “prediction appliances”.

To create a dedicated prediction instance, a user may specify a target computing infrastructure, such as, for example, whether it is a set of cloud instances or a set of dedicated hardware. The corresponding module is then provided and can be installed on the target computing infrastructure or packaged for installation. Users can construct instances from a set of initial predictive models or create “blank” instances. After initial installation, users can manage available predictive models by installing new ones or updating existing ones from the main installation.

For code-based prediction, deployment engine 140 may generate source code for calculating predictions based on a particular model, and users may incorporate the source code into other software. When the model is based on a technology whose leaf-level operations are all implemented in the same programming language as requested by the user, the placement engine 140 collects and analyzes the code for the leaf-level operations, thereby providing a source for the predictive model. code can be generated. If the model incorporates code from other languages or the language is different from the user's desired language, the deployment engine 140 may use a more complex approach.

One approach is to use a source-to-source compiler to translate the source code of the leaf-level work into the target language. Another approach is to create a function stub in the target language and then call the object code linked-in to the original language or access an emulator that executes this object code. The former approach may include using a cross-compiler to generate object code specifically for the user's target computing platform. The latter approach may involve the use of an emulator that will run on the user's target platform.

Another approach is to generate an abstract description of a particular model and then compile that description into the target language. To generate abstract descriptions, some embodiments of deployment engine 140 may use meta-models to describe a number of potential pre-processing, model-fitting, and post-processing steps. The batch engine can then extract the specific behavior for the complete model and encode it using the meta-model. In such an embodiment, a compiler for the target programming language may be used to translate the meta-model into the target language. So, if a user wants predictive code in a supported language, the compiler can generate it. For example, in the decision-tree model, the decision of a tree can be abstracted into logical if/then/else constructs that can be directly implemented in a wide variety of programming languages. Likewise, any set of mathematical operations supported in conventional programming languages may be used to implement a linear regression model.

However, disclosing the source code of a predictive model in any language may be undesirable in some cases (eg, where predictive modeling techniques or predictive models contain proprietary functionality or information). Accordingly, the placement engine 140 may transform the predictive model into a set of rules that preserve the predictive function of the predictive model without disclosing the details of its procedures. One approach is to apply an algorithm that generates these rules from a set of hypothetical predictions in which a predictive model is generated in response to hypothetical observations. Some of these algorithms can generate a set of if-then rules (eg RuleFit) for performing predictions. For these algorithms, the placement engine 140 may convert the resulting if-then rule into the target language instead of transforming the original predictive model. An additional advantage of transforming a predictive model into a set of if-then rules is that it targets a set of if-then rules rather than a predictive model with arbitrary control and data flow, since the underlying model of conditional logic in programming languages is more similar. Converting to a programming language is usually easier.

Once the model starts making predictions for new observations, the placement engine 140 can track these predictions, measure their accuracy, and use these results to improve the predictive modeling system 100 . In the case of service-based prediction, each observation and prediction can be stored through a data service layer, as the prediction occurs within the same distributed computing environment as the rest of the system. By providing an identifier for each prediction, some embodiments may allow the user or external software system to submit the actual value as and when recorded. For code-based prediction, some embodiments may include code to store observations and predictions back to an instance of the local system or data service layer. Again, providing an identifier for each prediction can facilitate the collection of model performance data for actual target values as they become available.

Information gathered directly by the placement engine 140 about the accuracy of predictions and/or observations obtained through other channels (e.g., to "refresh" an existing model, or to rebuild some or all of the modeling search space) It can be used to improve the model for prediction problems (to create a model by exploring). New data can be added by improving the model in the same way that the original data was added to create the model, or by submitting target values for data previously used for prediction.

Some models can be refreshed (e.g., refitted) by applying corresponding modeling techniques to new data and combining the resulting new model with existing models, while others apply corresponding modeling techniques to the original and new data. It can be refreshed by applying it to a combination. In some embodiments, when refreshing the model, only some of the model parameters may be recalculated (e.g., to refresh the model more quickly, or because new data provides information that is particularly relevant to a particular parameter). .

Alternatively or additionally, a new model may be created that partially or fully explores the modeling search space with new data included in the data set. A rescan of the search space may be limited to a portion of the search space (eg, limited to modeling techniques that performed well in the original search) or may cover the entire search space. In either case, the initial relevance score for the modeling technique(s) that produced the deployed model(s) may be recalculated to reflect the performance of the deployed model(s) on the prediction problem. Users can choose to exclude some old data to perform a recalculation. Some embodiments of placement engine 140 may keep track of different versions of the same logic model, including which subsets of data were used to train.

In some embodiments, this prediction data may be used to perform post-request analysis of the trend of the input parameters or predictions themselves over time, and to alert the user to potential problems with the input or quality of the model predictions. For example, if an aggregate measure of model performance begins to degrade over time, the system may alert the user to consider refreshing the model or examining whether the input itself is shifting. Such shifts can be caused by temporal changes in specific variables or by drifting of the entire population. In some embodiments, most of this analysis is performed after the prediction request is completed to avoid slowing down the prediction response. However, the system does some validation at prediction time to avoid particularly bad predictions (e.g., if the input is outside the range of values computed as valid given properties of the original training data, modeling technique, and final model fit state). can be done

Post hoc analysis can be important if the user has deployed the model to make extrapolations far beyond the population used for training. For example, although a model has been trained on data from one geographic area, it can be used to make predictions on a population in an entirely different geographic area. Sometimes, this extrapolation to a new population can result in substantially worse model performance than expected. In such a case, the placement engine 140 may automatically refresh the model and extend the original training data by alerting the user and/or re-customizing one or more modeling techniques using the new values.

Advantages of some embodiments

The predictive modeling system 100 can significantly improve the productivity of an analyst at any skill level and/or can significantly increase the accuracy of predictive models achievable with a given amount of resources. Automating procedures reduces the workload, and systematizing processes can be consistent, allowing analysts to spend more time generating unique intuitions. Three typical scenarios show benefits such as outcome prediction, feature prediction, and measurement inference.

Expect results

If an organization can accurately predict outcomes, it can plan more efficiently and improve its behavior. Thus, a common application of machine learning is to develop algorithms that generate predictions. For example, many industries face the problem of cost forecasting on large-scale, time-consuming projects.

In some embodiments, the techniques described herein may be used to anticipate cost overruns (eg, over software costs or over construction costs). For example, the techniques described herein can be applied to the problem of predicting cost overruns as follows:

1. Select the appropriate model fit metric for the response variable type (e.g., numeric or binary, approximately Gaussian or strongly non-Gaussian): predictive modeling system 100 It can recommend metrics based on data characteristics, requiring little skill and effort, but allows the user to make the final choice.

2. Pre-process the data to resolve outliers and missing data values: The predictive modeling system 100 provides a detailed overview of data characteristics, allowing users to develop better situational awareness of modeling problems and potential modeling Allows for more effective evaluation of tasks. Predictive modeling system 100 may include automated procedures for detection and processing of outliers detection and replacement, imputation of missing values, and other data anomalies that require less skill and effort by the user. Procedures for predictive modeling systems to address these challenges can be systematic, leading to more consistent modeling results across methods, data sets, and time than ad hoc data editing procedures.

3. Partition data for modeling and evaluation: The predictive modeling system 100 can automatically partition data into training, validation, and holdout sets. This partitioning may be more flexible than the train and test partitioning used by some data analysts, and may be consistent with widely accepted recommendations from the machine learning community. The use of a partitioning approach that is consistent across methods, data sets and time can make results more easily comparable, enabling more effective allocation of resource allocations in a commercial context.

4. Select model structures, generate derived features, select model tuning parameters, customize models, and evaluate: In some embodiments, predictive modeling system 100 includes decision trees, neural networks, support vectors Many different model types can be customized, including but not limited to machine models, regression models, boosted trees, random forests, deep learning neural networks, and the like. The predictive modeling system 100 may provide an option to automatically construct an ensemble from these component models that exhibit the best individual performance. Exploring a larger space of potential models can improve accuracy. The predictive modeling system can automatically generate various derived features suitable for different data types (eg, Box-Cox transformations, text pre-processing, key components, etc.). Exploring a larger space of potential transformations can improve accuracy. The predictive modeling system 100 can use mutual validation to select the best values for these tuning parameters as part of the model building process, thereby improving the selection of tuning parameters and how the selection of parameters affects the results. You can create an audit trail for The predictive modeling system 100 can customize and evaluate the different model structures considered as part of this automated process, ranking the results in terms of validation set performance.

5. Select the final model: The selection of the final model may be made by the predictive modeling system 100 or the user. In the latter case, the predictive modeling system may, for example, evaluate the performance of a graded validation set for the model, the option of comparing and grading performance by quality measures other than those used in the fitting process, and/or determining the best individual performance. Support can be provided to help users make these decisions, including the opportunity to build ensemble models from the component models they represent.

One important practical aspect of the model development process of a predictive modeling system is that once the initial data set is assembled, all subsequent operations can occur within the same software environment. This aspect represents a significant difference from conventional model-building efforts, which often involve a combination of different software environments. A significant practical disadvantage of this multi-platform analysis approach is the need to convert the results into a common data format that can be shared between different software environments. Often this conversion is done manually or with a custom "one-time" reformatting script. Errors in this process can lead to very serious data distortions. The predictive modeling system 100 can avoid such reformatting and data transmission errors by performing all operations in one software environment. More generally, because it is highly automated, and because it customizes and optimizes many different model structures, the predictive modeling system 100 is substantially faster, more systematic, and thus more easily descriptive and more repeatable routing to the final model. can be Also, as the predictive modeling system 100 explores more different modeling methods and includes possible predictors, the resulting model may be more accurate than that obtained by conventional methods.

property prediction

In many fields, organizations face uncertainty in the outcome of their production processes and want to predict how a given set of conditions will affect the final properties of the output. Therefore, a common application of machine learning is to develop algorithms to predict these properties. For example, concrete is a common building material whose final structural properties can vary significantly from situation to situation. Neither models developed from first principles nor models developed from traditional regression models provide adequate predictive accuracy because of the significant change in concrete properties over time and their dependence on highly variable composition.

In some embodiments, the techniques described herein may be used to predict properties of the outcome of a production process (eg, properties of concrete). For example, the techniques described herein may be applied to the problem of predicting properties of concrete such as:

1. Split the data set into training, validation, and test subsets.

2. Clean the modeling data set: The predictive modeling system 100 can automatically identify missing data, outliers and other important data anomalies, recommend processing strategies, and provide the user the option to accept or reject them. have. This approach may require less skill and effort by the user, and/or may provide more consistent results across methods, data sets and time.

3. Select a response variable and select a first-order fit metric: the user can select a predicted response variable from what is available in the modeling data set. Once the response variables have been selected, the predictive modeling system 100 can recommend compatible custom metrics that the user can accept or ignore. This approach may require little skill and effort by the user. A predictive modeling system is a set of predictive models, including conventional regression models, neural networks, and other machine learning models (eg, random forests, boosted trees, support vector machines), based on response variable types and selected custom metrics. can provide By automatically searching among the space of possible modeling approaches, the predictive modeling system 100 can increase the predictive accuracy of the final model. A default set of model selections can exclude certain model types from consideration, add other model types supported by the predictive modeling system but not part of the default list, or create your own custom model types (e.g. R or Python). (implemented in Python) can be overridden.

4. Generate input features, customize the model, optimize model-specific tuning parameters, and evaluate performance: In some embodiments, feature creation allows scaling for numerical covariates, Box-Cox transforms, key components, etc. may include Tuning parameters for the model can be optimized through mutual validation. Validation set performance measures may be computed and presented for each model along with other overview properties (eg, model parameters for regression models, variable importance measures for boosted trees or random forests).

5. Select the final model: The selection of the final model may be made by the predictive modeling system 100 or the user. In the latter case, the predictive modeling system may, for example, evaluate the performance of a graded validation set for the model, the option to compare and grade performance by other quality measures than those used in the fitting process, and/or determine the best individual performance. Support can be provided to help users make decisions, including the opportunity to build ensemble models from these component models they represent.

inference of measurement

Because some measures cost much more than others, organizations may want to substitute a cheaper metric for a more expensive metric. Thus, a common application of machine learning is to infer an expensive measurable output from the known output of a cheaper one. For example, "curl" is an attribute that captures the tendency of a paper product to deviate from its planar shape, but this can usually only be determined after the product is finished. Being able to infer the curl of a paper from mechanical properties that are easily measured during manufacturing can result in significant cost savings in achieving a given level of quality. For typical end-use attributes, the relationship between these attributes and manufacturing process conditions is not well understood.

In some embodiments, the techniques described herein can be used to infer measurements. For example, the techniques described herein can be applied to the problem of inferring measurements as follows:

1. Characterize the modeling data set: The predictive modeling system 100 provides key overview characteristics and can provide recommendations for processing over critical data that the user is free to accept, reject, or request more information. . For example, a key characteristic of a variable may be calculated and displayed, the appearance of missing data may be indicated, a processing strategy may be recommended, an outlier of a numeric variable may be detected and, if found, a processing strategy These recommendations may be made and/or other data anomalies may be automatically detected (eg, inliers, informational variables whose values do not change), and recommended actions may be made available to the user. .

2. Split the data set into training/validation/holdout subsets.

3. Feature Creation/Model Structure Selection/Model Fit: The predictive modeling system 100 can combine and automate these steps to allow for wide internal iterations. Multiple features can be automatically created and evaluated using both classic techniques such as key components and newer methods such as boosted trees. Many different model types can be customized and compared, including regression models, neural networks, support vector machines, random forests, boosted trees and others. Additionally, the user may have the option to include other model structures that are not part of this default collection. Model sub-structure selection (e.g., selection of the number of hidden units of a neural network, specification of other model-specific tuning parameters, etc.) can be done automatically by broad mutual validation as part of this model fitting and evaluation process. have.

4. Select the final model: The selection of the final model may be made by the predictive modeling system 100 or the user. In the latter case, the predictive modeling system may, for example, evaluate the performance of a graded validation set for the model, the option of comparing and grading performance by quality measures other than those used in the fitting process, and/or the best individual performance. Support can be provided to help users make these decisions, including the opportunity to build ensemble models from these component models that represent

In some embodiments, the predictive modeling system 100 automates and efficiently implements data pre-processing (eg, anomaly detection), data segmentation, multi-feature generation, model fitting, and model evaluation, so The time required may be much shorter than would be achieved in a conventional development cycle. Additionally, in some embodiments, the predictive modeling system is capable of detecting known data anomalies, such as missing data and outliers, and less broadly understood anomalies such as inliers (repeated observations consistent with data distribution but in error) and postdictors. ) (i.e., extremely predictive covariates resulting from missing information), the resulting model may be more accurate and useful because it automatically includes a data post-processing procedure to handle both. In some embodiments, the predictive modeling system 100 may explore a much broader range of model types and more specific models of each type than is conventionally feasible. The versatility of this model can greatly reduce the likelihood of unsatisfactory results even when applied to data sets of compromised quality.

Implementation of a predictive model system

5 , in some embodiments, predictive modeling system 500 (eg, one embodiment of predictive modeling system 100 ) includes at least one client computer 510 , at least one server 550 . and one or more processing nodes 570 . The configuration shown is for illustrative purposes only, and it is intended that there may be any number of clients 510 and/or servers 550 .

In some embodiments, predictive modeling system 500 may perform one or more (eg, all) steps of method 300 . In some embodiments, the client 510 may implement the user interface 120 , and the predictive modeling module 552 of the server 550 may be configured with another component of the predictive modeling system 100 (eg, a modeling space search engine). 110, a library of modeling techniques 130, a library of prediction problems, and/or a modeling placement engine 140). In some embodiments, the computational resources allocated by the search engine 110 for the search of the modeling search space may be those of one or more processing nodes 570, and the one or more processing nodes 570 according to a resource allocation schedule. Able to practice modeling techniques. However, embodiments are not limited by the manner in which components of predictive modeling system 100 or predictive modeling method 300 are distributed among clients 510 , server 550 and one or more processing nodes 570 . Also, in some embodiments, all components of predictive modeling system 100 may be implemented on a single computer (instead of being distributed among clients 510 , server 550 and processing node(s) 570 ). or implemented on two computers (eg, client 510 and server 550 ).

One or more communication networks 530 connect the client 510 with the server 550 , and one or more communication networks 580 connect the server 550 with the processing node(s) 570 . Communication can be any standard phone line, LAN or WAN link (eg T1, T3, 56kb, X.25), broadband access (ISDN, Frame Relay, ATM) and/or wireless link (IEEE 802.11, Bluetooth). This can be done through the medium of Preferably, the network 530/580 is capable of performing TCP/IP protocol communications and data transmitted by the client 510, server 550 and processing node(s) 570 (eg, HTTP /HTTPS requests, etc.) can be communicated over something like a TCP/IP network. However, the network type is not limited, and any suitable network may be used. Non-limiting examples of networks that may serve as or be part of a communications network 530/580 include a wireless or wired Ethernet-based intranet, a local or wide area network (LAN or WAN), and/or many other communications It includes a global communications network known as the Internet that can accommodate media and protocols.

The client 510 is preferably implemented as software 512 running on hardware. In some embodiments, the hardware is such as the MICROSOFT WINDOWS family of operating systems from Microsoft Corporation, Redmond, Wash., the MACINTOSH operating system from Apple Computer, Cupertino, CA, and/or SUN SOLARIS from SUN MICROSYSTEMS. various Unix families, and personal computers (eg, PC with INTEL processor or APPLE MACINTOSH) capable of running with an operating system such as GNU/Linux from RED HAT, INC. of Durham, NC, USA. can Client 510 may also be operated as a smart or dumb terminal, network computer, wireless device, wireless telephone, information appliance, workstation, minicomputer, mainframe computer, personal data assistant, tablet, smart phone or general purpose computer. It may be implemented on hardware, such as another computing device, or a special purpose hardware device used solely to serve as the client 510 .

Generally, in some embodiments, client 510 sends and receives electronic mail and/or instant messages, requests and views content available on the World Wide Web, participates in chat rooms, or uses a computer, portable device, or cellular It can be operated and used for a variety of activities, including using the phone to perform other tasks typically performed. The client 510 may also be operated by the user on behalf of others, such as an employer providing the client 510 to the user as part of their employment.

In various embodiments, software 512 of client computer 510 includes client software 514 and/or web browser 516 . Web browser 514 enables client 510 to request a web page or other downloadable program, applet, or document (eg, from server 550 ) in a web-page request. Examples of web pages include computer-executable or interpretable information, graphics, sound, text and/or video, which may be displayed, executed, reproduced, processed, streamed, and/or stored, and may be linked to other web pages. A data file that may contain links or pointers. Examples of commercially available web browser software 516 are INTERNET EXPLORER provided by Microsoft Corporation, NETSCAPE NAVIGATOR provided by AOL/Time Warner, FIREFOX provided by Mozilla Foundation, or CHROME provided by Google.

In some embodiments, software 512 includes client software 514 . The client software 514 provides functionality for the client 510, for example, allowing a user to send and receive e-mail, instant messages, phone calls, video messages, streaming audio or video, or other content. do. Examples of client software 514 include, but are not limited to, OUTLOOK and OUTLOOK EXPRESS provided by Microsoft Corporation, THUNDERBIRD provided by the Mozilla Foundation, and INSTANT MESSENGER provided by AOL/Time Warner. Standard components associated with the client computer are not shown, including central processing units, volatile and non-volatile storage, input/output devices, and displays.

In some embodiments, web browser software 516 and/or client software 514 may allow a client to access user interface 120 for predictive modeling system 100 .

Server 550 interacts with client 510 . Server 550 is implemented on one or more server-class computers running server-class operating systems (eg, SUN Solaris, GNU/Linux, and MICROSOFT WINDOWS operating system families) with sufficient memory, data storage, and processing capabilities. It is preferable to be Depending on the capabilities of the device and the size of the user base, other system hardware and software than those specifically described herein may also be used. For example, server 550 may be or be part of a logical grouping of one or more servers, such as a server farm or server network. As another example, there may be a plurality of servers 550 that are associated with or connected to each other, or the plurality of servers may operate independently, but have shared data. In further embodiments and as is common in large-scale systems, the application software may be implemented in components, with different components running on different server computers, the same server, or some combination.

In some embodiments, the server 550 includes a predictive modeling module 552 , a communication module 556 , and/or a data storage module 554 . In some embodiments, predictive modeling module 552 may implement modeling spatial search engine 110 , library of modeling techniques 130 , library of predictive problems, and/or modeling placement engine 140 . In some embodiments, the server 550 communicates the output of the predictive modeling module 552 to the client 510 using the communication model 556 and/or execution of the modeling technique on the processing node(s) 570 . can supervise Modules described throughout this specification may be implemented as software programs and/or hardware devices (eg, ASICs, FPGAs, processor, memory, storage, etc.), in whole or in part.

The data storage module 554 may store, for example, the predictive modeling library 130 and/or the library of predictive problems. The data storage module 554 may be, for example, MySQL Database Server by MySQL AB, Uppsala, Sweden, PostgreSQL Database Server by PostgreSQL Global Development Group, Berkeley, CA, or ORACLE, Redwood Shores, CA, USA. It can be implemented using the ORACLE Database Server provided by Copr.

6-8 show one possible implementation of the predictive modeling system 100 . The discussion of Figures 6-8 is given by way of illustration of some embodiments and is in no way limiting.

To carry out the procedures described above, the predictive modeling system 100 may use a distributed software architecture 600 running on a variety of client and server computers. The purpose of the software architecture 600 is to simultaneously deliver a rich user experience and computationally intensive processing. Software architecture 600 may implement a variation of the basic four-tier Internet architecture. As shown in Figure 6, we extend this foundation to leverage cloud-based computation coordinated through the application and data layers.

Similarities and differences between architecture 600 and the basic four-tier Internet architecture may include:

(1) Client 610. Architecture 600 makes essentially the same assumptions for any other Internet application and client 610 . The main use-cases involve frequent access over long periods of time to perform complex tasks. Thus, the target platform includes a rich web client running on a laptop or desktop. However, the user can access the architecture through a mobile device. Accordingly, the architecture is designed to accommodate native clients 612 accessing the Interface Service API directly using a relatively thin client-side library. Of course, any cross-platform GUI layer such as Java and Flash can access these APIs as well.

(2) interface service (620). This architectural layer is an extended version of the basic Internet presentation layer. Due to the complex user interactions that can be used to direct machine learning, alternative implementations, including static HTML, dynamic HTML, SVG visualization, executable Javascript code, and even self-contained IDEs, are available through this layer for a wide variety of content. can support Also, as new Internet technologies evolve, implementations may need to accommodate new types of content or change division of labor between client, presentation and application layers to execute user interaction logic. Thus, the interface service layer 620 can provide a flexible framework for integrating together multiple content delivery mechanisms of varying richness and common supporting facilities such as authentication, access control and input validation.

(3) Analysis Services (630). The architecture can be used to create predictive analytics solutions, so its application layer is focused on delivering analytics services. The computational intensity of machine learning drives major improvements over standard application layers, such as the dynamic allocation of machine-learning tasks to multiple virtual "workers" running in a cloud environment. For any type of logic operation request generated by the execution engine, the analytics service layer 630 accepts the request, decomposes the request into a task, assigns the task to a worker, provides the data needed to execute the task, Coordinate with other layers to match execution results. Associated differences from the standard application layer also exist. Predictive modeling system 100 may allow users to develop their own machine-learning techniques, and thus some implementations may provide one or more full IDEs, and may provide those across client, interface services, and analytics service layers. function is divided. The execution engine then incorporates new and improved technologies generated through these IDEs into future machine-learning computations.

(4) worker cloud (640). In order to efficiently perform modeling operations, the predictive modeling system 100 may decompose them into smaller tasks and assign them to virtual worker instances running in the cloud environment. Architecture 600 allows for different types of workers and different types of clouds. Each worker type corresponds to a specific virtual machine configuration. For example, the default worker type provides general machine-learning capabilities for reliable modeling code. However, another type enforces additional security "sandboxing" of user-developed code. Alternative types may provide configurations optimized for specific machine-learning techniques. As long as the analytics services layer 630 understands the purpose of each worker type, it can assign tasks appropriately. Similarly, the analytics services layer 630 may manage workers in different types of clouds. Organizations can maintain a pool of instances in a private cloud, as well as have the option of running instances in a public cloud. You may have different pools of instances running on different kinds of commercial cloud services or even proprietary internal services. As long as the analytics services layer 630 understands the trade-off between functionality and cost, it can allocate tasks appropriately.

(5) data service (650). Architecture 600 assumes that the various services running on the various tiers may benefit from the corresponding various storage options. Thus, for example, a framework for delivering a rich array of data services 650 such as file storage for any type of persistent data, temporary databases for purposes such as caching, and persistent databases for long-term record management. provides These services may also be specialized for certain types of content, such as virtual machine image files used for cloud workers and IDE servers. In some cases, implementations of data services layer 650 may enforce specific access idioms for specific types of data, allowing other layers to coordinate seamlessly. For example, standardizing the format for data sets and model results means that the analytics services layer 630 can simply pass a reference to the user's data set when assigning tasks to workers. The worker can then access this data set from the data service layer 650 and return a reference to the model results stored via the data service 650 , which in turn has.

(6) External system 660. As with any other Internet application, the use of the API allows external systems to be integrated with the predictive modeling system 100 at any layer of the architecture 600 . For example, the business dashboard application may access graphical visualization and modeling results through the interface service layer 620 . An external data warehouse or even a live business application may provide modeling data sets to the analytics service layer 630 via the data integration platform. The reporting application can access all modeling results from a specific time period through the data service layer 650 . However, in most situations the external system will not have direct access to the worker cloud 640 ; It will utilize this through the analytics service layer 630 .

As with all multi-layer architectures, the layers of architecture 600 are logical. Physically, services from different layers can run on the same system, different modules of the same layer can run on separate machines, and multiple instances of the same module can run across several machines. Likewise, services in one layer may run across multiple network segments, and services from another layer may or may not run on other network segments. However, the logical structure not only helps to reconcile developers and operators' expectations of how different modules will interact, but also gives operators the flexibility they need to balance service-level requirements such as scalability, reliability and security. .

Although the high-level layers appear to be quite similar to those of typical Internet applications, the addition of cloud-based operations can substantially change how information flows through the system.

Internet applications typically provide two distinct types of user interaction, synchronous and asynchronous. For conceptually synchronous operations, such as finding and booking an aircraft, the user makes a request and waits for a response before making the next request. For conceptually asynchronous operations, such as setting an alert for an online transaction that meets certain criteria, the user makes a request and expects the system to notify him of the result at some time in the future (usually, the system will notify the user Provides an initial request "ticket", and provides a notification over a designated communication channel).

Conversely, building and refining machine-learning models may involve interaction patterns at some point in the middle. Setting up a modeling problem may involve an initial series of conceptually synchronous steps. However, if a user instructs the system to initiate computation of an alternative solution, a user who understands the scale of the corresponding computation is less likely to expect an immediate response. Superficially, this anticipation of a delayed outcome makes this phase of the interaction seem asynchronous.

However, the predictive modeling system 100 does not force the user to “fire-and-forget” until receiving a notification, ie, stop engaging with the problem itself. In fact, I encourage him to continue exploring the data set and review preliminary results as soon as they arrive. This additional search or initial intelligence may inspire him to change the model-building parameters "in-flight". The system can then process the requested change and reallocate the processed task. The predictive modeling system 100 may continuously allow for such request-and-modification dynamics throughout a user's session.

Thus, the analytics service and data service layers can mediate between the request-response loop from the user on the one hand and the request-response loop to the worker cloud on the other hand. 7 shows this view:

Fig. 7 emphasizes that the predictive modeling system 100 is not necessarily tailored to a layered model, which assumes that each layer mostly depends only on the layer immediately below it. Rather, analytics service 630 and data service 650 coordinate users and operations collaboratively. From this perspective, there are three "columns" of information flow:

(1) Interface <-> analysis + data. Column 710 in the leftmost flow first transforms the user's raw data set and modeling requirements into a list of refined data sets and computational tasks, which are then combined to deliver the results to the user in a format that the user can easily understand. Thus, goals and constraints flow from interface service 620 to analysis service 630 , and progress and exceptions are reversed. In parallel, the raw data set and user annotations flow from the interface service 620 to the data service 650, and the trained model and its performance metrics flow in reverse. The user can initiate changes and enforcement adjustments by the analytics service 630 and data service 650 layers at any point. In addition to this dynamic cyclical flow, more typical linear interactions (eg, when Internet service 620 retrieves system state from analytics service 640 and retrieves static content from data service 650 ) Also note that it exists.

(2) analysis + data <-> worker. Column 730 of the rightmost flow provisions workers, assigns computational tasks, and provides data for these tasks. Thus, task assignments, their parameters, and data references flow from the analytics service 630 to the worker cloud 640 , with progress and exceptions flowing in reverse. The refined data set flows from the data service 650 to the worker cloud 640 , and the modeling results flow in the opposite direction. The updated instruction from the user will force the analytics service layer 630 to interrupt the in-flight workers and assign updated modeling tasks, as well as force a refresh of the data set from the data service layer 650 . can In turn, the updated assignments and data sets change the flow of results back from the worker.

(3) Analysis <-> data. The middle two layers mediate between themselves to mediate between the left and right columns. Most of this traffic 720 is related to tracking the execution progress and intermediate computations of cloud workers. However, the flow can be complicated, especially when responding to the aforementioned in-flight changes to model-building instructions; The analytics and data service evaluates the current computational state, determines which intermediate computations are still valid, and correctly constructs new computational tasks. Of course, there is also a more conventional linear interaction here (when the analytics service retrieves the rules and configuration for the cloud worker from the data service).

This conceptual model of information flow provides context for the arrangement of functional modules within hierarchies. It is not just a stateless block that provides application programming interfaces (APIs) to higher-level blocks and consumes APIs in lower-level blocks. Rather, it is a dynamic participant in the collaboration between the user and the operation. 8 shows the arrangement of these functional modules. From the user's point of view, the interface service layer provides several distinct areas of functionality.

(1) User/Project Manager (802). Each machine-learning project has at least one assigned administrator who can manage project-level parameters, responsibilities, and resources using the project management component of the interface. This functional component also supports system-level management functions.

(2) Monitoring (810). This module provides diagnostics on the computing infrastructure. It cooperates with the corresponding module 818 in the analytics service layer to track computational resource usage in both real-time for each worker instance and aggregate for each computational task.

(3) Technical Designer (804). This module supports a graphical interface for using the methods, techniques and task builders described above. An example of how this graphical interface can be implemented is Javascript running on the client 610 and communicating with the technical architect 804 via AJAX requests, rendering the graph graphically to the user and pushing changes back to the server. )do.

(4) Technology IDE (812). As noted above, some implementations of predictive modeling system 100 may provide technology developers with built-in access to an IDE to implement leaf-level tasks. These IDEs can support general-purpose programming languages used for machine-learning, such as special scientific computing environments such as Python or R. This function may be executed across the client 610 , interface service 620 and analytics service 630 layers. The client component 610 may download and execute a Javascript container for an IDE environment that first registers a session as an interface service component through AJAX. After authenticating and validating the registration request, the interface service component downloads the user's project data to the client 610 and passes the session to a dedicated IDE server instance running in the analytics service layer. This server instance then communicates directly with the client 610 via a web socket.

(5) Data Tools (806). This module enables model builders to specify data sets, understand them, and prepare them for model-building.

(6) Modeling Dashboard (814). Each project has its own modeling dashboard. Instances of this module provide control and metrics to model builders to launch the modeling process for a project, measure results as they arrive, and perform in-flight adjustments. It calculates which modeling techniques are running on which data sets and passes these requirements to the analytics service layer. Once the execution engine starts building the model, this module provides execution state and control.

(7) Intuition (808). Once the machine-learning process begins to produce practical results, this module provides model builders with deeper intuition. Examples include a text mining overview, predictor importance, and a one-way relationship between each predictor and target. Most of these intuitions are easy to understand and do not require deep knowledge of statistics.

(8) Prediction (816). Once the execution engine has built at least one model, this module provides an interface for making predictions based on new data.

An activity in the interface service layer triggers an activity in the analytics service layer. As mentioned above, the technology IDE and monitoring module are partitioned to run in part on the analytics service layer (see monitoring module 818 and technology IDE module 820 ). Other modules in this layer include:

(1) job queue (822). Each project may have its own work queue instance servicing model computation requests from the corresponding modeling dashboard instance. Tasks include references to partitions in the project's data set, modeling techniques, and priorities within the project. After that, the module constructs and maintains a prioritized list of tasks. When computing resources are available, the broker 824 requests the next job from the job queue. Users with sufficient privileges can add, remove, or re-prioritize modeling jobs from the queue at any time. The queue is persisted through the transient DB module 826, and its backend storage provides extremely fast response times.

(2) Brokers (824). These modules instantiate workers, assign tasks to them, and monitor their health. One broker can run for each worker cloud. The broker dynamically provisions and terminates worker instances to service the current request level from the open work queue plus secure buffers. Upon launch, each worker automatically registers with the broker for the cloud environment, and provides information on its computational function. Brokers and workers send heartbeat messages to each other every few seconds. The worker will automatically restart and re-register if it crashes or loses contact with that broker. The broker retires workers from the pool of available resources and logs an alert if too many heartbeat messages are missed. As new jobs arrive from the job queue and workers complete existing jobs, the broker continuously recalculates the number of workers and the assignment of jobs to these workers.

(3) worker cloud (640). This module contains a pool of workers. Each worker is a virtual machine instance or other unit of self-contained computational resources running within that cloud environment and receives work from its corresponding broker. From the operator's point of view, work involves reference to the project, segmentation of the project's data set, and modeling techniques. For each task in the assigned modeling skill, the operator can first query the file storage module 830, which has a special directory subtree for modeling results, to see if any other operator has completed the partitioning of that data set in the project. have. If it is the first worker processing the step, the first worker performs the calculation and stores it in file storage 830 so that other workers can reuse it. Because modeling techniques are assembled from the jobs in a common modeling job library, there can be a significant level of commonality in the execution of jobs across modeling techniques. Caching the results of task execution may allow implementations to significantly reduce the amount of computational resources consumed.

The data service layer 650 provides a variety of different storage mechanisms to support modules of different layers.

(1) Temporary DB (826). This module benefits from extremely fast access and/or provides an interface to and manages a storage mechanism for ephemeral data. Some implementations use an in-memory DBMS deployed in a master-slave configuration with automatic failover. This module provides an interface for storing objects as key-value pairs. Keys are linked to specific users and projects, but are still very small. Values can be strings, lists, or sets.

(2) Persistent DB (828). This module provides an interface to a storage mechanism for persistent data and manages the storage mechanism. In some implementations, the main types of data processed by this module may include JSON objects, and highly scalable non-SQL, deployed in clusters with automatic failover for both high availability and high performance. database can be used. Objects stored through this module typically range in size to several megabytes.

(3) Save file (830). This module provides an interface to the storage mechanism for files and manages the storage mechanism. Data types stored through this module include uploaded data sets, derived data, model computations and predictions. This module can overlap file directories and naming conventions on top of cloud storage. Additionally, when cloud workers access this module, they can also temporarily cache stored files in local storage.

(4) VM image storage (832). This module provides an interface to and manages storage for VM images used to run the IDE and worker instances. It stores the image in a self-contained VM container format. In the case of an IDE instance, a new worker instance is loaded as an empty copy from the template of that worker type, preserving the user's state across sessions.

Together, these services manage a wide variety of information, including:

(1) UI Session 834: Data that renders the current state of an active user session and performs simple request authentication and access control.

(2) UI object 836: content displayed by the UI.

(3) Cache 838: Cached application content.

(4) system configuration 840: configuration parameters for launching the computing infrastructure and running the model retrieval service.

(5) System Status 842: Real-time data collected from modules of system 600.

(6) User/Project Manager 844: Individual user settings as well as settings and user privileges for each project.

(7) data set 846: data files uploaded for the project by the user.

(8) Modeling Calculations (848): Interim modeling results, final customized models and calculated predictions.

(9) VM Image (850): An image used to launch a new IDE server.

Even in this case, the above-described specific modules 802-850 are logical configurations. Each module can contain executable code from many other source files, and a given source file can provide functionality to many other modules.

Out-of-hours forecast

In some embodiments, predictive modeling system 100 includes a time series model capable of predicting the value of target X at time t and optionally t+1,..., t+i, wherein X at times prior to t , and optionally other predictor variables P at times before t. In some embodiments, the predictive modeling system 100 partitions past observations to train a supervised learning model, measure its performance, and improve accuracy. In some embodiments, time-series models provide useful time-related prediction features, eg, for predicting previous values of a target at different lags. In some embodiments, the predictive modeling system 100 refreshes the time series model as time progresses and new observations arrive, taking into account the amount of new information in these observations and the cost of refitting the model.

An example illustrating the beneficial use of a time series model is described below. In this example, the supermarket chain wants to predict daily sales for the next six weeks for each location in the supermarket. Available data includes three years' worth of prior daily sales from 10,000 locations and other variables (e.g., population and economic growth around each location, history and planned dates of holidays and major social events, and planned dates of chain promotions). include In some embodiments, a time series model trained on the available data can accurately predict daily sales for the next 6 weeks for each location in the supermarket.

Some embodiments of techniques for creating and using time series models are described below. When a data scientist develops a predictive modeling technique with the modeling technique builder 220 , the data scientist may indicate that the modeling technique is specific to a time series prediction problem. The modeling technology builder 220 then encodes these properties in the modeling technology's metadata. The data set itself can also have time series specific metadata (eg, the date range for which the data was generated, the time resolution of the observations, down-sampling that has already occurred, etc.).

When the search engine 110 loads the data set (eg, at step 404 of the method 400 shown in FIG. 4 ), it may automatically detect whether the data set appears to contain time series data. and if detected, it appears that there is a time index. The time index may include a time resolution and a time step (“time interval”). Time resolution is the unit in which time is maintained (eg, seconds or days). For example, if a date (eg, all dates) is encoded in a standard date format (eg, mm/dd/yyyy), engine 110 may use the date as the resolution. As another example, if a date (eg, all dates) includes hours, minutes, and seconds, engine 110 may use seconds as the time resolution. Similarly, engine 110 may generate any suitable time metric (eg, thousand years, hundred years, ten years, years, quarters, seasons, months, weeks, days, hours, minutes, seconds, milliseconds, microseconds) as a common time resolution. seconds, nanoseconds, etc.) can be used. A time step is a period of time (eg, minimum period of time, most common period of time, user-specified period of time, etc.) between consecutive observations (eg, daily, weekly, or yearly data).

If the data set contains time series data with a mixture of temporal resolutions, the engine 110 may use the most common resolution as part of the temporal index, or "lowest common resolution" after converting all temporal data to the common resolution. can be used In some embodiments, engine 110 uses an internal objective function that weights the frequency and inconsistency of potential indexes to select an index (eg, an optimal index). For example, if 90% of the date variables are days resolution and 10% is seconds resolution, the objective function may determine that day resolution is the best choice. An inverse mix (90% resolution of seconds, 10% resolution of work) may yield a decision that the resolution of seconds is the best choice. At a 50% blend, the objective function can determine that the trade-off for resolution (eg, time) between days and seconds is optimal. The choice of the objective function can be further determined by meta-machine learning on the space of the objective function used previously and the accuracy for prediction problems with various characteristics.

In some embodiments, the user may override the search engine's determination (eg, at step 406 of method 400 ) as to whether or not the data set is a time series data set, and may override the time series processing of the data set. Block or force as a data set. When the user forcibly selects the processing of the data set as a time series data set, the user can specify which variables are time-based, how to interpret the data format, and/or the time index. In some cases, the user may provide information on how to transform a variable to use a given temporal index.

For an automatically detected time series data set, the user can accept the suggested time index, choose from any automatically detected alternatives, or specify a custom time index, which can also be used to use the custom time index. You may need to specify how to transform the variable. Once the temporal indices are accepted, selected or specified, the other temporal variables can be transformed into offsets from each other and used as predictive features. For example, if the data set uses mm/dd/yyyy as the time index and one of the variables is “birthday”, the engine 110 may trigger other events (eg, “marriage”, “child’s birthday”, etc.). ) from "birthday" (also known as "age").

After the user identifies or indicates a time series data set and identifies, selects or specifies a time index, the engine 110 evaluates the data set (eg, at step 408 of the method 400 ) and (eg, , at step 410 of method 400) present this rating to the user. This representation may include a diagram showing target and predictor variable values over time, individually or in groups. In some embodiments, the expression represents the dependence of the present value of each time-series variable on past values in one or more time lags. If the data set is a panel data set containing a mixture of time series and cross-sectional variables, the representation can represent dependencies between cross-sectional variables at the same time period or at different lags.

The user can select the end of the training window (e.g., the time range of data used for training) and the beginning of the validation window (e.g. the time range of the data used for verification) or holdout window (e.g., holdout). It can represent a "skip range" in data that is a gap between the time spans of the data used for testing. In some cases, there are operational or logistical reasons for the delay between the last historical observation and the initial prediction. In the supermarket example, data for the past week could all arrive at headquarters on a Sunday and be perfectly clean. However, even after the predictive model is complete, it can take several days for operations to deliver predictive information for all store locations and for stores to adjust operations in response to forecasts. Likewise, there may be a delay in receiving or cleaning historical data. In either case, the practical purpose may be to initiate predictions after a certain gap from the last available historical observation. Skip ranges allow engine 110 to evaluate candidate predictive models with this gap in situ.

The user may indicate a desired prediction range (eg, the number of future time periods to be predicted by the model or the number of distinct future events to be predicted by the model). In some cases, this range may be just one observation, for example next quarter sales. Among other things, as in the example of the supermarket chain, the range may include several future time periods, which in this example is 42 days.

In some cases, engine 110 may present an expected range based on the frequency of the data and/or the total number of periods of time in the data. For example, with daily data and a relatively small number of time periods, engine 110 may present a 7-day forecast, while with a relatively large number of time periods, it may offer a 30-day forecast. For monthly data, engine 110 may present 3-month, 6-month, or 12-month projections depending on the number of time periods. With quarterly data, the engine 110 may present a forecast for the fourth quarter, the eighth quarter, or the twelfth quarter. For annual data, engine 110 may present a five-year, ten-year, or twenty-year forecast.

In some cases, engine 110 uses a verification range equivalent to an observation range or multiple (eg, a logical multiple) of the expected range (eg, when the user specifies an unusually short expected range).

For time-series data, engine 110 generates a set of training ranges, a set of corresponding verification ranges offset by a skip range, and a holdout range (e.g., at steps 418 and 422 of method 400). ) can implement cross-validation and holdout. Engine 110 may have a default target number of training and verification ranges that can be adjusted depending on the amount of data available. A data set with a relatively small period of time may be partitioned into fewer training and validation spans, and a data set with a relatively large period of time may be partitioned into more training and validation spans.

Depending on the size of the data, the frequency of the data, the length of the skip range, and/or the expected range, the engine 110 may select a length for the training range. For example, if daily observations and skips and expected ranges are expressed in weeks, months, or years (or correspondingly multiples of days such as 7, 30, and 365), the engine will display the training range for the entire number of weeks, months, or years. can be selected. The length of this training window depends on the total number of observations, the total amount of data, the total amount of variance of the target and predictor variables over time, the amount of seasonal variance exhibited by the variable, the consistency of the variance of these variables over different time windows, and/or the expected It may depend on the target length of the period. The engine 110 may use an internal objective function that weights these factors to select a length (eg, an optimal length) of the training window. For example, the default may be 5 training and verification ranges that are equally divided over the entire data set. However, if the amount of variation over a longer period of time is low, the engine 110 may shorten the time window. Alternatively, if there is an annual seasonality in the data, the engine 110 uses only three windows to place several years of data into each range. Alternatively, if there are several specific time periods in the data set that exhibit high variability, the engine 110 may partition the data such that each window includes one of these time periods. The selection of the objective function can be determined by the meta-machine learning of the previously used space of the objective function and the accuracy of the prediction problem with various characteristics.

With the desired number of training and verification ranges, plus the length of these ranges plus the skip ranges, the engine 110 may partition the data set into a coherent series of training and verification ranges. For panel data, each training and validation pair can be further split into folds (eg, by randomly assigning section observations to folds). In the supermarket example, the training range is 30 weeks, the skip range is 1 week, and the validation range is 6 weeks, which can yield approximately 4 training sets over 3 years. However, since there are 10,000 stores, these stores can be further "down-sampled" as described below to improve performance. The engine may also reserve sub-windows within the training and validation windows to use to tune model hyper-parameters and mixed models.

In some embodiments, the holdout data is only in the last time window. However, when the data set is panel data and down-sampled, the holdout data may be data from the same period as the other data, but may be data from other non-overlapping samples.

As with a cross-sectional model (see, for example, the description of step 350 of method 300), engine 110 may iterate through a data set, training each model over a small portion of a training window, and , can evaluate its performance against that part, and decide whether to continue testing the model on additional data based on that performance. For time series data, each part may end at the last observation of the training window, the initial part may start such that the partial training window is a logical multiple of the validation window, and larger parts may use larger multiples. . For example, the verification window measured in weeks may use a first portion starting 4 weeks before the end of the training window, a second portion starting 8 weeks before, and a third portion starting 12 weeks before the end of the training window. The verification window, measured in months, has a first part starting March before the end of the training window, a second part starting before June, a third part starting before September, and a fourth part starting before December. Can be used. The validation window, measured in years, may use a first part starting 4 years before the end of the training window, a second part starting 8 years ago, and a third part starting 12 years ago; alternatively, the partial duration is It may be 5, 10 and 15 years before the end of the training window. A portion may be linear (eg 3, 6, 9, 12 period or 4, 8, 12, 16 period) or geometrically (eg 3, 6, 12, 24 period or 4, 8, 16 period) , 32 periods) can be increased. A specific schedule based on problem domains and/or data analysis is also possible with exponential growth of parts (eg 3, 6, 24, 192 periods or 4, 8, 32, 256 periods).

To improve execution speed and reduce resource consumption, engine 110 may suggest default down-sampling. There are two types of down-sampling: chronological and cross-sectional. In chronological down-sampling, the total number of observations within each partition can be reduced to a fixed percentage, or aggregated with a longer temporal resolution. For example, a time window containing 10 billion observations may be reduced to 10 million observations. Alternatively, a window of data with observations every millisecond may have observations aggregated every minute. In this case, all observations within the down-sampling window can be reduced to a single value for the aggregated observation. This aggregate value may be, for example, a starting value, an ending value, a maximum extreme value, an average value, or a value calculated through a weighting function to account for some variation characteristic. The engine 110 may use an objective function to weight the potential reduction in observations due to variability of the target and predictor within the corresponding time window. For example, a 1000x reduction in observations can be justified to 5% in variability. The choice of the objective function can be further determined by meta-machine learning on the space of the objective function used previously and the accuracy for prediction problems with various characteristics.

Chronological down-sampling may be used, for example, for extremely high frequency data where the variation of the data is lower than the frequency of the data. In general, cross-sectional down-sampling is more common because there can be a very large number of sections whose behavior does not change significantly over time. For the example of a supermarket chain, there are 10,000 sections, one for each store location. Most of the variability may be due to chronological factors, with relatively low cross-sectional variability across stores. For example, similar stores may actually belong to a much smaller number of representative groups. Accordingly, the engine 110 may merely construct a random subsample of a section, in this case a storage location, and train a predictive model for one or more subsamples. For example, instead of using all 10,000 stores, the engine 110 may randomly select and train 1,000 stores. Also, if cross-validation is used along the cross-sectional dimension, the engine 110 may apply such down-sampling. For example, for 5-fold cross-validation, the engine may use 5 folds of 400 stores instead of 5 folds of 2,000 stores. The percentage of down-sampling may be determined by balancing multiple characteristics of the sample or as a function of the number of sections determined by a user-specified objective function. In some cases, the selection of down-sampling percentages may be refined by meta-machine learning for many different prediction problems.

One challenge in generating accurate predictions from time series models is that these predictions may be sensitive to the choice of training and validation time windows. In some embodiments, the engine automatically evaluates this sensitivity. For example, the engine 110, when executed, may evaluate its sensitivity to time window selection for all modeling techniques. As another example, the engine 110 may evaluate this sensitivity after the model exceeds a certain threshold of prediction accuracy. A third option is to evaluate the sensitivity of the top model based on its relative prediction accuracy. A fourth option is to run a sensitivity analysis on demand when the user requests it. Sensitivity analysis may include fitting the model to a sliding training and validation window and then measuring how the model accuracy changes depending on the points included in the window. For example, the graph may plot model accuracy on the vertical axis and the starting observations of the training window on the horizontal axis.

Once the engine completes fitting the time series model and presents it to the user (eg, step 446 of method 400 ), the user can further explore the impact of different training data windows on model performance. . The user interface may allow the user to select a particular custom model or group of custom models, and adjust the training and validation windows, including any observations that are not in the holdout window. If the engine did not use sub-windows for hyper-parameter tuning, the optimal hyper-parameters calculated during the original fitting of the model can be used when refitting the model with the new window. If nested training and validation windows are available, the engine automatically recalculates the optimal hyper-parameters in the scheduled window and/or decides whether to recalculate the optimal hyper-parameters or use the original computed ones. You can allow the user to choose.

After the user has reviewed the holdout results (eg, step 450 of method 400 ), the user can use any combination of training, validation, and data from the holdout window to any subset of the model. can be customized again. Since the data from the holdout window is typically the most recent data, including it in the fit of the model can improve the accuracy of the model in predicting future values. Thus, before deploying the model to make new predictors, it is possible to refit the model to the training data whose last observations are the last observations in the holdout window. The selection of the first observation to use for this re-customization may depend on the size (eg, optimal size) of the training window computed during the sensitivity analysis. The user may override the calculated starting point based on the user's analysis of domain knowledge and/or the sensitivity of the prediction problem.

Some examples of time series modeling techniques

Referring to FIG. 9 , a method 900 for time-series predictive modeling may include steps 910 - 980 . In step 910, time series data is obtained. Time series data may include one or more data sets. Each data set may include a plurality of observations. Each observation may include (1) an indication of time associated with the observation and (2) a value of one or more variables. In step 920, a time interval of the time series data is determined. At step 930 , one or more variables of the time series data are identified as targets. Optionally, one or more variables of the time series data may also be identified as features. In step 940, an “expected range” and a “skip range” associated with the prediction problem represented by the time series data are determined. The expected range may indicate a duration of time when the value of the target is predicted. The skip range may represent a time lag between the time associated with the earliest prediction in the prediction range and the time associated with the most recent observation on which the prediction in the prediction range is based.

In step 950, training data is generated from time series data. The training data includes a first subset of observations in the at least one data set. The first subset of observations includes training-input and training-output collections of observations. The times associated with observations in the training-input and training-output collections correspond to the training-input time range and the training-output time range, respectively. A skip range separates the end of the training-input time range from the beginning of the training-output time range. The duration of the training-output time range is at least as long as the expected range. In step 960, testing data is generated from the time series data. The testing data includes a second subset of observations in the at least one data set. The second subset of observations includes a test-input and test-validation collection of observations. The times associated with the observations in the test-input and test-validation collections correspond to the test-input time span and the test-validation time span, respectively. The skip range separates the end of the test-input time range from the beginning of the test-validation check time range. The duration of the test-validation time span is at least as long as the expected span. In step 970, the predictive model is fitted to the training data. In step 980, the customized model is tested against the testing data.

In step 910, time series data is obtained. The time series data may be obtained from any suitable source using any suitable technique (measured using a sensor, received over a communication network, loaded from a computer-readable medium, etc.). Time series data may include one or more data sets, each of which may include one or more observations. A data set may correspond to each “section” of data, as described above. Each observation in the data set may include an indication (eg, a timestamp) of a time associated with that observation. The time associated with an observation may be the time when a value of the observation is measured, reported, received, etc. A “time” associated with an observation may include a date and/or time, or any other suitable temporal data.

In step 920, a time interval of the time series data is determined. In some embodiments, time intervals of data are explicitly indicated by metadata associated with the data. In some embodiments, the time interval of the data is determined by analyzing the data (eg, based on the temporal resolution of a time indicator associated with the observation and/or the interval between times associated with successive observations).

The time interval between observations in a data set can be uniform or non-uniform. If the time interval between observations in the data set is uniform, then the uniform interval may be the time interval in the data set. If the time interval between observations in the data set is non-uniform, the data set may be modified such that the time interval between observations in the modified data set is uniform. The time interval for the modified data set may be determined, for example, by applying an objective function (eg, a weighted objective function) to the original data set. The objective function may determine a modified time interval, for example, based on (1) each proportion of a pair of consecutive observations representing each of the non-uniform time intervals, and/or (2) the duration of the non-uniform time. . In some embodiments, the modified time interval is the shortest common period (eg, the shortest period that is an integer multiple of each non-uniform period).

An original data set with non-uniform time intervals may be transformed into a modified data set with modified uniform time intervals by sampling (eg, down-sampling) and/or aggregating observations in the original data set. Down-sampling the observations in the original data set includes, for each instance of the modified time interval of the time period covered by the observation in the original data set: the original data associated with a time corresponding to the instance of the time interval in the time series data. It may include identifying all observations in the set, aggregating the identified observations to produce aggregate observations, and inserting the aggregate observations into the revised data set. Aggregating a set of identified observations involves calculating the value of each variable in the aggregated observations: (1) the value of the corresponding variable included in the earliest identified observation, (2) the value of the corresponding variable included in the most recent identified observation, ( 3) the maximum value of the corresponding variable value contained in the identified observation, (4) the minimum value of the corresponding variable value contained in the identified observation, (5) the average of the corresponding variable value contained in the identified observation, (6) This can be done by setting it to the value of a function of the corresponding variable value included in the identified observation.

The time interval between data sets of time series data may be uniform or non-uniform. If the time interval between data sets is uniform, the uniform interval may be the time interval of time series data. If the time intervals between the data sets are non-uniform, one or more data sets may be modified such that the time intervals between the modified data sets are uniform. The corrected time interval of the time series data may be selected from intervals in the data set or determined using any other suitable technique. For example, the adjusted time interval of time series data is determined by (1) each proportion of observations included in the data set representing each of the non-uniform time intervals, and/or (2) the duration of the non-uniform time interval of the data set. based on the objective function (ie, a weighted objective function). In some embodiments, the corrected time interval of the time series data is the shortest common period (eg, the shortest period that is an integer multiple of each non-uniform time interval of the data set). Some embodiments of techniques for modifying time intervals of data sets have been described above.

In some embodiments of method 900, feature engineering may be performed on time series data. Such feature engineering may be performed, for example, before or after the time interval of the time series data is determined. In some embodiments, performing feature engineering on the time series data includes: identifying a first variable in the time series data having a value representing time; generating values of a second variable, each value of the second variable being an offset between a time value of the first variable and a reference time value; and adding the second variable to the time series data (eg, adding each value of the second variable to an observation including the value of the first variable from which the value of the second variable was derived). In some embodiments, the first variable may be removed from the time series data. In some embodiments, the reference time is the date of the event (eg, birth, marriage, graduating from school, starting employment with an employer, graduating from school, starting work in a particular position, etc.). This feature engineering technique can be used to transform absolute time values into relative time values, which will greatly facilitate the identification of patterns (e.g., age-related patterns of entities) in data from different data sets of different time periods. Therefore, it is very easy to accurately predict a value related to such a pattern.

In some embodiments, performing feature engineering includes temporally down-sampling the time series data. Several techniques for temporally down-sampling time series data have been described above. Temporally down-sampling the time series data may improve the efficiency of a time series modeling technique based on the time series data (eg, reduce computational resource usage).

In some embodiments of method 900, graphical information (eg, graphs or charts) may be presented (eg, displayed) via a user interface. The graphical information may represent a time lag between a change in the value of one variable and a correlated change in the value of another variable. Such correlation may be detected using any suitable technique.

At step 930 , one or more variables of the time series data are identified as targets. A target may be identified, for example, based on metadata associated with the time series data, a description of the prediction problem, and/or user input. Optionally, one or more variables of the time series data may also be identified as features. A feature may be identified, for example, based on metadata associated with the time series data, a description of the prediction problem, and/or user input.

In step 940, an expected range associated with the prediction problem represented by the time series data is determined. The expected range may indicate a duration of a period for which the value of the target is predicted. The expected range is (1) the time interval of the time series data, (2) the number of observations included in the time series data, (3) the period of time covered by the observations in the time series data, and (4) microseconds, milliseconds, seconds, and minutes. , hours, days, weeks, months, quarters, seasons, years, ten years, a natural period selected from the group consisting of a hundred years, and a thousand years, (5) a user input, and the like. In some embodiments, the expected range is an integer multiple of the time interval of the time series data. In general, the expected range can increase as the number of observations in time series data increases.

In step 940, a skip range associated with the prediction problem represented by the time series data is determined. The skip range may represent a time lag between the time associated with the earliest prediction in the expected range and the time associated with the most recent observation on which the prediction in the expected range is based. The skip range may be, at least in part, a latency in the collection of time series data, a latency in communication of the time series data, a latency in the analysis of time series data, a latency in the communication of the analysis of the time series data, and/or It may be determined based on the waiting time of the action implementation based on the analysis of the time series data. This waiting time may be determined, for example, based on user input and/or metadata associated with time series data. In some embodiments, the skip range is determined based on metadata associated with the time series data or specified by the user.

In step 950, training data is generated from time series data. The training data includes a first subset of observations in the at least one data set. The first subset of observations includes training-input and training-output collections of observations. The times associated with observations in the training-input and training-output collections correspond to the training-input time range and the training-output time range, respectively. A skip range separates the end of the training-input time range from the beginning of the training-output time range. The duration of the training-output time range is at least as long as the expected range.

In some embodiments, the duration of the training-input time range is the total number of observations in the time series data, the amount of change over time in the value of at least one variable, the amount of periodic change in the value of the at least one variable, over a plurality of time periods. is determined based on the consistency of the variation in the value of at least one of the variables across, and/or the duration of the expected range.

In some embodiments, rather than training the model on all training data, a subset of training data is identified for the purpose of training a predictive model on the subset. Training a model based on a subset of training data may use less computational resources (and time) than training a model based on all training data. In some embodiments, each subset of training data ends at the end time of the training-input time range. The training data subsets may begin at different times within the training-input time range and/or may sample observations in the training-input time range at different sampling rates. The duration of the training data subset may be an integer multiple of the duration of the expected range.

In some embodiments, the training data may be down-sampled (eg, temporally down-sampled or sectional down-sampled). The training data may be temporally down-sampled by selecting a down-sampled time interval and down-sampling each data set of training data according to the down-sampled time interval (e.g., using the techniques described above). have. The training data may be single-sided down-sampled by removing one or more data sets from the training data. In some embodiments, the training data is down-sampled both temporally and cross-sectionally.

In step 960, testing data is generated from the time series data. The testing data includes a second subset of observations in the at least one data set. The second subset of observations includes a test-input and test-validation collection of observations. The times associated with the observations in the test-input and test-validation collections correspond to the test-input time span and the test-validation time span, respectively. The skip range separates the end of the test-input time range from the beginning of the test-validation time range. The duration of the test-validation time span is at least as long as the expected span.

In some embodiments, the duration of the test-input time range is the total number of observations of the time series data, the amount of change over time in the value of at least one of the variables, the periodic amount of change in the value of the at least one variable, the variable over a plurality of time periods. the consistency of the change in the value of at least one of the variables, and/or the duration of the expected range.

In some embodiments, a subset of testing data is identified for the purpose of testing a predictive model on a subset rather than training the model on all training data. Testing the model on a subset of the testing data may use less computational resources (and time) than testing the model on all the training data. In some embodiments, each subset of testing data ends at the end time of the test-input time range. The testing data subset may be started at different times within the test-input time range, and/or may sample observations in the test-input time range at different sampling rates. The duration of the testing data subset may be an integer multiple of the duration of the expected range.

In some embodiments, the testing data may be down-sampled (eg, temporal down-sampled and/or sectional down-sampled). Some techniques for temporal and cross-sectional down-sampling have been described above.

In step 970, the predictive model is fitted to the training data. In step 980, the customized model is tested against testing data. Mutual validation (including but not limited to nested mutual validation) and holdout techniques may be used to customize and/or test predictive models. For mutual validation, time series data may be segmented in cross-section and/or in time.

Some embodiments of the method 300 for selecting a predictive model for a predictive problem may be used to select a time-series predictive model for a time-series predictive problem. In some embodiments, model-specific predictive values of one or more features of time series data may be determined (eg, using embodiments of method 1000 described below). In some embodiments, time series prediction models may be blended (eg, using techniques described herein). In some embodiments, the time series prediction model may be deployed and/or refreshed (eg, using techniques and/or performance enhancements described herein). In some embodiments, the strength of an interaction between two or more variables (eg, features) of time series data may be determined (eg, using techniques described herein).

universal Feature importance

When considering supervised machine learning problems, an important challenge is often to measure which features have the most predictable values for the target. Measuring the importance of these features can be useful in two separate stages of predictive modeling: (1) understanding the prediction problem in general, and (2) understanding how a particular customized model produces predictive outcomes. .

For purpose (1), the metric of feature importance is an evaluation of the data set (eg, step 408 of method 400), presentation of a rating to a user (eg, step 408 of method 400) 410)), how the user refines the data set (e.g., step 412 of method 400), and/or which modeling technique attempts or presents an attempt (e.g., step 422 and 424)) can be reported. For purpose (2), the metric of feature importance can inform the automated development of the blended model (eg, step 432 of method 400), and help the user to understand the relative performance of alternative models. (eg, step 446 of method 400).

Indeed, determining feature importance for a first purpose generally occurs within the context of a particular model or family of models (eg, a random forest). Of course, each model may have advantages and disadvantages as a device for measuring feature importance. Thus, calculating feature importance using several different types of models can deliver improved understanding. For example, random forests, generalized additive models, and support vector machines are fundamentally different types of models for machine learning and can produce different measures of importance of the same feature for the same data set. These differences can provide a deeper intuition into the structure of the prediction problem and provide avenues for further exploration. For purpose (2), feature importance is generally specific to the model(s) under consideration.

Some embodiments of techniques for calculating feature importance for an arbitrary model are described below.

The search engine 110 may calculate the importance of any feature, given a data set (or any sample thereof) and modeling techniques, using universal partial dependencies. First, the engine 110 uses modeling techniques to obtain accuracy metrics for the predictive model customized to the sample. The engine 110 may perform this fit from scratch or may use a previous fit. Then, for a given feature, engine 110 takes all values across all observations, shuffles them, and reassigns them to the observations (eg, randomly reassigns them). Such random shuffling may reduce (eg, discard) any predictions for that feature. The engine can then re-score the model against the data set with the shuffled feature values to generate new values for the accuracy metric. (Optionally, before rescoring the customized model against the shuffled data set, the engine can refit the model back to the shuffled data set.) The decrease in the accuracy of the model is determined by how much the predictions are lost. , thus indicating the importance of the feature to the model and/or within the scope of the modeling technique.

Using the modeling techniques and/or techniques described above for calculating the importance of a feature to a model, the engine iterates the features to determine the relative importance of the features in the model and/or modeling technique, and/or the model and/or It can be iterated over the model and/or model to determine the relative importance of features across modeling techniques or both.

To meet objective (1), engine 110 may generally maintain a list of modeling techniques that produce exemplary results for feature importance. When evaluating the data set, the engine 110 may automatically execute all or some of these modeling techniques. The engine can choose which modeling technique to run based on the properties of the data set. The user interface may display feature importance values individually for each modeling technique used, or relative across all modeling techniques used. Engine 110 may also allow a user to select any modeling technique from modeling technique library 130 and use it to measure feature importance.

In addition to helping the user understand the prediction problem, the engine 110 may use the results from this application of feature importance to guide further automated analysis. For example, for features that score high in overall importance, the engine may allocate more resources to exploring the interactions of those features. In the case of features that score low in overall importance, the engine can completely exclude those features from the data set. If some features are low with high importance to some models or modeling techniques and low importance to other models or modeling techniques, the engine instructs a deeper search for the predictive model, tries more modeling techniques and model space More data is available earlier in the discovery process.

To meet objective (2), the engine may calculate feature importance values for the customized model automatically or on demand. Automatically, engine 110 may calculate feature importance values for all models or portions thereof. The portion may be calculated in any suitable manner. For example, a portion may include the N top performing models, all models that meet at least some level of performance threshold for a particular performance metric, or all models whose performance metric is within a given portion of the top model's performance. have. In some embodiments, the engine may take into account available computational resources and adjust portions to be larger or smaller in proportion to these resources.

In some embodiments, a user may request feature importance calculations on demand. In some cases, a user may wish to ascertain the importance of some or all features for a given model. In other cases, a user may wish to ascertain the importance of some or all features across different models.

Using the techniques described above to provide feature importance values for prediction problems in general or specific models can have many advantages. for example:

(1) If a feature is not informative in general or at least in all accurate models, the collection of data corresponding to that feature may be stopped. In some cases, there is a real cost to making the feature available, such as labor to extract data from the source location or even the cost of paying the data provider.

(2) The difference between user expectations of feature importance and measured feature importance may warrant further investigation. The data set that accounts for the inconsistency may turn out to be erroneous, or it may prove that the differences are real and provide new intuitions for the prediction problem.

(3) In some cases, it may be desirable to create a model that makes predictions using as few features as possible. In this case, the user can redo the specific modeling technique, use only the N features of the most important importance among all modeling techniques, or rerun the search with only those features with importance values that exceed a certain threshold.

(4) Knowing which features are important aids can help users improve predictive models by experimenting with different ways of transforming and combining the most important features.

of one or more features in the data set. the predicted value part of the skill to decide Example

10 illustrates a method 1000 for determining predictive values (eg, “significance”) of one or more features of an initial data set representing an initial prediction problem. In some embodiments, the predictive modeling system 100 is configured during evaluation of the data set (eg, step 408 of method 400 ), and/or when processing the data set (eg, method 300 ). ), the method 1000 (or a portion thereof) may be performed. In some embodiments, method 1000 may be used to determine predictive values of any data set feature for any predictive model or predictive modeling technique.

At step 1010, the system 100 performs a plurality of predictive modeling procedures. Each predictive modeling procedure is associated with a predictive model. Performing each modeling procedure includes fitting an associated predictive model to at least a portion of an initial data set representing the initial prediction problem. The initial data set contains previous observations, and each observation generally contains values of at least some of the features of the initial data set.

At step 1020 , the system 100 determines a first accuracy score of each of the customized predictive models. The first accuracy score of the customized model indicates the accuracy with which the customized model predicts one or more outcomes of the initial prediction problem. Any suitable metrics and/or techniques for determining the accuracy of a model may be used, including, but not limited to, testing the model against the holdout portion of the initial data set.

At step 1030, the system 100 "shuffles" the values of a particular feature F over observations in the initial data set, thereby generating a modified data set representing the corrected prediction problem. In some embodiments, shuffling is performed by reallocating the value of feature F from the original observation to a different observation (eg, randomly). This shuffling operation may reduce (eg, discard) any predictions for feature F. Other techniques for reducing (e.g., discarding) the predicted value of feature F in a data set are possible, including, but not limited to, removing feature F from the data set or assigning each observation the same value for feature F do. In some embodiments, any technique that reduces the predicted value of feature F below a threshold may be used.

At step 1040, the system 100 determines a second accuracy score of the customized predictive model for the modified predictive problem. The second accuracy score indicates the accuracy with which the customized model predicts one or more outcomes of the modified prediction problem. Any suitable metric and/or technique for determining the accuracy of a model may be used, including but not limited to testing the model against the holdout portion of the modified data set. In some embodiments, the customized model is refitted to the modified data set prior to determining the second accuracy score.

At step 1050 , for each predictive modeling procedure (or customized model), the system 100 calculates a predicted value of feature F. In some embodiments, a predicted value of a feature F for a model or modeling procedure is calculated based on a change in accuracy (eg, based on a difference between the first and second accuracy scores for the model). In some embodiments, to determine a predictive value of a feature for a modeling procedure or model based on the first and second accuracy scores, the predictive value generally increases as the difference between the first accuracy score and the second accuracy increases. function is used. Because individual predictions may be specific to a particular modeling procedure or model, the predictions determined in step 1050 may be referred to herein as “model-specific predictions”.

At step 1060, the system 100 determines whether to analyze the predictions of other features. In some embodiments, this determination is made based on user input (eg, the system continues to analyze predicted values of features until all features specified by the user have been analyzed). In some embodiments, the system analyzes all features of the data set. In some embodiments, the system analyzes only a subset of the features in the data set. These features may be selected based on any suitable criteria. If the system determines in step 1060 that there are other features to analyze, the system repeats steps 1030-1050 for those features.

In some embodiments, method 1000 includes an additional step (not shown) of selecting a predictive modeling procedure to be performed at step 1010 . System 100 may select a modeling procedure from, for example, library 130 of predictive modeling techniques. In some embodiments, system 100 selects two or more modeling procedures from two or more different predictive modeling families. Examples of predictive modeling families include linear regression techniques (eg, generalized additive models), tree-based techniques (eg, random forests), support vector machines, neural networks (eg, multi-layer perceptrons), etc. may include For example, system 100 may select a tree family (e.g., a random forest modeling procedure), other modeling procedures from a linear regression family (e.g., a generalized additive model), and a support vector machine modeling procedure. .

The system 100 is configured to (1) process a data set based on model-specific predictions, (2) present an evaluation of the data set to a user, and (3) a predictive model for blending based on model-specific predictions. (4) present the evaluated model and associated model-specific predictions to the user, and (5) allocate resources during the process of evaluating the space of potential predictive modeling solutions to the prediction problem based on the model-specific predictions. and (6) one or more model-specific predictors determined via method 1000, including, but not limited to, calculating a model-independent predictive value (feature importance value) of a feature based on the model-specific predictive value of the feature. (feature importance value) can be used to perform any suitable operation (eg, predictive modeling operation). In some embodiments, after calculating the model-independent feature importance value, the system 100 uses the model-independent feature importance value to perform any suitable task (eg, tasks (1) as described in the previous sentence. )-(5)) can be performed.

In some embodiments, processing the data set (eg, as described above with reference to method 300 ) and/or during evaluation of the data set (eg, at step 408 of method 400 ) When doing so, as part of the predictive modeling procedure, the system 100 performs feature creation and/or feature engineering based on model-specific predictive values. For example, system 100 may organize “less important” features from a data set. In this context, when a feature's predicted value is below a threshold voltage, when a feature has one of the M lowest predicted values of a feature in the data set, when a feature does not have one of the N highest predicted values of a feature in the data set, etc. A feature may be classified as "less important". As another example, the system may generate features derived from “more important” features in a data set. In this context, when the predicted value of a feature is greater than a threshold, when a feature has one of the N highest predicted values of a feature in the data set, when a feature does not have one of the M lowest predicted values in the data set, etc. , the feature can be classified as "more important". In some embodiments, system 100 may calculate predictive values of features derived using method 1000 .

In some embodiments, the system 100 may present (eg, display) an evaluation of the data set to the user (eg, at step 410 of the method 400 ), wherein the presented evaluation is the data set. predicted values of features in and/or information derived therefrom. For example, for one or more modeling procedures or models, system 100 may (1) identify "more important" and/or "less important features", (2) display predicted values of the features, and (3) Rank the features by their predictions, and/or (4) recommend that the collection of less important features be stopped and/or less important features are removed from the data set. In response to presenting the user with an evaluation of the data set, system 100 may receive user input specifying a refinement of the data set (eg, at step 412 of method 400 ).

In some embodiments, system 100 selects a predictive model for blending based on model-specific predictive values and blends the selected model (eg, at step 432 of method 400 ). System 100 may use any suitable technique to select a predictive model for blending. For example, system 100 may select a “complementary top model” for blending. A "complementary top model" in this context may include an accurate model that achieves high accuracy through other mechanisms. The system 100 determines if the model's accuracy is greater than a threshold accuracy, if the model has one of the N highest accuracy values among the customized models, if the model does not have one of the M lowest accuracy values among the customized models. etc., the model can be classified as a "top" model. The system determines if (1) the most important features for the model (e.g., the features with the highest predicted values for the model) differ, or (2) features that are important to the first model are not important to the second model, and If a feature that is not important to one model is important to a second model, then the two models can be classified as "complementary" models. In this context, a feature may be "important" to the model if it has a high predictive value for the model (eg, the highest predictor, one of the best N predictors, a predictor greater than a threshold, etc.). In this context, if a feature has a low predictive value for the model, then the feature may be "not important" to the model (eg, lowest predicted value, one of lowest N predicted values, predicted value lower than a threshold, etc.) . In some embodiments, system 100 may use the classification techniques described above to select two or more complementary top models for blending.

In some cases, mixing complementary top models can yield a mixed model with very high accuracy compared to component models. Conversely, mixing non-complementary models may not yield mixed models with significantly higher accuracy than component models.

In some embodiments, system 100 may present the evaluated predictive model and associated model-specific predictive values to the user (eg, at step 446 of method 400 ). In some embodiments, system 100 calculates and/or displays feature importance values for only a subset of predictive models (eg, top models). Presenting feature importance values to the user may help the user understand the relative performance of the evaluated model. For example, based on the presented feature importance values, the user (or system 100) can identify a top model M that is superior to other top models and one or more features F that are important to model M but not to other top models. have. The user may conclude (or the system 100 may represent) that model M makes better use of the information represented by feature F, compared to other top models. Based on these findings, a party interested in the outcome expected by the model M may invest in a system to measure or control feature F, which may improve the outcome expected by the model.

In some embodiments, during the process of evaluating the space of a potential predictive modeling solution to a predictive problem (eg, while performing method 300 ), system 100 determines the number of features in a data set representing a predictive problem. Resources for evaluation of modeling procedures may be allocated based on the predicted values. For example, system 100 may select or present a predictive modeling procedure to evaluate (eg, at step 310 of method 300 , or at step 424 of method 400 ). As noted above, system 100 may select or suggest predictive modeling procedures that are predicted to be appropriate or very appropriate for a data set. Some techniques for determining the adequacy of a modeling procedure for data sets with particular characteristics have been described above. In determining the appropriateness of a predictive modeling procedure for a prediction problem based on the characteristics of the prediction problem, the system 100 may treat the characteristics of the more important features of the data set as characteristics of the prediction problem. In this way, the relevance score generated by the system 100 can be tailored to more important features of the data set. Because the system 100 allocates resources based on relevance scores, tailoring relevance scores to more important features of the data set may result in resources being allocated for evaluation of predictive modeling procedures based in part on feature importance. .

Additionally or alternatively, the system 100 may process the data set and/or during evaluation of the data set (eg, as described above with reference to step 408 of method 300 or method 400 ). allocating more resources to the feature creation task with more significant features of the data set, and/or more to the mixing of complementary top models (eg, at step 432 of method 400 ) resources can be allocated.

In some embodiments, system 100 may calculate a model-independent predictive value of feature F based on the model-specific predictive value of the feature. System 100 may (1) compute statistical measures of model-specific predictors (eg, mean, median, standard deviation, etc.), or (2) determine combinations of model-specific predictors. Any suitable technique can be used to compute the model-independent predictive value of a feature, including but not limited to, in the latter case, the combination can be a weighted combination. The model-specific feature values for the less accurate model may be weighted more heavily than the model-specific feature values for the less accurate models, In some embodiments, the model-specific feature values for the least accurate modeling procedures are described in this paragraph. may be excluded from calculated calculations and/or combinations.

2nd model

As noted above (e.g., in the section titled "Model Deployment Engine"), appropriate 2 It may be desirable to generate a model-of-model (“second-order model”) using a quadratic modeling technique (eg RuleFit), which also allows people to interpret the quadratic model more easily than the original model, ; a quadratic model can provide an intuition for a "black box" model that does not.

However, there are concerns that the quadratic model may be systematically less accurate for prediction than the source model. Techniques for reducing (eg, minimizing) any loss of accuracy associated with moving from a source model to a quadratic model (and in some cases to produce a quadratic model with higher accuracy than the source model) Some embodiments of is described below.

In some embodiments, the user customized model (eg, step 446 of method 400 ) and/or the result of a holdout test against a top model (eg, step 450 of method 400 ) )), the user (a) interprets the model to understand how to use the features to generate accurate predictions and/or (b) deploys the model using the deployment engine 140 may want Certain modeling techniques may use opaque and/or complex models to build a model, which may, in at least some cases, make the model difficult to understand and increase the difficulty of generating predictive code. Preventing disclosure of proprietary model-building techniques can make the model understandable and exacerbate the challenge of generating predictive code for the model.

In some embodiments, the predictive modeling system 100 addresses this problem by building a quadratic model of the source model. In some embodiments, predictive modeling system 100 builds quadratic models using one or more modeling techniques (eg, RuleFit, generalized additive models, etc.) that generally produce models that are relatively easy to interpret. And for this, the prediction code can be generated relatively easily. This technique is referred to herein as a “second-order modeling technique”. A “first-order” model to which the engine 110 is customized (eg, a first-order model that the user wants to better understand, a first-order model that the user wants to generate predictive code for, and/or a first-order model with proprietary features) After generating , the system 100 may generate a secondary model of the primary model using secondary modeling techniques.

For each feature in the first-order model, there is a corresponding set of feature values from or derived from the original data set. The secondary modeling technique may use the same features as the primary model, and thus, for training and testing data of the secondary modeling technique, the original values of these features or a subset thereof. In some embodiments, instead of using the actual values of the target from the data set, the second-order modeling technique uses the predicted values of the target from the first-order model.

In some cases, secondary modeling techniques may use alternative or supplemental training and/or testing data. These alternatives include other real-world data from the same or different data sources (e.g., for the purpose of covering a wider range of possibilities than exist in real-world samples) (e.g., via interpolation and extrapolation). ) real-world data combined with machine-generated data, or data entirely generated by machine-based probabilistic models. In some embodiments, the values of the target variables used to train the second-order model are predicted values from the first-order model.

One consideration is that any errors in the first-order model may be synthesized or magnified when building the second-order model, thereby systematically reducing the accuracy of the second-order model. First, the inventors recognize and understand that there are questions about whether this is empirically true. Second, if that is the case, we recognize and understand that using a more accurate first-order model has the potential to reduce the loss of accuracy. For example, as a mixture model is sometimes more accurate than any single model (as described at the end of the section titled "Modeling Space Search Engine," for example), a second-order model can be used in a mixture of first-order models. Customizing can reduce any loss of accuracy associated with quadratic modeling.

We have empirically determined that concerns about the accuracy of the quadratic model are largely wrong. Testing of the system was performed on 381 data sets resulting in 1195 classifications and 1849 regression first-order models. For the tested classification models, 43% of the second-order models were less accurate than the corresponding first-order model, but did not deteriorate more than 10% according to the log-loss measure of accuracy. 30% of the second model was actually more accurate than the first model. In only 27% of those cases, the second-order model was 10% less accurate than the first-order model. About one-third of these cases (about 9% of the total population) occurred when the data set was very small. One third of these cases (again about 9% of the total population) occurred when the primary model was very accurate at less than 0.1 log loss and the secondary model was also still very accurate below 0.1 log loss. Thus, in more than 90% of cases where the data set was large enough, the second-order model was either within 10% of the first-order model or the second-order model was very accurate according to the absolute standard. In 41% of the cases, the best secondary model was derived from a mixture of primary models.

For the regression models tested, 39% of the second-order models were less accurate than the corresponding first-order model, but deteriorated to less than 10% according to the residual mean square error measure of accuracy. 47% of the second model was actually more accurate than the first model. In only 14% of cases, the second-order model was 10% less accurate than the first-order model. About 10% of those cases (about 1.5% of the total population) occurred when the data set was very small. In 35% of all cases, the best secondary model was derived from a mixture of primary models.

Based on these empirical data, the inventors recognize and understand that quadratic models are generally relatively accurate or absolutely accurate when the original data set is suitably large. In many cases, the second-order model is actually more accurate than the first-order model. Finally, for more than one-third of all classification and regression problems tested, the most accurate secondary model was derived from a mix of primary models.

We believe that second-order models may be advantageous for (a) understanding complex first-order models, (b) simplifying the task of generating predictive codes for predictive models, and (c) protecting proprietary model-building techniques. recognized and understood that Mixed models often produce the best predictive results, but are also generally more complex because they combine the complexity of all components of all models involved in the mix. Also, generating predictive codes for a mixture model typically combines the task of generating predictive codes for all component models involved in the mixture. At the top of this component task, mixed models are generally slower to generate predictions (and/or require more computational resources to generate predictions), as all mixed models are typically computed to generate each prediction. ). Quadratic models typically reduce this time (and/or use of computational resources) to the time to compute a single model. In addition, the hybrid model includes proprietary features of each component model. Thus, the quadratic model and the mixed model behave in a highly complementary manner.

In addition, because the system 100 can automatically split the data set into training, testing, and holdout partitions, the system can easily determine whether any particular second-order model performs properly compared to the first-order model. have.

Some Examples of Secondary Modeling Techniques

Referring to FIG. 11A , a method 1100 for generating a quadratic predictive model may include steps 1110 and 1120 . In step 1110, a customized first-order predictive model is obtained. The first-order predictive model is configured to predict values of one or more output variables of the prediction problem based on values of one or more first input variables. The first-order model may be constructed using any suitable technique (e.g., executing machine-executable modules implementing predictive modeling procedures, performing embodiments of predictive modeling method 300, constructing two or more predictive models) mixing, etc.), and may be any suitable type of predictive model (eg, a time series model, etc.). At step 1120, a second order predictive modeling procedure is performed on the customized first order model. A quadratic modeling procedure involves a quadratic predictive model (eg, a RuleFit model, a generalized additive model, any model organized as a set of conditional rules, a mixture of two or more of previous types of models, etc.).

Referring to FIG. 11B , the method 1120 for performing the secondary predictive modeling procedure may include c steps 1122 , 1144 , 1126 and 1128 . At step 1122 , secondary input data is generated. The secondary input data includes a plurality of secondary observations. Each secondary observation includes the value of the output variable predicted by the primary model based on the observation of the one or more second input variables and the value of the first input variable corresponding to the value of the second input variable. Generating the secondary input data may include for each secondary observation: obtaining an observed value of the second input variable and a corresponding observed value of the first input variable, and predicting the output variable. Applying the first-order predictive model to the corresponding observed values of the first input variable to generate the observed values.

The first-order and second-order models can use the same set of input variables. Alternatively, the second-order model may use one or more (eg, all) of the input variables of the first-order model, and may also use one or more other input variables. Alternatively, the second-order model may use none of the input variables of the first-order model, but one or more other input variables.

In step 1124, secondary input data and secondary testing data are generated from the secondary input data. Some embodiments of techniques for generating training data and testing data from an input data set are described herein.

In step 1126 , a customized secondary predictive model of the customized primary model is generated by fitting the secondary predictive model to secondary training data. In step 1128, the customized secondary predictive model of the customized primary model is tested against secondary testing data. Mutual validation (including but not limited to nested mutual validation) and holdout techniques may be used to customize and/or test predictive models.

In some embodiments, primary and secondary customization models may be compared. For example, an accuracy score may be determined for each custom model. The accuracy score of each custom model may indicate the accuracy with which the custom model predicts the outcome of one or more prediction problems. (In some embodiments, the accuracy score of the second-order model is indicative of the accuracy with which the customized model predicts observations of the target variable, rather than the accuracy with which the customized model predicts the target variable values predicted by the first-order model. Other implementations In an example, the accuracy score of the second-order model is indicative of the accuracy with which the customized model predicts the target variable values predicted by the first-order model.) In some cases, the accuracy score of the customized second-order model is the customized first-order model. Exceeds the accuracy score of the model. As another example, the amount of computational resources by each customized predictive model is used to predict the outcome of one or more predictive problems. In some embodiments, the amount of computational resources used by the customized second-order model is less than the amount of computational resources used by the customized first-order model.

In some embodiments, a plurality of secondary modeling procedures are performed, and a plurality of secondary models of the primary model are generated. In some embodiments, accuracy scores and/or computational resource usage of the quadratic models are determined and compared. One or more secondary models may be selected for deployment based at least in part on the accuracy score and/or computational resource usage of the model.

Some embodiments of the method 300 for selecting a predictive model for a predictive problem may be used to select a quadratic predictive model for a predictive problem. In some embodiments, model-specific predictive values of one or more features of the input data may be determined for a quadratic model (eg, using embodiments of method 1000 described below). In some embodiments, quadratic predictive models may be blended (eg, using techniques described herein). In some embodiments, the quadratic predictive model may be deployed and/or refreshed (eg, using techniques and/or performance enhancements described herein).

Some embodiments of the method 300 for selecting a predictive model for a predictive problem may be used to select a quadratic predictive model for a predictive problem.

text language condition

Some features of a data set may contain unstructured text blocks. Since different languages have different characteristics, the best predictive modeling step may be depending on which language or languages are present in the unstructured text feature.

In some embodiments, system 100 detects a language as a text string, or more generally calculates the likelihood that the string is in a given language. System 100 may incorporate knowledge of the text language into a predictive modeling process through conditional text processing. For example, modeling techniques may include pre-processing steps that include extracting structured features from text. The feature extraction procedure may vary based on language. As another example, some modeling techniques may be specific to a particular language or perform differently for other languages.

For example, the character n-gram is effective in Chinese text. Usually, n-grams are preferred for processing English texts because they often produce good results but are inconsistent in English texts. There are also "synthetic" languages, such as German and Finnish, that form a word from many morphemes. Since word n-grams can lose much of the information in this structure, processing texts in these languages typically require additional specialized tokenization to extract morphemes from words. Further, processing of text from other languages may benefit from other processing for stopword removal, stemming, and lemmatization.

In some embodiments, individual modeling techniques may have conditional logic for text processing that depends on the detected language. For example, the modeling technique may use a word n-gram for a European language and a character n-gram for a Chinese language. Engine 110 may pass the detected language or languages to a modeling technique when dispatched for execution.

In some embodiments, engine 110 may exclude a modeling technique or family of modeling techniques from evaluation based on language, or change its resource allocation (eg, processing priority) based on language. In some embodiments, a modeling technique may have associated metadata that specifies the language or languages for which the description is appropriate. In some embodiments, a modeling technique may have associated metadata that indicates how well the technique will perform for a set of languages. These performance estimates may be based on previous information about modeling techniques, or they may be computed through meta-machine learning based on how well different modeling techniques perform in different languages.

interaction strength

In a predictive model, two or more predictor variables (eg, features) taken together may influence a target variable in addition to or in lieu of those variables taken separately. In some embodiments, the system determines values of a metric or set of metrics that represent such interactions across a wide variety of prediction problems. In some embodiments, engine 110 incorporates metrics into a process that automatically applies appropriate interaction modeling techniques. In some embodiments, engine 110 allows a user to select from alternative pre-packaged interaction modeling techniques or apply their own custom techniques.

For example, in the context of auto insurance, there may be an interaction between "age" and "gender" features. At some age, women are safer drivers than men. At other ages, men and women are equally safe. And at other ages, men are safer. As another example, in the context of geographic analysis, there may be an interaction between latitude and longitude. Using only features of latitude and longitude without interaction, the model can predict that Singapore has low GDP per capita instead of very high GDP per capita, but using interaction information, the model can predict that Singapore has high GDP per capita. can be predicted correctly.

In general, detecting an interaction includes performing a method of detecting a statistically significant interaction between two or more features. In some embodiments, engine 110 may detect an interaction when evaluating the data set (eg, at step 408 of method 400 ). In some embodiments, individual modeling techniques may be configured to be individually tested for interaction (eg, by modeling technique builder 220 ). The former approach may provide a general solution, while the latter approach may provide more flexibility in dealing with specific situations.

In the general case, the engine 110 determines (a) which subset of features are candidates for interaction, (b) which interaction order to test in, and (c) which specific combination of features to test. can For each decision, the engine 100 may use either the results of empirical or meta-machine learning on past prediction problems. For example, some studies have shown that testing interactions over three-way yields little improvement in prediction accuracy. Thus, in some embodiments, the engine 110 determines that a particular problem has a significant probability that it is not explicitly overridden by the user, or is the exceptional case that modeling with higher order interactions will improve accuracy. If the meta-machine learning algorithm does not indicate, it can default to testing only for 2-way interactions or only testing for 2-way interactions.

After determining the set of potential interactions, engine 110 may select an interaction detection method. In some cases, engine 110 may use the same interaction detection method for all possible interactions. In other cases, engine 110 may select different interaction detection methods for different combinations of features based on the characteristics of the features. Some examples of interaction methods are ANOVA, Partial Dependence Functions (J.H. Friedman and B.E. Popescu, Predictive Learning with Rule Ensembles, 2005), GUIDE (W. Loh, Unbiased Variable Selection and Interaction Detection). Regression trees with , Statistica Sinica 12(2) 2002), Grove (D. Sorokina et al., detecting statistical interactions with additive grooves in trees, ICML, 2008), and FAST (Yin Lou et al., paired interactions). Accurate intelligent model with having, KDD, 2013).

In some embodiments, system 100 performs these techniques of detecting interactions using the same mechanisms described above to implement other machine learning techniques. After the engine 110 obtains the interaction detection result, the engine (1) constructs new features based on the important interactions and adds them to the data set and/or (2) the modeling technique can handle the interactions directly. If you have the ability to do this, you can pass a list of important interactions as parameters to the modeling technique. Execution of the modeling technique may continue in the manner described above.

Better predictive performance

A user may use the placement engine (eg, using the model placement engine 140 ) to potentially customize the model to an independent prediction service. In some cases, calculating a prediction target from observations of a feature using a particular model may require significant computational resources. In some embodiments, engine 110 provides timely responses while limiting (eg, minimizing) resource consumption, particularly when the prediction service handles requests for many models and/or frequent requests for one or more models. is configured to provide

An example is described below. In online games, game providers can support many different types of games, with many instances of each type of game and many users playing in each instance. In order to increase (eg, optimize) user satisfaction and revenue from games, such providers may want to predict the behavior of users based on the outcome of the games they play. These providers may use these predictions to provide to players or adjust future gaming experiences. Thus, such providers may rely on tens to hundreds of different models, each of which can be used to make, for example, thousands to millions of predictions per day. In addition, the relative demand for prediction across models can change significantly over time due to demographic effects, changes in popularity, differences in average game completion times, etc.

To make operational predictions, users may want a standalone prediction service that is completely separate from the model building computing infrastructure. The independent prediction service may run in different computing environments, or may be managed as separate components within the shared computing environment. Once instantiated, the execution, security, and monitoring of the service is completely decoupled from the model building environment, allowing users to deploy and manage it independently.

After instantiating the service, the deployment engine may allow the user to install a customized model into the service. To improve (eg, optimize) performance, implementation of modeling techniques appropriate for model customization may be sub-optimal for prediction. Customizing a model, for example, requires executing the same algorithm repeatedly, so it is often worth investing a significant amount of overhead to enable fast parallel execution of the algorithm. However, if the expected rate of prediction requests is not very high, the same overhead may not be worthwhile for an independent prediction service. In some cases, modeling technology developers may provide specialized versions of one or more of the component execution tasks that provide better performance characteristics in a predictive environment. In particular, implementations designed for highly parallel execution or execution on specialized processors may benefit predictive performance. Similarly, if the modeling technique involves tasks specified in a programming language, pre-compiling the tasks upon service instantiation rather than waiting for service startup or an initial request for predictions from that model may provide a performance improvement. .

Additionally, model fitting tasks typically use a different computing infrastructure than prediction services. To protect the cloud infrastructure from errors during the modeling technique execution and to prevent access to the modeling technique from other users in the cloud, the modeling technique can run in a secure computing container during model customization. However, prediction services often run on dedicated machines or clusters. Thus, eliminating the secure container layer can reduce overhead without any practical disadvantages.

Thus, based on the specific tasks executed by the model's modeling techniques, expected loads, and the characteristics of the target computing environment for prediction, the deployment engine may use a set of rules for packaging and deploying the model. These rules can optimize execution.

Because a given prediction service may run multiple models, the service may allocate computing resources across prediction requests for each model. There are two basic cases of deploying on one or more server machines and deploying on a compute cluster.

In the case of deployment to servers, the challenge is how to allocate requests among multiple servers. A prediction service may have several types of prior information. This information may include (a) an estimate of the time it will take to execute the predictions for each model constructed, (b) the expected frequency of requests for each model constructed at different times, and (c) the desired priority of model execution. The estimate of the execution time may be calculated based on measuring the actual execution speed of the predictive code for each model under one or more conditions. The desired priority of model execution may be specified by the service manager. The expected frequency of requests can be calculated from historical data of that model, predictions based on meta-machine learning models, or provided by administrators.

The service may include an objective function that combines some or all of these factors to compute a portion of the aggregate computing power of all available servers that may be initially allocated to each model. When a service receives a request and executes it, it naturally gets updated information about the execution time and expected frequency of the request. Thus, the service can recompute these parts and reallocate the model to the server accordingly.

A deployed prediction service can have two different types of server processes: routers and workers. One or more routers may form a routing service that accepts requests for predictions and assigns them to workers. An incoming request may have one or more vectors of a model identifier indicating which predictive model to use, a user or client identifier indicating which user or software system is making the request, and predictor variables for that model.

When a request comes in to a dedicated prediction service, that routing service may examine some combination of a model identifier, a user or client identifier, and a number of vectors of predictor variables. The routing service then issues requests to (1) execute the given model and/or (2) increase (eg, maximize) server cache hits for instructions and data used for a given user or client. can be assigned to workers. The routing service may also consider the number of vectors of predictor variables to achieve a mix of batch sizes submitted to each worker to balance latency and throughput.

Examples of algorithms for allocating requests for models across workers include round-robin, weighted round robin based on model computational strength and/or computational power of workers, and dynamic assignment based on reported load. can do. To facilitate rapid routing of requests to a specified server, the routing service may use a hash function that selects the same server for a given identical set of observed characteristics (eg, model identifiers). The hash function may be a simple hash function or a consistent hash function. A consistent hash function requires less overhead when the number of nodes (corresponding to workers in this case) changes. Thus, if a worker goes down or a new worker is added, a consistent hash function can reduce the number of hash keys that must be re-computed.

In addition to improving (e.g. optimizing) performance by intelligently distributing prediction requests among available services, the prediction service can also improve the performance of individual models by intelligently configuring how each worker runs each model. can be improved (eg, optimized). For example, if a given server receives a mix of requests for several different models, loading and unloading models for each request can incur significant overhead. However, aggregating requests for batch processing can introduce significant latency. In some embodiments, once an administrator specifies a latency tolerance for a model, the service can intelligently make this trade-off. For example, an urgent request may have a latency tolerance of only 100 milliseconds, in which case the server may only process one or at most a few requests. Conversely, a two-second latency tolerance can enable hundreds of batch sizes. Due to overhead, doubling the latency tolerance can increase throughput by 10x to 100x.

Similarly, using operating system threads can improve throughput while increasing latency due to thread setup and initialization overhead. In some cases, predictions may be extremely sensitive to latency. If all requests for a given model are likely to be latency sensitive, the service can configure the server handling these requests to operate in single-threaded mode. Also, if only a subset of the requests are likely to be latency sensitive, the service may allow the requester to flag a given request as sensitive. In this case, the server can only operate in single-threaded mode while servicing specific requests.

In some cases, a user's organization may have a batch of predictions that the organization would like to use a distributed computing cluster to compute as quickly as possible. Distributed computing frameworks (e.g. Apache Spark) generally allow organizations to set up clusters that run the framework, and any program designed to work with the framework is then configured with data and executable instructions. You can submit your work.

Because the execution of one prediction on a model does not affect the results of another prediction on the results of that model or any other model, prediction is a stateless operation in the context of cluster computing, and thus is generally easy to parallelize. do. Thus, given the arrangement of data and executable instructions, the normal behavior of the partitioning and allocation algorithm of the framework can result in linear scaling.

In some cases, making predictions can be part of a larger workflow where data is generated and consumed in many steps. In this case, the prediction task can be integrated with other behaviors through a publish-subscribe mechanism such as Apache Kafka. Prediction services subscribe to channels that generate new observations that need predictions. After the service makes a prediction, it publishes it on one or more channels where other programs can consume it.

performance improvement

Searching among fitting modeling techniques and/or multiple alternative techniques may be computationally intensive. Computational resources can be expensive. Some embodiments of the system 100 for generating a predictive model identify opportunities to reduce resource consumption.

Based on user preferences, engine 110 may adjust the search for the model to reduce execution time and consumption of computing resources. In some cases, a prediction problem may involve a lot of training data. The benefit of mutual validation in such cases is usually lower in terms of reducing model bias. Thus, since one operation time, typically 5 to 10 times the data amount, is much less than 5 to 10 times the data amount 1/5 to 1/10, the user can read all but each cross-validation fold. You may prefer to fit the model to the training data all at once.

Even if the user does not have a relatively large training set, the user may still want to save time and resources. In this case, engine 110 may provide a “greedier” option that uses some more aggressive search approach. First, engine 110 may try a smaller subset of possible modeling techniques (eg, only those with relatively high expected performance). Second, engine 110 can more aggressively remove poorer models in each round of training and evaluation. Third, the engine 110 may take larger steps than when searching for the optimal hyper-parameter for each model.

In general, searching for better (eg, optimal) hyper-parameters can be expensive. Thus, even if the user wants the engine 110 to evaluate a broad spectrum of potential models and not actively remove them, the engine will still save resources by limiting (eg, optimizing) hyper-parameter searches. can The cost of this search is usually proportional to the size of the data set. One strategy is to tune hyper-parameters to a small part of the data set and then extrapolate these parameters to the entire data set. In some cases, adjustments are made to account for larger amounts of data. In some embodiments, engine 110 may use one of two strategies. First, the engine 110 may make adjustments based on its experience with its modeling techniques. Second, the engine 110 participates in meta-machine learning to track how the hyper-parameters of each modeling technique change with the data set size, builds a meta-prediction model of the hyper-parameters, and then asks the user to make trade-offs. You can apply a meta model if you want.

When working with categorical prediction problems, there can be a minority class and a majority class. A minority class can be much smaller, but relatively more important, as in the case of fraud detection. In some embodiments, engine 110 "down-samples" the majority class so that the number of training observations for that class is more similar to that for the minority class. In some cases, modeling techniques can directly accommodate these weights automatically during model customization. If the modeling technique does not accommodate these weights, the engine 110 may make a post-fit adjustment proportional to the amount of down-sampling. This approach may sacrifice some accuracy for a much shorter execution time and lower resource consumption.

Some modeling techniques can be implemented more efficiently than others. For example, some modeling techniques may be optimized to run on parallel computing clusters or servers with specialized processors. The metadata of each modeling technique may represent any of these performance benefits. When engine 110 assigns computing tasks, it may detect tasks for modeling techniques that benefit from currently available computing environments. Then, during each search round, the engine 110 may use a larger chunk of the data set for this task. After that, these modeling techniques can be completed more quickly. Also, if the accuracy is good enough, there may be no need to test other modeling techniques that perform relatively poorly.

User Interface (UI) Enhancements

The engine 110 may extract more information from the user prior to model building to help the user create a better predictive model, and may provide the user with a better understanding of model performance after model fitting.

In some cases, the user may have additional information about the appropriate data set to better direct the search for an accurate predictive model. For example, a user may know that a particular observation has special importance and would like to indicate that importance. Engine 110 may allow a user to easily create new variables for this purpose. For example, one synthetic variable may indicate that the engine should use a particular observation as part of a training, validation, or holdout data partition instead of randomly assigning it to this partition. This feature can be useful in situations where certain values occur infrequently and corresponding observations must be carefully assigned to different partitions. This feature can be useful in situations where a user wants to train a model using another machine learning system and wants to perform comparisons where training, validation, and holdout partitions are identical.

Likewise, certain observations may indicate particularly important events for which the user wishes to assign additional weights. Thus, an additional variable inserted into the data set can represent the relative weight of each observation. Engine 110 may use these weights when training a model and calculating its accuracy for the purpose of producing more accurate predictions under higher weighted conditions.

In other cases, the user may have prior information about how a particular feature should behave in the model. For example, a user may know that a particular feature should have a monotonic effect on the prediction target in a certain range. In auto insurance, it is generally believed that the chance of an accident increases monotonically with age after 30. Another example is to create bands for non-continuous variables. Personal income continues, but there are analysis rules that assign values to bands such as increments from $10K to $100K, bands from $25K to $250K, and bands of all income above $250K. Then there are cases where constraints on the data set require constraints on specific features. Sometimes, a categorical variable can have a very large number of values compared to the size of the data set. A user may wish to indicate that the engine 110 should ignore category features greater than a certain number of possible categories, limit the number of categories to the most frequent X, and assign all other values to "other" categories. . In all such situations, the user interface may present the user with an option to specify this information for each detected feature (eg, at step 412 of method 400 ).

The user interface may provide guided assistance in feature transformation. For example, a user may want to convert a continuous variable to a categorical variable, but there may not be a standard rule for that variable. By analyzing the shape of the variance, the engine 110 can select an optimal number of categorical bands and points to place a “knot” in the variance that defines the boundary between each band. Optionally, the user can override this default of the user interface by adding or deleting knots as well as moving the position of the knots.

Likewise, for features that are already categorical, engine 110 may simplify the presentation by combining one or more categories into a single category. Based on the relative frequency of each observed category and the frequency it appears for values of other features, engine 110 may calculate an optimal way to combine categories. Optionally, the user can override this calculation by removing the original category from the combined category and/or putting the existing category into the combined category.

In certain cases, prediction problems may include events that occur at irregular intervals. In such cases, it can be useful to automatically create new features that capture how many times these events have occurred within a particular time frame. For example, in an insurance forecasting problem, a data set may hold a record of each time a policyholder makes an insurance claim. However, when building a model that predicts future risk, it may be more useful to consider how much insurance the policyholder had in the past X years. The engine may detect this situation when evaluating a data set (eg, at step 408 of method 400 ) by detecting a data structure relationship between a record corresponding to an entity and another record corresponding to an event. . Upon presenting the data set to the user (eg, at step 410 ), the user interface may automatically create or present such features. Calculated to maximize the statistical dependence between occurrences of future events of these variables, or some other experience may also be used to suggest time frame thresholds based on the frequency at which events occur. The user interface may also allow the user to override the creation of such features, force the creation of such features, and override the presented time frame threshold.

When the system makes predictions based on the model, users can review these predictions and search for outliers. For example, the user interface may provide a list of all or a subset of the predictions for the model and indicate which ones were extreme in terms of the magnitude of the predictor's value or the low probability of having that value. It is also possible to provide an intuition as to the reasons for the extreme values. For example, a particularly high value in an auto insurance risk model may have the reason "age < 25 marital status = single".

Further description of some embodiments

Although the examples provided herein may describe modules residing on separate computers or operations performed by separate computers, the functionality of these components may be implemented on a single computer or any larger number of computers in a distributed fashion. It should be understood that there is

The above-described embodiments may be implemented in any number of ways. For example, embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or collection of processors, whether provided on a single computer or distributed among multiple computers. It should also be understood that the computer may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Also, a computer is not generally considered a computer, but may be embedded in a device having suitable processing capabilities, including a Personal Digital Assistant (PDA), a smart phone, or any other suitable portable or stationary electronic device.

These computers may be interconnected by one or more networks of any suitable form, including local area networks or enterprise networks or wide area networks such as the Internet. Such networks may be based on any suitable technology, may operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.

In addition, the various methods or processes described herein may be coded as software executable on one or more processors employing any one of a variety of operating systems or platforms. Further, such software may be written using any of a number of suitable programming languages and/or programming or scripting tools, and may also be compiled as executable machine language code or intermediate code running on a framework or virtual machine. .

In this regard, some embodiments include computer readable media (or a plurality of computer readable media) encoded with one or more programs that, when executed on one or more computers or other processors, perform the methods of implementing the various embodiments described above ( for example, computer memory, one or more floppy disks, compact disks, optical disks, magnetic tapes, flash memory, field programmable gate arrays or other semiconductor devices, or circuit configurations of other types of computer storage media). The computer-readable medium or media may be non-transitory. The computer-readable medium or media may be transportable such that a program or programs stored on one or more other computers or other processors may be loaded to implement various aspects of predictive modeling as described above. The terms “program” or “software” are used herein in their generic sense to refer to any type of computer code or computer-executable instructions that may be employed to program a computer or other processor to implement the various aspects described herein. is used in Further, in accordance with an aspect of the present invention, one or more computer programs that perform the predictive modeling method when executed need not reside on a single computer or processor, but among a number of different computers or processors to implement the various aspects of predictive modeling. It should be understood that it can be distributed in a modular fashion.

Computer-executable instructions can be in many forms, such as program modules, being executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functions of the program modules may be combined or distributed as required in various embodiments.

Further, the data structures may be stored in computer-readable media in any suitable form. For simplicity of illustration, a data structure may be represented as having fields associated through the location of the data structure. This relationship may be similarly achieved by allocating locations on a computer-readable medium conveying the relationship between the fields to the storage for the fields. However, any suitable mechanism may be used to establish relationships between information in fields of a data structure, including the use of pointers, tags, or other mechanisms for establishing relationships between data elements.

Also, predictive modeling techniques may be implemented as a method in which examples are provided. The actions performed as part of this method may be ordered in any suitable manner. Thus, although shown as sequential operations in the illustrated embodiments, embodiments may be constructed in which the operations are performed in an order other than that shown, which may include performing several operations simultaneously.

In some embodiments, the method(s) may be implemented as computer instructions stored in a portion of a computer's random access memory to provide control logic affecting the processes described above. In such an embodiment, the program may be written in any of a number of high-level languages such as FORTRAN, PASCAL, C, C++, C#, Java, JavaScript, Tcl, or BASIC. In addition, programs can be recorded as scripts, macros or functions included in commercial software such as EXCEL or VISUAL BASIC. Additionally, the software may be implemented in assembly language for a microprocessor resident on a computer. For example, software may be implemented in Intel 80x86 assembly language when configured to run on an IBM PC or PC clone. The software may be embodied in an article of manufacture including, but not limited to, “computer-readable program means” such as a floppy disk, hard disk, optical disk, magnetic tape, PROM, EPROM or CD-ROM.

The various aspects of the invention may be used alone, in combination, or in various arrangements not specifically described above, and thus the invention finds its application in the details and arrangement of components set forth in the foregoing description or in the drawings shown. do not limit For example, an aspect described in one embodiment may be combined in any manner with an aspect described in another embodiment.

Terms

The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting.

As used herein and in the claims, the indefinite articles “a” and “an” should be understood to mean “at least one” unless explicitly indicated to the contrary. The phrase "and/or" as used in the specification and claims should be understood to mean "either one or two" of the elements so combined, i.e., the elements are present in combination in some cases and separate in other cases. exist as an enemy. A plurality of elements listed as “and/or” should be construed in the same way, ie, as “one or more” of the combined elements. Other elements may optionally be present other than those specifically identified by the "and/or" clause, whether related or unrelated to the specifically identified element. Thus, as a non-limiting example, reference to "A and/or B" when used in conjunction with a non-limiting language such as "comprising" may in one embodiment refer to only A (optionally elements other than B). include); in other embodiments, with only B (optionally including elements other than A); In another embodiment, both A and B (optionally including other elements); etc.

As used in the specification and claims, “or” should be understood to have the same meaning as “and/or” as defined above. For example, when separating items from a list, "or" or "and/or" should be construed as inclusive, i.e., including one or more numbers or lists of one or more elements, optionally including an unlisted addition. There is an item. When used in a claim, such as "only one" or "exactly one", it shall indicate the inclusion of exactly one element of a number or list of elements, unless expressly indicated to the contrary, such as "consisting of". In general, the term "or," as used, refers to an exclusive alternative (i.e., "one or the other one but not both"). "Consisting essentially of" when used in the claims has the meaning commonly used in the field of patent law.

As used in the specification and claims, the phrase “at least one” in reference to a list of one or more components should be understood to mean at least one element selected from the elements of the list of any one or more elements, but the list of elements It is not necessary to include at least one of each element and list of elements specifically listed in This definition also allows for elements to optionally be present other than those specifically identified within the list of elements in which the phrase "at least one" is referenced, with or without relation to the specifically identified element. Thus, by way of non-limiting example, “at least one of A and B” (or equivalently, “at least one of A or B” or equivalently “at least one of A and/or B”) means, in one or more embodiments, B optionally comprising at least one A that does not contain (and optionally includes elements other than B); In other embodiments A comprises at least one B absent (and optionally comprising elements other than A), optionally at least one B; In another embodiment, one or more A and optionally one or more B may be combined with one or more (and optionally other elements) one or more A's, and the like.

Uses of "including," "comprising," "having," "containing," "involving," and variations thereof, refer to the enumerated item thereafter. and additional items.

The use of ordinal terms such as “first,” “second,” “third, etc.” in a claim to modify a claim element means that the use of an ordinal term in one claim element relative to the temporal order or other element in which the operation of the method is performed. does not imply precedence, precedence, or order of An ordinal term is only used as a label to distinguish one claim element with a particular name from another element with the same name (but use the ordinal term) to distinguish claim elements.

equivalent

Accordingly, having described various aspects of at least one embodiment of the present invention, it should be understood that various changes, modifications, and improvements will readily occur to those skilled in the art. Such changes, modifications and improvements are intended to be a part of this invention and are intended to be within the spirit and scope of the invention. Accordingly, the foregoing description and drawings are by way of example only.

Claims (131)

A predictive modeling method comprising:
performing a predictive modeling procedure, the predictive modeling procedure comprising:
(a) obtaining time series data comprising one or more data sets, each data set comprising a plurality of observations, each observation comprising (1) an indication of a time associated with the observation and (2) one or more variables obtaining the time series data including respective values of ;
(b) determining a time interval of the time series data;
(c) identifying the one or more variables as targets and zero or more other variables as features;
(d) determining an expected range and a skip range associated with a prediction problem represented by the time series data, wherein the expected range represents a duration of time for which values of the targets are to be predicted, and wherein the skip range is the expected range and the skip range, representing the time lag between the time associated with the earliest prediction in the expected range and the time associated with the latest observation on which the predictions in the expected range are based. determining;
(e) generating training data from the time series data, the training data comprising a first subset of observations in at least one of the data sets, the first subset of observations comprising the observations training-input and training-output collections of generating the training data, wherein a range separates an end of the training-input time range from a beginning of the training-output time range, and a duration of the training-output time range is at least as long as the expected range;
(f) generating testing data from the time series data, the testing data comprising a second subset of observations in at least one of the data sets, the second subset of observations comprising the observations and test-input and test-validation collections of , the skip range separates the end of the test-input time range from the beginning of the test-validation time range, and the duration of the test-validation time range is at least as long as the expected range. to do;
(g) fitting a predictive model to the training data; and
(h) testing the customized model against the testing data. According to claim 1,
wherein the time interval of the time series data is determined based at least in part on times associated with at least a subset of observations included in at least one of the data sets. 3. The method of claim 2,
The step of determining the time interval of the time series data includes:
determining, for each of the data sets, a respective time interval of the data set;
determining that the time intervals of the data sets are constant; and
and setting a time interval of the time series data to a time interval of the data sets. 4. The method of claim 3,
Determining the time interval of the data set comprises:
determining, for one or more pairs of successive observations included in the data set, a respective time period between the successive observations;
determining that the period of time between the pair of consecutive observations is constant; and
and setting a time interval of the data set to a time interval between the pair of consecutive observations. 3. The method of claim 2,
The step of determining the time interval of the time series data includes:
determining, for each of the data sets, a respective time interval of the data set; and
determining that the time intervals of at least two of the data sets are different;
The time interval of the time series data is at least partially in at least one of (1) respective portions of the observations included in each of the data sets, and (2) each of the respective time intervals of the data sets. A predictive modeling method that is determined based on 6. The method of claim 5,
Determining each time interval of the data set comprises:
determining each time period between each successive pair of observations included in the data set;
If the time period between the pair of consecutive observations represents a plurality of non-uniform durations, then the time interval of the data set is: (1) respective portions of the pair of consecutive observations representing each of the non-uniform durations, and ( 2) is determined based at least in part on at least one of the durations of the time period;
where the time period between the pair of consecutive observations is a constant duration, the time interval of the data set is the respective duration of the time period. 6. The method of claim 5,
wherein the time interval of the time series data is the shortest time interval that is an integer multiple of each of the respective time intervals of the data sets. 8. The method of claim 7,
For each data set, if the time interval of the data set is shorter than the time interval of the time series data, convert the time interval of the data set to the time interval of the time series data by down-sampling the observations in the data set A predictive modeling method further comprising the step of: 9. The method of claim 8,
Down-sampling the observations in the data set comprises: for each instance of a time interval of the time series data in a time period corresponding to the data set:
identifying all observations in the data set associated with times corresponding to the respective instances of the time interval of the time series data;
aggregating the identified observations to produce an aggregate observation; and
replacing the identified observations in the data set with the aggregate observation. 10. The method of claim 9,
and the number of identified observations corresponding to instances of the time interval of the time series data is equal to a ratio between the time interval of the time series data and the time interval of the data set. 10. The method of claim 9,
Aggregating the identified observations comprises: calculating the value of each variable of the aggregated observation, (1) the value of the corresponding variable included in the earliest of the identified observations, (2) the most recent of the identified observations. (3) the maximum value of the corresponding variable values included in the identified observations, (4) the minimum value of the corresponding variable values included in the identified observations, (5) the corresponding variable values included in the identified observations; and setting to the mean of the corresponding variable values included in the observations, or (6) the value of a function of the corresponding variable values included in the identified observations. 6. The method of claim 5,
wherein the time interval of the time series data is selected from the group consisting of time intervals of the data sets. 13. The method of claim 12,
the data sets include a first data set representing a first time interval and a second data set representing a second time interval greater than the first time interval, wherein the second time interval is selected as the time interval of the time series data become,
The method is:
converting a time interval of the first data set to a time interval of the time series data by down-sampling the observations in the first data set. 6. The method of claim 5,
wherein the time interval of the time series data is different from each of the time intervals of the data set. According to claim 1,
At least one group of observations of the time series data includes respective values of a first variable, the method comprising:
Before fitting the predictive model to the training data and testing the customized model against the testing data:
determining that values of the first variable include time values;
for each observation in the group, generating a respective value of a second variable, wherein the value of the second variable comprises an offset between the time value of the first variable and a reference time value generating each value of the second variable; and
and adding the values of the second variable to respective observations in the group. 16. The method of claim 15,
and removing values of the first variable from observations within the group. 16. The method of claim 15,
wherein the reference time includes the date of the event. 18. The method of claim 17,
wherein the event comprises birth, marriage, graduation from school, commencement of employment with an employer, or commencement of work in a particular position. According to claim 1,
The variables include a first variable and a second variable, the method comprising:
determining that changes in values of the first variable and the second variable are correlated with a time lag between changes in the value of the first variable and the correlated changes in the value of the second variable; ; and
and displaying, via a graphical user interface, graphical content representing a duration of a time lag between changes in the value of the first variable and correlated changes in the value of the second variable. . According to claim 1,
The expected range includes (1) a time interval of the time series data, (2) the number of observations included in the time series data, (3) a period of time corresponding to the time series data, and (4) microseconds, milliseconds, seconds , minutes, hours, days, weeks, months, quarters, seasons, years, decades, hundred years and millennia, the predictive modeling is determined based at least in part on at least one of a natural time period selected from the group consisting of: Way. 21. The method of claim 20,
The predicted range is an integer multiple of a time interval of the time series data. 22. The method of claim 21,
and an interval between times associated with successive predictions of the expected range is equal to a time interval of the time series data. According to claim 1,
The skip range includes a latency in the collection of the time series data, a waiting time in communication of the time series data, a waiting time in the analysis of the time series data, a waiting time in communication of the analysis of the time series data, and a waiting time in implementation of actions based on the analysis of the time series data. According to claim 1,
The duration of the training-input time range is defined as the total number of observations included in the time series data, the amount of variation in values of at least one of the variables over time, and the periodicity in at least one of the values of the variables. determining based, at least in part, on at least one of an amount of variation, a consistency of variation in at least one values of the variables over a plurality of time periods, and a duration of the expected range. Way. 25. The method of claim 24,
fitting the predictive model to training data comprises fitting the predictive model to a subset of training data corresponding to a portion of the training-input time range;
wherein the portion of the training-input time range begins at a time subsequent to a start time of the training-input time range and ends at an end time of the training-input time range. 26. The method of claim 25,
wherein the duration of the portion of the training-input time range is an integer multiple of the duration of the expected range. According to claim 1,
and down-sampling the training data prior to fitting the predictive model to the training data. 28. The method of claim 27,
Down-sampling the training data comprises:
removing, from the training data, all observations obtained from at least one of the data sets. 28. The method of claim 27,
Down-sampling the training data comprises:
setting the down-sampled time interval of the training data to an integer multiple of the time interval of the time series data; and
For each instance of the down-sampled time interval of the training data:
identifying all observations in the training data associated with times corresponding to the respective instances of the down-sampled time interval of the training data;
aggregating the identified observations to aggregate an aggregate observation; and
replacing the identified observations in the training data with the aggregate observation. According to claim 1,
and down-sampling the testing data prior to testing the customized model against the testing data. According to claim 1,
The predictive modeling method further comprising the step of performing cross-validation of the predictive model. 32. The method of claim 31,
wherein the training data is first training data, the testing data is first testing data, the customized model is a first customized model, and performing mutual validation of the predictive model includes:
(i) generating second training data and second testing data from the time series data, the second training data comprising a third subset of observations in at least one of the data sets; generating the second training data and second testing data, wherein the second testing data comprises a fourth subset of observations in at least one of the data sets;
(j) fitting the predictive model to the second training data to obtain a second customized model; and
(k) testing the second customized model against the second testing data. 33. The method of claim 32,
the first subset of observations corresponds to a sliding training window covering a first range of training times, each observation included in the first subset is associated with a time within the first range of training times, wherein the third subset of s corresponds to a sliding training window covering a second range of training times, each observation included in the third subset is associated with a time within the second range of training times; and the earliest time of the first range of s is earlier than the earliest time of the second range of training times. 34. The method of claim 33,
wherein the second subset of observations corresponds to a sliding test window covering a first range of testing times, each observation included in the second subset is associated with a time within the first range of testing times; wherein the fourth subset of s corresponds to a sliding test window covering a second range of testing times, each observation included in the fourth subset is associated with a time within the second range of testing times; and the earliest time of the first range of s is earlier than the earliest time of the second range of testing times. 35. The method of claim 34,
and the first testing time range partially overlaps the second training time range. 36. The method of claim 35,
and the second testing time range does not overlap any portion of the first training time range and does not overlap any portion of the second training time range. 33. The method of claim 32,
and partitioning the time series data into a plurality of partitions including at least a first partition and a second partition. 38. The method of claim 37,
and partitioning the time series data into a plurality of partitions comprises assigning each of the data sets to a corresponding partition. 38. The method of claim 37,
Partitioning the time-series data into the plurality of partitions includes temporally partitioning the time-series data. 40. The method of claim 39,
wherein each of the partitions corresponds to a respective portion of a period associated with the time series data, and each observation included in the time series data is assigned to a partition corresponding to a portion of a period of time that matches the time associated with the observation. modeling method. 38. The method of claim 37,
the first training data comprises a subset of observations included in the first partition of the time series data;
the first testing data includes respective subsets of observations included in all partitions of the time series data except for the first partition;
the second training data comprises a subset of observations included in the second partition of the time series data; and
and the second testing data includes respective subsets of observations included in all partitions of the time series data except for the second partition. 33. The method of claim 32,
The first division of the time series data includes the first training data and the second training data and the first testing data and the second testing data, and the second division of the time series data includes holdout data. comprising, the method comprising:
and testing the first customized model and the second customized model against the holdout data. 43. The method of claim 42,
The predictive modeling method, wherein the predictive model is not customized to the holdout data. According to claim 1,
and performing nested mutual validation of the predictive model. 45. The method of claim 44,
The step of performing nested mutual validation of the predictive model includes:
dividing the time series data into a first plurality of divisions including at least a first division of the time series data and a second division of the time series data; and
the first division of the time series data into a plurality of divisions of the first division of the time series data comprising at least a first division of the first division of the time series data and a second division of the first division of the time series data partitioning the partition;
wherein the training data comprises the first partition of the first partition of the time series data, and the testing data includes at least the first partition of the time series data other than the first partition of the first partition of the time series data. A predictive modeling method comprising a plurality of partitions. 46. The method of claim 45,
wherein the training data is first training data, the testing data is first testing data, the customized model is a first customized model, and performing nested mutual validation of the predictive model comprises:
(i) generating second training data and second testing data from the first partition of the time series data, the second training data comprising the second partition of the first partition of the time series data; wherein the second testing data comprises at least a plurality of partitions of the first partition of the data set other than the second partition of the first partition of the time series data. step;
(j) fitting the predictive model to the second training data to obtain a second customized model; and
(k) testing the second customized model against the second testing data. 47. The method of claim 46,
The step of performing the nested mutual validation includes:
testing the first customized model and the second customized model for the second partition of the time series data; and
comparing the first customized model and the second customized model based on a result of testing the first customized model and the second customized model on the second partition of the time series data; A predictive modeling method comprising: According to claim 1,
determining, for the customized model, model-specific predictive values of one or more features of the time series data. 49. The method of claim 48,
pruning a feature from the time series data based at least in part on model-specific predicted values of the features, generating a feature derived from two or more features in the time series data, and comparing the derived feature to the from the group consisting of at least one of adding to time series data, blending the predictive model with other predictive models, and allocating resources during the process of evaluating the adequacy of predictive modeling procedures for the predictive problem. The predictive modeling method, further comprising: performing at least one selected action. According to claim 1,
determining, based, at least in part, on at least one of characteristics of the predictive problem and attributes of respective predictive modeling procedures, appropriateness of the plurality of predictive modeling procedures for the predictive problem;
selecting one or more predictive modeling procedures from the plurality of predictive modeling procedures based on the determined suitability of the selected modeling procedures for the predictive problem; and
and performing the one or more predictive modeling procedures. 51. The method of claim 50,
Performing the one or more predictive modeling procedures comprises:
sending an instruction to a plurality of processing nodes, the instruction comprising a resource allocation schedule allocating resources of the processing nodes for execution of the selected modeling procedures, wherein the resource allocation schedule is configured for the prediction problem. based at least in part on the relevance of selected modeling procedures;
receiving results of execution of the selected modeling procedures by the plurality of processing nodes according to the resource allocation schedule, the results comprising: predictive models generated by the selected modeling procedures; and the prediction problem; the receiving comprising at least one of the scores of the generated models for associated time series data; and
selecting, from the generated models, a predictive model for the predictive problem based at least in part on a score of the selected predictive model. According to claim 1,
and mixing the customized model with other customized models to create a blended predictive model. According to claim 1,
and deploying the customized model. 54. The method of claim 53,
wherein the time series data is first time series data, and placing the customized model comprises generating one or more predictions by applying the customized model to second time series data representing one or more instances of the prediction problem. and wherein the first time series data does not include the second time series data. 54. The method of claim 53,
wherein the time series data is first time series data, and wherein deploying the customized model comprises refreshing the customized model based at least in part on second time series data. 56. The method of claim 55,
wherein the customized model is a first customized model, and refreshing the customized model based at least in part on the second time series data comprises:
performing the predictive modeling procedure on the second time series data to generate a second customized model; and
mixing the first customized model and the second customized model to generate a refreshed predictive model. 56. The method of claim 55,
Refreshing the customized model based at least in part on the second time series data comprises:
and performing the predictive modeling procedure on third time series data including at least a portion of the first time series data and at least a portion of the second time series data to generate a refreshed predictive model. 54. The method of claim 53,
The customized model is deployed on one or more servers, and other customized models are also deployed on the one or more servers, and a prediction request for the customized model and the other customized model generates a prediction: (1) among the servers based at least in part on at least one of an estimate of the amount of time used by each of the customized models to Assigned, predictive modeling method. 59. The method of claim 58,
Each prediction request is assigned to a respective thread, each prediction request has an associated latency-sensitivity value, and the number of threads executing on a particular server is the latency of the threads executing on the particular server. - a predictive modeling method, determined based at least in part on the sensitivity values. According to claim 1,
determining a value of a metric representing an interaction strength of two or more features included in the time series data; and
When the value of the metric exceeds a threshold, generating time-series values of a new feature based on the values of the two or more features, and adding the new feature to the time-series data. . According to claim 1,
Further comprising the step of determining the temporal resolution of the time series data, predictive modeling method. According to claim 1,
wherein the targets are identified based on user input. In the predictive modeling apparatus,
A memory configured to store a machine-executable module encoding a predictive modeling procedure, the predictive modeling procedure comprising a plurality of operations including at least one pre-processing operation and at least one model-fitting operation Memory; and
at least one processor configured to execute the machine-executable module;
execution of the machine-executable module causes the apparatus to perform the predictive modeling procedure, the predictive modeling procedure comprising performing the pre-processing operation and performing the model-fitting operation;
Performing the pre-processing task is:
(a) obtaining time series data comprising one or more data sets, each data set comprising a plurality of observations, each observation comprising (1) an indication of a time associated with the observation and (2) one or more variables obtaining the time series data including the respective values of
(b) determining a time interval of the time series data;
(c) identifying the one or more variables as targets and zero or more other variables as features; and
(d) determining an expected range and a skip range associated with a prediction problem represented by the time series data, wherein the expected range indicates a duration of a period for which values of the targets are to be predicted, and wherein the skip range is the expected range indicating a time lag between a time associated with the earliest prediction in , and a time associated with the most recent observation upon which the prediction in the expected range is based;
Performing the model-fitting task is:
(e) generating training data from the time series data, the training data comprising a first subset of observations in at least one of the data sets, the first subset of observations comprising the observations training-input and training-output collections of generating the training data, wherein a range separates the end of the training-input time range from the beginning of the training-output time range, and wherein a duration of the training-output time range is at least as long as the expected range;
(f) generating testing data from the time series data, the testing data comprising a second subset of observations in at least one of the data sets, the second subset of observations comprising the observations and test-input and test-validation collections of , the skip range separates the end of the test-input time range from the beginning of the test-validation time range, and the duration of the test-validation time range is at least as long as the expected range. step to do,
(g) fitting a predictive model to the training data; and
(h) testing the customized model against the testing data. 64. The method of claim 63,
wherein the machine-executable module comprises a directed graph representing dependencies between the tasks. delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete

Images