JP2023520313A - Generating Performance Forecasts with Uncertainty Intervals - Google Patents

Abstract

Generating performance predictions with uncertainty intervals. A technique for generating a performance prediction for a machine learning model with uncertainty intervals includes obtaining a first model configured to perform a task and a manufacturing data set. At least one metric that predicts performance of the first model when performing a task on the manufacturing data set is generated using the second model. The second model is a meta-model associated with the first model. At least one value predicting uncertainty of at least one metric predicting performance of the first model when performing a task on the manufacturing data set is generated using the third model. A third model is a meta-meta model associated with the second model. A representation of at least one metric predicting performance of the first model and at least one value predicting uncertainty of the at least one metric is provided.

Description

The present invention relates generally to machine learning, and more particularly to techniques for generating performance predictions for machine learning (ML) models with uncertainty intervals.

In recent years, ML models have been increasingly used for various tasks including image recognition, speech processing, language translation and object classification, to name a few. In general, the ML models used for these tasks have become increasingly complex and expensive in terms of computational resources and time required to train and maintain the models. Furthermore, due to the different goals of each task, models trained for one task or domain are typically unusable for other domains, even if the models are closely related. The models themselves can differ dramatically from each other, such that . Because of this possibility, attempts have been made to predict a model's performance on a set of data for a given task. Performance prediction, however, suffers from multiple sources of uncertainty, which can affect the accuracy of the performance prediction.

One embodiment presented herein includes a computer-implemented method for generating performance predictions for machine learning (ML) models. The computer-implemented method generally includes obtaining a first model configured to perform a task and a manufacturing data set including unlabeled data. The computer-implemented method also includes using the second model to generate at least one metric that predicts performance of the first model when performing a task on the manufacturing data set. The second model is a meta-model associated with the first model. The computer-implemented method uses one or more third models to predict uncertainty in at least one metric that predicts performance of the first model when performing a task on the manufacturing data set. Further comprising generating at least one value. Each of the one or more third models is a meta-meta-model associated with the second model. The computer-implemented method further includes providing a representation of at least one metric that predicts performance of the first model and at least one value that predicts uncertainty of the at least one metric.

In another embodiment, a computer program product comprising a storage medium having computer readable program code that enables a processing unit to perform one or more aspects of the disclosed methods and the disclosed methods. and systems having processors, memory and application programs configured to implement one or more of

1 is a block diagram illustrating a networked system used to predict the performance of a model on a task and the uncertainty of the model's performance on a task, according to one embodiment; FIG. FIG. 4 illustrates a stacked meta-modeling workflow for generating model performance predictions and uncertainty intervals for model performance predictions, according to one embodiment. FIG. FIG. 4 illustrates a stacked meta-modeling workflow for generating model performance predictions and uncertainty intervals for model performance predictions, according to one embodiment. FIG. 1 illustrates an example procedure for generating one or more datasets for training a meta-meta-model, according to one embodiment. 4 illustrates an example of generating features for training a meta-meta model, according to one embodiment. 4 is a flowchart of a method for predicting the performance of a model on a task and the uncertainty for the performance of the model on a task, according to one embodiment. A method for training a meta-model and a meta-meta-model of a stacked meta-model workflow for generating model performance predictions and uncertainty intervals for model performance predictions, according to one embodiment is a flow chart. 4 is a flowchart of a method for training a meta-meta model, according to one embodiment. FIG. 11 illustrates a simulation of an example performance prediction and uncertainty interval, according to one embodiment. FIG.

ML tools are increasingly used for a variety of tasks including, but not limited to, regression and classification, optimization, prediction, and so on. However, in some cases the performance of a given model on a task may change over time. For example, a model's ability to predict accurately may deteriorate over time depending on how much the underlying data differs from the model's training data. Many conventional techniques can now be used to predict the performance of a model on a given task. For example, one conventional technique involves detecting the amount of movement of one or more features of the underlying data. Other prior art techniques measure the accuracy of the model by measuring the rate of correct predictions (e.g., the number of times any class was correctly predicted, the area under the receiver operating characteristic (ROC) curve, etc.) over time. including tracking However, these prior art techniques are only used for performance prediction and do not capture model uncertainty.

In addition, many conventional techniques exist for predicting uncertainties only (eg, without considering the underlying performance prediction task). Examples of such techniques include, but are not limited to, confidence intervals and probability (p) values. Additionally, in Bayesian modeling, there are different types of model uncertainty, including, for example, chance uncertainty and epistemological uncertainty. Anecdotal uncertainty captures the inherent uncertainty of data, examples of which can include noise, various data omissions, confusion, and the like. Epistemological uncertainty captures uncertainty due to models (eg, model architecture, model parameters, model assumptions, parameter estimates, insufficient training, etc.).

One problem with the methods described above is that they generally assume that choices exist over the architecture of the base model, allowing predictions of model performance and/or model uncertainty. In the absence of this choice, some methods, such as aggregation techniques, can be used to capture the uncertainty, but these methods assume that the base model is a white box (e.g., internal architecture and model) whose parameters are visible or known). However, in many situations, neither of the above assumptions are acceptable when a customer-provided model is used. Accordingly, it may be desirable to provide techniques for generating both model performance predictions and model uncertainty predictions.

Embodiments of the present disclosure provide techniques for predicting the performance of the ML model on the underlying task along with uncertainty intervals for the predicted performance of the ML model. More specifically, given a trained ML model and a set of (unlabeled) manufacturing data (which may or may not be significantly different from the training data for the model), embodiments perform Predict model performance (e.g., accuracy or other performance or quality-related metrics) and uncertainty intervals (e.g., bands or error bars around predictions) on a set, for specific test instances or test instances can explain the uncertainty associated with the model in batches of

In one embodiment, detailed further below, a stacked meta-modeling method is used to generate performance and uncertainty predictions. For example, embodiments include two levels of meta-models: (1) a first level meta-model that predicts the performance of the base model; ) Generate a second level meta-meta model that predicts the uncertainty of the first level meta model. By using the stacked meta-modeling method, embodiments can operate models of arbitrary architectures (e.g., the stacked meta-modeling method is agnostic to the model architecture, and the inner parameters of the model ), multiple types of uncertainty (eg, anecdotal uncertainty, epistemological uncertainty, etc.) can be captured.

As used herein, "meta-model" generally means a model of (lower-level) models, and "meta-meta-model" generally means a model of meta-models. Meta-models, for example, capture patterns observed in lower-level models that interact with data. In various embodiments of the present disclosure, visual tasks (e.g., processing images for action recognition, object detection, face recognition, number or letter classification, etc.) are used as examples to illustrate the functionality of ML models. Note that However, embodiments of the present disclosure are readily applicable to any number of domains, using any input (eg, video, audio, text, etc.). Further, as used herein, the "performance" of a model may refer to one or more accuracy-related metrics of the model on unlabeled data. Similarly, "performance prediction" may refer to a model-based method of predicting one or more performance- or quality-related metrics of an underlying base model in unlabeled data.

The description of various embodiments of the invention is presented for purposes of illustration and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein are used to best describe principles of embodiments, practical applications, or technical improvements over technology found on the market or disclosed herein by a person of ordinary skill in the art. It was chosen to make it possible to understand the embodiment.

Reference will now be made to embodiments presented in this disclosure. However, the scope of this disclosure is not limited to particular described embodiments. Instead, any combination of the following features and elements, whether related to different embodiments or not, is contemplated to implement and practice the discussed embodiments. Furthermore, although the embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether particular advantages are achieved by a given embodiment is subject to the present invention. It is not a limitation on the scope of disclosure. Accordingly, the following aspects, features, embodiments and advantages are merely illustrative and should not be considered elements or limitations of the appended claims, except where expressly recited in the claims. Similarly, references to the "present invention" should not be construed as a generalization of any inventive subject matter disclosed herein, except as expressly recited in the claims, rather than to the appended claims. should not be considered an element or limitation of the scope of

FIG. 1 is a block diagram illustrating a networked system 100 used to generate performance predictions and model uncertainty predictions, according to one embodiment. System 100 includes computing system 102 . Computing system 102 may also be connected to other computers through network 140 . Network 140 may include one or more networks of various types, such as a local area or local area network (LAN), a general wide area network (WAN), a telecommunications or cellular network or a public network. Including networks (eg, the Internet) or combinations thereof.

Computing system 102 generally includes one or more processors 104 , storage 108 , network interface 110 , input devices 152 and output devices 154 connected to memory 106 via bus 150 . Computing system 102 is generally under the control of an operating system (not shown). Examples of operating systems include the UNIX(R) operating system, versions of the Microsoft Windows(R) operating system and distributions of the Linux(R) operating system (UNIX stands for The Open Group, Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both, and Linux is a trademark of Linus Torvalds in the United States, other countries, or both). More generally, any operating system that supports the functionality disclosed herein may be used. Processor 104 is included to represent a single CPU, multiple CPUs, a single CPU with multiple processing cores, and the like.

Memory 106 may include a variety of computer readable media selected for performance or other capabilities, ie, volatile and/or nonvolatile media, removable and/or non-removable media, and the like. Memory 106 may include cache, random access memory (RAM), and the like. Storage 108 may be a disk drive or flash storage device. Although shown as a single unit, storage 108 may be fixed and/or removable storage devices such as fixed disk drives, solid state drives, removable memory cards, optical storage devices, network It may be a combination of Attached Storage (NAS) or Storage Area Network (SAN). Network interface 110 may be any type of network communication device that enables computing system 102 to communicate with other computing systems or devices over network 140 .

Input device 152 provides input to computing system 102 . For example, a keyboard and/or mouse may be used. Output device 154 may be any device for providing output to a user of computing system 102 . For example, output device 154 may be any conventional display screen. Although shown separately from input device 152, output device 154 and input device 152 may be combined. For example, a display screen with an integrated touch screen may be used.

Here, storage 108 includes manufacturing dataset 130 , base model 132 , test dataset 134 , training dataset 136 and offline model and dataset 138 . A base model 132 is a model used to analyze a particular task (eg, a customer model for image classification). In general, any suitable type of model can be used for base model 132, examples of which include artificial neural networks, decision trees, support vector machines, regression analysis, Bayesian networks, genetic algorithm, etc., but is not limited to these. The base model 132 can be a black box model (eg, a model whose architecture and/or parameters are unknown) or a white box model (eg, a model whose architecture and/or parameters are known). can do.

The base model 132 may be trained/developed on a particular training dataset (e.g., training dataset 136) and using a labeled test dataset (e.g., test dataset 134 containing classification labels). may be evaluated. Manufacturing dataset 130 is an unlabeled dataset for which no ground truth (eg, classification labels) is available. After base model 132 is developed, manufacturing data set 130 may be gathered from operating conditions. Offline models and/or datasets 138 include various models and datasets, some of which may be relevant to the original task of the base model 132, some of which may be 132 may be unrelated to the original task. Off-line models and/or datasets 138 are discussed in further detail below with respect to FIGS. 2B and 3. FIG.

Memory 106 includes a prediction engine 120 configured to implement one or more of the techniques described herein for generating uncertainty intervals for performance predictions and models. be done. Prediction engine 120 includes performance component 122 and uncertainty component 124, each of which may include software, hardware, or a combination thereof. The prediction engine 120 generates performance predictions (or predicts performance) of the base ML model 132 (via the performance component 122) and (via the uncertainty component 124) e.g., stacked meta-modeling A method is configured to generate uncertainty intervals for performance prediction. For example, the performance component 122 uses the meta-model (of the base model 132) to estimate the performance (eg, one or more accuracy-related metrics) of the base model 132 on the unlabeled manufacturing dataset 130. can be predicted. The meta-model may be trained using one or more of base model 132 , test dataset 134 and training dataset 136 .

Similarly, the uncertainty component 124 can use the meta-meta-model (of the base model 132 meta-model) to predict the uncertainty of the performance component 122 meta-model. The meta-meta-model is one of the off-line model/dataset 138, the meta-model of the performance component 122, the base model 132, the manufacturing dataset 130, the test dataset 134 and the training dataset 136 collection or May be trained with multiple. In one embodiment, the uncertainty component 124 can generate uncertainty intervals (also known as error bars) that represent the uncertainty of the meta-model. The uncertainty interval details a band (or tolerance) around the output of performance component 122 (eg, the predicted performance of base model 132). This band (or tolerance) around the prediction can explain the uncertainty associated with the base model 132 in a particular test instance (eg, of the manufacturing data set 130). Performance component 122 and uncertainty component 124 are described in further detail below with respect to FIGS. 2A and 2B.

Because FIG. 1 merely illustrates one illustrative example of a system 100 that can generate performance predictions with model uncertainty intervals, other configurations of system 100 generate performance predictions with model uncertainty intervals. Note that it can be applied to For example, in some embodiments, one or more of the contents of storage 108 (eg, manufacturing dataset 130, base model 132, test dataset 134, training dataset 136, and offline model/dataset 138) or one or more of the content (eg, performance component 122 and uncertainty component 124) or both across one or more computing systems 102 in a network (eg, a cloud computing environment) May be distributed. In such an embodiment, the first computing system 102 receives one or more of the content used to generate performance predictions for the base model 132 from one or more second computing systems 102 . You can search for more than one. Similarly, the first computing system 102 can generate uncertainty intervals for the predicted performance of the base model 132 from one or more second computing systems 102 . may be searched for one or more of In still other embodiments, performance predictions and/or uncertainty intervals can be generated by a single computing system 102 or multiple computing systems 102 .

2A-2B illustrate stacked meta-modeling for generating a performance prediction of a base model (e.g., base model 132) and uncertainty intervals for the performance prediction of the base model, according to one embodiment. • A workflow 200 is shown. Here, the stacked meta-modeling workflow 200 includes a training phase 210 and a use (or deployment) phase 212 (shown in FIG. 2A) and a training phase 214 and a use phase 216 (shown in FIG. 2B).

As shown in FIG. 2A, performance component 122 includes meta-model 202, which is generally intended to predict the probability of success and failure of base model 132 on an instance by instance basis. learn. Workflow 200 shows many different pieces of information that can be considered during training phase 210 . In the depicted embodiment, base model 132 and (labeled) test dataset 134 are processed and used as inputs for training meta-model 202 . As further indicated, in some embodiments, training dataset 136 can also be used as additional input for training meta-model 202 .

Training stage 210 depicts a supervised learning method for training meta-model 202 . Training phase 210 can be performed offline (eg, when meta-model 202 is not in use or deployed) or can be performed (eg, when meta-model 202 is in use or deployed). when) is executable online. More generally, however, any suitable machine learning training mechanism consistent with the functionality described herein can be used. Although base model 132 , test dataset 134 or training dataset 136 or a combination thereof are shown as entering directly into meta-model 202 during training phase 210 , those skilled in the art will appreciate that any input data may be input to meta-model 202 . Note further that it recognizes that this information can be processed in a variety of ways before being supplied to the .

Once meta-model 202 has been trained, the (unlabeled) manufacturing dataset 130 (e.g., unclassified labels) can be fed to meta-model 202 as an input, and meta-model 202 can (e.g., During the usage (deployment) phase 212) performance predictions 204 can be output. In some cases, the output prediction scores (eg, softmax, logit) of base model 132 and/or the original input features of base model 132 can also be provided as inputs to meta-model 202 . Although manufacturing data set 130 is shown as entering directly into meta-model 202, those skilled in the art will appreciate that this information can be processed in a variety of ways before any input data is provided to meta-model 202. Note to recognize

Performance prediction 204 is a predicted measure of how well (or accurately) the base model 132 will perform at its task (eg, image classification) on the (unlabeled) manufacturing dataset 130 . For example, assuming base model 132 is a model for object classification, performance prediction 204 is a percentage value (e.g., 74 %) can be included. More generally, performance prediction 204 may include one or more accuracy-related metrics of base model 132 in manufacturing dataset 130 . Examples of this type of accuracy-related metric are area under ROC score, true positive rate (TPR), F1 value, false positive rate (FPR), R-squared score, accuracy score (e.g. (percentage of predictions that the

Performance prediction 204 can be a batch accuracy prediction or a pointwise accuracy prediction. In some cases, performance predictions 204 may be used for pre-deployment quality checks of base model 132 (eg, determining whether base model 132 should be deployed). In some cases, performance prediction 204 is used for quality control/inspection (e.g., filtering inaccurate predictions, identifying production reductions, etc.) during runtime or deployment of base model 132. can be used. However, as noted above, performance prediction 204 can be adversely affected by multiple sources of uncertainty (eg, anecdotal uncertainty, epistemological uncertainty, etc.) that are not captured in accuracy-related metrics. To address this, embodiments use uncertainty component 124 to predict uncertainty intervals for performance prediction 204 .

As shown in FIG. 2B, the uncertainty component 124 includes a meta-meta-model 206, which includes errors in the meta-model 202 and actual observed performance (e.g., predicted learn to predict the function of the actual accuracy minus the absolute value of the actual accuracy). Workflow 200 shows many different pieces of information that can be considered during training phase 214 . In the depicted embodiment, offline model/dataset 138 and meta-model 202 are processed and used as inputs for training meta-meta-model 206 . As further indicated, in some embodiments, manufacturing dataset 130 may also be used as additional or sole input for training meta-meta-model 206 .

A training phase 214 depicts a supervised learning method for training the meta-meta-model 206 . Training phase 214 can be performed offline for one or more runs. More generally, however, any suitable machine learning training mechanism consistent with the functionality described herein can be used. Although offline models/datasets 138, meta-models 202, or manufacturing datasets 130, or a combination thereof, are shown directly entering meta-meta-models 206, those skilled in the art will appreciate that any input data may be entered into meta-meta-models 206. Note further that it recognizes that this information can be processed in a variety of ways before being supplied to the .

Meta-meta-model 206 is configured to learn from one or more offline runs of training phase 214 when and how wrong meta-model 202 is likely to be. In one embodiment, meta-meta-model 206 is trained on multiple background offline models/datasets 138 (excluding manufacturing dataset 130) to represent the expected uncertainty of meta-model 202. General features can be learned. In other embodiments, meta-meta-model 206 can be trained on manufacturing dataset 130 (eg, without off-line model/dataset 138). In still other embodiments, meta-meta-model 206 can be trained on offline/model/dataset 138 and manufacturing dataset 130 .

Offline models/datasets 138 may be labeled datasets that are relevant to the base model 132 task (e.g., object classification) or unrelated to the base model 132 task (e.g., voice-to-text conversion) or both. In some embodiments, at least one (first) dataset of offline models/datasets 138 used to train meta-meta-model 206 is one Or it can be generated dynamically from multiple (second) data sets. For example, a first data set can be generated based on resampling one or more features in a second data set.

FIG. 3 illustrates an example of generating one or more datasets for training meta-meta-model 206, according to one embodiment. As shown, given a (base) labeled dataset 302 (which may be one of the offline models/datasets 138), the prediction engine 120, to train the meta-meta-model 206: One or more training/test data sets 304-1 through 304-6 and one or more manufacturing data sets 306-1 through 306-6 may be generated. In one embodiment, prediction engine 120 resamples (base) labeled dataset 302 based on one or more features of labeled dataset 302 to generate training/test dataset 304 and A manufacturing data set 306 can be generated.

In one example, the training/test dataset 304 and production dataset 306 can be generated by resampling the (base) labeled dataset 302 based on various features to introduce covariate shifts. be. In this example, certain features of the (base) labeled dataset 302 used to resample the distributions of the training/test dataset 304 and the production dataset 306 are determined by (e.g., feature importance )) can be dynamically selected by the prediction engine 120 . In another example, the training/test dataset 304 and production dataset 306 are resampled from the (base) labeled dataset 302 based on the actual class labels to introduce conventional stochastic shifts. can be generated by

As shown in FIG. 3, assuming the feature is "feature A" (e.g. "dog") (or "feature B" is e.g. "cat"), the ratio of "feature A" (or "feature B") is Each of the training/test datasets 304-1 through 304-6 and the production datasets 306-1 through 306-6 may be varied (eg, by resampling features in the labeled dataset 302). Here, for example, training/test data set 304-1 contains a statistical distribution of 100% of "feature A" (or 0% of "feature B"), and the corresponding manufacturing data set 306-1 is: Contains the statistical distribution of 0% of "feature A" (or 100% of "feature B"). Similarly, at the opposite end, the training/test data set 304-6 contains a statistical distribution of 0% of "feature A" (or 100% of "feature B") and the corresponding manufacturing data set 306-6 contains the statistical distribution of 100% of "feature A" (or 0% of "feature B").

Referring back to FIG. 2B, in some embodiments, training meta-meta-model 206 (eg, during training phase 214) includes metadata about the input dataset (eg, training/test parameters). Generating one or more features based on the offline model/dataset 138) and/or other distributed feature spatial characteristics of the input dataset, which may include dataset 304, manufacturing dataset 306, etc. may include In one embodiment, one or more features may be based on base model 132 and off-line models/datasets 138 (including training and manufacturing datasets). For example, the prediction engine 120 may calculate various distributions in the test dataset and the manufacturing dataset and compare the distances between the various distributions used to train the meta-meta model 206. one or more features may be created.

As shown in FIG. 4, for example, a first distribution (histogram 402) can be calculated based on a test data set (eg, test data set 304), and a second distribution (histogram 404) is: It can be calculated based on a manufacturing data set (eg, manufacturing data set 306). The distance 406 (or more generally the difference) between the distributions is computed and can be used as features (also called feature values) 408 . Generating features 408 to train meta-meta-model 206 in this manner results in significantly noisy (e.g., above threshold) performance in scenarios where the average output from meta-model 202 is stable. Variation is high (eg, above a threshold) in scenarios such as where the uncertainty of the sample formula is too large due to the error bars of the batch formula, although it may account for the predictions.

In one embodiment, example feature 408 may be based on the highest confidence score from base model 132 . For example, base model 132 can be used to score samples in test and manufacturing data sets. Assuming base model 132 is used for object classification (e.g., between class A and class B), the output prediction score using base model 132 is for each sample (or data points) and for each sample (or data point) in the manufacturing data set. As an example, for the first sample in the test dataset, the base model 132 may output 95% class A, and for the first sample in the manufacturing dataset, the base model 132 may output 90% class A.

After all samples in each of the test and manufacturing data sets have been scored, a subset of the scored samples in both the test and manufacturing data sets can be used to create histograms. . For example, the highest confidence score (e.g., between a certain threshold range, e.g., between 90-95%, 95-98%, 98-100%, etc.) from the scored samples in the test dataset ( First) a histogram can be generated. Similarly, a (second) histogram of the highest confidence scores (eg, between certain threshold ranges) from the scored samples of the manufacturing data set can be generated. The two histograms can then be compared with a divergence function (eg, similarity metric, dissimilarity metric). In one embodiment, the distance 406 (e.g., Hellinger distance) between two histograms can be calculated, and the value of distance 406 (e.g., in [0, 1]) is the value of meta-meta model 206. can be used as feature 408 for

In one embodiment, an example feature 408 can be based on the shadow model's highest confidence score. Compared to the base model's highest confidence score, in this embodiment, the other (proxy) model is be trained. The highest confidence scores from the test and manufacturing datasets are then calculated using the proxy model (eg, as opposed to the base model 132). Histograms can be generated based on the highest confidence scores, and the distance between histograms can be used as feature 408 for meta-meta-model 206 .

In one embodiment, an example feature 408 can be based on a class frequency distance. In this embodiment, for example, a histogram of the percentage of samples in the test and manufacturing data sets predicted to be in each class by the base model 132 may be generated. A distance 406 (eg, Hellinger distance) between histograms can be calculated and the distance value can be used as a feature 408 for the meta-meta model 206 .

In one embodiment, the example feature 408 can be based on the highest feature distance. In this embodiment, for example, a Shadow Random Forest model can be trained and used to identify data features with the highest feature importance (eg, meeting a predetermined condition). Once identified, the histograms of the test and manufacturing datasets can be projected into this dimension (eg, a compressed one-dimensional feature space). A distance 406 (eg, Hellinger distance) between histograms can be calculated and the distance value can be used as a feature 408 for the meta-meta model 206 .

In one embodiment, example features 408 may be based on meta-model predictions (eg, performance predictions 204). In this embodiment, the change in accuracy of the base model between the test dataset and the manufacturing dataset as predicted by meta-model 202 is used as feature 408 for meta-meta-model 206. can be done. In one embodiment, an example feature 408 is based on a statistical hypothesis test between a first statistical distribution (of the first data set) and a second statistical distribution (of the second data set). be able to.

The above features train a meta-meta model 206 that can use any combination of other features or features based on the meta data of the input dataset that match the functionality described herein. Note that it is provided merely as an example of features that can be used to. In some embodiments, for example, multiple meta-meta-models can be generated/trained based on different features. In these embodiments, the particular meta-meta-model used during the execution stage 216 is based on the characteristics of the data (e.g., metadata about the dataset, number of features, distribution feature space characteristics, etc.). Sometimes dynamically selectable.

Once meta-meta-model 206 is trained, one or more of base model 132, test dataset 134, training dataset 136, meta-model 202, production dataset 130, and performance prediction 204 may receive input , and meta-meta-model 206 can output uncertainty predictions 208 (eg, during the use (deployment) phase 216). Uncertainty prediction 208 is the predicted amount of uncertainty in performance prediction 204 that is output from meta-model 202 . For example, assuming base model 132 is a model for object classification, uncertainty prediction 208 is an interval (or tolerance) representing the amount of uncertainty in performance prediction 204 (e.g., ±4, ±7, +2) can be shown. In some embodiments, the uncertainty prediction 208 is used for additional pre-deployment quality inspection of the base model 132 or for quality control/inspection during runtime or during deployment of the base model 132 or the like. Can be used for both.

FIG. 5 is a flowchart of a method 500 for predicting a model's performance on a task and uncertainty for the model's performance on a task, according to one embodiment. Method 500 may be performed by a prediction engine of a computing system (eg, prediction engine 120 of computing system 102).

Method 500 may begin at block 502 where a prediction engine obtains a base model (eg, base model 132) and a set of manufacturing data (eg, manufacturing data set 130). At block 504, the prediction engine uses the meta-model (eg, meta-model 202) to predict the performance of the base model (eg, performance prediction 204) in the set of manufacturing data. For example, the prediction engine may generate one or more accuracy-related metrics of the base model in the set of manufacturing data. In one particular example, an accuracy-related metric may indicate the success/failure probability (eg, X% success) of a base model at its task using a set of manufacturing data.

At block 506, the prediction engine uses the meta-meta-model (e.g., meta-meta-model 206) to generate one or more uncertainty intervals (e.g., , uncertainty prediction 208). For example, the prediction engine uses a set of manufacturing data to generate a tolerance (or error band) (e.g., ±Y) that indicates the amount of uncertainty in the predicted performance of the base model on that task. can do.

In one embodiment, the uncertainty interval (predicted in block 506) may be asymmetric. For example, the prediction engine can predict a signed value of error (eg, + or -) that expresses the uncertainty of predicted performance. In another example, a first uncertainty interval can be generated for the upper band/range around the predicted performance and a second uncertainty interval can be generated for the lower band/range around the predicted performance. band/range of .

In some embodiments, multiple meta-meta-models are used to generate multiple uncertainty intervals (e.g., a first meta-meta-model for a first (upper) uncertainty interval and a second A second meta-meta model for the (lower) uncertainty interval of ) can be generated. In some embodiments, multiple uncertainty intervals can be generated by a single meta-meta-model. For example, a single meta-meta-model can include two sub-modules (or components) for predicting upper and lower uncertainty intervals.

At block 508, the prediction engine provides an indication of the performance of the base model and one or more uncertainty intervals. In one embodiment, the prediction engine may provide a representation on a user interface of a display of a computing device (eg, computing system 102). In one embodiment, the prediction engine can provide an indication in response to requests for base model performance and uncertainty predictions for a set of manufacturing data. For example, a user may use a prediction engine to determine which base model (among multiple base models) should be used in a set of manufacturing data. In another example, the prediction engine may monitor the deployment/runtime of the base model and continually provide an indication to the computing device (eg, at one or more predetermined time intervals). This representation can then be used to determine when the base model should be improved and/or replaced. In some cases, the prediction engine may run the meta-meta model as often as the meta model.

FIG. 6 is a stacked meta-model workflow for generating model performance predictions and uncertainty intervals for model performance predictions, according to one embodiment. 6 is a flowchart of a method 600 for Method 600 may be performed by a prediction engine of a computing system (eg, prediction engine 120 of computing system 102).

Method 600 is a method in which a prediction engine uses a base model (eg, base model 132), a test dataset (eg, test dataset 134), and one or more additional datasets (eg, offline models/datasets). set 138) may begin at block 602. At block 604, the prediction engine trains a meta-model (eg, meta-model 202) to predict the performance of the base model on the task based on the base model and the test dataset. At block 606, the prediction engine trains a meta-meta-model to predict the uncertainty of the performance of the base model on the task based on the meta-model and one or more additional datasets. do.

FIG. 7 is a flowchart of a method 700 for training a meta-meta model, according to one embodiment. Method 700 may be performed by a prediction engine of a computing system (eg, prediction engine 120 of computing system 102).

Method 700 may begin at block 702 where a prediction engine obtains a labeled dataset (eg, labeled dataset 302). At block 704, the prediction engine generates one or more additional datasets (eg, training/test dataset 304, manufacturing dataset 306) based on the labeled dataset. At block 706, the prediction engine determines (or calculates) one or more features (eg, feature 408) based on evaluation of one or more additional data sets. At block 708, the prediction engine trains one or more meta-meta-models (e.g., meta-meta-model 206), each based at least in part on one or more features. • It is configured to predict the uncertainty interval of the model (eg, meta-model 202).

FIG. 8 shows simulations 802 and 804 of example uncertainty intervals for performance prediction and performance prediction of a base model, according to one embodiment.

More specifically, simulation 802 depicts performance predictions of the base model across different test/manufacturing data sets 1-K. As shown, the base model accuracy (e.g., the actual accuracy of the base model) is represented by line 808, the predicted accuracy of the base model is represented by line 810, and the error margin for the predicted accuracy. A bar is represented by 806 . Each test/manufacturing data set 1-K may have a different statistical distribution of features/labels. For example, test/manufacturing data set 1 may include a statistical distribution of 100% "feature A" in the test data set and 0% "feature A" in the manufacturing data set. Similarly, at the opposite end, test/manufacturing data set K may include a statistical distribution of 0% "feature A" in the test data set and 100% "feature A" in the manufacturing data set.

Simulation 804, which corresponds to simulation 802, shows the meta model error (represented by 840), fixed error bars (or uncertainty values) 820 and the meta model error versus the actual base model output. delta 830 between. As shown, using a stacked meta-model workflow to generate uncertainty intervals for model performance prediction, embodiments compared to using a fixed uncertainty value 820: A more accurate uncertainty interval can be generated.

Reference is made above to embodiments presented in this disclosure. However, the scope of this disclosure is not limited to particular described embodiments. Instead, any combination of features and elements, whether related to different embodiments or not, is contemplated for implementing and practicing the discussed embodiments. Furthermore, although the embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether particular advantages are achieved by a given embodiment is subject to the present invention. It is not a limitation on the scope of disclosure. Accordingly, the aspects, features, embodiments and advantages set forth herein are merely illustrative and shall not be considered elements or limitations of the appended claims, unless expressly recited in the claims. . Similarly, references to the "present invention" should not be construed as a generalization of any inventive subject matter disclosed herein, except as expressly recited in the claims, rather than to the appended claims. should not be considered an element or limitation of the scope of

Aspects of the present invention may be an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or, as referred to herein, a "circuit," "module," or "system." Embodiments may take the form of combining software and hardware aspects, which may generally be referred to as.

The invention may be a system, method or computer program product or combination thereof. The computer program product may include a computer readable storage medium having computer readable program instructions for causing a processor to carry out aspects of the present invention.

A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction-executing device. A computer-readable storage medium may be, for example, without limitation, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory. (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy (R) - Mechanically encoded devices such as discs, punch cards or raised structures in grooves on which instructions are recorded and any suitable combination of the foregoing. Computer-readable storage media as used herein include radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables). ) or an electrical signal transmitted over a wire, per se, should not be construed as a transient signal.

The computer readable program instructions described herein can be transferred from a computer readable storage medium to a respective computing/processing device or over a network such as the Internet, local area network, wide area network or wireless network or combinations thereof. can be downloaded to an external computer or external storage device via A network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers or edge servers, or a combination thereof. A network adapter card or network interface in each computing/processing device for receiving computer-readable program instructions from the network and storing the computer-readable program instructions on a computer-readable storage medium within the respective computing/processing device. transfer to

Computer readable program instructions for performing operations of the present invention may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data or Smalltalk(R), C++. source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Either The computer-readable program instructions reside entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the user's computer. may be executed on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN) (e.g., the Internet). It may also be connected to an external computer (via the Internet using a service provider). In some embodiments, electronic circuits, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), can be used with a computer to carry out aspects of the invention. Computer readable program instructions can be executed by personalizing an electronic circuit using the state information of the readable program instructions.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart and/or block diagrams, and combinations of blocks in the flowchart and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions are the means by which instructions executed through a processor of a computer or other programmable data processing apparatus perform the functions/acts specified in the flowchart illustrations and/or block diagrams. can be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to create a machine. These computer readable program instructions are such that a computer readable storage medium in which the instructions are stored comprises an article of manufacture that implements aspects of the functions/acts specified in the flowchart and/or block diagram blocks. , a computer, programmable data processing device, or other device, or combination thereof, to function in a specific manner.

Also, computer readable program instructions may be used to cause instructions executed on a computer, other programmable device, or other device to perform the functions/acts specified in the flowchart illustrations and/or block diagrams. It can be loaded into another programmable data processing apparatus or other device to cause a series of operational steps to be executed on the computer, other programmable device or other device to produce a computer-implemented process.

The flowcharts and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block of a flowchart or block diagram can represent a module, segment, or portion of instructions containing one or more executable instructions for performing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. good. Each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, represent specialized hardware and/or combinations of computer instructions that perform the specified functions or acts or implement combinations of specialized hardware and computer instructions. Note also that it can be implemented by the base system.

Embodiments of the present invention may be provided to end users through a cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing may be defined as computing power that provides an abstraction between computational resources and their underlying technical architecture (e.g., servers, storage, networks), with minimal management It enables convenient, on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released by work or service provider interaction. Thus, regardless of the underlying physical systems used to provide the computing resources (or the location of those systems), cloud computing allows users to create virtual computing resources (e.g., storage, data, applications and even fully virtualized computing systems).

Typically, cloud computing resources are provided to users on a pay-as-you-go basis, and users pay for the computational resources actually used (e.g., the amount of storage space consumed by the user or instantiated by the user). number of virtualization systems connected). Users can access any of the resources in the cloud at any time and from anywhere across the Internet. In the context of the present invention, a user can access available applications in the cloud (e.g., prediction engine 120) or related data (e.g., base model 132, test dataset 134, training dataset 136, offline models/data). set 138, manufacturing data set 130, etc.) can be accessed. For example, the prediction engine 120 can run on a computing system in the cloud and can predict the performance of the base model along with uncertainty intervals for the predicted performance of the base model. In this type of case, the application retrieves one or more of the input information used to generate the performance and/or uncertainty predictions from storage locations in the cloud and stores the performance predictions in storage locations in the cloud. Predictions and/or uncertainty predictions can be stored.

While the above is directed to embodiments of the invention, other and further embodiments of the invention may be devised without departing from its basic scope, which scope is determined by the following claims. .

Claims (21)

A computer-implemented method for generating a performance prediction for a machine learning (ML) model, comprising:
obtaining a manufacturing data set including a first model configured to perform a task and unlabeled data;
generating at least one metric that predicts the performance of the first model when performing the task on the manufacturing data set using a second model, the second model comprising: , a meta-model associated with the first model;
predicting uncertainty of the at least one metric that predicts the performance of the first model when performing the task on the manufacturing data set using one or more third models generating at least one value, wherein each of said one or more third models is a meta-meta-model associated with said second model;
providing an indication of the at least one metric that predicts the performance of the first model and the at least one value that predicts the uncertainty of the at least one metric;
A computer-implemented method comprising: wherein the first model is a black box model;
2. The computer-implemented method of claim 1. wherein the first model is a white box model;
2. The computer-implemented method of claim 1. at least one of the one or more third models is trained on (i) a training data set for the first model and (ii) the second model;
2. The computer-implemented method of claim 1. At least one of the one or more third models includes (i) a training data set for the first model, (ii) one or more additional data sets and (iii) the second is trained on a model of
2. The computer-implemented method of claim 1. at least one of the one or more third models is trained on (i) one or more additional datasets and (ii) the second model;
2. The computer-implemented method of claim 1. said at least one of said one or more third models is trained offline on said one or more additional datasets and said second model;
7. The computer-implemented method of claim 6. said one or more additional data sets comprising at least one data set comprising a first set of labels for classifying data in said at least one data set;
the first set of labels is different from a second set of labels for classifying data in a dataset used to train the second model;
7. The computer-implemented method of claim 6. The one or more additional datasets comprise at least one labeled dataset, the computer-implemented method comprising:
generating a plurality of test datasets and a plurality of manufacturing datasets based on the labeled datasets;
including the plurality of test datasets and the plurality of manufacturing datasets in the one or more additional datasets;
7. The computer-implemented method of claim 6. Generating the plurality of test data sets and the plurality of manufacturing data sets may include each test data set and corresponding manufacturing data set having a different ratio of the one or more features to the labeled data set. resampling one or more features of the labeled dataset to include
10. The computer-implemented method of claim 9. The computer-implemented method includes a first statistical distribution of a first data set in the one or more additional data sets and a second statistical distribution of a second data set in the one or more additional data sets. generating at least one feature value for training the one or more third models based at least in part on a statistical distribution; said at least one is further trained with said one or more feature values;
7. The computer-implemented method of claim 6. the at least one feature value is further generated based on a divergence function between the first statistical distribution and the second statistical distribution;
12. The computer-implemented method of claim 11. the at least one feature value is further generated based on a statistical hypothesis test between the first statistical distribution and the second statistical distribution;
12. The computer-implemented method of claim 11. The one or more third models are a subset of the plurality of third models, and the computer-implemented method comprises at least partially characterizing one or more features of the one or more additional data sets. further comprising selecting the one or more third models from the plurality of third models based on
2. The computer-implemented method of claim 1. the at least one value predicting the uncertainty of the at least one metric comprises an interval range;
2. The computer-implemented method of claim 1. wherein the interval range is an asymmetric interval range comprising a first upper extent and a second lower extent;
16. The computer-implemented method of claim 15. the first upper range and the second lower range are generated via one of the one or more third models;
17. The computer-implemented method of claim 16. the one or more third models comprises a plurality of third models;
said first upper range is generated via a first of said plurality of third models;
the second lower range is generated via a second of the plurality of third models;
17. The computer-implemented method of claim 16. said first upper range is generated via a first component of said one or more third models;
the second lower range is generated via the one second component of the one or more third models;
17. The computer-implemented method of claim 16. a system,
one or more computer processors;
a memory containing a program;
The program, when executed by the one or more computer processors, performs acts for generating a performance prediction for a machine learning (ML) model, the acts comprising:
obtaining a manufacturing data set including a first model configured to perform a task and unlabeled data;
generating at least one metric that predicts the performance of the first model when performing the task on the manufacturing data set using a second model, the second model comprising: , a meta-model associated with the first model;
at least one value that predicts the uncertainty of the at least one metric that predicts the performance of the first model when performing the task on the manufacturing data set using a third model; generating, wherein the third model is a meta-meta model associated with the second model;
providing an indication of the at least one metric that predicts the performance of the first model and the at least one value that predicts the uncertainty of the at least one metric;
system including. A computer readable storage medium having computer readable program code embedded therein, the computer readable program code executable by one or more computer processors for generating a performance prediction of a machine learning (ML) model. perform the actions of
obtaining a manufacturing data set including a first model configured to perform a task and unlabeled data;
generating at least one metric that predicts the performance of the first model when performing the task on the manufacturing data set using a second model, the second model comprising: , a meta-model associated with the first model;
at least one value that predicts the uncertainty of the at least one metric that predicts the performance of the first model when performing the task on the manufacturing data set using a third model; generating, wherein the third model is a meta-meta model associated with the second model;
providing an indication of the at least one metric that predicts the performance of the first model and the at least one value that predicts the uncertainty of the at least one metric;
A computer readable storage medium comprising

Images