LU100491B1 - Method of selecting a prover - Google Patents

Abstract

The present invention proposes a computer implemented method of selecting a prover among a plurality of provers for a design to be verified. The method comprises collecting, by a data module, raw data relating to the design, and extracting from the raw data a plurality of input features, transforming, by a transformer module, the plurality of input features, wherein trans- forming the plurality of features comprises applying a linear regression to the plurality of fea- tures, classifying using a classification module, the provers from the plurality of provers, in which the classification module is adapted to predict a best prover being the prover which solves a property faster than the remaining provers of the plurality of provers, selecting one or more provers based on the results of the classification. Fig. 1A

Description

TITLE: METHOD OF SELECTING A PROVER

CROSS REFERENCE TO RELATED APPLICATIONS

None.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

None.

BACKGROUND OF THE INVENTION

Field Of The Invention [0003] The present invention relates to formal verification, and more particularly, to a computer-implemented method of selecting a prover among a plurality of provers for a design to be verified.

Brief Description Of The Related Art [0004] The invention relates to a method for verifying or disproving the correctness of underlying algorithms.

[0005] Verification engines use different algorithms for formal verification, search as binary decision diagrams (BDD) based reachability, satisfiability (SAT) based bounded model checking (BMC), interpolation, and property-directed reachability (PDR). The verification engines check whether a certain state, e.g. an improper state or a valid state. The property can not be proofed or falsifies the property with a trace.

U-referred to as a "prover".

[0007] Different verification engines may have drastically different performances on different problems depending on the verification algorithms used by the verification engines. Indeed, the ability to prove or disprove properties, as well as the resources and CPU time needed for the verification, vary considerably, depending on the verification engine, design and property.

Properties and designs may be described by a set of characteristics or features. For example, design control points include design input, in-out ports, un-driven wires, user-defined cut points. Another example are the Design Structural elements, as the total number of design counters, RAM. Yet another example is the design state. Complexity defining a total number of design states elements, including the total number of design latches and flip-flops. Another example is the Property Logical Complexity, comprising at least one of a sequential depth, a total number of flip-flops, latches and combinational gates.

The runtime on the other hand, and to minimize the runtime on the other hand, to make an efficient use of CPU time.

An automated scheduling strategy for verification engines may be defined in order to have a predictable and limited amount of resources invested in a set of designs / properties, i.e. run time (real time / CPU time), number of cores to run in parallel, memory consumption. The automated scheduling strategy may or may not be run sequentially or in parallel, and how long to run (in terms of time, steps or Boolean constraint propagation (BCP) counts), for what type of property, on different designs.

[00011] In order to define an automated scheduling strategy, it is necessary to select the selection engine or prover suitable for the given formal verification to be done. Prover parameter configuration is therefore needed to limit the time spent. Provers can be configured by limiting the maximum number of steps, if available or on the BCP counter.

[00012] State of the art approaches for verification engine selection are primarily based on brute force or on verification engineers' feelings.

[00013] It has been proposed to use linear regression to estimate the CPU time based on design and property features. A regression model of the runtime vs. the design and property features, using LASSO, regularization, and compare against K Nearest Neighbors.

[00014] Machine learning is the subfield of computer science that gives the ability to learn without knowledge. Machine learning algorithms that can learn and make predictions on data - search algorithms Machine learning has been applied for integrated circuit verification, for e.g. to cope with process variations in semiconductor production, for fault injection in the verification of safety-critical systems.

[00015] In machine learning, a pipeline is a sequential chain of operations, in particular of transformers and estimators, which is trained and trained in used to make predictions.

A first approach to the verification of parallel execution engines has been proposed in which Principal Component Analysis (PCA) and linear regression are applied to estimate the time of four different engines. PCA is a statistical procedure known in the field of machine learning, which uses orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variable called principal components. In this approach, both polynomial and exponential transformations of features are used, thereafter a performance function is defined and a weight per engine is calculated which correlates to the engine runtime. The correlation of selected engines is thereafter minimized.

[00017] Yet another approach applies to SAT solver, which optimally performs on the different constraints, based on attributes of the problem instances (i.e., to find the implementation of. ku meta-classifier is proposed, which combines multiple classification strategies for a resulting better one. A misclassification penalty is proposed, whichever is to the additional CPU time required to solve a problem.

[00018] The patent application US 2014/0330758 discloses a method for training a classifier to create a prediction model. The approach described in patent application US'758 relies on machine learning. More precisely, US'758 applies Support Vector Machine (SVM) classification to prover selection. Features can be collected by a BMC SAT solver, and include statistics for the CNF generated at the various steps of the BMC algorithm. The prediction model uses the feature data of the design verification problem to make a result prediction for the design verification model.

[00019] There is a need to improve the efficiency of a system and method for design verification.

[00020] The invention relates to a method and method for selecting a verification engine for design verification.

SUMMARY OF THE INVENTION

[00021] The present invention proposes a computer averaged over a raw material of a raw material, a data module, a data module, a raw data relating to the design and extraction of raw data using input modules, transforming, by a transformer module, by the process of transforming the processes of the invention, by using a classification module The classification module is based on the results of the proverbial process and the proverbial analysis of the proverbs.

Therefore, the present invention proposes using a pipeline made of multiple "11 transformers and estimators exported to online use in formal verification. Using the Journey Description Each Design and Checking Loss of Property or the Design of the Prover Selection Module.

The invention is aiming at a machine learning based method and verification.

In an aspect of the invention, the step of extracting the parameters as input features for the transformer module.

[00025] In a further aspect of the invention, the transforming of the collected data. By a standard scaler module, the input features of the process of applying a standard component analysis module to orthogonal transformation as a principal component analysis to having the standardized features, and keeping the components resulting from the principal component analysis.

[00026] In an aspect of the invention, the classification module is pre-trained by collecting a set of training data, design parameters and runtimes for a process of extracting and extracting, transforming the set of training data, and classifying the provers using the transformed set of training data.

The classifying step may include assigning, for a property, a probability to a prover using a multi-class logistic regression based on design features and an expected result of the property.

[00028] In addition to the invention, the method of configuring the proverbs of the production process using linear regression modeling dependency of a step size on the design and an expected time of the prover to check the assertion.

[00029] By providing a prover configuration setting, it becomes possible to interrupt the provers when they reach the configured step size for a property subsequent properties can be reached and per vers do not spend too much time on this package. While a large step size is needed, there is no such thing as a design, with no meaningful result for many properties, and no big gain in time , Therefore a step size depending on the design features must be set. The prediction of the step size is realized.

A prover may be configured using a LASSO regression model having dependent variables and independent variables, in which the dependent variable of the regression model is a number of steps and the independent variable of the regression model is design statistics and expected check time.

In an aspect of the invention, the method of grouping pro vers in a peculiarity of prover groups and grouping properties in a p urn of properties, based on runtimes for each property and prover.

[00032] In an aspect of the invention, the method of assigning to a property group, in particular in which the assigned prover group has a minimum sum of runtimes. The method may further comprising, for each assigned prover group, the prover of the prover group having the minimum runtime of the property group to which the assigned prover group is assigned.

It should be understood that there are representative properties in the data that are clustered; In this case, among the pro vers selected as optimal for the property clusters, there will be optimally for each type of check, i.e. for each type of property.

In an aspect of the invention, the method comprises the selected prover groups in parallel.

[00036] A computer program product comprising a non-transitory computer mediuAP having a method to be used therefor.

[00037] A further benefit from the invention is further analyzed, treated and returned without particular design information and thus emergency allowing any reverse engineering. The method is proven to be very powerful in remote verification applications or for remote computing.

[00038] The scikit-leam Python libraries may be used to train and validate the pipeline.

Still other aspects, features, and advantages of the present invention are not apparent from the following detailed description, simply by way of illustrating a preferred and implementations. The present invention is thus capable of being modified in various respects, all without departing from the scope of the present invention. Accordingly, the drawings and descriptions are to be regarded as illustrative in nature, and not as restrictive. The invention also seeks to further clarify and further explain the invention.

LETTER OF THE DRAWINGS

For a more complete understanding of the present invention and the advantages thereof, reference is now made to the following description and the accompanying drawings, in which: FIG. 1A is a schematic view of a pipeline in accordance with the invention. FIG. 2 shows a correlation plot of design parameters in accordance with a preferred embodiment of the invention. FIG. 3 shows a correlation plot of transformed features in accordance with the invention.

FIG. 4A is an exemplary scatter plot of predicted probability vs. real ratio with a classifier according to an embodiment of the present invention and FIG. FIG. 4B is a performance plot of a classifier in accordance with a preferred embodiment of the present invention.

FIG. 5 shows a confusion matrix in accordance with a preferred embodiment of the invention.

FIG. 6 shows a multiclass receiver operating characteristic plot in accordance with a preferred embodiment of the present invention.

FIG. 7 shows the number of steps for different designs in accordance with a preferred embodiment of the present invention.

FIG. 8 shows comparative data between real time, predicted numbers of steps and numbers of steps in accordance with a preferred embodiment of the present invention. FIG. 9A is an illustration of a method for selecting a prover in accordance with the present invention.

FIG. 10 shows a method for selecting a prover according to the present invention.

FIG. 11 shows a plot of heatmap runtimes checks for different provers in accordance with a preferred embodiment of the invention.

FIG. 12 shows a dendrogram of clusters of provers in accordance with a preferred embodiment of the present invention.

FIG. 13 shows another dendrogram of clusters of provers in accordance with a preferred embodiment of the invention.

FIG. 14 shows a table of times a pair of cluster properties and cluster provers in accordance with a preferred embodiment of the invention.

FIG. 15 shows dendrograms of hierarchical clusters of provers in accordance with a preferred embodiment of the present invention.

FIG. FIG. 16 shows another example of the dendrograms of hierarchical clusters of provers in accordance with a preferred embodiment of the present invention. [00057] FIG. 17 shows a heat map and dendrograms of clustered provers for clustered properties in accordance with a preferred embodiment of the invention. LU1

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[00058] As will be explained, the present invention proposes a pipeline for online selection of provers, based on design and property parameters. A pipeline is a chained set of operations which can be fit and globally optimized. The pipeline, which standardize, transforms the features, classify, rank and select best performing provers among a set of provers. The method and method of the present invention further predict the size of the execution time. The grouping of provers is thus to minimize the runtime over a set of checks, based on runtimes of provers. The pipeline can be used for dynamic prediction on new designs, and will be used again.

[00059] A general architecture for a system and method for building a pipeline in accordance with a preferred embodiment of the present invention is shown in FIG. 1A and IB. The system includes a computing device 1, which may be a computer or server having one or more processors, a memory and a non-transitory storage medium, as well as a hard drive or a solid state drive. The computing device 1 comprises a data module 40, a pipeline module, estimation module 60 and a prover selection module 70.

[00060] The data module 40 is provided for collecting the data. 50. Collecting data is the first step. SI to build the pipeline.

[00061] Raw data may include features of properties, features of designs, runtime of each property on each property, the result of the property - hold / fail, wining prover per property, and for each prover - the status, runtime and number of steps needed.

In the present invention, a predefined numbers of checks, i.e. D ran properties,,, set set designs designs designs pro pro pro pro pro pro pro pro pro P. P. P. P. P. P. P. P. P. P. P. P. P. P. P. P. P. P. P. P. P. P. P. P. P. P. Each P. P. P. P. P. P. P. DP DP DP DP DP DP DP DP DP DP DP DP DP DP DP DP DP DP DP DP DP DP DP DP. In the example described, there are 1176 checks run from 10 designs. 6 provers were used and 42 design parameters were taken from the design, for each design. Thleu skilled person understands the number of checks, designs, provers and design parameters are given as examples and are in no way limiting the invention.

[00063] The data module 40 is therefore ready to be used in the pipeline module 40. This is the second step S2 of the method for building the pipeline. For each property, the winning provers WP among the set of provers are labeled accordingly, if they converge within the shortest runtime compared to the rest of provers and if the difference is significant - e.g. bigger than 5 minutes. Runtimes may therefore be aggregated to be used as input for regression model and distance matrix for the hierarchical clustering.

[00064] It may happen that different problems occur in almost the same time, or because of different loads on the machines, there are small variations in the recorded runtime. To avoid labeling the winning prover in a random way, the labeling can account for the actual difference in solve time, and label as the winning provers. 5 minutes. If too much points lie in the "gray zone" another solution can be used instead of winning provers. In this case the prover selection wants to keep out of the selection of the worst provers.

It should be noted that only the provers are capable of failing.

Another way to label the provers is to label them as fast or slow, compared to the fastest prover, rather than labeling the prover as a winning prover or not. These alternative labeling avoids the risk of labeling provers randomly when more provers are almost and close to each other, and so avoids labeling many samples a piece of information worth modeling as well. Therefore, in some cases, it may or may not be on a pro-level basis, whether or not it is near or slow. An alternative is to use the same winning prover. LU1 [00067] In machine learning, a feature is an individual measurable property of a phenomenon being observed. In the present invention, prover performance is a dependent variable of machine learning, i. modeled on output. The present invention also proposes extracting design parameters. Therefore, the data module 40 outputs a set of features F, using machine learning features.

In a preferred embodiment of the present invention, the pipeline module is provided to fit (train) the pipeline. The fitting of the pipeline is done by running, by a transformer module 50, a succession of transformers, i.e. operators which may clean, reduce, expand or generate feature representations (at step S3), followed by estimation, by a classification module 60, of the probability of each prover to solve a property, on the transformed features (step S4).

[00069] The transformer module 50 is adapted to standardize each feature from the set of features to a unit variance. This can be done by a standard scaler 52, which allows to improve the quality of the final estimation, by weighting equally to the machine learning features of the set of features F (design parameters DP).

[00070] Component Analyzer module 54 is adapted to apply PCA to the set of design parameters DP. Principal Component Analysis (PCA) is a statistical methodology known to be used in the field of machine learning, which uses orthogonal transformation to convert a set of observations into a set of values of linearly uncorrelated variable called principal components.

The present invention recognizes that the design parameters DP, i.e. the machine learning features, as a heatmap with blue for maximum positive correlation. P (42 parameters in the example) and red for minimally negative correlation in the example of the present application.

[00072] The present invention recognizes that PCA may be used to transform the with-U "chine learning space into an orthogonal reduced space. PCA is provided to derive the set of features values a set of linearly independent components - called principal components - which can then be used to reconstruct the initial set of features values. The PCA module 54 includes a set of design parameters. DP to a set of orthogonal components which contains a high percentage of the variance of the reconstructed input space, from the PCA components.

[00073] If a subset of the PCA components is used only a subset of the variability of the initial components is reconstructed. The PCA module is PCA components, i.e. the ones which ensure most of the variability of the initial components is preserved. When the optimal orthogonal transformation of the set of features has been found, only the Principal Analysis Components (PCA). In other words, the transformer module 50 with its PCA module 54 can be seen as a dimensionality reduction module.

[00074] Figure 3 shows the results of the PCA transformation in the example of the present specification. PCA has been. The parameters of the DPA have been changed Applied. In the example of Figure 3, a total of 9 PCA components are kept, all the rest having an insignificant ratio of explained variance in the total. The PCA module 54 has removed the cross-correlation. 2. In other words, without this step of PCA, strongly correlated parameters would not be distinguishable, and the final model would have randomly chosen parameters out of cross correlated sets. It should be noted that the number of PCA components which are used as an example only and not limiting the invention. PCA is a requirement for the next step of building and fitting the pipeline.

[00075] BP, as the winning prover (s), among the set of provers P for each property (step S4).

[00076] Classification as separate type into distinct classes. Applied to the present application, the target class is the winning prover the prover of the set of provers P which solves the property faster than the other provers of the set of provers. A classification model CM run by the classification module 60 is built by inputting a set of training data. In the present application the training data are the runtime of the property checks and the corresponding machine learning features.

[00077] It is understood that the classification model can be used by a different dataset. In the present application, the different data set are new properties ..

[00078] A multiclass One-vs-remainder logistic regression may be used to predict probabilities of each prover of the set of provers. The classification module 60 is capable of predicting only the winning prover, but also calibrated probabilities per prover, i.e. chances of the provers to win.

It should be noted that using multi-class logistic regression classifiers has more advantages. First, the probability of each prover can not be predicted, and not only as winner or not. This allows ranking the provers, and choosing more to run. the ones whose predicted probabilities exceed a threshold. Further, if there are any features are known, then each property has one winning prover with probability 1 and the rest 0. If there are features missing, the classifier wants to predict a set of winning provers, with different probabilities, so that for all the properties, a set of It can be selected almost by one of the selected provers. Another advantage is that it validates the classifier, a natural comparison. The closer the probability to the ratio, the better the classifier. An example scatter plot of predicted probability versus real ratio is shown in FIG. 4A, where only a subset of features are used. Each point is a design-prover combination, and is assigned a probability. The predicted probability and real ratio are close to each other and the ranking of the top 2 provers per design is correct (can predict each design their top 2 winning provers).

[00080] The classification model CM may be validated using classification accuracy, at step S5. [00081] Several scores may quantify the performance of the classification module 60. Accuracy is the number of correct predictions made by the total number of predictions made. The classification accuracy with different sample sizes is shown in Figure 4B which shows a performance plot of a classifier in accordance with a preferred embodiment of the invention.

[00082] Confusion matrix is another tool to help quantify the performance of a prover. Each column of the confusion matrix represents the instances in a predicted class, while each row of the confusion matrix represents the instances in an actual class.

[00083] An example of confusion matrix is shown in FIG. 5, with the number of labeled samples versus the real number of labeled samples, for each prover pair. In case one winning prover is selected by property, the confusion matrix plot shows the number of correctly classified versus misclassified properties. As can be seen in the example of FIG. 5, approver 3 is misclassified, while disproved completely correctly classified. This misclassification happens because approver 3 is not the winning prover for most properties, but is the winning prover for some properties.

[00084] To avoid this effect of misclassification, the classification module 60 Comparing the predicted probabilities to the real ratio of each prover per design and yielding good results: less than 10% have an absolute difference of 0.1 and no case above 0.3. The coverage score indicates how many provers to select to win all properties.] [00085] Receiver operating characteristic (ROC) is another score to quantify the performance of the classification module 60. ROC curves have true positive rate on the Y axis, and false positive rate on the X axis. This means that the top left of a ROC curve is the "ideal" point - a false positive rate of zero, and a true positive rate of one. Having a larger area under the curve (AUC) is better. FIG. 6 shows a ROC curve in the example of the present application. As can be seen on fig. 6, the ROC curve thus indicates acceptable AUC, despite Approver3 classifier's poor performance.

[00086] Thus, the validation may be done on designs not used for training, which is reach 90% if one design is left for validation. The validation design parameters should be within the parameters of the training designs.

[00087] It should be noted that other classifiers may be used as K Nearest Neighbors Classifiers or Random Forest Classifiers. K Nearest Neighbors Classifier is simple to implement, with the advantage of linear performance of the probability of the transformed feature space. However, K Nearest Neighbors Classifier has the risk of not scaling when the dataset is too big. On the other hand, Random Forest Classifier might be overfit, while being easily exported to productive use. Of course, it may be possible to combine more classifiers, to get a better prediction.

[00088] The skilled person wants to understand that the present application uses multi-class logistic regression to assign probabilities to optimal prover selection is a classification problem.

It should be noted that a method of classification CM can be im-U1 plemented. If the classifier or repressor is not accurate enough, then the estimators' fit, the data preparation or the data collection steps have to be revisited. More specifically, when the test score is poor compared to the train score, the model is probably over-fit, which can happen when the training data is not sufficient, or the chosen model was too complex. The second cause is unlikely, since a logistic regression model is used with only linear dependencies features-class probability. When both test and train scores are poor, there may be The unaccounted features may be known but unusable during model deployment. because they are too expensive, or unknown. The 2nd case can also be diagnosed by isolating cases where the winning prover varies when none of the known features vary. During the process of dynamic deployment of the model.

[00091] The pipeline can be optimized by a search e.g. grid, on the complete parameter space of the operators involved, e.g. Optimization of the classification the regularization of linear regression.

[00092] The present invention is concerned with configuring the provers, in particular, setting the maximum number of steps the prover runs before the prover will be interrupted. Search a configuration is needed because it can not be used directly as a limit and a step size is needed because different machines have different runtimes. Which is the prover's internal counter-step size or BCP counter.

Therefore, a prover configuration setting is needed to interrupt the provers when they reach the configured step size for a property properties can not be reached and provers do not spend too much time on this particular property. While a large step size is needed, there is no such thing as a design, with no meaningful result for many properties, and no big gain in time , Therefore a step size depending on the design features must be set.

CD LU1 [00094] The present invention proposes a regression module 64, which is provided to model the dependency of the step size on the design parameters (features) and expected time to check the assertion. This is done by the regression module 64, at step S6 of the method for building the pipeline. A prediction is made by setting an expected runtime (on the reference machine) and specifying the design features values, as inputs, and getting the corresponding step size for a particular prover as output.

[00095] RM to optimally predict the step size in order to limit the time spent per check (property). As soon as possible, with the regression model RM with linear regression for configuring the provers P allow avoiding the drawbacks of the prior art approaches which proposed to add multiple instances of the same prover with different settings and compare them to select the best setting. This prior art approach is not efficient for continuous setting.

[00096] Ordinary Least Squares Linear Regression is an approach for modeling the relationship between a scalar dependent variable, one or more explanatory variables (or independent variables). The Ordinary Least Squares Linear Regression fits a linear model with coefficients to minimize the residual sum of squares between the observed responses in the dataset and the responses predicted by the linear approximation. Because the present invention is concerned with the configuration of the provers, the dependent variable of the linear regression may be the number of steps, and the independent variables of the linear regression may be design statistics and expected check time.

Fig. 7 shows the result of a linear regression done by the regression module 64 for two provers, namely, disproving and disprover 3. More precisely, Fig. 7 shows the linearized step on log 10 (1 + Real Time) with different colors for different designs, the data being measured on the prover Disproverl for designs where step size is bigger than 1, and runtime is bigger than 10s. At least for the two provers disproved and disproved, there is an exponential dependency of the time for the check to converge, on the required number of steps. The modeling is split into fitting a linear function F so that steps = F (logl0 (l + real time)). The resulting coefficients for different designs are fitted as linear regression models on the transformed features. LU1 [00098] The regression of linear coefficients as shown in fig. 7 what to do using a Least Angle Regression LASSO model. Least absolute shrinkage and selection operator (LASSO) is a linear model that estimates sparse coefficients. LASSO is useful due to its tendency to prefer solutions with fewer parameter values, effectively reducing the number of variables. Least Angle Regression LASSO is the model selection algorithm which less greedy than traditional forward selection methods. In the present embodiment, a LASSO model is of interest for the following reasons. There are many samples as the number of designs. Because PCA was first applied, as explained above, the correlation of features was removed. Therefore the problem of randomly chosen features does not appear here in the model. This problem of randomly chosen features has not been applied before. It is also known that there are other linear regression models that could be applied, such as the nearest neighbor regressor which is an alternative model, with comparable performance.

[00099] The regression model RM is validated at step S7. The regression module may be validated using maximum or average absolute error on step size prediction.

The step size prediction versus the actual step size can be seen in FIG. 8 showing a pair plot. This type of plot shows scatters of each dimension against each other (real time, number of steps and prediction on the number of steps in the present application), and histograms of the data on the main diagonal, colored by the design in the present application. One design is left for validation ('lsu_package'). 94% for Lasso. The pair plot shows an approximate linear dependency and equal range between the actual and the predicted number of steps. Predictedlarslasso, as well as predictedknnlall, hence the prediction follows the real value and 95% for Nearest Neighbor Regressor.

[000101] The maximum absolute error on the coefficients of the estimated dependency for the designs used for train, less than 0.15 and 6 for the test design. 5, both on training and validation. The validation design has a much higher error of the linear coefficients than the training designs. This indicates an overfit. The nearest neighbor regressor can be added to increase the confident1 on prediction.

[000102] Once the pipeline has been built and validated, the pipeline may be used in a method for verifying a design under test, i.e. which prover (s) should be used for the verification of the design under test, and optionally define a test scheduling. Optionally, new batches of data may be collected from new designs, to refine the pipeline, e.g. by partial fits, using Stochastic Gradient Descent.

[000103] The method comprises a first step of collecting data (S21). The data module 40 can collect and prepare data in the pipeline. The data design parameters DP of the designs under test to be used as features F.

[000104] The expected result of the property (hold / fail) may be used to separate the data before using the pipeline.

[000105] The features are then used by the validated pipeline to classify the prover, as a winning prover or not. It should be understood that the prediction of the step size is realized. This is useful to schedule, and predict the step size which is a long time duration (on the reference machine).

[000106] The prediction functions of the invention are the transformations and linear functions.

The prediction functions are done by the transformer module 50 and the classification module 60. As explained above with reference to Figs. 1A and IB, the transformer module 50 is adapted to transform the functions and apply linear functions (step S22) to standardize each feature from the set of features to a unit variance. This is done by the standard scaler module 52 and the principal component analyzer module 54, adapted to apply PCA to the set of design parameters DP.

[000108] PCA may be used to transform the machine into an orthogonal reduced space. The PCA module 54 is adapted to transform and reduce the set of design parameters DP to a set of orthogonal components which contains a high percentage of the variance of tfW ^ output. When the optimal orthogonal transformation of the set of features has been found, only the Principal Analysis Components (PCA) which preserve most of the variance of the reconstructed input space are selected. In other words, the transformer module 50 with its PCA module 54 can be seen as a dimensionality reduction module.

[000109] The classification module 60 is used to predict the best prover (s) BP as the winning prover WP among the set of provers P for each property (step S23). The classification model CM is run by the classification module 60 by inputting the new properties of the design under test.

[000110] A multiclass one-vs-remainder logistic regression may be used to predict probabilities of each prover of the set of provers. The classification module 60 is capable of predicting only the winning prover, but also calibrated probabilities per prover, i.e. chances of the provers to win.

[000111] Based on the classification of the products obtained by the classification module 60, a prover clustering module 70 may provide clustering capabilities (step S24). Clustering is the task of grouping a group of objects in the same group (called a cluster) are more similar to those in other groups (clusters). In the present invention, the process of clustering is the same as that of a proverb in a similar manner, based on runtimes with different properties in our case , Clustering can thus be applied on properties, to group them based on similar runtimes with different provers.

[000112] The prover clustering module 70 can group provers based on similarity of the runtime on different checks when more provers can run in parallel.

[000113] The prover clustering module 66 can parallelize the provers by selecting provers from different clusters. Clusters indicates which groups of provers are orthogonal in their behavior on sets of checks / designs. In parallel to minimize the runtime for the overall set of checks.

f D

I

[000114] With the present invention proposing using clustering for prover scheduling, by grouping provers based on runtime similarity and choosing provers from different groups to optimize the runtime on sets of checks. 10 shows a method for selecting provers to run in parallel according to the present invention.

[000115] In a first step S31, the prover clustering module 70 gets the runtime for each property and each prover. The result of this step can be seen in FIG. 11, which shows a plot of heatmap runtime checks for different provers.

[000116] The heatmap shows by the color gradient the runtimes for a simple example with 120 checks and 6 provers. The numbers of checks and numbers are for illustrative purpose only. The timeouts may be replaced by higher runtime values. In the example of fig. 11, the timeouts were replaced by double the maximum runtime (seen by dark gradient).

[000117] The provers and checks are grouped together into prover clusters and property clusters in the next steps. To do so, cluster properties are grouped in a limited number of clusters at step S32, and the provers are clustered in a defined number of prover clusters m. The defined numbers of prover clusters represents the maximum number of a user would like to have in parallel.

[000118] FIG. 12 shows a dendogram of clusters of provers and FIG. 13 shows a dendogram of clusters of properties. A dendrogram is a tree-like diagram of the distance between clusters with iteratively merged clusters from the bottom - where each sample is one cluster, upwards. The dendrogram assists in selecting an optimal number of clusters. The optimal number of clusters can be identified by deciding a cut-off point in the dendrogram. This is applied separately on the properties which hold and on properties which fail.

[000119] Once the clusters have been created, the prover clustering module 70 is adapted to assign a prover cluster for each property cluster (step S34). The prover cluster has a cluster of prover clusters having no time out and / or a minimum sum of runtimes.

[000120] The skilled person will understand that this method of assigning a prover cluster instead of a single property is faster. Indeed, the clustering approach, the minimum runtime of the current combination for the property, for each property, and then the minimum runtime of all the properties.

[000121] Finally, for each prover cluster, the best prover from the cluster may be selected.

[000122] The result can be seen on FIG. 14 shows a table of run times per property cluster and provers cluster. More precisely, the table of FIG. 14 contains the sum of runtimes for the selected best prover in each prover cluster, and for each cluster of properties. NA indicates there are timeouts in the respective cluster. The runtimes of the selected clusters are marked yellow. There are two clusters of provers converge, and one clusters of provers converge, and one clusters of provers converge, and one clusters of provers converge, and one clusters of provers converges provers converges, which is chosen as the only option.

[000123] In the example described with reference to Figs. 10-14, the skilled person understands that the cluster of the cluster 1 and cluster 3 may be selected as being the best provers to run in parallel. In this example, the best provers are those with the smallest overall runtime.

[000124] The clustering can be validated by ensuring a good enough distance between clusters.

[000125] Agglomerative hierarchical clustering based on the difference between runtimes of different provers on different checks. A linkage matrix with pairwise distances between points, according to a chosen linkage metric is first calculated. The second step is an iterative clustering which maximizes the distance hM-1 tween clusters.

[000126] It should be noted that the linkage metric defines the distance between newly formed clusters depending on the coordinates of the points in the clusters. The linkage can affect the resulting clusters. To identify the best linkage metric, it can be summarized in terms of cophenetic correlation and how it compares to a known set of properties. The cophenetic correlation is a measure of how faithfully a dendrogram preserves the pairwise distances between the original unmodeled data points. In variance minimization algorithm, which is defined here [https://docs.scipy.org/doc/], the linkage metric which maximizes the cophenetic correlation and grouped scipy-0.14.0 / reference / generated / scipy.cluster.hierarchy.linkage.html # scipy.cluster.hierarchy.linkag e] · [000127] The dendrograms of the runtimes of 841 holding properties and 179 failing properties are shown in Figures 15 and 16. Different clusters are colored using a default cutoff level of 70% from maximum intra-cluster distance. As can be seen on Figure 15, the grouping is according to expectations, similar strategies of prover2, e.g. strategies 0-8 vs. Strategies 9-15 falling in different clusters if choosing a cut-off intra-cluster distance of 2000, and approvers 3 and 4 having as well as differences in case of different clusters. A different clustering is obtained for failing assertions, as seen on FIG. 16.

[000128] Finally, Figure 17 shows the heatmap of the runtimes, where the center of gravity is on the left, the dendrogram on top of the dendrogram on the left the rows and columns are reordered as the dendrograms indicate. A reduced set of properties is shown (properties taking longer than 5 minutes).

[000129] A pattern is visible which indicates that prover clusters behave similarly as property clusters. This can be used for scheduling by selecting provers to run in parallel from clusters.

[000130] It should be noted that the provers and the property runtime is needed for each prover and property combination. But sometimes data is missing and clustering must be a way to work with it. In no case should the data be replaced by 0, because this would consider the prover to be almost solver for the property. So, removing the properties where some of the time is not recommended, because it's just removing them, is almost too slow on some others. ,

[000131] The missing data should be treated. If the data is missing because the prover was not recorded, then the observations could have gone completely to avoid assuming which would be the prover behavior. The results of the calculations include: 1) clustering the filled-in data, and 2) assigning to the missing values. Yet another solution would be to assign the average coordinate values of the nearest neighbors in the other dimensions. If the data is missing because of the prover timed out, then there are more options: A solution consists in filling in the missing data with fixed values, higher than the time limit used, but not too high, e.g. 1.5 * time limit. In this way, the missing values would determine with many missing values to the ones without, but would avoid distancing the provers with different amounts of too much from each other. The resulting clusters can be validated by comparing against clusters with varying values.

It should be noted that using the prover clusters as a target class for the prover selection classifier, instead of the actual provers, can improve the quality of the prover selection because it reduces the provers to groups with significant differences, thus making it easier be separated by a classifier.

[000133] The checks can thus be clustered by similarity on runtimes with different provers. This can be performed on the features after the PCA transformation. And the best prover.

[000134] For an efficient online use, by some types of classifiers. Random Forests, which features are most important to be evaluated first, and to check if the decision of the model is already confident enough.

[000135] Scheduling of the provers in parallel can be done by clustering and selecting provers from alternative clusters. Alternative to clustering the runtimes and choosing provers from alternative clusters, the scheduling of which provers to run in parallel can minimize the number of timeouts for a given time limit, and candidate groups which have the same number of timeouts are sorted by probability to solve within a time limit. In a further aspect, the provers can be scheduled sequentially. The scheduling can define provers from clusters which are almost on some types of checks (properties) first, with a parameter setting corresponding to a small time limit, which may result in subsequent groups to and clusters having generally higher run times.

[000136] In summary, the present invention provides a portfolio of provers based on the actual design and verification task at hand that maximizes the chance of success while staying within resource bounds. This approach eliminates the need for expert knowledge on the user side.

[000137] The innovation in embedding machine learns to be a formal verification tool. A verification challenge (designs and properties). The verification problem. Examples for such features may include design control points such as design input, in-out ports, un-driven wires, and user-defined cut points, Design Structural Elements as the total number of counters in the designs or the number of RAM blocks , Design State Complexity measuring the total number of design state elements, including the total number of design latches and flip-flops, Property Complexity comprising the sequential depth, a total number of flip flops, latches, and combinational gates of the property.

[000138] Further, the present invention provides a methodology for evaluating and minimizing resource consumption. This predictor can be read by the extracted features and available user data. The automated scheduling includes the selection of a set of verification engines, the selection of runtime parameters of verification engines, and resource limits (eg, in terms of real time, steps in Bounded Model Checking, or Boolean constraint propagation counts in verification engines based on SAT).

[000139] Machine learning is used for optimal engine selection, and provides a tool that automatically detects a verification schedule at the verification challenge at hand.

[000140] The description of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit itself to the precise form disclosed, and "modifications and variations are possible in light of the above teachings. The embodiment of the invention is characterized by the fact that it is used in a particular way. It is intended that the scope of the invention be appended hereto, and their equivalents. The entirety of each of the aforementioned documents is incorporated by reference. It is thus to be understood that individual aspects may be arbitrarily combined with each other.

Claims (13)

A computer-implemented method of selecting a demonstrator from among a plurality of demonstrators (P) for a design to be verified (D), comprising: - collecting, by a data module (40), raw data relating to designing and extracting, from the raw data, a plurality of input characteristics (F); transforming, by a transformer module (50), the plurality of input characteristics, wherein the transformation of the plurality of input characteristics comprises applying a linear regression to the plurality of input characteristics (F); classifying, using a classification module (60), the demonstrators of the plurality of demonstrators, in which the classification module (60) is adapted to predict a better demonstrator (P), being the demonstrator that solves a property faster the remaining demonstrators of the plurality of demonstrators (P), and select one or more demonstrators based on the results of the classification. The method of claim 1, wherein the step of extracting a plurality of characteristics (F) comprises the step of retrieving design parameters (DP) as input characteristics (F) for the module transformer (50). The method of claim 1, wherein the transformation of the collected data comprises removing the cross-correlation of the input characteristics (F). The method of claim 3, wherein the step of removing the cross-correlation of the input characteristics (F) comprises: standardizing, by a standard scalar module (52), the input characteristics of the plurality of characteristics input (F) to a unit variance to standardized characteristics (F); and applying, by a main component analysis module (54), an orthogonal transformation as a main component analysis with standardized characteristics (F), and keeping the components resulting from the main component analysis having a ratio significant variance explained. The method of claim 1, wherein the classification module (60) is pre-trained by collecting a set of training or training data relating to a set of designs (D), design parameters and durations of time. performing for a plurality of demonstrators, transforming the training or training data set, and classifying the demonstrators using the transformed set of training or learning data. The method of claim 1, wherein the classifying step comprises assigning a probability to a demonstrator, for a property, using a multi-class logistic regression based on design features and an expected result of the property. The method of claim 1, further comprising configuring the demonstrators of the plurality of demonstrators using a linear regression model dependence of a step size on the design and an expected time for verification of the assertion by the demonstrator. The method of claim 7, wherein a demonstrator is configured using a LASSO regression model having dependent variables and independent variables, wherein the dependent variable of the regression model is a number of steps and the variable independent of the regression model are design statistics and expected audit time. The method of claim 1, further comprising grouping demonstrators into a plurality of demonstrator groups and grouping properties into a plurality of properties, based on the execution times for each property and demonstrator. The method of claim 9, including assigning a demonstrator group to a property group, in particular wherein the assigned demonstrator group of the plurality of demonstrator groups has a minimum amount of run time. The method of claim 10, comprising selecting, for each assigned demonstrator group, the demonstrator group demonstrator having the minimum execution times on the properties of the property group to which the assigned demo group is assigned. The method of claim 10, further comprising executing the selected demonstrators of the parallel assigned demonstrator groups. A computer program product comprising a non-transitory computer support with a computer program stored therein for causing a computer to perform the steps of a method according to claim 1.