WO2017123683A1 - Automatic recommendations for deployments in a data center - Google Patents
Automatic recommendations for deployments in a data center Download PDFInfo
- Publication number
- WO2017123683A1 WO2017123683A1 PCT/US2017/013082 US2017013082W WO2017123683A1 WO 2017123683 A1 WO2017123683 A1 WO 2017123683A1 US 2017013082 W US2017013082 W US 2017013082W WO 2017123683 A1 WO2017123683 A1 WO 2017123683A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- target
- deployment
- feature
- features
- target feature
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/06—Buying, selling or leasing transactions
- G06Q30/0601—Electronic shopping [e-shopping]
- G06Q30/0631—Item recommendations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/145—Network analysis or design involving simulating, designing, planning or modelling of a network
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/34—Network arrangements or protocols for supporting network services or applications involving the movement of software or configuration parameters
Definitions
- the configuration of the deployment includes the identification of the various deployed systems of the deployment such as the servers, the operating systems and other system software, the applications (e.g., appliances), the security systems, the storage systems, the networking systems, and so on.
- An operating system for example, may be identified by name of the operating system, version number, patch level, and so on.
- Each of the deployed systems may have various sub-systems.
- a security system may have a firewall appliance, an anti-virus appliance, an encryption appliance, and so on.
- the configuration of the deployment includes the identifications of such sub -sub systems, any sub-sub-systems, and so on.
- Each of these deployed systems and sub-systems may have various parameters or configuration settings which are also part of the configuration of the deployment.
- an anti-virus appliance may have parameters that indicate which types of documents to scan and when to scan those types of documents.
- the types of documents may include emails, email attachments, web pages, newly created documents, and so on.
- the configuration settings may indicate to scan email attachments when they are received or when they are to be opened.
- the identifications of the systems of a deployment, the parameters of the systems and sub-systems, and other settings describing a current configuration of a system or subsystem are referred to as features of the deployment.
- a large deployment can have hundreds and even thousands of features.
- it is important that they be established at optimal values to satisfy objectives for the deployment such a minimum response time, highly secure, and so on.
- the establishing of the feature values can be a very complex and time-consuming process.
- a certain firewall appliance may not work well with a certain patch level of a certain version of an operating system. So, an administrator of a deployment would want to ensure that the certain firewall appliance and the certain patch level are not part of the deployment either during initial configuration of the deployment or as part of an upgrade to the deployment.
- a method and system for recommending a target feature value for a target feature for a target deployment is provided.
- a recommendation system for each of a plurality of deployments, collects feature values for the features (including the target feature) of that deployment.
- the recommendation system then generates a model for recommending a target feature value for the target feature based on the collected feature values of the features for the deployments.
- the recommendation system applies the model to the features of the target deployment to identify a target feature value for the target feature.
- the recommendation system then provides the identified target feature value as a recommendation for the target feature for the target deployment.
- Figure 1 illustrates a display page that allows a user to select target features to be recommended.
- Figure 2 illustrates a display page that provides a recommendation for a target feature.
- Figure 3 illustrates a display page for an administrator of a deployment to restrict the type of data that is collected by the recommendation system.
- Figure 4 illustrates a display page for an administrator to select the resources whose configuration information is not to be collected.
- Figure 5 illustrates a display page for an administrator to set constraints on or policies for recommendations.
- Figure 6 illustrates a display page for an administrator to set constraints for appliances of various companies.
- Figure 7 is a block diagram illustrating components of a recommendation system in some embodiments.
- Figure 8 is a flow diagram that illustrates processing of a control recommendation component in some embodiments.
- Figure 9 is a flow diagram that illustrates processing of a collect deployment data component in some embodiments.
- Figure 10 is a flow diagram that illustrates processing of a generate models component of the recommendation system in some embodiments.
- Figure 11 is a flow diagram that illustrates the processing of a train model component in some embodiments.
- Figure 12 is a flow diagram that illustrates processing of a generate recommendation component in some embodiments.
- Figure 13 is a flow diagram that illustrates processing of an analyze recommendation component in some embodiments.
- a recommendation system for recommending a feature value for a target feature for a target deployment within a data center is provided.
- the recommendation system recommends a feature value for the target feature based on similarity between the features of the target deployment and the features of other deployments in the data center. For example, if all the deployments with similar features use a certain firewall, the recommendation system may recommend that certain firewall for the target deployment.
- the recommendation system collects feature values for features, including the target feature, of deployments within the data center.
- a data center may provide computer resources at a single geographic location or at multiple geographic locations.
- the features may relate to data center resources (e.g., virtual machines and storage), appliances (e.g., firewalls and packet filters), configuration information and log files of the data center resources and appliances, and so on.
- data center resources e.g., virtual machines and storage
- appliances e.g., firewalls and packet filters
- configuration information and log files of the data center resources and appliances e.g., configuration information and log files of the data center resources and appliances, and so on.
- the recommendation system After collecting the feature values, the recommendation system generates a model for recommending a feature value for the target feature based on the collected feature values of the features for the deployments with the feature value of the target feature being a label for each deployment. For example, if the target feature is a firewall, then then the feature value may be the name of a firewall.
- the recommendation system may generate a feature vector for each deployment and label each feature vector with the name of the firewall used by the deployment.
- the model may be generated using one or more machine learning techniques. After generating the model, the recommendation system applies the model to the features of the target deployment to identify a feature value for the target feature and provides the identified feature value as a recommendation for the target feature for the target deployment.
- the recommendation system recommends that the target deployment use the firewall with that certain name.
- the recommendation system thus performs collaborative filtering to base recommendations on deployments with similar features.
- the recommendation system may generate a model for each possible target feature using the features of all the deployments as training data for the model. For example, the recommendation system first collects the feature values of the features of all deployments. For each possible target feature, the recommendation system generates a feature vector for each deployment, labels each feature vector with the feature value of the possible target feature for that deployment, and trains a model using feature vectors and labels as training data. To make a recommendation for a target feature for a target deployment, the recommendation system generates a feature vector for the target deployment and applies the model for that target feature to the feature vector to determine the feature value for the target feature to recommend.
- the recommendation system may generate separate models for clusters of similar deployments.
- the recommendation system first clusters the deployments based on similarity between features of the deployments. For example, the recommendation system may use a k-means clustering algorithm or a k-nearest neighbor clustering algorithm.
- the recommendation system may select a subset of the features for generating the clusters.
- the recommendation system may then generate a model for each cluster using training data that includes for each deployment, a feature vector of feature values of features of that deployment that is labeled with the feature value of the target feature (i.e., target feature value) of that deployment.
- the recommendation system For example, if 10 clusters are generated, the recommendation system generates 10 models (e.g., classifiers) for the target feature.
- the recommendation system may use different sets of features when generating the clusters and the models. Indeed, the recommendation system may even use different sets of features to generate the models for the different clusters. For example, it may be that all the deployments in a certain cluster have the same feature value for a certain feature. In such a case, that certain feature is not informative to distinguish feature values for the target feature. Another cluster, however, may have different feature values for that certain feature and thus may be informative to distinguish feature values for the target feature.
- the recommendation model may generate a model for each cluster for each possible target feature. For example, if there are five target features and 10 clusters, the recommendation system generates 50 models.
- the recommendation system may even generate different clusterings of deployments for different target features. For example, the recommendation system may use one set of features to generate six clusters for one target feature and use another set of features to generate 11 clusters for another target feature.
- the recommendation system when a model is generated for each cluster, the recommendation system generates a recommendation by first identifying the cluster of deployments to which the target deployment is most similar. For example, the recommendation system may generate a representative feature vector for each cluster. If the feature value of a feature is numeric (e.g., number of virtual machines), then the representative feature value for that feature may be an average, mean, or median of the feature values within the cluster. If the feature value is non-numeric (e.g., name of a firewall), then the representative feature value may indicate the percentage of deployments in the cluster that have each non-numeric value. The recommendation system may identify the most similar cluster based on cosine similarity between the representative feature vector for a cluster and the feature vector for the target deployment. After identifying the most similar cluster, the recommendation system applies the model for that cluster to the feature vector of the target deployment to generate the recommendation for the target feature.
- the feature value of a feature is numeric (e.g., number of virtual machines)
- the representative feature value for that feature may be an
- the recommendation system may monitor the features of the deployments in real time and provide recommendations based on changes in the features of the deployments. For example, if several deployments are the subject of a certain type of attack, some administrators of deployments may quickly change the configuration of their firewalls in an attempt to thwart the attack.
- the recommendation system may use such changes in the configurations as a trigger to launch the generating of recommendations for changing the configurations of firewalls of other deployments.
- the trigger may be based on a recommendation criterion such as a percentage of deployments that change their configuration in a certain time period, the sophistication of the deployments, the reputation of administrators of the deployments, and so on.
- the recommendation system generates a recommendation relating to the configuration of the firewall for other deployments and provides the recommendation for each deployment.
- a deployment may authorize the automatic implementing of recommendations.
- the recommendation system may direct a configuration manager of the deployment to change the configuration to automatically implement the recommended configuration without the explicit approval of an administrator of the deployment.
- the recommendation system may also regenerate models because of changes in the features of the deployments. As the features of the deployments change, the current models may not be as effective in making appropriate recommendations.
- the recommendation system may regenerate models, for example, based on when a certain time period has elapsed, the percentage of deployments that change a feature used in clustering or generating a model, and other factors such as those used to trigger the generation of recommendations.
- the recommendation system provides a solution to various technical problems encountered when configuring a deployment.
- a target deployment may use a default configuration for a firewall that is not effective at thwarting certain types of attacks.
- the recommendation system may direct the target deployment to change its configuration accordingly.
- the response time for users of a target deployment may become unacceptably long.
- an administrator of the target deployment may request recommendations for various target features. Since the features of a deployment may include usage patterns of users, the recommendation system may recommend, for example, increasing the number of virtual machines or the amount of storage based on deployments with similar usage patterns.
- Figures 1-6 illustrate aspects of the user experience of the recommendation system in some embodiments.
- Figure 1 illustrates a display page that allows a user to select target features to be recommended.
- a display page 100 lists various target features (or categories of target features), such as firewall, firewall configuration, database configuration, number of virtual machines, and so on.
- a user may select one or more of the target features for which the user wants a recommendation for their deployment.
- Figure 2 illustrates a display page that provides a recommendation for a target feature.
- a display page 200 provides a recommendation for a target feature that is a firewall.
- the recommendation system recommended that the deployment use firewall X.
- the recommendation system may also generate and provide an analysis of why firewall X was recommended.
- the analysis indicates that 60% of similar deployments use firewall Y and 40% use firewall X.
- the recommendation system recommended firewall X because the target deployment uses packet filter B, and 100%) of the deployments that use firewall X also use packet filter B while only 10%> of the deployments that use firewall Y also use packet filter B.
- Figure 3 illustrates a display page for an administrator of a deployment to restrict the type of data that is collected by the recommendation system.
- a data center may request that deployments make certain types of data available for collection by the recommendation system in exchange for using the recommendation system, some deployments may not want to make all the requested data available. For example, certain governments may require that permission be requested and granted before certain types of data can be collected. As another example, a deployment may consider usage patterns of its users to be confidential and may not want the usage patterns to be collected even though they are used anonymously.
- a display page 300 lists categories of data. To restrict the collection of data, the administrator selects the category to view the types of data whose collection may be restricted.
- FIG. 4 illustrates a display page for an administrator to select the resources whose configuration information is not to be collected.
- a display page 400 lists various resources such as virtual machines, storage, platform as a service (“PaaS”) products, subscriptions (e.g., collections of resources), groups (e.g., group of resources in a collection), and so on.
- PaaS platform as a service
- subscriptions e.g., collections of resources
- groups e.g., group of resources in a collection
- an administrator may want to restrict the collection of configuration information for group b of subscription 1 because it represents a group of resources that processes highly confidential information.
- Figure 5 illustrates a display page for an administrator to set constraints on or policies for recommendations. For example, an administrator may want a deployment to use firewalls only from certain companies.
- a display page 500 lists categories of target features whose recommendations can be constrained. The target features that may be constrained may include features relating to resources, appliances, security packages, and so on.
- Figure 6 illustrates a display page for an administrator to set constraints for appliances of various companies.
- a display page 600 lists companies and their appliances.
- the recommendation system may input a list of companies and their appliances or may generate the list from analysis of the collected deployment data. An administrator can select to allow recommendations on a company-by-company basis or on an appliance-by- appliance basis.
- FIG. 7 is a block diagram illustrating components of a recommendation system in some embodiments.
- the recommendation system 700 may include various modules or components 710-720 that implement the recommendation system and various data storages 731-736 to store data relating to the recommendations.
- a control recommendation component 710 controls the overall process of collecting data, generating models, and providing recommendations.
- the control recommendation component may invoke a collect deployment data component 711, a generate models component 712, a generate recommendation component 713, an analyze recommendation component 714, and a monitor changes component 715.
- the collect deployment data component collects data from the various deployments of a data center.
- the generate models component generates models for target features and invokes a train model component 716 to train each model.
- the generate recommendation component generates a recommendation for a target feature of a target deployment using a generated model.
- the analyze recommendation component generates analysis of why a certain target feature value was recommended.
- the monitor changes component monitors the changes in feature values of the deployments in a data center and triggers the making of recommendations when the changes satisfy a recommendation criterion.
- a present user experience component 720 coordinates the presenting of the user interface to an administrator of a deployment.
- the data storages include a deployment data storage 731, a deployment restriction storage 732, a deployment policy storage 733, a cluster storage 734, a model storage 735, and a recommendation storage 736.
- the deployment data storage stores the deployment data that is collected by the recommendation system.
- the deployment restriction storage stores any restrictions placed on the collection of data by deployments.
- the deployment policy storage stores the policies or constraints on recommendations specified by each deployment.
- the cluster storage stores the information describing each cluster, such as the deployments within each cluster and a representative feature vector of each cluster.
- the recommendation system may also have multiple clusterings of the deployments for use in recommending different sets of target features. For example, the deployments may be clustered using one set of feature vectors and clustered again using a second set of feature vectors.
- the model storage stores the information describing each model. The information may include weights for the various features used by the model.
- the recommendation storage stores the recommendations that have been provided to various deployments.
- the computing systems on which the recommendation system may be implemented may include a central processing unit, input devices, output devices (e.g., display devices and speakers), storage devices (e.g., memory and disk drives), network interfaces, graphics processing units, accelerometers, cellular radio link interfaces, global positioning system devices, and so on.
- the input devices may include keyboards, pointing devices, touch screens, gesture recognition devices (e.g., for air gestures), head and eye tracking devices, microphones for voice recognition, and so on.
- the computing systems may include servers of a data center, massively parallel systems, and so on.
- the computing systems may access computer-readable media that include computer-readable storage media and data transmission media.
- the computer-readable storage media are tangible storage means that do not include a transitory, propagating signal.
- Examples of computer-readable storage media include memory such as primary memory, cache memory, and secondary memory (e.g., DVD) and other storage.
- the computer-readable storage media may have recorded on them or may be encoded with computer-executable instructions or logic that implements the recommendation system.
- the data transmission media are used for transmitting data via transitory, propagating signals or carrier waves (e.g., electromagnetism) via a wired or wireless connection.
- the computing systems may include a secure cryptoprocessor as part of a central processing unit for generating and securely storing keys and for encrypting and decrypting deployment data using the keys.
- the recommendation system may be described in the general context of computer-executable instructions, such as program modules and components, executed by one or more computers, processors, or other devices.
- program modules or components include routines, programs, objects, data structures, and so on that perform particular tasks or implement particular data types.
- the functionality of the program modules may be combined or distributed as desired in various examples.
- aspects of the recommendation system may be implemented in hardware using, for example, an application-specific integrated circuit (ASIC).
- ASIC application-specific integrated circuit
- the recommendation system may use various machine learning techniques, such as a support vector machine, a Bayesian network, learning regression, and a neural network, when generating models. For example, after clustering deployments, the recommendation system may employ a support vector machine to train classifiers for each cluster that implement the model. To train a classifier for a certain target feature value (e.g., certain firewall), the recommendation system may use the feature vectors of the deployments that have that target feature value as positive examples of training data and the feature vectors of the deployments that do not have that target feature value as negative examples of training data.
- a support vector machine e.g., a Bayesian network
- learning regression e.g., a neural network
- a support vector machine operates by finding a hypersurface in the space of possible inputs.
- the hypersurface attempts to split the positive examples (e.g., filter of company X) from the negative examples (e.g., filter of company Y) by maximizing the distance between the nearest of the positive and negative examples and the hypersurface.
- a support vector machine simultaneously minimizes an empirical classification error and maximizes a geometric margin. This allows for correct classification of data that is similar to but not identical to the training data.
- Various techniques can be used to train a support vector machine.
- One technique uses a sequential minimal optimization algorithm that breaks the large quadratic programming problem down into a series of small quadratic programming problems that can be solved analytically. (See Piatt, J., "Sequential Minimal Optimization," 1998 http://research.microsoft.com/pubs/69644/tr-98-14.pdf)
- a support vector machine is provided training data represented by y x ) where x l represents a feature vector and y l represents a label for page .
- a support vector machine may be used to o timize the following:
- the recommendation system uses a radial basis function ("RBF") kernel as represented by the following:
- the recommendation system may also use a polynomial Gaussian RBF or a sigmoid kernel.
- the recommendation system may use cross-validation and grid search to find optimal values for parameters y and C . (See Hsu, C. W., Chang, C. C, and Lin, C. J.,
- FIG. 8 is a flow diagram that illustrates processing of a control recommendation component in some embodiments.
- a control recommendation component 800 controls the overall processing of the recommendation system.
- the component loops collecting the deployment data from the deployments of a data center.
- the component selects the next deployment.
- decision block 802 if all the deployments have already been selected, the component continues at block 804, else the component continues at block 803.
- the component invokes a collect deployment data component to collect the deployment data for the selected deployment and then loops to block 801 to select the next deployment.
- the various deployments may store their deployment data at a central repository so that the recommendation system can use the data without having to collect the data individually from the various deployments.
- the component receives target feature sets with each target feature set specifying one or more target features.
- a target feature set is a set of target features whose recommendation would likely be made as a group or are related in some other way.
- one target feature set may specify configuration features of a firewall, and another target feature set may specify features such as number of virtual machines and amount of storage.
- the recommendation system generates separate models for each target feature set.
- the component invokes a generate models component to generate models for each target feature set.
- the component loops providing recommendations for target deployments.
- the component selects the next target deployment. In decision block 807, if all the target deployments have already been selected, then the component completes, else the component continues at block 808.
- the component selects the next target feature set.
- decision block 809 if all the target feature sets have already been selected, then the component loops to block 806 to select the next target deployment, else the component continues at block 810.
- the component invokes a generate recommendation component to generate a recommendation for the selected target deployment for the selected target feature set.
- the component invokes an analyze recommendation component to analyze the generated recommendation and then loops to block 808 to select the next target feature set.
- FIG. 9 is a flow diagram that illustrates processing of a collect deployment data component in some embodiments.
- a collect deployment data component 900 is passed an indication of a deployment and collects the deployment data for that deployment.
- the component identifies data center resources of the deployment and a history of changes to the resources. For example, the fact that a deployment used a certain configuration with a small number of virtual machines but changed that configuration when the number of virtual machines was increased may be useful in recommending changes to configuration when a target deployment has increased its number of virtual machines.
- the component collects configuration information for the resources (e.g., applications executed on virtual machines, amount of memory of virtual machines, and so on).
- the component collects usage patterns of the resources.
- the usage patterns may include patterns of users accessing the deployment, patterns of applications accessing resources of the deployment, and so on.
- the component collects network data of the deployment.
- the network data may include the types of network traffic (e.g., video), amount of network traffic, and so on.
- the component identifies the appliances used by the deployment.
- the component collects the logs of the resources and the appliances. For example, the logs may indicate when each user logged on to an application of a deployment.
- the component collects various indicators of compromise regarding attacks and threat intelligence from the deployment and then completes.
- the component may, for example, analyze usage patterns to determine whether an attack (e.g., denial of service) was attempted.
- a deployment may purchase threat intelligence that, for example, identifies Internet Protocol ("IP”) addresses whose access should be blocked by a firewall.
- IP Internet Protocol
- FIG 10 is a flow diagram that illustrates processing of a generate models component of the recommendation system in some embodiments.
- a generate models component 1000 is provided target feature sets and trains a model for each target feature set.
- the component selects the next target feature set.
- decision block 1002 if all the target feature sets have already been selected, then the component completes, else the component continues at block 1003.
- the component selects the features for use in training the model.
- the component may employ a feature selection algorithm that searches for a subset of features that can effectively model the target feature set.
- the feature selection algorithm may train models using various subsets of features of certain deployments and score each model based on a set of deployments that were not used to train the model (e.g., cross-validation).
- a feature selection algorithm may start out with all features and repeatedly generate models and remove the least significant features until some termination criterion is satisfied.
- the component may use a principal component analysis technique to identify the informative features for selection of the target feature set.
- the component invokes a train model component to train a model and then loops to block 1001 to select the next target feature set.
- FIG 11 is a flow diagram that illustrates the processing of a train model component in some embodiments.
- a train model component 1100 is passed an indication of features and a target feature set and trains a model for the target feature set using the features.
- the component generates, for each deployment, a feature vector from the features.
- the component generates clusters of deployments based on the feature vectors.
- the component loops training a model for each cluster.
- the component selects the next cluster.
- decision block 1104 if all the clusters have already been selected, then the component completes, else the component continues at block 1105.
- the component trains a model for the cluster using the feature vectors of the deployments in the cluster and values of the target features of the target feature set as labels.
- the component generates a representative feature vector for the cluster and then loops to block 1103 to select the next cluster.
- the component may also train a separate model for each target feature of the target feature set.
- FIG 12 is a flow diagram that illustrates processing of a generate recommendation component in some embodiments.
- a generate recommendation component 1200 is invoked passing an indication of a target deployment and a target feature set. The component recommends a value for the target deployment for each target feature in the target feature set.
- the component generates a feature vector for the target deployment.
- the component selects a clustering of the deployments that was generated for the target feature set.
- the component identifies the cluster that is most similar to the feature vector based on the representative feature vectors of the clusters.
- the component applies the model of the identified cluster to the feature vector to generate a recommendation for feature values for the target feature set and then completes.
- Figure 13 is a flow diagram that illustrates processing of an analyze recommendation component in some embodiments.
- An analyze recommendation component 1300 is invoked passing a recommended feature value of a target feature and an indication of the cluster whose model was used to recommend that feature value.
- the component identifies significant features that were used in providing the recommendation for the target feature and then computes various statistics to explain the recommendation.
- the component identifies the significant features used in making the recommendation.
- the significant features may have been identified by a feature selection algorithm.
- the feature selection algorithm may generate an evaluation score for different subsets of features that indicates how well a subset of features models the target feature.
- the component selects the next distinct target feature value of a deployment in the cluster.
- the component may select distinct target feature values for all deployments regardless of the clustering so that the statistics are based on all deployments.
- the component may also generate statistics based on both of the deployments of the cluster and all deployments.
- the component generates a percentage for the selected feature value as the percentage of deployments in the cluster that have that selected feature value.
- the component selects the next significant feature starting with the most significant feature.
- decision block 1306 if all the significant features have already been selected, then the component loops to block 1302 to select the next feature value of the target feature, else the component continues at block 1307.
- block 1307 the component selects the next feature value of the selected significant feature.
- decision block 1308 if all the feature values have already been selected, then the component loops to block 1305 to select the next significant feature, else the component continues at block 1309.
- block 1309 the component generates a percentage for the selected feature value for the selected significant feature and then loops to block 1307 to select the next feature value.
- An implementation of the recommendation system may employ any combination of the embodiments.
- the processing described below may be performed by a computing device with a processor that executes computer-executable instructions stored on a computer-readable storage medium that implements the recommendation system.
- a method performed by a computing system for recommending a target feature value for a target feature for a target deployment within a data center is provided.
- the method for each of a plurality of deployments within the data center, collects feature values for features of that deployment, the features including the target feature.
- the method generates a model for recommending a target feature value for the target feature based on the collected feature values of the features for the deployments.
- the feature value for the target feature of a deployment is a label for that deployment.
- the method applies the model to the features of the target deployment to identify a target feature value for the target feature.
- the method provides the identified target feature value as a recommendation for the target feature for the target deployment.
- a method clusters the deployments based on similarity between features of the deployments and for each cluster, generates a model for that cluster based on the collected feature values of the features for the deployments within the cluster, with the target feature value for the target feature being a label for a deployment.
- a method selects a model of a cluster based on similarity of the features of the target deployment to the features of the deployments within the cluster.
- a method generates an analysis of the recommendation based on significant features that contributed to the recommendation of the identified target feature value.
- the features are selected from a group consisting of data center resource information, appliance information, network information, and indicators of compromise.
- the data center resource information includes one or more of identification of data center resources, configuration of data center resources, logs of data center resources, and usage patterns of data center resources.
- the appliance information includes one or more of identification of appliances, configuration of appliances, and logs of appliances.
- a method receives an indication of a restricted feature for a certain deployment and suppresses the collecting of the restricted feature from that certain deployment.
- a method receives an indication of a policy for the target feature of the target deployment and ensures that the identified target feature value is consistent with the policy of the target deployment.
- the features used to generate the model are selected using a feature selection algorithm that generates an evaluation score for different subsets of features, the evaluation score indicating how well a subset of features models the target feature.
- a computing system for identifying a target feature value for a target feature for a target deployment comprises a processor for executing computer-executable instructions and a computer- readable storage medium storing computer-executable instructions.
- the instructions generate training data that includes for each of a plurality of deployments, a feature vector of feature values of features of the deployment.
- the instructions generate a model for recommending a target feature value for the target feature based on the training data.
- the instructions apply the model to a feature vector of feature values of the target deployment to identify the target feature value for the target feature for the target deployment.
- the instructions that generate the training data further label the feature vector of a deployment with the feature value of the deployment for the target feature.
- the instructions further cluster the deployments based on similarity between features of the deployments and wherein the instructions that generate a model generate a model for each cluster using the feature vectors of the deployments in the cluster.
- the instructions that apply the model select a model of a cluster based on similarity of the feature vector of the target deployment to the feature vectors of the deployments within the cluster.
- the instructions generate an analysis of the identified target feature value based on significant features that contributed to the identification of the identified target feature value.
- the instructions automatically change a current target feature value of the target deployment to the identified target feature value.
- a method performed by a computing system for providing a recommendation of a target feature value for a target feature of a target deployment monitors changes in feature values of features of deployments. When the changes satisfy a recommendation criterion, the method generates a model for recommending a target feature value for the target feature based on feature values of features of the deployments and applies the model to feature values of the target deployment to identify the target feature value for the target feature for the target deployment.
- a changed feature value is for the target feature.
- a changed feature value is for a feature other than the target feature.
- the method automatically implements the identified target feature value for the target deployment.
- the feature values may be collected from deployments of multiple data centers that may include deployments in private data centers of organizations, deployments in cloud-based data centers, and so.
- provider of security appliances may request that its customers provide their feature values of the security features of their installations of the security appliances. The provider can then, based on those feature values, make recommendations to its current customers for changes in feature values, new customers for initial feature values, and so on. Accordingly, the invention is not limited except as by the appended claims.
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Finance (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Accounting & Taxation (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Marketing (AREA)
- Economics (AREA)
- Development Economics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A recommendation system for recommending a target feature value for a target feature for a target deployment is provided. The recommendation system, for each of a plurality of deployments, collects feature values for the features of that deployment. The recommendation system then generates a model for recommending a target feature value for the target feature based on the collected feature values of the features for the deployments. The recommendation system applies the model to the features of the target deployment to identify a target feature value for the target feature. The recommendation system then provides the identified target feature value as a recommendation for the target feature for the target deployment.
Description
AUTOMATIC RECOMMENDATIONS FOR DEPLOYMENTS IN A DATA
CENTER
BACKGROUND
[0001] When software is to be deployed in a data center, the configuration of the deployment needs to be established. The configuration of the deployment includes the identification of the various deployed systems of the deployment such as the servers, the operating systems and other system software, the applications (e.g., appliances), the security systems, the storage systems, the networking systems, and so on. An operating system, for example, may be identified by name of the operating system, version number, patch level, and so on. Each of the deployed systems may have various sub-systems. For example, a security system may have a firewall appliance, an anti-virus appliance, an encryption appliance, and so on. The configuration of the deployment includes the identifications of such sub -sub systems, any sub-sub-systems, and so on. Each of these deployed systems and sub-systems may have various parameters or configuration settings which are also part of the configuration of the deployment. For example, an anti-virus appliance may have parameters that indicate which types of documents to scan and when to scan those types of documents. The types of documents may include emails, email attachments, web pages, newly created documents, and so on. The configuration settings may indicate to scan email attachments when they are received or when they are to be opened. The identifications of the systems of a deployment, the parameters of the systems and sub-systems, and other settings describing a current configuration of a system or subsystem are referred to as features of the deployment.
[0002] A large deployment can have hundreds and even thousands of features. When establishing the feature values for the features, it is important that they be established at optimal values to satisfy objectives for the deployment such a minimum response time, highly secure, and so on. The establishing of the feature values, however, can be a very complex and time-consuming process. For example, a certain firewall appliance may not work well with a certain patch level of a certain version of an operating system. So, an administrator of a deployment would want to ensure that the certain firewall appliance and the certain patch level are not part of the deployment either during initial configuration of the deployment or as part of an upgrade to the deployment. It can require a very high-level of skill to establish the optimal feature values for a deployment and to re-establish feature values when problems arise such as a system failure, software is
infected with a vims, an intrusion is detected, and so on. Even when an administrator has such a high-level of skill, the administrator may not have all the information available to establish the optimal values. For example, the administrator may be unaware of a recent intrusion of a deployment of another organization. Even if the administrator did have all the information, the number of different combinations of feature values for the features can be astronomically high making the understanding the effects, which can be subtle, of each combination very difficult.
SUMMARY
[0003] A method and system for recommending a target feature value for a target feature for a target deployment is provided. In some embodiments, a recommendation system, for each of a plurality of deployments, collects feature values for the features (including the target feature) of that deployment. The recommendation system then generates a model for recommending a target feature value for the target feature based on the collected feature values of the features for the deployments. The recommendation system applies the model to the features of the target deployment to identify a target feature value for the target feature. The recommendation system then provides the identified target feature value as a recommendation for the target feature for the target deployment.
[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
BRIEF DESCRIPTION OF DRAWINGS
[0005] Figure 1 illustrates a display page that allows a user to select target features to be recommended.
[0006] Figure 2 illustrates a display page that provides a recommendation for a target feature.
[0007] Figure 3 illustrates a display page for an administrator of a deployment to restrict the type of data that is collected by the recommendation system.
[0008] Figure 4 illustrates a display page for an administrator to select the resources whose configuration information is not to be collected.
[0009] Figure 5 illustrates a display page for an administrator to set constraints on or policies for recommendations.
[0010] Figure 6 illustrates a display page for an administrator to set constraints for appliances of various companies.
[0011] Figure 7 is a block diagram illustrating components of a recommendation system in some embodiments.
[0012] Figure 8 is a flow diagram that illustrates processing of a control recommendation component in some embodiments.
[0013] Figure 9 is a flow diagram that illustrates processing of a collect deployment data component in some embodiments.
[0014] Figure 10 is a flow diagram that illustrates processing of a generate models component of the recommendation system in some embodiments.
[0015] Figure 11 is a flow diagram that illustrates the processing of a train model component in some embodiments.
[0016] Figure 12 is a flow diagram that illustrates processing of a generate recommendation component in some embodiments.
[0017] Figure 13 is a flow diagram that illustrates processing of an analyze recommendation component in some embodiments.
DETAILED DESCRIPTION
[0018] A recommendation system for recommending a feature value for a target feature for a target deployment within a data center is provided. In some embodiments, the recommendation system recommends a feature value for the target feature based on similarity between the features of the target deployment and the features of other deployments in the data center. For example, if all the deployments with similar features use a certain firewall, the recommendation system may recommend that certain firewall for the target deployment. To support making recommendations, the recommendation system collects feature values for features, including the target feature, of deployments within the data center. (A data center may provide computer resources at a single geographic location or at multiple geographic locations.) The features may relate to data center resources (e.g., virtual machines and storage), appliances (e.g., firewalls and packet filters), configuration information and log files of the data center resources and appliances, and so on.
[0019] After collecting the feature values, the recommendation system generates a model for recommending a feature value for the target feature based on the collected feature values of the features for the deployments with the feature value of the target feature being a label for each deployment. For example, if the target feature is a firewall,
then then the feature value may be the name of a firewall. The recommendation system may generate a feature vector for each deployment and label each feature vector with the name of the firewall used by the deployment. The model may be generated using one or more machine learning techniques. After generating the model, the recommendation system applies the model to the features of the target deployment to identify a feature value for the target feature and provides the identified feature value as a recommendation for the target feature for the target deployment. For example, if the features of the target deployment are most similar to features of deployments that use a firewall with a certain name, then the recommendation system recommends that the target deployment use the firewall with that certain name. The recommendation system thus performs collaborative filtering to base recommendations on deployments with similar features.
[0020] In some embodiments, the recommendation system may generate a model for each possible target feature using the features of all the deployments as training data for the model. For example, the recommendation system first collects the feature values of the features of all deployments. For each possible target feature, the recommendation system generates a feature vector for each deployment, labels each feature vector with the feature value of the possible target feature for that deployment, and trains a model using feature vectors and labels as training data. To make a recommendation for a target feature for a target deployment, the recommendation system generates a feature vector for the target deployment and applies the model for that target feature to the feature vector to determine the feature value for the target feature to recommend.
[0021] Alternatively, rather than having only one model for each target feature, the recommendation system may generate separate models for clusters of similar deployments. The recommendation system first clusters the deployments based on similarity between features of the deployments. For example, the recommendation system may use a k-means clustering algorithm or a k-nearest neighbor clustering algorithm. The recommendation system may select a subset of the features for generating the clusters. The recommendation system may then generate a model for each cluster using training data that includes for each deployment, a feature vector of feature values of features of that deployment that is labeled with the feature value of the target feature (i.e., target feature value) of that deployment. For example, if 10 clusters are generated, the recommendation system generates 10 models (e.g., classifiers) for the target feature. The recommendation system may use different sets of features when generating the clusters and the models. Indeed, the recommendation system may even use different sets of
features to generate the models for the different clusters. For example, it may be that all the deployments in a certain cluster have the same feature value for a certain feature. In such a case, that certain feature is not informative to distinguish feature values for the target feature. Another cluster, however, may have different feature values for that certain feature and thus may be informative to distinguish feature values for the target feature. The recommendation model may generate a model for each cluster for each possible target feature. For example, if there are five target features and 10 clusters, the recommendation system generates 50 models. The recommendation system may even generate different clusterings of deployments for different target features. For example, the recommendation system may use one set of features to generate six clusters for one target feature and use another set of features to generate 11 clusters for another target feature.
[0022] In some embodiments, when a model is generated for each cluster, the recommendation system generates a recommendation by first identifying the cluster of deployments to which the target deployment is most similar. For example, the recommendation system may generate a representative feature vector for each cluster. If the feature value of a feature is numeric (e.g., number of virtual machines), then the representative feature value for that feature may be an average, mean, or median of the feature values within the cluster. If the feature value is non-numeric (e.g., name of a firewall), then the representative feature value may indicate the percentage of deployments in the cluster that have each non-numeric value. The recommendation system may identify the most similar cluster based on cosine similarity between the representative feature vector for a cluster and the feature vector for the target deployment. After identifying the most similar cluster, the recommendation system applies the model for that cluster to the feature vector of the target deployment to generate the recommendation for the target feature.
[0023] In some embodiments, the recommendation system may monitor the features of the deployments in real time and provide recommendations based on changes in the features of the deployments. For example, if several deployments are the subject of a certain type of attack, some administrators of deployments may quickly change the configuration of their firewalls in an attempt to thwart the attack. The recommendation system may use such changes in the configurations as a trigger to launch the generating of recommendations for changing the configurations of firewalls of other deployments. The trigger may be based on a recommendation criterion such as a percentage of deployments that change their configuration in a certain time period, the sophistication of the
deployments, the reputation of administrators of the deployments, and so on. The recommendation system generates a recommendation relating to the configuration of the firewall for other deployments and provides the recommendation for each deployment. In some embodiments, a deployment may authorize the automatic implementing of recommendations. In such a case, the recommendation system may direct a configuration manager of the deployment to change the configuration to automatically implement the recommended configuration without the explicit approval of an administrator of the deployment. The recommendation system may also regenerate models because of changes in the features of the deployments. As the features of the deployments change, the current models may not be as effective in making appropriate recommendations. The recommendation system may regenerate models, for example, based on when a certain time period has elapsed, the percentage of deployments that change a feature used in clustering or generating a model, and other factors such as those used to trigger the generation of recommendations.
[0024] The recommendation system provides a solution to various technical problems encountered when configuring a deployment. For example, a target deployment may use a default configuration for a firewall that is not effective at thwarting certain types of attacks. As other deployments within the data center change their configurations to thwart an attack, the recommendation system may direct the target deployment to change its configuration accordingly. As another example, the response time for users of a target deployment may become unacceptably long. Because of the complexities of the target deployment, it can be difficult to identify a solution that would improve response time. In such a case, an administrator of the target deployment may request recommendations for various target features. Since the features of a deployment may include usage patterns of users, the recommendation system may recommend, for example, increasing the number of virtual machines or the amount of storage based on deployments with similar usage patterns.
[0025] Figures 1-6 illustrate aspects of the user experience of the recommendation system in some embodiments. Figure 1 illustrates a display page that allows a user to select target features to be recommended. A display page 100 lists various target features (or categories of target features), such as firewall, firewall configuration, database configuration, number of virtual machines, and so on. A user may select one or more of the target features for which the user wants a recommendation for their deployment.
[0026] Figure 2 illustrates a display page that provides a recommendation for a target feature. A display page 200 provides a recommendation for a target feature that is a firewall. In this example, the recommendation system recommended that the deployment use firewall X. The recommendation system may also generate and provide an analysis of why firewall X was recommended. In this example, the analysis indicates that 60% of similar deployments use firewall Y and 40% use firewall X. The recommendation system recommended firewall X, however, because the target deployment uses packet filter B, and 100%) of the deployments that use firewall X also use packet filter B while only 10%> of the deployments that use firewall Y also use packet filter B.
[0027] Figure 3 illustrates a display page for an administrator of a deployment to restrict the type of data that is collected by the recommendation system. Although a data center may request that deployments make certain types of data available for collection by the recommendation system in exchange for using the recommendation system, some deployments may not want to make all the requested data available. For example, certain governments may require that permission be requested and granted before certain types of data can be collected. As another example, a deployment may consider usage patterns of its users to be confidential and may not want the usage patterns to be collected even though they are used anonymously. A display page 300 lists categories of data. To restrict the collection of data, the administrator selects the category to view the types of data whose collection may be restricted.
[0028] Figure 4 illustrates a display page for an administrator to select the resources whose configuration information is not to be collected. A display page 400 lists various resources such as virtual machines, storage, platform as a service ("PaaS") products, subscriptions (e.g., collections of resources), groups (e.g., group of resources in a collection), and so on. In this example, an administrator may want to restrict the collection of configuration information for group b of subscription 1 because it represents a group of resources that processes highly confidential information.
[0029] Figure 5 illustrates a display page for an administrator to set constraints on or policies for recommendations. For example, an administrator may want a deployment to use firewalls only from certain companies. A display page 500 lists categories of target features whose recommendations can be constrained. The target features that may be constrained may include features relating to resources, appliances, security packages, and so on.
[0030] Figure 6 illustrates a display page for an administrator to set constraints for appliances of various companies. A display page 600 lists companies and their appliances. The recommendation system may input a list of companies and their appliances or may generate the list from analysis of the collected deployment data. An administrator can select to allow recommendations on a company-by-company basis or on an appliance-by- appliance basis.
[0031] Figure 7 is a block diagram illustrating components of a recommendation system in some embodiments. The recommendation system 700 may include various modules or components 710-720 that implement the recommendation system and various data storages 731-736 to store data relating to the recommendations. A control recommendation component 710 controls the overall process of collecting data, generating models, and providing recommendations. The control recommendation component may invoke a collect deployment data component 711, a generate models component 712, a generate recommendation component 713, an analyze recommendation component 714, and a monitor changes component 715. The collect deployment data component collects data from the various deployments of a data center. The generate models component generates models for target features and invokes a train model component 716 to train each model. The generate recommendation component generates a recommendation for a target feature of a target deployment using a generated model. The analyze recommendation component generates analysis of why a certain target feature value was recommended. The monitor changes component monitors the changes in feature values of the deployments in a data center and triggers the making of recommendations when the changes satisfy a recommendation criterion. A present user experience component 720 coordinates the presenting of the user interface to an administrator of a deployment. The data storages include a deployment data storage 731, a deployment restriction storage 732, a deployment policy storage 733, a cluster storage 734, a model storage 735, and a recommendation storage 736. The deployment data storage stores the deployment data that is collected by the recommendation system. The deployment restriction storage stores any restrictions placed on the collection of data by deployments. The deployment policy storage stores the policies or constraints on recommendations specified by each deployment. The cluster storage stores the information describing each cluster, such as the deployments within each cluster and a representative feature vector of each cluster. The recommendation system may also have multiple clusterings of the deployments for use in recommending different sets of target features. For example, the deployments may be
clustered using one set of feature vectors and clustered again using a second set of feature vectors. The model storage stores the information describing each model. The information may include weights for the various features used by the model. The recommendation storage stores the recommendations that have been provided to various deployments.
[0032] The computing systems on which the recommendation system may be implemented may include a central processing unit, input devices, output devices (e.g., display devices and speakers), storage devices (e.g., memory and disk drives), network interfaces, graphics processing units, accelerometers, cellular radio link interfaces, global positioning system devices, and so on. The input devices may include keyboards, pointing devices, touch screens, gesture recognition devices (e.g., for air gestures), head and eye tracking devices, microphones for voice recognition, and so on. The computing systems may include servers of a data center, massively parallel systems, and so on. The computing systems may access computer-readable media that include computer-readable storage media and data transmission media. The computer-readable storage media are tangible storage means that do not include a transitory, propagating signal. Examples of computer-readable storage media include memory such as primary memory, cache memory, and secondary memory (e.g., DVD) and other storage. The computer-readable storage media may have recorded on them or may be encoded with computer-executable instructions or logic that implements the recommendation system. The data transmission media are used for transmitting data via transitory, propagating signals or carrier waves (e.g., electromagnetism) via a wired or wireless connection. The computing systems may include a secure cryptoprocessor as part of a central processing unit for generating and securely storing keys and for encrypting and decrypting deployment data using the keys.
[0033] The recommendation system may be described in the general context of computer-executable instructions, such as program modules and components, executed by one or more computers, processors, or other devices. Generally, program modules or components include routines, programs, objects, data structures, and so on that perform particular tasks or implement particular data types. Typically, the functionality of the program modules may be combined or distributed as desired in various examples. Aspects of the recommendation system may be implemented in hardware using, for example, an application-specific integrated circuit (ASIC).
[0034] In some embodiments, the recommendation system may use various machine learning techniques, such as a support vector machine, a Bayesian network, learning
regression, and a neural network, when generating models. For example, after clustering deployments, the recommendation system may employ a support vector machine to train classifiers for each cluster that implement the model. To train a classifier for a certain target feature value (e.g., certain firewall), the recommendation system may use the feature vectors of the deployments that have that target feature value as positive examples of training data and the feature vectors of the deployments that do not have that target feature value as negative examples of training data.
[0035] A support vector machine operates by finding a hypersurface in the space of possible inputs. The hypersurface attempts to split the positive examples (e.g., filter of company X) from the negative examples (e.g., filter of company Y) by maximizing the distance between the nearest of the positive and negative examples and the hypersurface. A support vector machine simultaneously minimizes an empirical classification error and maximizes a geometric margin. This allows for correct classification of data that is similar to but not identical to the training data. Various techniques can be used to train a support vector machine. One technique uses a sequential minimal optimization algorithm that breaks the large quadratic programming problem down into a series of small quadratic programming problems that can be solved analytically. (See Piatt, J., "Sequential Minimal Optimization," 1998 http://research.microsoft.com/pubs/69644/tr-98-14.pdf)
[0036] A support vector machine is provided training data represented by yx ) where xl represents a feature vector and yl represents a label for page . A support vector machine may be used to o timize the following:
where vector w is perpendicular to the separating hypersurface, the offset variable b is used to increase the margin, the slack variable £l represents the degree of misclassification of xt , the function φ maps the vector Xl into a higher dimensional space, and C represents a penalty parameter of the error term. A support vector machine supports linear classification but can be adapted to perform nonlinear classification by modifying the kernel function as re resented by the following:
[0037] In some embodiments, the recommendation system uses a radial basis function ("RBF") kernel as represented by the following:
K (Xl , Xj ) Qxv\ ~y\\ xi X , \, y >
The recommendation system may also use a polynomial Gaussian RBF or a sigmoid kernel. The recommendation system may use cross-validation and grid search to find optimal values for parameters y and C . (See Hsu, C. W., Chang, C. C, and Lin, C. J.,
"A Practical Guide to Support Vector Classification," Technical Report, Dept. of Computer Science and Information Engineering, National Taiwan University, Taipei, 2010.)
[0038] Figure 8 is a flow diagram that illustrates processing of a control recommendation component in some embodiments. A control recommendation component 800 controls the overall processing of the recommendation system. In blocks 801-803, the component loops collecting the deployment data from the deployments of a data center. In block 801, the component selects the next deployment. In decision block 802, if all the deployments have already been selected, the component continues at block 804, else the component continues at block 803. In block 803, the component invokes a collect deployment data component to collect the deployment data for the selected deployment and then loops to block 801 to select the next deployment. In some embodiments, the various deployments may store their deployment data at a central repository so that the recommendation system can use the data without having to collect the data individually from the various deployments. In block 804, the component receives target feature sets with each target feature set specifying one or more target features. A target feature set is a set of target features whose recommendation would likely be made as a group or are related in some other way. For example, one target feature set may specify configuration features of a firewall, and another target feature set may specify features such as number of virtual machines and amount of storage. The recommendation system generates separate models for each target feature set. In block 805, the component invokes a generate models component to generate models for each target feature set. In blocks 806-811, the component loops providing recommendations for target deployments. In block 806, the component selects the next target deployment. In decision block 807, if all the target deployments have already been selected, then the component completes, else the component continues at block 808. In block 808, the component selects the next target
feature set. In decision block 809, if all the target feature sets have already been selected, then the component loops to block 806 to select the next target deployment, else the component continues at block 810. In block 810, the component invokes a generate recommendation component to generate a recommendation for the selected target deployment for the selected target feature set. In block 811, the component invokes an analyze recommendation component to analyze the generated recommendation and then loops to block 808 to select the next target feature set.
[0039] Figure 9 is a flow diagram that illustrates processing of a collect deployment data component in some embodiments. A collect deployment data component 900 is passed an indication of a deployment and collects the deployment data for that deployment. In block 901, the component identifies data center resources of the deployment and a history of changes to the resources. For example, the fact that a deployment used a certain configuration with a small number of virtual machines but changed that configuration when the number of virtual machines was increased may be useful in recommending changes to configuration when a target deployment has increased its number of virtual machines. In block 902, the component collects configuration information for the resources (e.g., applications executed on virtual machines, amount of memory of virtual machines, and so on). In block 903, the component collects usage patterns of the resources. The usage patterns may include patterns of users accessing the deployment, patterns of applications accessing resources of the deployment, and so on. In block 904, the component collects network data of the deployment. For example, the network data may include the types of network traffic (e.g., video), amount of network traffic, and so on. In block 905, the component identifies the appliances used by the deployment. In block 906, the component collects the logs of the resources and the appliances. For example, the logs may indicate when each user logged on to an application of a deployment. In block 907, the component collects various indicators of compromise regarding attacks and threat intelligence from the deployment and then completes. The component may, for example, analyze usage patterns to determine whether an attack (e.g., denial of service) was attempted. A deployment may purchase threat intelligence that, for example, identifies Internet Protocol ("IP") addresses whose access should be blocked by a firewall.
[0040] Figure 10 is a flow diagram that illustrates processing of a generate models component of the recommendation system in some embodiments. A generate models component 1000 is provided target feature sets and trains a model for each target feature
set. In block 1001, the component selects the next target feature set. In decision block 1002, if all the target feature sets have already been selected, then the component completes, else the component continues at block 1003. In block 1003, the component selects the features for use in training the model. The component may employ a feature selection algorithm that searches for a subset of features that can effectively model the target feature set. The feature selection algorithm may train models using various subsets of features of certain deployments and score each model based on a set of deployments that were not used to train the model (e.g., cross-validation). In some embodiments, a feature selection algorithm may start out with all features and repeatedly generate models and remove the least significant features until some termination criterion is satisfied. Alternatively, or in addition, the component may use a principal component analysis technique to identify the informative features for selection of the target feature set. In block 1004, the component invokes a train model component to train a model and then loops to block 1001 to select the next target feature set.
[0041] Figure 11 is a flow diagram that illustrates the processing of a train model component in some embodiments. A train model component 1100 is passed an indication of features and a target feature set and trains a model for the target feature set using the features. In block 1101, the component generates, for each deployment, a feature vector from the features. In block 1102, the component generates clusters of deployments based on the feature vectors. In blocks 1103-1106, the component loops training a model for each cluster. In block 1103, the component selects the next cluster. In decision block 1104, if all the clusters have already been selected, then the component completes, else the component continues at block 1105. In block 1105, the component trains a model for the cluster using the feature vectors of the deployments in the cluster and values of the target features of the target feature set as labels. In block 1106, the component generates a representative feature vector for the cluster and then loops to block 1103 to select the next cluster. The component may also train a separate model for each target feature of the target feature set.
[0042] Figure 12 is a flow diagram that illustrates processing of a generate recommendation component in some embodiments. A generate recommendation component 1200 is invoked passing an indication of a target deployment and a target feature set. The component recommends a value for the target deployment for each target feature in the target feature set. In block 1201, the component generates a feature vector for the target deployment. In block 1202, the component selects a clustering of the
deployments that was generated for the target feature set. In block 1203, the component identifies the cluster that is most similar to the feature vector based on the representative feature vectors of the clusters. In block 1204, the component applies the model of the identified cluster to the feature vector to generate a recommendation for feature values for the target feature set and then completes.
[0043] Figure 13 is a flow diagram that illustrates processing of an analyze recommendation component in some embodiments. An analyze recommendation component 1300 is invoked passing a recommended feature value of a target feature and an indication of the cluster whose model was used to recommend that feature value. The component identifies significant features that were used in providing the recommendation for the target feature and then computes various statistics to explain the recommendation. In block 1301, the component identifies the significant features used in making the recommendation. The significant features may have been identified by a feature selection algorithm. The feature selection algorithm may generate an evaluation score for different subsets of features that indicates how well a subset of features models the target feature. In block 1302, the component selects the next distinct target feature value of a deployment in the cluster. For example, if 20 deployments have the target feature value of A, 10 deployments have the target feature value of B, and one deployment has the target feature value of C, then there would be three distinct target feature values. Alternatively, the component may select distinct target feature values for all deployments regardless of the clustering so that the statistics are based on all deployments. The component may also generate statistics based on both of the deployments of the cluster and all deployments. In block 1303, if all the feature values for the target feature have already been selected, then the component completes, else the component continues at block 1304. In block 1304, the component generates a percentage for the selected feature value as the percentage of deployments in the cluster that have that selected feature value. In block 1305, the component selects the next significant feature starting with the most significant feature. In decision block 1306, if all the significant features have already been selected, then the component loops to block 1302 to select the next feature value of the target feature, else the component continues at block 1307. In block 1307, the component selects the next feature value of the selected significant feature. In decision block 1308, if all the feature values have already been selected, then the component loops to block 1305 to select the next significant feature, else the component continues at block 1309. In block 1309, the
component generates a percentage for the selected feature value for the selected significant feature and then loops to block 1307 to select the next feature value.
[0044] The following paragraphs describe various embodiments of aspects of the recommendation system. An implementation of the recommendation system may employ any combination of the embodiments. The processing described below may be performed by a computing device with a processor that executes computer-executable instructions stored on a computer-readable storage medium that implements the recommendation system.
[0045] In some embodiments, a method performed by a computing system for recommending a target feature value for a target feature for a target deployment within a data center is provided. The method, for each of a plurality of deployments within the data center, collects feature values for features of that deployment, the features including the target feature. The method generates a model for recommending a target feature value for the target feature based on the collected feature values of the features for the deployments. The feature value for the target feature of a deployment is a label for that deployment. The method applies the model to the features of the target deployment to identify a target feature value for the target feature. The method provides the identified target feature value as a recommendation for the target feature for the target deployment. In some embodiments, a method clusters the deployments based on similarity between features of the deployments and for each cluster, generates a model for that cluster based on the collected feature values of the features for the deployments within the cluster, with the target feature value for the target feature being a label for a deployment. In some embodiments, a method selects a model of a cluster based on similarity of the features of the target deployment to the features of the deployments within the cluster. In some embodiments, a method generates an analysis of the recommendation based on significant features that contributed to the recommendation of the identified target feature value. In some embodiments, the features are selected from a group consisting of data center resource information, appliance information, network information, and indicators of compromise. In some embodiments, the data center resource information includes one or more of identification of data center resources, configuration of data center resources, logs of data center resources, and usage patterns of data center resources. In some embodiments, the appliance information includes one or more of identification of appliances, configuration of appliances, and logs of appliances. In some embodiments, a method receives an indication of a restricted feature for a certain deployment and
suppresses the collecting of the restricted feature from that certain deployment. In some embodiments, a method receives an indication of a policy for the target feature of the target deployment and ensures that the identified target feature value is consistent with the policy of the target deployment. In some embodiments, the features used to generate the model are selected using a feature selection algorithm that generates an evaluation score for different subsets of features, the evaluation score indicating how well a subset of features models the target feature.
[0046] In some embodiments, a computing system for identifying a target feature value for a target feature for a target deployment is provided. The computing system comprises a processor for executing computer-executable instructions and a computer- readable storage medium storing computer-executable instructions. The instructions generate training data that includes for each of a plurality of deployments, a feature vector of feature values of features of the deployment. The instructions generate a model for recommending a target feature value for the target feature based on the training data. The instructions apply the model to a feature vector of feature values of the target deployment to identify the target feature value for the target feature for the target deployment. In some embodiments, the instructions that generate the training data further label the feature vector of a deployment with the feature value of the deployment for the target feature. In some embodiments, the instructions further cluster the deployments based on similarity between features of the deployments and wherein the instructions that generate a model generate a model for each cluster using the feature vectors of the deployments in the cluster. In some embodiments, the instructions that apply the model select a model of a cluster based on similarity of the feature vector of the target deployment to the feature vectors of the deployments within the cluster. In some embodiments, the instructions generate an analysis of the identified target feature value based on significant features that contributed to the identification of the identified target feature value. In some embodiments, the instructions automatically change a current target feature value of the target deployment to the identified target feature value.
[0047] In some embodiments, a method performed by a computing system for providing a recommendation of a target feature value for a target feature of a target deployment is provided. The method monitors changes in feature values of features of deployments. When the changes satisfy a recommendation criterion, the method generates a model for recommending a target feature value for the target feature based on feature values of features of the deployments and applies the model to feature values of the target
deployment to identify the target feature value for the target feature for the target deployment. In some embodiments, a changed feature value is for the target feature. In some embodiments, a changed feature value is for a feature other than the target feature. In some embodiments, the method automatically implements the identified target feature value for the target deployment.
[0048] Although the subject matter has been described in language specific to structural features and/or acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. For example, the feature values may be collected from deployments of multiple data centers that may include deployments in private data centers of organizations, deployments in cloud-based data centers, and so. As an example, provider of security appliances may request that its customers provide their feature values of the security features of their installations of the security appliances. The provider can then, based on those feature values, make recommendations to its current customers for changes in feature values, new customers for initial feature values, and so on. Accordingly, the invention is not limited except as by the appended claims.
Claims
1. A method performed by a computing system for recommending a target feature value for a target feature for a target deployment within a data center, the method comprising:
for each of a plurality of deployments within the data center, collecting feature values for features of that deployment, the features including the target feature;
generating a model for recommending a target feature value for the target feature based on the collected feature values of the features for the deployments, with the feature value of the target feature for a deployment being a label for that deployment;
applying the model to the features of the target deployment to identify a target feature value for the target feature; and
providing the identified target feature value as a recommendation for the target feature for the target deployment.
2. The method of claim 1 wherein the generating of the model includes:
clustering the deployments based on similarity between features of the deployments; and
for each cluster, generating a model for that cluster based on the collected feature values of the features for the deployments within the cluster, with the feature value for the target feature for a deployment being a label for a deployment.
3. The method of claim 2 wherein the applying of the model includes selecting a model of a cluster based on similarity of the features of the target deployment to the features of the deployments within the cluster.
4. The method of claim 1 further comprising generating an analysis of the recommendation based on significant features that contributed to the recommendation of the identified target feature value.
5. The method of claim 1 wherein the features are selected from a group consisting of data center resource information, appliance information, network information, and indications of compromise.
6. The method of claim 1 further comprising:
receiving an indication of a restricted feature for a certain deployment; and suppressing the collecting of the restricted feature from that certain deployment.
7. The method of claim 1 further comprising:
receiving an indication of a policy for the target feature of the target deployment; and
ensuring that the identified target feature value is consistent with the policy of the target deployment.
8. The method of claim 1 wherein the features used to generate the model are selected using a feature selection algorithm that generates an evaluation score for different subsets of features, the evaluation score indicating how well a subset of features models the target feature.
9. A computing system for identifying a target feature value for a target feature for a target deployment, the computing system comprising:
a processor for executing computer-executable instructions; and
a computer-readable storage medium storing computer-executable instructions that, when executed by the processor,
generate training data that includes for each of a plurality of deployments, a feature vector of feature values of features of the deployment;
generate a model for recommending a target feature value for the target feature based on the training data; and
apply the model to a feature vector of feature values of the target deployment to identify the target feature value for the target feature for the target deployment.
10. The computing system of claim 9 wherein the computer-executable instructions that generate the training data further label the feature vector of a deployment with the feature value of the deployment for the target feature.
11. The computing system of claim 9 wherein the computer-executable instructions further include instructions that, when executed by the processor, cluster the deployments based on similarity between features of the deployments and wherein the computer-executable instructions that generate a model generate a model for each cluster using the feature vectors of the deployments in the cluster.
12. The computing system of claim 9 wherein the computer-executable instructions further include instructions that, when executed by the processor, generate an analysis of the identified target feature value based on significant features that contributed to the identification of the identified target feature value.
13. The computing system of claim 9 wherein the computer-executable instructions further include instructions that, when executed by the processor, automatically change a current target feature value of the target deployment to the identified target feature value.
14. A method performed by a computing system for providing a recommendation of a target feature value for a target feature of a target deployment, the method comprising:
monitoring changes in feature values of features of deployments;
when the changes satisfy a recommendation criterion,
generate a model for recommending a target feature value for the target feature based on feature values of features of the deployments; and apply the model to feature values of the target deployment to identify the target feature value for the target feature for the target deployment.
15. The method of claim 14 wherein a changed feature value is for the target feature.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP17701410.7A EP3403232A1 (en) | 2016-01-15 | 2017-01-12 | Automatic recommendations for deployments in a data center |
CN201780004250.4A CN108292414B (en) | 2016-01-15 | 2017-01-12 | Automatic recommendation of deployments in a data center |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201662279435P | 2016-01-15 | 2016-01-15 | |
US62/279,435 | 2016-01-15 | ||
US15/156,182 US11533240B2 (en) | 2016-01-15 | 2016-05-16 | Automatic recommendations for deployments in a data center |
US15/156,182 | 2016-05-16 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2017123683A1 true WO2017123683A1 (en) | 2017-07-20 |
Family
ID=57882197
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2017/013082 WO2017123683A1 (en) | 2016-01-15 | 2017-01-12 | Automatic recommendations for deployments in a data center |
Country Status (4)
Country | Link |
---|---|
US (2) | US11533240B2 (en) |
EP (1) | EP3403232A1 (en) |
CN (1) | CN108292414B (en) |
WO (1) | WO2017123683A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2019067167A1 (en) * | 2017-09-29 | 2019-04-04 | Oracle International Corporation | Artificial intelligence driven configuration management |
US10789065B2 (en) | 2018-05-07 | 2020-09-29 | Oracle lnternational Corporation | Method for automatically selecting configuration clustering parameters |
Families Citing this family (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10055489B2 (en) * | 2016-02-08 | 2018-08-21 | Ebay Inc. | System and method for content-based media analysis |
US10459450B2 (en) | 2017-05-12 | 2019-10-29 | Autonomy Squared Llc | Robot delivery system |
US10944723B2 (en) * | 2017-11-17 | 2021-03-09 | ShieldX Networks, Inc. | Systems and methods for managing endpoints and security policies in a networked environment |
US11159553B2 (en) * | 2018-04-13 | 2021-10-26 | Webroot Inc. | Determining exploit prevention using machine learning |
US10824412B2 (en) * | 2018-04-27 | 2020-11-03 | Nutanix, Inc. | Method and apparatus for data driven and cluster specific version/update control |
US20200401904A1 (en) * | 2019-06-24 | 2020-12-24 | GE Precision Healthcare LLC | Adaptive medical imaging device configuration using artificial intelligence |
US11888686B2 (en) * | 2019-11-01 | 2024-01-30 | Microsoft Technology Licensing, Llc | Admin change recommendation in an enterprise |
US11882095B2 (en) * | 2020-04-13 | 2024-01-23 | Google Llc | Firewall insights processing and machine learning |
US11283688B2 (en) * | 2020-05-19 | 2022-03-22 | Cisco Technology, Inc. | Delayed recomputation of formal network topology models based on modifications to deployed network topologies |
US10992540B1 (en) | 2020-05-19 | 2021-04-27 | Cisco Technology, Inc. | Determining formal models using weighting factors for computing elements in multi-cloud environments |
US11424989B2 (en) | 2020-06-15 | 2022-08-23 | Cisco Technology, Inc. | Machine-learning infused network topology generation and deployment |
US11398948B2 (en) | 2020-06-29 | 2022-07-26 | Cisco Technology, Inc. | Generation and deployment of inherited network topology models |
US11074058B1 (en) * | 2020-06-30 | 2021-07-27 | Microsoft Technology Licensing, Llc | Deployment operations based on deployment profiles in a deployment system |
CN112001442B (en) * | 2020-08-24 | 2024-03-19 | 北京达佳互联信息技术有限公司 | Feature detection method, device, computer equipment and storage medium |
US11418397B1 (en) * | 2021-02-01 | 2022-08-16 | Cisco Technology, Inc. | Automated generation of standard network device configurations |
US20220382818A1 (en) * | 2021-05-26 | 2022-12-01 | Walmart Apollo, Llc | Methods and apparatus for correcting search queries |
US11757888B2 (en) | 2021-06-15 | 2023-09-12 | Fortinet, Inc. | Systems and methods for fine grained forward testing for a ZTNA environment |
US11888870B2 (en) | 2021-10-04 | 2024-01-30 | Microsoft Technology Licensing, Llc | Multitenant sharing anomaly cyberattack campaign detection |
US11861352B2 (en) * | 2021-12-29 | 2024-01-02 | Microsoft Technology Licensing, Llc | Smart deployment using graph optimization |
CN115033718B (en) * | 2022-08-15 | 2022-10-25 | 浙江大学 | Service application deployment method, device and equipment |
Family Cites Families (27)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7644051B1 (en) * | 2006-07-28 | 2010-01-05 | Hewlett-Packard Development Company, L.P. | Management of data centers using a model |
US20080040455A1 (en) * | 2006-08-08 | 2008-02-14 | Microsoft Corporation | Model-based deployment and configuration of software in a distributed environment |
US7974827B2 (en) * | 2007-04-23 | 2011-07-05 | Microsoft Corporation | Resource model training |
US20090276403A1 (en) * | 2008-04-30 | 2009-11-05 | Pablo Tamayo | Projection mining for advanced recommendation systems and data mining |
US8098590B2 (en) | 2008-06-13 | 2012-01-17 | Qualcomm Incorporated | Apparatus and method for generating performance measurements in wireless networks |
US8504504B2 (en) | 2008-09-26 | 2013-08-06 | Oracle America, Inc. | System and method for distributed denial of service identification and prevention |
US8255529B2 (en) | 2010-02-26 | 2012-08-28 | Red Hat, Inc. | Methods and systems for providing deployment architectures in cloud computing environments |
US8918439B2 (en) | 2010-06-17 | 2014-12-23 | International Business Machines Corporation | Data lifecycle management within a cloud computing environment |
US8606924B2 (en) | 2011-03-29 | 2013-12-10 | Bmc Software, Inc. | Pre-bursting to external clouds |
US9594149B2 (en) | 2011-10-28 | 2017-03-14 | Telefonaktiebolaget L M Ericsson (Publ) | Fingerprinting with radio channel related information |
US8949162B1 (en) | 2012-06-28 | 2015-02-03 | Amazon Technologies, Inc. | Customer experience in network-based services |
US8813235B2 (en) | 2012-08-10 | 2014-08-19 | Nopsec Inc. | Expert system for detecting software security threats |
US9258321B2 (en) | 2012-08-23 | 2016-02-09 | Raytheon Foreground Security, Inc. | Automated internet threat detection and mitigation system and associated methods |
US9588820B2 (en) | 2012-09-04 | 2017-03-07 | Oracle International Corporation | Cloud architecture recommender system using automated workload instrumentation |
US9208041B2 (en) * | 2012-10-05 | 2015-12-08 | International Business Machines Corporation | Dynamic protection of a master operating system image |
US20140136360A1 (en) | 2012-11-09 | 2014-05-15 | Sap Ag | Economic decision system for consumption of security services in cloud environments |
TWI474213B (en) | 2013-01-09 | 2015-02-21 | Hope Bay Technologies Inc | Cloud system for threat protection and protection method using for the same |
US9324119B2 (en) | 2013-03-15 | 2016-04-26 | Alert Enterprise | Identity and asset risk score intelligence and threat mitigation |
US20140281741A1 (en) | 2013-03-15 | 2014-09-18 | Khushboo Shah Bohacek | Method, user interface and apparatus for cloud service confidence level generation and display |
US9189224B2 (en) | 2013-07-11 | 2015-11-17 | Oracle International Corporation | Forming an upgrade recommendation in a cloud computing environment |
US9355088B2 (en) * | 2013-07-12 | 2016-05-31 | Microsoft Technology Licensing, Llc | Feature completion in computer-human interactive learning |
US20150066599A1 (en) | 2013-08-28 | 2015-03-05 | Connectloud, Inc. | Method and apparatus for periodic diagnostics of tenant event streams |
WO2015070189A1 (en) | 2013-11-10 | 2015-05-14 | Cloudweaver, Inc. | Methods and system for automated or user-assisted grouping and management of groups in cloud infrastructure and network |
US9692789B2 (en) | 2013-12-13 | 2017-06-27 | Oracle International Corporation | Techniques for cloud security monitoring and threat intelligence |
US10318882B2 (en) * | 2014-09-11 | 2019-06-11 | Amazon Technologies, Inc. | Optimized training of linear machine learning models |
CN104765793B (en) | 2015-03-24 | 2018-12-11 | 广东欧珀移动通信有限公司 | A kind of software recommendation method and server |
CN105119956B (en) | 2015-07-09 | 2018-08-17 | 传成文化传媒(上海)有限公司 | Network application system and dispositions method |
-
2016
- 2016-05-16 US US15/156,182 patent/US11533240B2/en active Active
-
2017
- 2017-01-12 WO PCT/US2017/013082 patent/WO2017123683A1/en unknown
- 2017-01-12 CN CN201780004250.4A patent/CN108292414B/en active Active
- 2017-01-12 EP EP17701410.7A patent/EP3403232A1/en not_active Ceased
-
2022
- 2022-11-14 US US17/986,626 patent/US20230071347A1/en active Pending
Non-Patent Citations (8)
Title |
---|
FRANCESCO RICCI ET AL: "Recommender Systems Handbook", 28 October 2010, SPRINGER, ISBN: 978-0-387-85820-3, pages: ToC,Ch01 - Ch07, XP055260525 * |
HSU, C. W.; CHANG, C. C.; LIN, C. J.: "A Practical Guide to Support Vector Classification", 2010, NATIONAL TAIWAN UNIVERSITY |
MITCH TULLOCH: "Microsoft System Center - Deploying Hyper-V with Software-Defined Storage & Networking", 16 March 2015, MICROSOFT PRESS, ISBN: 978-0-7356-9567-2, XP055353981 * |
NELSON L. S. DA FONSECA ET AL: "Cloud Services, Networking, and Management", 20 April 2015, WILEY, ISBN: 978-1-118-84594-3, pages: ToC - Pre,Ch06,Ind, XP055353811 * |
NIGEL CAIN ET AL: "Microsoft System Center - Building a Virtualized Network Solution", 28 July 2015, MICROSOFT PRESS, ISBN: 978-0-7356-9580-1, XP055353951 * |
PLATT, J., SEQUENTIAL MINIMAL OPTIMIZATION, 1998, Retrieved from the Internet <URL:http://research.microsoft.com/pubs/69644/tr-98-14.pdf> |
WIKIPEDIA: "Event-driven architecture", INTERNET ARTICLE, 25 December 2015 (2015-12-25), XP055354271, Retrieved from the Internet <URL:https://en.wikipedia.org/w/index.php?title=Event-driven_architecture&oldid=696694134> [retrieved on 20170313] * |
WIKIPEDIA: "Sequential minimal optimization", INTERNET ARTICLE, 2 November 2015 (2015-11-02), XP055353931, Retrieved from the Internet <URL:https://en.wikipedia.org/w/index.php?title=Sequential_minimal_optimization&oldid=688732632> [retrieved on 20170313] * |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2019067167A1 (en) * | 2017-09-29 | 2019-04-04 | Oracle International Corporation | Artificial intelligence driven configuration management |
US10496396B2 (en) | 2017-09-29 | 2019-12-03 | Oracle International Corporation | Scalable artificial intelligence driven configuration management |
US10592230B2 (en) | 2017-09-29 | 2020-03-17 | Oracle International Corporation | Scalable artificial intelligence driven configuration management |
US10664264B2 (en) | 2017-09-29 | 2020-05-26 | Oracle International Corporation | Artificial intelligence driven configuration management |
CN111316231A (en) * | 2017-09-29 | 2020-06-19 | 甲骨文国际公司 | Artificial intelligence driven configuration management |
US11023221B2 (en) | 2017-09-29 | 2021-06-01 | Oracle International Corporation | Artificial intelligence driven configuration management |
US10789065B2 (en) | 2018-05-07 | 2020-09-29 | Oracle lnternational Corporation | Method for automatically selecting configuration clustering parameters |
Also Published As
Publication number | Publication date |
---|---|
US11533240B2 (en) | 2022-12-20 |
CN108292414A (en) | 2018-07-17 |
EP3403232A1 (en) | 2018-11-21 |
US20170207980A1 (en) | 2017-07-20 |
US20230071347A1 (en) | 2023-03-09 |
CN108292414B (en) | 2023-02-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20230071347A1 (en) | Recommendation for deployment based on feature value changes | |
Ullah et al. | Architectural tactics for big data cybersecurity analytics systems: a review | |
US11425161B2 (en) | Computer-implemented method, a system and a computer program for identifying malicious URI data items | |
CN112567367B (en) | Similarity-based method for clustering and accelerating multiple incident surveys | |
Gardiner et al. | On the security of machine learning in malware c&c detection: A survey | |
US11558408B2 (en) | Anomaly detection based on evaluation of user behavior using multi-context machine learning | |
Biggio et al. | Poisoning complete-linkage hierarchical clustering | |
WO2018022259A1 (en) | Remediation for ransomware attacks on cloud drive folders | |
WO2020197681A1 (en) | Detecting a missing security alert using a machine learning model | |
Ndichu et al. | A machine learning approach to detection of critical alerts from imbalanced multi-appliance threat alert logs | |
Dabas et al. | An effective malware detection method using hybrid feature selection and machine learning algorithms | |
Liu et al. | SCADS: A scalable approach using spark in cloud for host-based intrusion detection system with system calls | |
Tariq et al. | A review on intrusion detection in cloud computing | |
Hatcher et al. | Machine learning-based mobile threat monitoring and detection | |
Nezhadkamali et al. | Android malware detection based on overlapping of static features | |
Rahmat et al. | Network traffic-based hybrid malware detection for smartphone and traditional networked systems | |
Adigun et al. | SwarmDroid: Swarm optimized intrusion detection system for the Android mobile enterprise | |
Karthick et al. | Improved Intelligent Malware Detection Model in Cloud Environment | |
Gautam et al. | Comparative analysis of classification techniques in network based intrusion detection systems | |
Xinning | Android Malware Detection through Machine Learning on Kernel Task Structure | |
Cao et al. | Data Security and Malware Detection in Cloud Storage Services | |
Asgharzadeh et al. | A survey on intrusion detection system based support vector machine algorithm | |
Trung et al. | On the effectiveness of transferability of adversarial Android malware samples against learning-based detectors | |
Solanky et al. | A review on several vulnerabilities detection techniques in Android Mobile | |
Ghorbani et al. | Toward a policy-based distributed intruison detection system in cloud computing using data mining approaches |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 17701410 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |