CN115442242A - Workflow arrangement system and method based on importance ordering - Google Patents
Workflow arrangement system and method based on importance ordering Download PDFInfo
- Publication number
- CN115442242A CN115442242A CN202211038247.3A CN202211038247A CN115442242A CN 115442242 A CN115442242 A CN 115442242A CN 202211038247 A CN202211038247 A CN 202211038247A CN 115442242 A CN115442242 A CN 115442242A
- Authority
- CN
- China
- Prior art keywords
- service
- resource
- workflow
- importance
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 66
- 238000003860 storage Methods 0.000 claims abstract description 116
- 238000011156 evaluation Methods 0.000 claims abstract description 77
- 230000002776 aggregation Effects 0.000 claims abstract description 45
- 238000004220 aggregation Methods 0.000 claims abstract description 45
- 238000004458 analytical method Methods 0.000 claims abstract description 37
- 238000012163 sequencing technique Methods 0.000 claims abstract description 35
- 230000001419 dependent effect Effects 0.000 claims abstract description 24
- 230000008569 process Effects 0.000 claims abstract description 18
- 238000012544 monitoring process Methods 0.000 claims abstract description 11
- 238000004364 calculation method Methods 0.000 claims abstract description 4
- 230000010354 integration Effects 0.000 claims abstract description 4
- 230000002159 abnormal effect Effects 0.000 claims description 23
- 238000012545 processing Methods 0.000 claims description 16
- 230000004044 response Effects 0.000 claims description 16
- 230000006870 function Effects 0.000 claims description 12
- 230000007246 mechanism Effects 0.000 claims description 12
- 239000011159 matrix material Substances 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 9
- 238000001514 detection method Methods 0.000 claims description 8
- 238000005457 optimization Methods 0.000 claims description 8
- 238000012549 training Methods 0.000 claims description 7
- 238000010276 construction Methods 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 4
- 238000005520 cutting process Methods 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 2
- 238000002372 labelling Methods 0.000 claims description 2
- 238000010606 normalization Methods 0.000 claims description 2
- 238000005070 sampling Methods 0.000 claims description 2
- 230000004931 aggregating effect Effects 0.000 claims 1
- 230000002547 anomalous effect Effects 0.000 claims 1
- 230000003993 interaction Effects 0.000 claims 1
- 238000004422 calculation algorithm Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000007726 management method Methods 0.000 description 3
- 230000002452 interceptive effect Effects 0.000 description 2
- 102100029469 WD repeat and HMG-box DNA-binding protein 1 Human genes 0.000 description 1
- 101710097421 WD repeat and HMG-box DNA-binding protein 1 Proteins 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/02—Knowledge representation; Symbolic representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention provides a workflow service arrangement system based on importance ordering.A resource information aggregation analysis module is used for aggregation analysis of workflow dependent resource information, including resource registration and discovery, resource runtime monitoring and resource index aggregation analysis, so as to support the workflow dependent resource importance evaluation ordering process; the resource importance evaluation sequencing module is used for importance evaluation sequencing of workflow dependent resources, including service resource importance evaluation sequencing and storage resource importance evaluation sequencing, so as to automatically recommend high-quality resource instances to support the intelligent workflow resource arrangement process; the online workflow intelligent arrangement module is used for intelligently arranging workflow dependent resources, including service resource arrangement, storage resource arrangement, configuration resource arrangement and calculation resource arrangement, so as to realize workflow service capability integration. The invention realizes the intelligent process of workflow arrangement through online recommendation of high-quality resources, and improves the efficiency of workflow release and the performance of workflow service.
Description
Technical Field
The invention relates to a service resource integration technology, in particular to a workflow arrangement system based on importance sequencing and an implementation method thereof.
Background
Although the service-oriented workflow can realize the automation of the business process in the computing environment, with the expansion of the application field and the increase of the application function requirements, the number of workflow nodes is rapidly increased, the dependency relationship is more and more complex, and the difficulty of workflow service integrated management is brought. Especially, in a cloud computing environment, resources such as large-scale service resources and storage resources are distributed in different clusters or servers, and resources with the same functional attributes often have different system architectures, deployment modes, operating states, access performances and the like, so that the difficulty in selecting and arranging workflow resources is increased. Due to design defects, the existing system faces some new problems in the aspects of continuous access and integrated use of multi-source workflow services. Aiming at the operation scene of workflow service in multiple modes, multiple dependencies and multiple instances, the existing platform lacks an intelligent arrangement auxiliary support mechanism at a system level. More importantly, most platforms only pay attention to the static information of data and service resources related to the field, but neglect the influence of the resource use state and runtime information on the service performance and the platform resource utilization efficiency. Due to the fact that cloud computing, particularly container cloud environment, is high in randomness, resource load fluctuation and even resource stability are caused, and unreasonable resource selection and arrangement directly influences the overall service performance of the workflow. In summary, currently, there is no research on a service-oriented workflow intelligent arrangement management method and system in the related field, that is, an intelligent arrangement auxiliary support mechanism based on resource importance evaluation ranking is established on the basis of resource runtime analysis.
Disclosure of Invention
The invention aims to provide a workflow arrangement system based on importance sorting and an implementation method thereof.
The technical solution for realizing the purpose of the invention is as follows: a workflow services orchestration system based on importance ranking, comprising:
and the resource information aggregation analysis module is used for performing aggregation analysis on the workflow dependent resource information and supporting the workflow dependent resource importance evaluation sequencing process, and comprises three sub-functional modules of resource registration and discovery, resource runtime monitoring and resource index aggregation analysis. Firstly, establishing a resource directory and a routing mechanism to realize the convergence and sharing of resource information; then, an online monitoring mechanism is established to realize the acquisition of resource operation information. For service resources, a call link is further remodeled by analyzing context log information called by workflow service, and access information such as service response time, call times and the like is acquired. And finally, through resource aggregation, the aggregation analysis of resource operation information, access information and the like is realized, and the aggregation analysis is used as a decision factor for resource importance evaluation and provides data support for a dependency resource importance evaluation sequencing module.
The resource importance evaluation sequencing module is used for importance evaluation sequencing of workflow dependent resources and automatically recommending high-quality resource instances to support an intelligent workflow resource arrangement process and comprises two sub-function modules of service resource importance evaluation sequencing and storage resource importance evaluation sequencing. The service resource importance evaluation ranking comprises three steps of service pre-selection, service optimization and service ranking. Firstly, service preselection is carried out by constructing a service characteristic space, carrying out service resource abnormity analysis based on an isolated forest model and rejecting abnormal services to prevent workflow performance hidden danger caused by participation and arrangement of abnormal service resources. And then, carrying out service optimization on the basis of service preselection, improving the index weight of a multi-criterion decision method through an entropy weight method, and comprehensively considering the functions of various characteristics of the service to realize more accurate service importance evaluation. Finally, services are sorted according to the importance of the service resources, top-k operation is carried out, and a near-real-time service resource recommendation function is provided by establishing a service push mechanism of an online workflow arrangement module; the storage resource importance evaluation ranking comprises two steps of storage evaluation and storage ranking. On the basis of constructing a storage feature space, the importance of storage resources is evaluated by using an improved multi-criterion decision method. And sorting and storing according to the importance of the storage resources and recommending the top-k.
The on-line workflow intelligent arrangement module is used for intelligently arranging workflow dependent resources and realizing workflow service capability integration and comprises four sub-function modules of service resource arrangement, storage resource arrangement, configuration resource arrangement and calculation resource arrangement. Firstly, a graphical interactive interface is provided to build a workflow model. Then, the assembly of the workflow-dependent resources is automatically performed in a recursive workflow model. For service resource arrangement, optional service resource instances are retrieved through a service resource database, if the optional service resource instances exist, service resource importance evaluation ordering is carried out, recommended service resource instances are automatically assembled, otherwise, the dependent services are arranged firstly, and after the arrangement is finished, backtracking is carried out to the current arrangement service; for storage resource arrangement, retrieving selectable storage resource instances through a storage resource database, if the selectable storage resource instances exist, performing storage resource importance evaluation sequencing, automatically assembling recommended storage resource instances, and if the selectable storage resource instances do not exist, automatically creating the recommended storage resource instances according to a storage template; for configuration resource arrangement, a relevant configuration instance set is retrieved through a configuration resource database, if the configuration instance set exists, the currently activated configuration instance is obtained, the connection information of the arranged dependent service resources and storage resources is updated to the configuration instance, and a configuration mounting method is used for injecting the configuration instance into a service container. If not, automatically creating according to the configuration template; and for the arrangement of the computing resources, the computing resources are adaptively distributed through clustering analysis, and the operation parameters are configured to complete the automatic arrangement process of the workflow.
The workflow service arrangement method based on the importance sorting realizes the workflow service arrangement based on the importance sorting through the workflow service arrangement based on the importance sorting.
A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement a workflow services orchestration based on importance ranking using the workflow services orchestration based on importance ranking. .
A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements importance-ranked based workflow services orchestration from said importance-ranked based workflow services orchestration.
Compared with the prior art, the invention has the following remarkable advantages: the intelligent process of workflow arrangement is realized through online recommendation of high-quality resources, and the efficiency of workflow release and the performance of workflow services are improved.
Drawings
FIG. 1 is a workflow orchestration system architecture diagram of the present invention based on importance ranking.
FIG. 2 is a schematic view of a workflow model.
Fig. 3 is a schematic diagram of service resource importance evaluation ranking.
Fig. 4 is a flowchart of a service resource importance evaluation ranking algorithm.
FIG. 5 is a flow chart of a storage resource importance evaluation ranking algorithm.
FIG. 6 is a flow diagram of workflow automation orchestration.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
According to the invention, the automatic arrangement management process of the workflow in the cloud computing environment is realized by integrating the service evaluation sequencing technology, the service intelligent arrangement technology and the like, the system architecture is shown in figure 1, and the workflow model is shown in figure 2. The workflow is composed of service nodes (circles) and storage nodes (squares), and the arrow direction is the task flow direction. The workflow automatic arrangement is characterized in that high-quality resources are selected and assembled into a workflow example according to a workflow model structure by evaluating service resources and storage resources in a cloud environment. A workflow arranging system and method based on importance sorting comprises a resource information aggregation analysis module, a dependency resource importance evaluation sorting module and an online workflow intelligent arranging module. The execution flow of the workflow service orchestration system comprises the following steps:
a. the resource information aggregation and analysis module is used for establishing a resource directory and a routing mechanism to realize resource information aggregation, realizing resource operation index acquisition through an online monitoring mechanism, realizing resource state representation through aggregation analysis of operation indexes, and supporting a process of relying on resource importance evaluation and ranking.
b. And the resource importance evaluation sorting module is relied on, the screening of high-quality resources is realized through a service resource importance evaluation sorting algorithm and a storage resource importance evaluation sorting algorithm, and the intelligent arrangement process of the online workflow is supported through high-quality resource recommendation.
c. And the online workflow intelligent arrangement module constructs a workflow model according to requirements, and recursively performs automatic arrangement processes of relying on service resources, storage resources, configuration resources and computing resources by combining a resource importance evaluation ordering algorithm.
The composition and function of each module will be described in detail below with reference to fig. 2-6.
The resource information aggregation and analysis module is used for aggregation analysis of service resources and stored resource information, supports a process of relying on resource importance evaluation and sequencing, and comprises three sub-function modules of resource registration and discovery, resource runtime monitoring and resource index aggregation analysis. Refer to the resource information aggregation analysis module portion shown in fig. 1. The specific implementation steps are as follows:
(1) Resource registration and discovery
And a resource directory and a routing mechanism are established, and a resource provider registers the access address to the resource directory in an active or passive mode to realize the convergence of resource information. The resource information includes service resources and storage resources.
(2) Resource runtime monitoring
And acquiring resource operation indexes in real time through an online monitoring mechanism. For service resources, acquiring basic indexes such as CPU (Central processing Unit) occupancy rate, memory occupancy rate, disk occupancy rate, network bandwidth occupancy rate and the like, remolding a service calling link by analyzing context log information of workflow service calling, and acquiring basic indexes such as service response time, service calling times (including calling failure times) and the like; for the storage resources, basic indexes such as storage usage, query processing number per second, transaction processing number per second, disk I/O operand processing per second and the like are collected.
(3) Resource indicator aggregation analysis
And performing aggregation analysis on basic index information of service resources and storage resources, serving as a decision factor for resource importance evaluation, and providing data support for the dependency resource importance evaluation sequencing module. And (3) further constructing a resource aggregation index on the basis of the basic index shown in the step (2). The service resource aggregation index includes an average resource occupancy (RUR), a resource usage imbalance (RID), a service Throughput (TH), an average Response Time (RT), a service level agreement violation rate (SLA), an access error rate (ERR), and the like. The RUR aggregation index is defined as the average value of occupancy rates of various resources (including CPU, memory and the like). The RID aggregation index is defined as the variance of the occupancy rates of various types of resources. The TH aggregation indicator is defined as the total number of accesses of the service within an observation time window. The SLA aggregation index is defined as the ratio of service response violation time to average response time RT, a service response threshold value delta is set, the violation is determined when the average response time RT is larger than delta, and the violation time is determined as RT-delta. The ERR index is defined as the ratio of the number of successful accesses of the service request in the observation time window to the total number of accesses; the storage resource aggregation indicators include the remaining available capacity of the workflow storage (CAP), the average transaction number (QPS), the average transaction number (TPS), and the average processing disk I/O operand (IOPS). The CAP aggregation index is defined as a difference value between a storage resource quota and a storage resource usage amount in an observation time window. Other aggregate indicators including QPS, TPS, IOPS are defined as the mean of the respective indicators within the observation time window.
The resource importance evaluation sequencing module is used for evaluating the importance of workflow dependent resources, supports an intelligent arrangement process of the workflow by automatically recommending high-quality resource instances, and comprises two sub-functional modules of service resource importance evaluation sequencing and storage resource importance evaluation sequencing. The service resource importance evaluation ranking schematic diagram refers to fig. 3, the algorithm flow of the service resource importance evaluation ranking refers to fig. 4, the algorithm flow of the storage resource importance evaluation ranking refers to fig. 5, and the specific implementation steps are as follows:
(1) Service resource importance evaluation ranking
And evaluating and sequencing the importance of the service resources, analyzing the importance value of the service resources with the same function through an algorithm, and automatically pushing the service resources with high value to participate in workflow arrangement. Referring to fig. 3 and 4, the method comprises three steps of service resource pre-selection, service resource optimization and service resource sequencing. The overall process is as follows: firstly, abnormal service instances are removed through service resource preselection. The service importance value is then evaluated by the service resource preference. And finally, the service instances are sequenced according to the importance value, and the instances with the maximum importance value are pushed to participate in the arrangement of the workflow service resources. The detailed implementation steps are as follows:
a. service resource pre-selection
And performing service resource abnormity analysis based on the isolated forest model, and eliminating abnormal services to prevent workflow performance hidden danger caused by participation of abnormal service resources in arrangement. Compared with normal service, the abnormal service generally shows the characteristics of large resource consumption, long response time, high SLA violation rate and the like, and has obvious outlier. The service resource preselection method based on the isolated forest model isolates sample outliers by segmenting the service feature space for multiple times, and further realizes the detection of service abnormity. Referring to fig. 4, the specific implementation steps are as follows:
(4) constructing a service feature space
On the basis of resource information aggregation analysis, six types of typical aggregation characteristics are selected to construct a workflow service characteristic space SF as shown in a formula (1), and the workflow service characteristic space SF is used as a decision factor for service resource abnormity analysis. Wherein, the RUR, RID, SLA, QPS, RT and ERR are respectively the average resource occupancy rate, the resource usage imbalance degree, the SLA violation rate, the service throughput, the service average response time and the service access error rate of the workflow service.
SF＝(RUR,RID,SLA,QPS,RT,ERR) (1)
Based on the service feature space of the formula (1), acquiring service feature sample data in a historical environment to construct a training set S T The method is used for training the isolated forest model. To eliminate the influence of different dimensions, S T Feature normalization processing has been performed, namely: feature raw value/feature maximum value. Similarly, collecting service characteristic sample data in the current environment to construct a test set S I And the service set is used as the service set to be evaluated.
(5) Abnormal detection model based on isolated forest is constructed
Using training set S T The structure has t subtrees with the size of
Isolated forest of each sub-tree, height limit of each sub-tree l and sub-tree size
In a relationship of
The construction of the isolated forest model comprises the following four steps:
step 1: from S T Random sampling
And (4) taking the sample as a root node of an isolated tree.
And 2, step: and randomly selecting an attribute q belonging to SF, and randomly generating a cutting point p in the value range of the attribute q. The selection of the cut point generates a hyperplane, and the data space of the current node is divided into 2 subspaces. And constructing data with the attribute value smaller than p in the current node data as a left branch, and constructing data with the attribute value larger than or equal to p as a right branch.
And step 3: recursion step 2 is repeated on the left and right branches of the node, and new leaf nodes are continuously constructed until only one sample data or tree on the leaf node has grown to the limit height l.
And 4, step 4: and (3) iterating the steps 1-3, and constructing an isolated forest model with t subtrees as an anomaly detection model of the workflow service.
(6) Abnormal service detection filtering
Introducing outlier function
For detecting abnormal states of service, as in equation (2). E (h (SF)) is the expected value of the path length of the feature vector SF of the current detection service in the isolated forest sub-tree.
For inclusion in isolated forests
The average path length of the isolated tree of the individual sample data is expressed by equation (3), and ξ is an euler constant. When in use
I.e. the anomaly score
When the value approaches 1, the service is determined to be abnormal.
Evaluation and labeling of test set S by equation (2) I The abnormal state of the middle sample (1 represents normal, and-1 represents abnormal), and a preselected service characteristic set S' is obtained after the abnormal service is further eliminated.
b. Service resource optimization
On the basis of service resource pre-selection, the importance of the service resources is further evaluated by improving a multi-criterion decision method, and service resource optimization is realized. Since the traditional method uses a service characteristic value weighting method to evaluate the importance of the service, a phenomenon that the individual attribute value is relatively superior, which leads to the service ranking to be advanced, may occur. Therefore, the method improves the index weight coefficient of the multi-criterion decision method by using the entropy weight method, comprehensively considers the functions of various characteristics and realizes more accurate service importance evaluation. Referring to fig. 4, the specific implementation steps are as follows:
(3) computing service feature weight coefficients
First, the entropy E of service feature information is calculated by a set S' of preselected service features using equation (4) j . Where N is the capacity of the preselected service feature set S', p ij Is the probability of pre-selecting a service feature.
Then, a service characteristic weight coefficient W is calculated using equation (5) based on the service characteristic information entropy j . Where K is the service feature space dimension.
(4) Evaluation service importance of improved multi-criterion decision method
Firstly, using the service feature weight W described in step (1) j The weight coefficient is used as a decision index in a multi-criterion decision method to realize the enhancement of the method. Then, using an improved multi-criterion decision method to evaluate the service importance, comprising the following five steps:
step 1: construction of a service importance decision matrix DM with N rows and K columns using a set S' of preselected service features N×K N is the capacity of S', and K is the characteristic space dimension. DM N×K Matrix element value g ij Is a normalized service characteristic value. Each row of the matrix is a feature vector of a preselected service, also called an evaluation object.
And 2, step: computing a regularized service importance decision matrix NDM n×k ，NDM n×k Matrix elements
And step 3: determining an optimal solution
And worst case scenario
The determination method of the optimal scheme comprises the following steps: the maximum of the corresponding features in all samples is taken for positive features and the minimum of the corresponding features in all samples is taken for negative features. For example, if the resource occupancy rate indicator RUR is a negative characteristic, the minimum value of the indicators in all samples is taken as the corresponding characteristic value of the optimal scheme. The worst case is the opposite, with the positive features taking the minimum and the negative features taking the maximum.
And 4, step 4: calculating the distance D between each evaluation object and the optimal scheme i + Distance D of the worst case i - Formula (6) wherein W is j The service feature weight coefficient described for equation (5).
And 5: calculating service importance value V according to optimal and worst distances i As shown in formula (7). V i The larger the service, the less important the service, and vice versa.
After evaluation by equation (7), the pre-selected service feature set S' is converted into a preferred service feature set S ″.
c. Service resource ordering
B, according to the service importance value from large to small, carrying out service importance ranking on the preferable service feature set S' and selecting K with the maximum importance value 1 The individual services serve as a set S of recommended services. Meanwhile, a service push mechanism of an online workflow assembly module is established, and a near-real-time service resource recommendation function is provided.
(2) Storage resource importance evaluation ranking
And evaluating and sequencing the importance of the storage resources, analyzing the importance value of the storage resources with the same function through an algorithm, and automatically pushing the storage resources with high value to participate in workflow arrangement. Referring to fig. 5, the method includes two steps of storage evaluation and storage sorting, and the specific implementation steps are as follows:
a. storage resource evaluation
On the basis of the construction of the storage feature space, the importance of the storage resources is evaluated by using an improved multi-criterion decision method. Referring to fig. 5, the specific implementation steps are as follows:
(1) constructing a storage feature space
On the basis of storage resource information aggregation analysis, four types of typical index features are selected to construct a workflow storage feature space, as shown in formula (8), and the workflow storage feature space is used as a decision factor for storage resource abnormality analysis. In the formula, CAP, QPS, TPS, IOPS are the capacity of the workflow storage, the average number of processing queries, the average number of processing transactions, and the average number of processing disk I/O operands, respectively.
DF＝(CAP,QPS,TPS,IOPS) (8)
Based on the storage characteristic space of the formula (8), a current environment storage characteristic sample data set D is acquired I As a storage resource evaluation set.
(2) Computing storage feature weight coefficients
The feature weight coefficients are calculated using an entropy weight method. The method comprises the steps of calculating storage characteristic information entropy by using a characteristic sample data set and calculating a storage characteristic weight coefficient based on the characteristic information entropy. Specifically, a method of calculating a weight coefficient of a service feature is referred to.
(3) Multi-criterion decision method for evaluating storage importance
The feature weight coefficients based on the entropy weight method are used for improving the multi-criterion decision method so as to evaluate the importance of the storage resources. The method comprises the five steps of constructing a storage importance decision matrix by using a storage characteristic sample data set, regularizing the storage importance decision matrix, determining an optimal scheme and a worst method, calculating the distance between an evaluation object and the optimal scheme and the worst scheme, and calculating the importance value of storage resources. Specifically, the service resource importance evaluation method is not described herein again.
b. Memory ordering
According to the step a, the importance value of the storage resources is reduced from large to small for storing the characteristic sample set D I Sorting the storage importance, and selecting the K with the maximum importance value 2 The stores are referred to as a set of recommended stores D. Meanwhile, a storage pushing mechanism of an online workflow assembly module is established, and a near-real-time storage resource recommendation function is provided.
The online workflow intelligent assembly module is used for intelligently arranging workflow dependent resources and comprises four sub-function modules of service resource arrangement, storage resource arrangement, configuration resource arrangement and calculation resource arrangement. Referring to fig. 6, the specific implementation steps are as follows:
firstly, a graphical interactive interface is provided to build a workflow model. Referring to fig. 2, the workflow model includes service nodes and storage nodes, an association relationship is established between the nodes through resource identifiers, and an arrow points to a depended resource; then, based on the visualized workflow model, various resources which are depended by the workflow are automatically arranged in a recursive workflow model mode, including:
(1) Service resource orchestration
Referring to path (1) in fig. 6, searching whether an optional service resource instance exists in the service resource database by depending on the service resource identifier SvcID, if yes, performing service resource importance evaluation ranking through the resource evaluation ranking module, and automatically assembling the service resource instance with the maximum recommended importance value (recording the access address of the service resource instance, and then injecting a configuration file for arranging services). If the relevant service resource instance is not retrieved, the dependent service is arranged, and the current arranging service is traced back after the arranging is finished.
(2) Storage resource orchestration
Referring to path (2) in fig. 6, searching whether an optional storage resource instance exists in the storage resource database by depending on the storage resource identifier DataID, if yes, performing storage resource importance evaluation sorting through the resource evaluation sorting module, and automatically assembling the storage resource instance with the highest recommended importance value (recording the access address of the storage resource instance, and then injecting a configuration file for arranging services). And if the relevant storage resource instance is not retrieved, creating a storage instance according to a storage template provided by the service.
(3) Configuring resource orchestration
Referring to path (3) in fig. 6, retrieving whether there is an activated configuration instance in the configuration resource database by using the configuration resource identifier CfgID, injecting the connection information of the dependent service resource and storage resource in steps (1) and (2) into the configuration instance, and injecting into the service container by using the configuration mount method. And if the relevant configuration example is not searched, performing new construction.
(4) Computing resource orchestration
Referring to a path (4) in the graph, computing resources are distributed to undeployed service nodes in the workflow, operation parameters are configured, and container technology is used for deployment and operation; and for the deployed service nodes, updating the service configuration and refreshing the dependent resource connection information. Service node computing resource allocation Q' average usage of computing resources by historical services of the same type
Determine the relationship of
The adjusting coefficient delta is determined according to the actual application sceneAnd (4) determining.
The invention also provides a workflow service arrangement method based on importance sequencing, and the workflow service arrangement based on importance sequencing is realized through the workflow service arrangement based on importance sequencing.
A computer device comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein when the processor executes the computer program, the workflow service arrangement based on importance sorting is realized by utilizing the workflow service arrangement based on importance sorting. .
A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements importance-ranked based workflow services orchestration from said importance-ranked based workflow services orchestration.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several implementation modes of the present application, and the description thereof is specific and detailed, but not to be construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, and these are all within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.
Claims (10)
1. A workflow services orchestration system based on importance ranking, comprising:
the resource information aggregation analysis module is used for performing aggregation analysis on workflow dependent resource information, including resource registration and discovery, resource runtime monitoring and resource index aggregation analysis, so as to support a workflow dependent resource importance evaluation sequencing process;
the resource importance evaluation sequencing module is used for importance evaluation sequencing of workflow dependent resources, including service resource importance evaluation sequencing and storage resource importance evaluation sequencing, so as to automatically recommend high-quality resource instances to support the intelligent workflow resource arrangement process;
the online workflow intelligent arrangement module is used for intelligently arranging workflow dependent resources, including service resource arrangement, storage resource arrangement, configuration resource arrangement and calculation resource arrangement, so as to realize workflow service capability integration.
2. The workflow service orchestration system based on importance ranking according to claim 1, wherein the resource information aggregation analysis module comprises a resource registration and discovery sub-function module, a resource runtime monitoring sub-function module, and a resource index aggregation analysis sub-function module, wherein:
the resource registration and discovery sub-function module is used for establishing a resource directory and realizing the convergence and sharing of service resources and storage resources by registering a resource address to the resource directory;
the resource operation monitoring sub-function module is used for monitoring the operation state of the resource, realizing the acquisition and storage of resource operation index information, acquiring the CPU occupancy rate, the memory occupancy rate, the disk occupancy rate and the network bandwidth occupancy rate of the service resource, reshaping a service calling link by analyzing the context log information called by the workflow service, and acquiring the service response time and the service calling times; for storage resources, collecting storage usage, query processing number per second, transaction processing number per second and disk I/O operand processing per second;
and the resource index aggregation analysis sub-function module is used for aggregating resources, realizing the aggregation of resource operation indexes and access indexes, taking the aggregation as a decision factor for resource importance evaluation, and providing data support for the resource importance evaluation-dependent sorting module.
3. The workflow service orchestration system based on importance ranking of claim 2, wherein the resource index aggregation analysis sub-function module performs resource aggregation by constructing a resource aggregation index, wherein:
the service resource aggregation indexes comprise average resource occupancy (RUR), resource usage unbalance (RID), service Throughput (TH), average Response Time (RT), service level agreement violation rate (SLA) and access error rate (ERR), wherein the resource usage unbalance is the variance of the resource occupancy of CPU and memory; the service level agreement violation rate is the ratio of the service response violation time to the average response time, a service response threshold value delta is set, the violation is determined when the average response time is larger than delta, and the violation time is determined as RT-delta; the access error rate is the ratio of the number of successful accesses of the service requests in the observation time window to the total number of accesses;
the storage resource aggregation index comprises a workflow storage available Capacity (CAP), an average processing query number (QPS), an average processing transaction number (TPS) and an average processing disk I/O operand (IOPS), wherein the storage available capacity is a difference value between a storage resource quota and a storage resource usage amount in an observation time window, and the QPS, the TPS and the IOPS are average values of corresponding indexes in the observation time window respectively.
4. The workflow services orchestration system based on importance ranking of claim 1, wherein the resource importance evaluation ranking module comprises a service resource importance evaluation ranking sub-function module and a storage resource importance evaluation ranking sub-function module, wherein:
the service resource importance evaluation sequencing sub-function module comprises a service preselection sub-function module, a service optimization sub-function module and a service sequencing sub-function module, wherein the service preselection sub-function module is used for constructing a service characteristic space, performing service resource abnormity analysis based on an isolated forest model and rejecting abnormal services so as to prevent the potential hazards of workflow performance caused by the participation and the arrangement of the abnormal service resources; the service optimization sub-function module is used for carrying out service optimization on the basis of service pre-selection, and adjusting the index weight of the multi-criterion decision method through an entropy weight method so as to comprehensively consider the functions of various characteristics of the service and realize more accurate service importance evaluation; the service sequencing sub-function module is used for sequencing services according to the importance of the service resources, performing top-k operation and providing a near-real-time service resource recommendation function by establishing a service push mechanism with the online workflow arrangement module;
the storage resource importance evaluation sequencing sub-function module comprises a storage evaluation sub-function module and a storage sequencing sub-function module, wherein the storage evaluation sub-function module is used for adjusting the index weight of the multi-criterion decision method through an entropy weight method on the basis of constructing a storage feature space so as to comprehensively consider the function of each feature of the storage resource and realize more accurate storage resource importance evaluation; and the storage sorting sub-function module is used for sorting the storage according to the importance of the storage resource and recommending the top-k.
5. The workflow service orchestration system based on importance ranking according to claim 4, wherein the service pre-selection sub-function module is configured to construct a service feature space, perform service resource anomaly analysis based on an isolated forest model, and reject anomalous services, and the implementation steps are as follows:
(1) constructing a service feature space
Firstly, on the basis of resource information aggregation analysis, six types of typical aggregation characteristics are selected to construct a workflow service characteristic space SF as a formula (1), and the workflow service characteristic space SF is used as a decision factor for service resource abnormity analysis;
SF＝(RUR,RID,SLA,QPS,RT,ERR) (1)
wherein, RUR, RID, SLA, QPS, RT and ERR are the average resource occupancy rate, the resource use imbalance degree, the SLA violation rate, the service throughput, the service average response time and the service access error rate of the workflow service;
then, based on the service feature space of the formula (1), acquiring service feature sample data in a historical environment to construct a training set S T For training of isolated forest models, wherein the training set S T Feature normalization processing has been performed, namely: and similarly, collecting service characteristic sample data under the current environment to construct a test set S I As a service set to be evaluated;
(2) abnormal detection model based on isolated forest is constructed
Using training set S T The structure has t subtrees with the size of
Isolated forest of each sub-tree, height limit of each sub-tree l and sub-tree size
In a relationship of
The construction of the isolated forest model comprises the following four steps:
step 1: from S T Random sampling
Samples as root nodes of an isolated tree;
step 2: randomly selecting an attribute q belonging to SF, randomly generating a cutting point p in the value range of the attribute q, selecting the cutting point to generate a hyperplane, dividing the data space of the current node into 2 subspaces, constructing data with the attribute value less than p in the data of the current node as a left branch, and constructing data with the attribute value more than or equal to p as a right branch;
and step 3: in the recursion step 2 of the left branch and the right branch of the node, new leaf nodes are continuously constructed until only one sample data or tree on the leaf nodes grows to the limit height l;
and 4, step 4: iterating the steps 1-3, constructing an isolated forest model with t subtrees, and taking the isolated forest model as an abnormal detection model of the workflow service;
(3) abnormal service detection filtering
Introducing outlier function
For detecting abnormal states of a service, E (h (SF)) is a feature vector of the currently detected service as in equation (2)The expected value of the path length of the SF in an isolated forest sub-tree,
for inclusion in isolated forests
The average path length of the isolated tree of the individual sample data is expressed as formula (3), and xi is the Euler constant when
I.e. the anomaly score
When the number is close to 1, the service is judged to be abnormal;
evaluation and labeling of test set S by equation (2) I The abnormal state of the middle sample is ' 1 ' indicating normal, and ' 1 ' indicating abnormal, and a preselected service characteristic set S ' is obtained after the abnormal service is further eliminated.
6. The workflow service orchestration system according to claim 4, wherein the service preference sub-function module is configured to perform service preference based on service preselection, and adjust index weights of a multi-criteria decision method by an entropy weight method, and the implementation steps are as follows:
(1) computing service feature weight coefficients
First, service feature information entropy E is calculated from a preselected service feature set S' using equation (4) j Where N is the capacity of the preselected service feature set S', p ij Probability of being a preselected service feature;
then, a service characteristic weight coefficient W is calculated using equation (5) based on the service characteristic information entropy j Wherein K is a service characteristic space dimension;
(2) evaluation of service importance by improved multi-criterion decision method
Firstly, using the service feature weight W described in step (1) j The method is enhanced as a weight coefficient of a decision index in a multi-criterion decision method, and then the service importance evaluation is carried out by using the improved multi-criterion decision method, which comprises the following five steps:
step 1: construction of a service importance decision matrix DM with N rows and K columns using a set S' of preselected service features N×K N is the capacity of S', K is the characteristic spatial dimension, DM N×K Matrix element value g ij For normalized service feature values, each row of the matrix is a feature vector of a preselected service, also called an evaluation object;
step 2: computing a regularized service importance decision matrix NDM n×k ，NDM n×k Of (2) element(s)
And step 3: taking the maximum value of the corresponding features in all samples for positive features, taking the minimum value of the corresponding features in all samples for negative features, and determining the optimal scheme
And worst case scenario
And 4, step 4: calculating the distance D between each evaluation object and the optimal scheme i + Distance D of the worst case i - Formula (6) wherein W is j The service characteristic weight coefficient described by equation (5);
and 5: calculating service importance value V according to optimal and worst distances i As shown in formula (7), V i The larger the service, the more important the service, otherwise the less important the service;
after evaluation by equation (7), the preselected service feature set S' is converted into a preferred service feature set S ″.
7. The system of claim 1, wherein the online workflow intelligent orchestration module comprises a service resource orchestration sub-function module, a storage resource orchestration sub-function module, a configuration resource orchestration sub-function module, and a computing resource orchestration sub-function module, wherein:
the service resource arrangement sub-function module is used for providing a graphical interaction interface to construct a workflow model, retrieving optional service resource examples through a service resource database, evaluating and ordering the importance of the service resources if the optional service resource examples exist, automatically assembling recommended service resource examples, arranging the dependent services if the optional service resource examples do not exist, and backtracking to the current arrangement service after the arrangement is finished;
the storage resource arrangement sub-functional module is used for automatically assembling the workflow dependent resources in a recursive workflow model, retrieving optional storage resource examples through a storage resource database, evaluating and ordering the importance of the storage resources if the optional storage resource examples exist, automatically assembling recommended storage resource examples, and automatically creating the recommended storage resource examples according to a storage template if the optional storage resource examples do not exist;
the configuration resource arranging sub-function module is used for retrieving a relevant configuration instance set through a configuration resource database, acquiring a currently activated configuration instance if the configuration instance exists, updating connection information of the arranged dependent service resources and storage resources to the configuration instance, injecting the configuration instance into a service container by using a configuration mounting method, and automatically creating the configuration instance if the configuration instance does not exist;
and the computing resource arrangement subfunction module is used for self-adaptively distributing computing resources through clustering analysis, configuring operation parameters and finishing the automatic arrangement process of the workflow.
8. A workflow service orchestration method based on importance ranking, characterized in that the workflow service orchestration based on importance ranking is implemented by using the workflow service orchestration system based on importance ranking according to any one of claims 1 to 7.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor when executing the computer program implementing a ranking of importance based workflow services orchestration system according to any one of claims 1-7.
10. A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements importance-based ranking of workflow services using the importance-based ranking workflow services orchestration system of any of claims 1-7.
Priority Applications (5)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211038247.3A CN115442242A (en) | 2022-08-29 | 2022-08-29 | Workflow arrangement system and method based on importance ordering |
| CN202211625573.4A CN115858168B (en) | 2022-08-29 | 2022-12-16 | Earth application model arrangement system and method based on importance ranking |
| CN202211731819.6A CN116127190B (en) | 2022-08-29 | 2022-12-30 | Digital earth resource recommendation system and method |
| CN202211729747.1A CN116186133A (en) | 2022-08-29 | 2022-12-30 | Electronic document management method integrating forward index and backward index |
| CN202310035450.3A CN116542326A (en) | 2022-08-29 | 2023-01-10 | Knowledge representation method and system based on time sequence convolution |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211038247.3A CN115442242A (en) | 2022-08-29 | 2022-08-29 | Workflow arrangement system and method based on importance ordering |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115442242A true CN115442242A (en) | 2022-12-06 |
Family
ID=84245199
Family Applications (5)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211038247.3A Pending CN115442242A (en) | 2022-08-29 | 2022-08-29 | Workflow arrangement system and method based on importance ordering |
| CN202211625573.4A Active CN115858168B (en) | 2022-08-29 | 2022-12-16 | Earth application model arrangement system and method based on importance ranking |
| CN202211729747.1A Pending CN116186133A (en) | 2022-08-29 | 2022-12-30 | Electronic document management method integrating forward index and backward index |
| CN202211731819.6A Active CN116127190B (en) | 2022-08-29 | 2022-12-30 | Digital earth resource recommendation system and method |
| CN202310035450.3A Pending CN116542326A (en) | 2022-08-29 | 2023-01-10 | Knowledge representation method and system based on time sequence convolution |
Family Applications After (4)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211625573.4A Active CN115858168B (en) | 2022-08-29 | 2022-12-16 | Earth application model arrangement system and method based on importance ranking |
| CN202211729747.1A Pending CN116186133A (en) | 2022-08-29 | 2022-12-30 | Electronic document management method integrating forward index and backward index |
| CN202211731819.6A Active CN116127190B (en) | 2022-08-29 | 2022-12-30 | Digital earth resource recommendation system and method |
| CN202310035450.3A Pending CN116542326A (en) | 2022-08-29 | 2023-01-10 | Knowledge representation method and system based on time sequence convolution |
Country Status (1)
| Country | Link |
|---|---|
| CN (5) | CN115442242A (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117370424B (en) * | 2023-12-07 | 2024-02-13 | 深圳市易图资讯股份有限公司 | Mobile application comment data analysis mining method and system for economic information analysis |
| CN117421487B (en) * | 2023-12-19 | 2024-03-08 | 西安康奈网络科技有限公司 | Multiple network information screening management system based on artificial intelligence |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110078516A1 (en) * | 2009-09-28 | 2011-03-31 | International Business Machines Corporation | Method and a system for performing a two-phase commit protocol |
| CN109684537A (en) * | 2018-10-29 | 2019-04-26 | 昆明理工大学 | A kind of the knowledge resource intelligently pushing system and its method for pushing of Business Process-oriented |
| US11537446B2 (en) * | 2019-08-14 | 2022-12-27 | Microsoft Technology Licensing, Llc | Orchestration and scheduling of services |
| CN110795219B (en) * | 2019-10-24 | 2022-03-18 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Resource scheduling method and system suitable for multiple computing frameworks |
| CN111079009B (en) * | 2019-12-11 | 2023-05-26 | 中国地质大学（武汉） | User interest detection method and system for government map service |
| CN111752723B (en) * | 2020-06-06 | 2021-05-04 | 中国科学院电子学研究所苏州研究院 | Visual multi-source service management system and implementation method thereof |
| CN112463363B (en) * | 2020-11-06 | 2022-08-26 | 苏州浪潮智能科技有限公司 | Resource arranging method, device, equipment and storage medium |
| CN114138486B (en) * | 2021-12-02 | 2024-03-26 | 中国人民解放军国防科技大学 | Method, system and medium for arranging containerized micro-services for cloud edge heterogeneous environment |
| CN114422582B (en) * | 2022-01-20 | 2023-05-16 | 中国科学院软件研究所 | Dynamic service combination method and device for technological resources |
| CN114638021A (en) * | 2022-03-18 | 2022-06-17 | 北京邮电大学 | Internet of things lightweight block chain system security evaluation method |
| CN114756170B (en) * | 2022-04-02 | 2023-03-24 | 苏州空天信息研究院 | Storage isolation system and method for container application |
-
2022
- 2022-08-29 CN CN202211038247.3A patent/CN115442242A/en active Pending
- 2022-12-16 CN CN202211625573.4A patent/CN115858168B/en active Active
- 2022-12-30 CN CN202211729747.1A patent/CN116186133A/en active Pending
- 2022-12-30 CN CN202211731819.6A patent/CN116127190B/en active Active
-
2023
- 2023-01-10 CN CN202310035450.3A patent/CN116542326A/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| CN115858168A (en) | 2023-03-28 |
| CN116186133A (en) | 2023-05-30 |
| CN116542326A (en) | 2023-08-04 |
| CN116127190A (en) | 2023-05-16 |
| CN116127190B (en) | 2023-07-28 |
| CN115858168B (en) | 2023-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10726153B2 (en) | Differentially private machine learning using a random forest classifier | |
| CN115858168B (en) | Earth application model arrangement system and method based on importance ranking | |
| JP6782782B2 (en) | Outlier detection of streaming data | |
| Ouadah et al. | SEFAP: an efficient approach for ranking skyline web services | |
| US20150019530A1 (en) | Query language for unstructed data | |
| JP2016197431A (en) | Determination for connectivity in community | |
| CN104077723B (en) | A kind of social networks commending system and method | |
| US10748166B2 (en) | Method and system for mining churn factor causing user churn for network application | |
| CN107908536B (en) | Performance evaluation method and system for GPU application in CPU-GPU heterogeneous environment | |
| US11887013B2 (en) | System and method for facilitating model-based classification of transactions | |
| US20190149344A1 (en) | Intelligent search system for service cost and method thereof | |
| CN112800115B (en) | Data processing method and data processing device | |
| CN116663938B (en) | Informatization management method based on enterprise data center system and related device thereof | |
| CN112527448A (en) | Openstack-based dynamic load adjustment method and system | |
| CN113835874A (en) | Deep learning service scheduling method, system, terminal and storage medium | |
| CN114995964A (en) | Combination service reconstruction method, device, equipment and computer readable medium | |
| Xiao et al. | ORHRC: Optimized recommendations of heterogeneous resource configurations in cloud-fog orchestrated computing environments | |
| CN107066328A (en) | The construction method of large-scale data processing platform | |
| CN107093092A (en) | Data analysing method and device | |
| Abdullah et al. | An integrated-model QoS-based graph for web service recommendation | |
| Zhang et al. | Risk management of commodity trade business based on deep learning and parallel processing of visual multimedia big data | |
| CN113722141B (en) | Method and device for determining delay reason of data task, electronic equipment and medium | |
| CN108280224A (en) | Ten thousand grades of dimension data generation methods, device, equipment and storage medium | |
| CN107103095A (en) | Method for computing data based on high performance network framework | |
| Su et al. | GBA: A Tuning-free Approach to Switch between Synchronous and Asynchronous Training for Recommendation Models |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20221206 |
|
| WD01 | Invention patent application deemed withdrawn after publication |