CN116644922A - FaaS workflow partitioning and scheduling method and system based on fine-grained data dependence - Google Patents

FaaS workflow partitioning and scheduling method and system based on fine-grained data dependence Download PDF

Info

Publication number
CN116644922A
CN116644922A CN202310623714.7A CN202310623714A CN116644922A CN 116644922 A CN116644922 A CN 116644922A CN 202310623714 A CN202310623714 A CN 202310623714A CN 116644922 A CN116644922 A CN 116644922A
Authority
CN
China
Prior art keywords
workflow
fine
functions
node
grouping
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310623714.7A
Other languages
Chinese (zh)
Inventor
陈伟
魏峻
吴国全
刘璐
高浩城
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Institute of Software of CAS
Original Assignee
Institute of Software of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Institute of Software of CAS filed Critical Institute of Software of CAS
Priority to CN202310623714.7A priority Critical patent/CN116644922A/en
Publication of CN116644922A publication Critical patent/CN116644922A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
    • G06Q10/06311—Scheduling, planning or task assignment for a person or group
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F30/00—Computer-aided design [CAD]
    • G06F30/20—Design optimisation, verification or simulation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0633—Workflow analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/067—Enterprise or organisation modelling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/10—Office automation; Time management
    • G06Q10/103—Workflow collaboration or project management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2111/00—Details relating to CAD techniques
    • G06F2111/04—Constraint-based CAD
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Business, Economics & Management (AREA)
  • Human Resources & Organizations (AREA)
  • Engineering & Computer Science (AREA)
  • Strategic Management (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Economics (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Operations Research (AREA)
  • General Business, Economics & Management (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • Marketing (AREA)
  • Development Economics (AREA)
  • Educational Administration (AREA)
  • Game Theory and Decision Science (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Hardware Design (AREA)
  • Evolutionary Computation (AREA)
  • Geometry (AREA)
  • General Engineering & Computer Science (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention relates to a FaaS workflow partitioning and scheduling method and system based on fine-grained data dependence. The method comprises the following steps: performing fine-grained data dependency analysis on the FaaS workflow; according to the result of the fine-grained data dependency analysis, a computing node model and a workflow model are established; initially grouping functions in the workflow, and placing the groups on each node according to the resource use condition of the current node; identifying a critical path in the workflow according to the packet placement condition, and sequencing critical edges in the critical path according to the data transmission time to obtain a critical edge sequencing list; and merging the grouping of the functions according to the key edge ordering list. The invention can effectively optimize the data transmission cost in the FaaS workflow execution process, further effectively optimize the end-to-end time delay of the workflow execution and improve the application service quality.

Description

FaaS workflow partitioning and scheduling method and system based on fine-grained data dependence
Technical Field
The invention relates to a function-as-a-service (FaaS) workflow partitioning and scheduling method, in particular to a FaaS workflow partitioning and scheduling method and system based on fine-grained data dependence, and belongs to the technical field of computer software.
Background
Serverless (Serverless) computing is a Service mode in cloud computing, and mainly consists of Function-as-a-Service (FaaS) and back-end-as-Service (BaaS). Among them, faaS plays the most central role in the server architecture, so FaaS is also called Serverless function, and existing research works often use FaaS to refer to server. The serverless computing can enable a developer to write and run an application program in a function-as-a-service (FaaS) mode, and the developer only needs to write codes to realize the function without concern about the operation environment of the infrastructure resource operation and deployment.
Because a single function is difficult to meet the complex business needs in a real scenario, emerging applications have begun to develop and use FaaS workflows composed of multiple functions to provide complex application functions by coordinated control of the different functions. Industry and academia agree that several types of complex applications, including image processing, will eventually be widely used on the FaaS platform. The FaaS workflow may describe the process of the business, giving the engine an automated execution of the task. The FaaS workflow system manages and runs applications with functions as services. FaaS workflow products are currently offered by various cloud providers, such as AWS Step Functions, microsoft Durable Functions, google works, amoeba services workflow, etc.
Studies have shown that FaaS workflow platforms have a total running time far in excess of the actual time required for function execution, indicating that the overhead in the system for connecting functions is significant. This overhead severely affects the efficiency of the FaaS workflow execution. By analyzing the existing work and system execution mechanisms, the overhead is found to be divided into data transmission delay, scheduling delay and cold start delay. The existing research data show that for data intensive applications, the cold start delay is removed and the data transmission delay accounts for 95.13% of the overhead on average.
The problem of large data transmission overhead of the current FaaS workflow system is focused by academia, and some related achievements are formed. The existing research work mainly adopts corresponding solutions to optimize from the aspects of scheduling and deployment of workflow functions, containerization encapsulation, an execution framework of a workflow system, an access mechanism of data among functions and the like. Research work (see: li Z, liu Y, guo L, et al FaaSFlow: enable efficient workflow execution for function-as-a-service [ C ]// Proceedings of International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). ACM, 2022:782-796.) employs a data dependency analysis approach of function granularity to identify critical paths for workflow execution, deploy functions with execution dependencies on the critical paths to the same node, and exploit data locality to reduce data transmission overhead. The work can optimize the end-to-end delay of workflow execution to some extent. However, for workflows with fine-grained data dependence, this approach can create additional data transfer overhead and memory space usage.
Disclosure of Invention
The invention aims to solve the problem of high data transmission overhead of a FaaS workflow system, and provides a FaaS workflow partitioning and scheduling method, which effectively utilizes the data locality, reduces the data transmission delay executed by FaaS workflow application with fine-grained data dependence, thereby optimizing the end-to-end delay and improving the service quality.
The technical scheme adopted by the invention is as follows:
a FaaS workflow partitioning and scheduling method based on fine-grained data dependence comprises the following steps:
performing fine-grained data dependency analysis on the FaaS workflow;
according to the result of the fine-grained data dependency analysis, a computing node model and a workflow model are established;
initially grouping functions in the workflow, and placing the groups on each node according to the resource use condition of the current node;
identifying a critical path in the workflow according to the packet placement condition, and sequencing critical edges in the critical path according to the data transmission time to obtain a critical edge sequencing list;
and merging the grouping of the functions according to the key edge ordering list.
Further, performing fine-grained data dependency analysis on the FaaS workflow, wherein the fine-grained data dependency analysis comprises analyzing meta-information and dependency information of the workflow; the meta information is basic information, and comprises the name and the type of the function; the dependency information comprises an execution dependency relationship and a parameter fine granularity dependency relationship; the parameter fine-grained dependency is for each specific output parameter in the function, indicating which functions are to be used as input parameters by a subsequent one.
Further, the initially grouping the functions in the workflow includes: each function is divided into individual packets and the packets are placed randomly on top of the nodes on a decentralized average.
Further, the sorting of the critical edges in the critical path according to the data transmission time is to arrange the critical edges in the critical path in a reverse order according to the data transmission time, and the longer the data transmission time is, the more front the sorting is.
Further, the merging the grouping of the functions according to the critical edge ordering list includes:
carrying out merging constraint judgment on two groups of two functions on the key edge, if the merging constraint is not met, continuing to traverse the next key edge, and if the merging constraint is met, entering a new group scheduling stage;
in the new grouping scheduling stage, selecting and optimizing optional nodes, if a preselected node set has no available nodes, continuing to traverse the next key edge, if the number of the screened available nodes is greater than 1, entering the optimizing stage, and finally determining deployment nodes of the new grouping;
after the merging is completed, the current workflow information is updated, new critical path identification is performed, and the iterative process of merging packets is completed.
Further, the merging constraint judgment includes:
judging the grouping constraint, judging whether the two functions belong to the same grouping, and if so, not meeting the grouping constraint;
and (3) carrying out resource constraint judgment, judging whether the functions in the two groups have conflict in resource use, and judging whether the size of data transmitted by the two functions exceeds the total memory use limit of the workflow.
Further, the preselection is to filter nodes according to node resources; the node with the highest score is selected from the preselected nodes as the node for new packet deployment.
A fine-grained data-dependent FaaS workflow partitioning and scheduling system, comprising:
the fine granularity analysis module is used for carrying out fine granularity data dependency analysis on the FaaS workflow;
the model construction module is used for establishing a calculation node model and a workflow model according to the result of the fine-granularity data dependency analysis;
the initial grouping placement module is used for initially grouping the functions in the workflow and placing the grouping on each node according to the resource use condition of the current node;
the critical path identification module is used for identifying critical paths in the workflow according to the packet placement condition;
the key path sorting module is used for sorting key edges in the key paths according to the data transmission time to obtain a key edge sorting list;
and the grouping and combining module is used for combining the groupings of the functions according to the key edge ordering list.
The beneficial effects of the invention are as follows:
by adopting the method of the invention, the data transmission cost in the FaaS workflow execution process can be effectively optimized, the end-to-end time delay of the workflow execution can be further effectively optimized, and the application service quality can be improved.
Drawings
Fig. 1 is a picture recognition application FaaS workflow diagram.
Fig. 2 is a flow chart of a FaaS workflow partition scheduling method.
Fig. 3 is a diagram of a result of workflow partitioning and scheduling for picture recognition application.
Detailed Description
The present invention will be further described in detail with reference to the following examples and drawings, so that the above objects, features and advantages of the present invention can be more clearly understood.
In the invention, fine-grained function dependency analysis means that two function tasks of F1 and F2 exist, and if the output of F1 is taken as the input of F2, then F1 and F2 are called as two functions with dependency relationship, and F2 depends on F1. When F2 only needs to be part of the data output by F1, then F1, F2 has fine-grained data dependence.
The invention discloses a fine-grained data-dependent FaaS workflow partitioning and scheduling method, which comprises the following steps of:
1) Workflow fine-grained data dependent analysis. The step analyzes meta information and dependency information of the workflow according to the constructed workflow model. Meta information is basic information including the name, type, etc. of a function. The dependency information includes execution dependencies of predecessor, successor, etc., and data dependencies of fine granularity of parameters. Wherein the parameter fine-grained dependency is for each specific output parameter in the function, indicating which functions to use as input parameters each specific parameter will be followed.
2) And carrying out calculation node modeling and workflow modeling according to the fine-granularity dependency analysis result, namely establishing a calculation node model and a workflow model. Computing node modeling is primarily modeled with respect to its resource supply capabilities. Workflow modeling mainly models fine-grained data dependencies among functions.
3) Workflow initial packet placement. In the initial condition, each function is divided into a separate group. And placing the packet according to the resource use condition of the current node, namely placing the packet on each node.
4) Workflow critical path identification and sequencing. And identifying the critical paths in the workflow according to the current workflow division condition, sequencing the critical paths, and arranging the critical edges in the critical paths in reverse order according to the data transmission time, wherein the longer the data transmission time is, the more the sequencing is. And after the sequencing, a key edge sequencing list is obtained, and the influence of the key edge which is more forward on the end-to-end time delay of the workflow execution is more. Wherein the critical path refers to the longest path used in all paths from the start node to the end node.
5) And combining the function packets. Traversing the key edge sequencing list in sequence, and merging the groups where the two functions are located on the key edge. The merging process comprises the following steps:
a) And carrying out merging constraint judgment on two groups of two functions on the key edge. If the packet cannot be merged (i.e., does not meet the merge constraint), then the next critical edge is traversed. If the packet meets the merging constraint, a new packet scheduling stage is entered.
b) During the new packet scheduling phase, the algorithm will pre-select and prefer the optional nodes. If the preselected set of nodes has no nodes available, then the next critical edge is traversed. If the number of the selected available nodes is greater than 1, entering a preferred stage, and finally determining the deployment node of the new packet.
c) After the merging is completed, the current workflow division information is updated, and the step 4) is returned to perform new critical path recognition. If the traversed key edge ordered list has no function grouping which can be successfully combined, the partition scheduling is finished. After the completion, according to the scheduling result, the storage component selects a reasonable access mode for each parameter data during execution.
The whole process of the method is a process of iteratively combining packets.
In one embodiment of the invention, in the compute node model, the cluster is represented as n= { N 1 ,N 2 ,...N m Computing node i is denoted N i ＝{c i ,rc i ,m i ,rm i ,p i ,rp i }，1≤i≤m。c i Representing the total core number of CPU of node, rc i The unit vCPU represents the core, representing the number of cores that the CPU of the node has been occupied. m is m i Representing the total memory capacity of a node, rm i Representing the occupied memory capacity of the node in GB. P is p i Representing the maximum number of containers a node can deploy, rp i Representing the number of containers that a node has deployed, in units of one. The memory and CPU capacity available to the node for allocation and the number of remaining deployable containers can be calculated from the above-mentioned metrics. The network bandwidth from the computing node to the storage node is W, representing the speed of data transmission in the network in Mb/s.
In one embodiment of the invention, in the Workflow model, the Workflow is represented as workflow= { func 1 ，func 2 ，...func n Assume a function func j Has O as output parameter j The input parameters are I j And its output is expressed asThe input is denoted +.>Function func j And func k The fine-grained data dependency between is expressed as func j All dependent func in (a) k The parameter alpha constitutes the set A jk . Wherein inputa represents func j Is an input parameter of (output) β Representation func k Is provided for the output parameter.
Taking the picture identification FaaS workflow application shown in fig. 1 as an example, the technical scheme of the invention is shown in fig. 2, and specifically comprises the following steps:
the image recognition workflow contains five functions, the upload function is responsible for uploading the picture, and the execution time is about 4s. The extraimagemetadata function is responsible for analyzing the picture and extracting the picture meta-information, including geographic location, size, format, and time, with an execution time of about 2s. The reconning function is responsible for recognizing the picture, with an execution time of about 10s. The dataclassing function is responsible for data cleansing of metadata, with an execution time of about 3s. The storeImageInfo function is responsible for storing the image processed data for about 3 seconds. The memory resource requirement for each function is 0.25GB and the CPU resource requirement is 0.1vCPU.
The data dependency relationship of the parameter fine granularity among the functions is analyzed as follows: the extraimagemetadata function takes the output parameter image of the upload function as an input parameter, and the parameter image data size is 8MB. The recongintion function takes an output parameter, image, of the extraimagemetadata function as an input parameter, and the parameter, image, has a data size of 10MB. The dataconversion function takes the output parameter metadata of the extraImageMetadata function as an input parameter, and the data size of the parameter metadata is 10MB. The storeImageInfo function takes the output parameters readata of the dataclearingfunction and the output parameters result of the reconnion function as input parameters, the size of the parameter data readata is 3MB, and the size of the parameter data result is 5MB.
And establishing a calculation node model and a workflow model according to the resource use condition of the current node and the data dependency relationship with fine granularity of parameters among functions analyzed in the previous step. Wherein the function func j 、func k Time of data transmission between jk The calculation formula is as follows:
initially, each function is divided into separate groupings: { upload }, { extractImageMetadata }, { recording }, { data cloning }, { stock ImageInfo }. Performing hash calculation on each packet according to the current resource use condition of each node so as to randomly and dispersedly place the packets on each node: { upload } →N i1 ，{extractImageMetadata}→N i2 ，{recongintion}→N i3 ，{datacleansing}→N i4 ，{storeImageInfo}→N i5 I1, i2, i3, i4, i5 and m are equal to or less than 1. Note i1. i5 and following ix may be equal, indicating that the packet is placed on the same node.
Based on the current initial division placement condition of the workflow, obtaining the key path of the image recognition workflow as uplink- & gt extractImagemetadata- & gt reconning- & gt storeImageInfo, and sequencing the key path from high to low according to the calculated data transmission time as follows: < extraimagemetadata, recongintion >, < upload, extraimagemetadata >, < recogination, storeiageinfo >.
Traversing the key edge ordered list in order, and preferentially merging the groups where the two functions on the key edge are located. The groupings { extraimagemetadata } and { reconnino } where the two functions are located in the key edge < extraimagemetadata > are extracted. The merging process comprises the following steps:
1) And (5) carrying out merging constraint judgment. Firstly, judging grouping constraint, judging whether two functions belong to the same grouping, and if so, not meeting the grouping constraint. And then, judging whether the functions in the two groups have conflict in resource use or not by carrying out resource constraint judgment, and judging whether the size of data transmitted by the two functions exceeds the total memory use limit of the workflow or not. If the above constraints are met, it means that two packets can be combined into one packet.
2) The appropriate deployment node is selected for the new packet, and the selection phase is divided into two phases, pre-selection and preference.
2.1 A pre-selection stage. The filtering is mainly carried out according to the node resources, three resource indexes of the memory, the CPU and the maximum number of the deployable containers are considered, and when any resource of the node does not meet the requirement of a new packet, the node is filtered.
2.2 A) a preferred stage. Node resources passing through the pre-selected stage can support the deployment of new packets. Scoring is preferably used, and the node with the highest score is selected from the preselected nodes as the node for the new packet deployment. Using biThe n-pack scoring strategy MostRequestedPriority is that the node with the highest resource utilization rate at present is selected. Score i The calculation formula (full 10 minutes) is as follows:
wherein, the liquid crystal display device comprises a liquid crystal display device,representing cpu resource usage score, +.>Representing the memory resource usage score.
After the combination of the { extraimagemetadata } and { reconnion } packets is successful, the new packet will be deployed to node N with the highest score i6 And (3) the following steps: { extractImageMetadata, reconnion } →N i6 I6 is more than or equal to 1 and less than or equal to m. The data transmission of the two functions adopts a local memory, and the data transmission time between the two functions in the workflow is updated according to the multiple alpha of the memory access speed and the remote access speed.
3) Critical edge transmission time in merged workflow jk Updated toAfter the updating is completed, a new critical path of the workflow is identified, and the next round of iterative merging is carried out. And (5) until no combinable critical path function packet exists, and the partition scheduling is finished. The final division result is shown in fig. 3.
The embodiment realizes the division scheduling of the FaaS workflow through a series of measures, optimizes the data transmission delay 76.79 percent and optimizes the end-to-end delay 42.03 percent. Compared with the existing function granularity scheduling method, the data transmission delay optimizing effect of the embodiment is 36.72% higher, and the end-to-end delay optimizing effect is 15.94% higher. The result shows that the parameter fine granularity data dependence analysis mode has better optimization effect.
Another embodiment of the present invention provides a fine-grained data-dependent FaaS workflow partitioning and scheduling system, comprising:
the fine granularity analysis module is used for carrying out fine granularity data dependency analysis on the FaaS workflow;
the model construction module is used for establishing a calculation node model and a workflow model according to the result of the fine-granularity data dependency analysis;
the initial grouping placement module is used for initially grouping the functions in the workflow and placing the grouping on each node according to the resource use condition of the current node;
the critical path identification module is used for identifying critical paths in the workflow according to the packet placement condition;
the key path sorting module is used for sorting key edges in the key paths according to the data transmission time to obtain a key edge sorting list;
and the grouping and combining module is used for combining the groupings of the functions according to the key edge ordering list.
Wherein the specific implementation of each module is referred to the previous description of the method of the present invention.
Based on the same inventive concept, another embodiment of the present invention provides a computer device (computer, server, smart phone, etc.) comprising a memory storing a computer program configured to be executed by the processor, and a processor, the computer program comprising instructions for performing the steps in the inventive method.
Based on the same inventive concept, another embodiment of the present invention provides a computer readable storage medium (e.g., ROM/RAM, magnetic disk, optical disk) storing a computer program which, when executed by a computer, implements the steps of the inventive method.
What is not described in detail in the technical solutions of the invention can be realized by means of prior art.
The above embodiments are only for illustrating the technical solution of the present invention and not for limiting the same, and those skilled in the art may modify or substitute the technical solution of the present invention without departing from the principle and scope of the present invention, and the protection scope of the present invention shall be defined by the claims.

Claims (10)

1. The FaaS workflow partitioning and scheduling method based on fine-grained data dependence is characterized by comprising the following steps of:
performing fine-grained data dependency analysis on the FaaS workflow;
according to the result of the fine-grained data dependency analysis, a computing node model and a workflow model are established;
initially grouping functions in the workflow, and placing the groups on each node according to the resource use condition of the current node;
identifying a critical path in the workflow according to the packet placement condition, and sequencing critical edges in the critical path according to the data transmission time to obtain a critical edge sequencing list;
and merging the grouping of the functions according to the key edge ordering list.
2. The method of claim 1, wherein the fine-grained data-dependent analysis of the FaaS workflow includes parsing meta-information and dependency information of the workflow; the meta information is basic information, and comprises the name and the type of the function; the dependency information comprises an execution dependency relationship and a parameter fine granularity dependency relationship; the parameter fine-grained dependency is for each specific output parameter in the function, indicating which functions are to be used as input parameters by a subsequent one.
3. The method of claim 1, wherein the initially grouping functions in the workflow comprises:
each function is divided into individual packets and the packets are placed randomly on top of the nodes on a decentralized average.
4. The method of claim 1, wherein the ranking the critical edges in the critical path by data transfer time size is ranking the critical edges in the critical path in reverse order by data transfer time size, the longer the data transfer time the earlier.
5. The method of claim 1, wherein the merging the groupings of functions according to the critical edge ordering list comprises:
carrying out merging constraint judgment on two groups of two functions on the key edge, if the merging constraint is not met, continuing to traverse the next key edge, and if the merging constraint is met, entering a new group scheduling stage;
in the new grouping scheduling stage, selecting and optimizing optional nodes, if a preselected node set has no available nodes, continuing to traverse the next key edge, if the number of the screened available nodes is greater than 1, entering the optimizing stage, and finally determining deployment nodes of the new grouping;
after the merging is completed, the current workflow information is updated, new critical path identification is performed, and the iterative process of merging packets is completed.
6. The method of claim 5, wherein the merge constraint determination comprises:
judging the grouping constraint, judging whether the two functions belong to the same grouping, and if so, not meeting the grouping constraint;
and (3) carrying out resource constraint judgment, judging whether the functions in the two groups have conflict in resource use, and judging whether the size of data transmitted by the two functions exceeds the total memory use limit of the workflow.
7. The method of claim 5, wherein the preselection is filtering nodes based on node resources; the node with the highest score is selected from the preselected nodes as the node for new packet deployment.
8. A fine-grained data-dependent FaaS workflow partitioning and scheduling system, comprising:
the fine granularity analysis module is used for carrying out fine granularity data dependency analysis on the FaaS workflow;
the model construction module is used for establishing a calculation node model and a workflow model according to the result of the fine-granularity data dependency analysis;
the initial grouping placement module is used for initially grouping the functions in the workflow and placing the grouping on each node according to the resource use condition of the current node;
the critical path identification module is used for identifying critical paths in the workflow according to the packet placement condition;
the key path sorting module is used for sorting key edges in the key paths according to the data transmission time to obtain a key edge sorting list;
and the grouping and combining module is used for combining the groupings of the functions according to the key edge ordering list.
9. A computer device comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program comprising instructions for performing the method of any of claims 1-7.
10. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when executed by a computer, implements the method of any one of claims 1-7.
CN202310623714.7A 2023-05-30 2023-05-30 FaaS workflow partitioning and scheduling method and system based on fine-grained data dependence Pending CN116644922A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310623714.7A CN116644922A (en) 2023-05-30 2023-05-30 FaaS workflow partitioning and scheduling method and system based on fine-grained data dependence

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310623714.7A CN116644922A (en) 2023-05-30 2023-05-30 FaaS workflow partitioning and scheduling method and system based on fine-grained data dependence

Publications (1)

Publication Number Publication Date
CN116644922A true CN116644922A (en) 2023-08-25

Family

ID=87622589

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310623714.7A Pending CN116644922A (en) 2023-05-30 2023-05-30 FaaS workflow partitioning and scheduling method and system based on fine-grained data dependence

Country Status (1)

Country Link
CN (1) CN116644922A (en)

Similar Documents

Publication Publication Date Title
US20190122136A1 (en) Feature processing tradeoff management
US10764370B2 (en) Hybrid cloud migration delay risk prediction engine
US10776170B2 (en) Software service execution apparatus, system, and method
US20150379426A1 (en) Optimized decision tree based models
CN113015970A (en) Partitioning knowledge graph
CN112416585A (en) GPU resource management and intelligent scheduling method for deep learning
CN109062700A (en) A kind of method for managing resource and server based on distributed system
CN110740079B (en) Full link benchmark test system for distributed scheduling system
CN106815254A (en) A kind of data processing method and device
CN113391913A (en) Distributed scheduling method and device based on prediction
CN108829505A (en) A kind of distributed scheduling system and method
US20230176905A1 (en) Automatic driving simulation task scheduling method and apparatus, device, and readable medium
CN105740249B (en) Processing method and system in parallel scheduling process of big data job
CN114187300A (en) Image processing method, system and storage medium
CN107168795B (en) Codon deviation factor model method based on CPU-GPU isomery combined type parallel computation frame
CN113407343A (en) Service processing method, device and equipment based on resource allocation
CN112559525A (en) Data checking system, method, device and server
CN114756629B (en) Multi-source heterogeneous data interaction analysis engine and method based on SQL
Gu et al. Characterizing job-task dependency in cloud workloads using graph learning
EP4109366A1 (en) Method and device for managing project by using data merging
CN116644922A (en) FaaS workflow partitioning and scheduling method and system based on fine-grained data dependence
CN113625967B (en) Data storage method, data query method and server
CN113010296B (en) Formalized model based task analysis and resource allocation method and system
US20080147221A1 (en) Grid modeling tool
CN114153427A (en) Optimization method and system of continuous integration assembly line

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination