CN114416315A - Dynamic task scheduling method and system in continuous integrated system - Google Patents
Dynamic task scheduling method and system in continuous integrated system Download PDFInfo
- Publication number
- CN114416315A CN114416315A CN202210054155.8A CN202210054155A CN114416315A CN 114416315 A CN114416315 A CN 114416315A CN 202210054155 A CN202210054155 A CN 202210054155A CN 114416315 A CN114416315 A CN 114416315A
- Authority
- CN
- China
- Prior art keywords
- task
- construction
- tasks
- execution
- dynamic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a method and a system for dynamically scheduling tasks in a continuous integrated system, which solve the problem of efficiency caused by unreasonable task sequence construction and simultaneously improve the overall performance of the continuous integrated system. The technical scheme is as follows: the system comprises: the construction node distribution module is used for distributing the construction tasks to corresponding construction nodes according to the required resource labels of the construction tasks; the task basic priority distribution module is used for quantifying the basic value of the construction task according to the parameters of the construction task and distributing different weight parameters to each construction task so as to adjust the influence of each task attribute on the total basic value; the task dynamic priority scheduling module is used for dynamically scheduling the constructed tasks according to the residual value density and execution urgency of the tasks; and the construction resource load analysis module is used for automatically realizing the adjustment scheme including capacity expansion and expansion of the construction resources according to the task execution load condition of each construction node.
Description
Technical Field
The invention relates to a task scheduling technology, in particular to a task dynamic scheduling technology applied to a continuous integrated system.
Background
In enterprise-level DevOps practice, efficiency of continuous integration is very important, and most of the continuous integration systems at present decide the sequence according to the starting sequence of the construction tasks, and this mode is not flexible enough in dealing with various construction scenes, which may cause many problems, for example: the construction tasks waiting in the queue may have important and urgent construction tasks which cannot be executed preferentially, the construction tasks being executed may have tasks with lower importance occupying system resources, and the like, and an unreasonable construction task sequence is allocated, so that the construction resources are easily preempted and wasted, and the performance and the efficiency of a continuous integrated system are influenced.
Meanwhile, in a continuous integrated system, the problem of insufficient construction resources is usually encountered, operation and maintenance personnel are required to manually add machine nodes to reduce load pressure, the requirement of manual intervention is time-consuming and labor-consuming, the overall construction efficiency is also influenced, and how to automatically and reasonably adjust construction resources is also a problem to be solved.
Disclosure of Invention
The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.
The invention aims to solve the problems and provides a method and a system for dynamically scheduling tasks in a continuous integrated system, so that the problem of efficiency caused by unreasonable task sequence construction is solved, and the overall performance of the continuous integrated system is improved.
The technical scheme of the invention is as follows: the invention discloses a task dynamic scheduling system of a continuous integrated system, which comprises a construction node distribution module, a task basic priority distribution module, a task dynamic priority scheduling module and a construction resource load analysis module, wherein:
the construction node distribution module is used for distributing the construction tasks to corresponding construction nodes according to the required resource labels of the construction tasks;
the task basic priority distribution module is used for quantifying the basic value of the construction task according to the parameters of the construction task and distributing different weight parameters to each construction task so as to adjust the influence of each task attribute on the total basic value;
the task dynamic priority scheduling module is used for dynamically scheduling the constructed tasks according to the residual value density and execution urgency of the tasks;
and the construction resource load analysis module is used for automatically realizing the adjustment scheme including capacity expansion and expansion of the construction resources according to the task execution load condition of each construction node.
According to an embodiment of the task dynamic scheduling system of the persistent integration system of the present invention, the construction node allocating module is further configured to: the method comprises the steps of creating a construction task through template steps, setting a required environment resource label for each template step to identify a corresponding environment on which each step depends, distributing the construction task to the corresponding construction node to execute when the construction node has all resource label sets of the construction task, and distributing the construction node with the lowest load to execute the construction task when a plurality of construction nodes meet the construction task.
According to an embodiment of the task dynamic scheduling system of the persistent integration system of the present invention, the parameters of the build task in the task basic priority assignment module include: task use, importance degree, task type, task construction state and the number of depended tasks.
According to an embodiment of the task dynamic scheduling system of the persistent integration system of the present invention, the remaining value density of the tasks in the task dynamic priority scheduling module is calculated by the following formula:
wherein, T represents the predicted execution time of the task, T represents the executed time, V represents the basic value of the task, and m represents the variation of the instant value generation speed of the task.
According to an embodiment of the task dynamic scheduling system of the persistent integration system of the present invention, the urgency of execution of the task in the task dynamic priority scheduling module is calculated by the following formula:
wherein T represents the predicted execution time of the task, T represents the executed time, n is used for adjusting the influence of the task urgency on the dynamic priority of the task, d represents the task ending time, and tau represents the current time.
According to an embodiment of the task dynamic scheduling system of the persistent integration system, the influence weight of the residual value density and the execution urgency on the dynamic priority of the task is adjusted by adjusting the sizes of the m parameter and the n parameter.
According to an embodiment of the task dynamic scheduling system of the persistent integration system, a task dynamic priority scheduling module sets a preemption threshold coefficient k in the process of dynamically scheduling the constructed tasks, and only when the dynamic priority of the waiting tasks is larger than the product value of the currently executed task dynamic priority multiplied by the preemption threshold coefficient k, the resources can be preempted for execution, otherwise, the currently executed tasks are continuously executed.
According to an embodiment of the task dynamic scheduling system of the persistent integration system of the present invention, the build resource load analysis module is further configured to: and setting a background data acquisition program at each construction node, recording the load condition of the construction node, pushing the load condition of the construction node to an administrator interface, simultaneously recording the task execution success rate and the execution efficiency of each construction node by the background data acquisition program, and automatically expanding the construction nodes with the task queuing time length higher than a set value.
The invention also discloses a task dynamic scheduling method of the continuous integration system, which comprises the following steps:
step 1: the construction task is triggered and started, and a task queue is constructed;
step 2: distributing the construction tasks to corresponding construction nodes meeting the conditions according to the environment resource labels of the construction tasks;
and step 3: grouping according to the construction nodes, and calculating the basic value and the expected execution time of each construction task according to the task attributes;
and 4, step 4: calculating the dynamic priority of the tasks, and dynamically scheduling the construction tasks according to the priority, wherein the calculation of the dynamic priority of the tasks is obtained by the residual value density and execution urgency of the construction tasks;
and 5: entering a waiting queue, judging whether the number of the tasks which are not executed is larger than a set capacity expansion threshold, if so, automatically expanding the capacity of the resources and adding nodes, reallocating the tasks which are not executed to the nodes and returning to the step 3, and if not, returning to the step 4.
According to an embodiment of the method for dynamically scheduling tasks of the persistent integration system of the present invention, step 3 further includes:
the basic value of the construction task is calculated by summing up the products of each parameter of the construction task and the corresponding influence weight;
calculating the predicted execution time of the construction task by adopting the following steps: and if the construction is carried out before, the average value of the historical construction time of the construction task is taken, and if the construction is not carried out before, the average value of the predicted execution time of the task of the same construction node is taken.
According to an embodiment of the method for dynamically scheduling tasks of the persistent integration system of the present invention, step 4 further includes:
step 3, calculating the dynamic priority of all executed tasks and waiting tasks, wherein the structure with the highest basic value in step 3The task building method comprises the steps of firstly obtaining execution right, and then calculating the dynamic priority of each built task in a real-time computing system, wherein the dynamic priority of the task is
The method comprises the following steps that T represents predicted execution time of a task, T represents executed time, V represents basic value of the task, m represents change size of instant value generation speed of the task, n is used for adjusting influence of task urgency on dynamic priority of the task, d represents task ending time, and tau represents current time;
and secondly, sequencing the dynamic priorities of the tasks calculated in the previous step, comparing the current execution task priority multiplied by a preemption threshold coefficient k with the task of the current highest dynamic priority, entering step 5 if the product is greater than the highest dynamic priority, starting execution by the task preemption system resource if the product is less than the highest dynamic priority, and calculating the dynamic priority simultaneously with the waiting task after the task preemption system resource starts execution to perform dynamic scheduling.
Compared with the prior art, the invention has the following beneficial effects: the invention carries out dynamic priority scheduling on the construction tasks of the continuous integrated system, solves the efficiency problem caused by unreasonable construction task sequence, simultaneously realizes automatic adjustment of construction resources by analyzing the construction node load, and improves the overall performance of the continuous integrated system.
Drawings
The above features and advantages of the present disclosure will be better understood upon reading the detailed description of embodiments of the disclosure in conjunction with the following drawings. In the drawings, components are not necessarily drawn to scale, and components having similar relative characteristics or features may have the same or similar reference numerals.
Fig. 1A and 1B are schematic architectural diagrams illustrating an embodiment of a task dynamic scheduling system of a persistent integration system of the present invention.
FIG. 2 is a flowchart illustrating an embodiment of a method for dynamically scheduling tasks of a persistent integration system according to the present invention.
Detailed Description
The invention is described in detail below with reference to the figures and specific embodiments. It is noted that the aspects described below in connection with the figures and the specific embodiments are only exemplary and should not be construed as imposing any limitation on the scope of the present invention.
Fig. 1A and 1B show an architecture of an embodiment of a task dynamic scheduling system of a persistent integration system of the present invention. Referring to fig. 1A and 1B, the task dynamic scheduling system of the present embodiment includes the following modules: the system comprises a construction node distribution module, a task basic priority distribution module, a task dynamic priority scheduling module and a construction resource load analysis module.
The construction node distribution module is used for distributing the construction tasks to corresponding construction nodes according to the required resource labels of the construction tasks.
The specific implementation of the construction node distribution module is as follows: the method comprises the steps of creating a construction task through templating steps, wherein each step can be executed only depending on different environment dependencies, for example, maven construction requires a maven environment, docker mirror construction requires a docker environment and the like, therefore, a system sets required environment resource labels (namely the maven environment, the docker environment and the like in the example) for each templating step, when a construction node has all resource label collections of the construction task, the construction task can be distributed to the construction node for execution, and when a plurality of construction nodes can meet the construction task, the construction node with the lowest load at present is distributed for executing the construction task.
The task basic priority distribution module is used for quantifying the basic value of the construction task according to the parameters of the construction task by combining the characteristics of the construction task in the continuous integrated system, distributing different weight parameters to each construction task, and adjusting the influence of each attribute on the total basic value.
The parameters of the construction task include:
1. task purpose, importance (emergency release construction > batch test > general construction);
2. task type (value: timing trigger < MR trigger, PUSH trigger < manual trigger), where MR refers to the Merge Request in git.
3. A task construction state (the last construction condition: construction after failure > first construction > construction after success);
4. the number of tasks that are relied upon (the more tasks that are relied upon, the higher the importance of the task).
And the task dynamic priority scheduling module is used for dynamically scheduling the constructed tasks according to the residual value density and execution urgency of the tasks.
The task dynamic priority scheduling module is specifically realized as follows: in a real-time application system, the execution time of a task with high value is not necessarily urgent, but the value of a task with urgent time is not necessarily high, so in order to take fairness and efficiency into consideration, the value density and the execution urgency of a task queue are synchronously calculated in the execution process of the task by comprehensively considering factors such as the task value, deadline, free time and the like, and dynamic scheduling is realized.
The task value density mentioned above means: the value attribute of the task is an important judgment factor of the task execution priority, the constructed basic value is already calculated when the basic priority is distributed by the task basic priority distribution module, namely the constructed task is executed and finished to bring benefits to the system, however, the value of the task is not generated at the moment of task execution but is gradually accumulated along with the task execution, so that the system comprehensively considers the value and the execution time of the task, and preferably uses the residual value density to quantify the importance degree of the constructed task, namely the value of the task generated in unit time in the residual execution time.
The above-mentioned task
Wherein, T represents the predicted execution time of the task, T represents the executed time, V represents the basic value of the task, and m represents the variation of the instant value generation speed of the task.
The performance urgency of the above task refers to: in a continuous integrated system, some urgent construction requirements require that the system can meet the time constraint of tasks as far as possible to ensure that the construction tasks can be successfully executed before the deadline, and a traditional scheduling method based on task time attributes usually judges the urgency of task execution according to the deadline of the tasks, but has certain limitation only according to the deadline, so that the system analyzes the relationship between the deadline of the tasks, the residual execution time and the idle time, judges the urgency of the construction tasks by using the execution strength, namely the ratio of the execution time required by the completion of the tasks to the idle time of the tasks, and the higher the ratio indicates that the execution urgency of the tasks is higher.
The above-mentioned task
Wherein T represents the predicted execution time of the task, T represents the executed time, n is used for adjusting the influence of the task urgency on the dynamic priority of the task, d represents the task ending time, and tau represents the current time.
The system sets appropriate adjusting parameters m and n for the two evaluation indexes (the residual value density of the task and the execution urgency of the task), adjusts the influence weight of the residual value density and the execution urgency of the task on the dynamic priority of the task by adjusting the sizes of the m and the n, and improves the adaptability to different continuous integration systems.
And meanwhile, a reasonable preemption threshold coefficient k is set, the resource execution can be preempted only when the dynamic priority of the waiting task is larger than the product value of the dynamic priority of the currently executed task multiplied by the preemption threshold coefficient k, otherwise, the currently executed task is continuously executed, and the situation that the tasks are repeatedly preempted due to the alternate rise of the priorities of two or more tasks is avoided.
The construction resource load analysis module is used for automatically realizing the adjustment scheme of construction resources including capacity expansion and expansion according to the task execution load condition of each construction node.
Specifically, a background data acquisition program is arranged on each construction node, the load condition of the construction node is recorded and pushed to an administrator interface, meanwhile, the task execution success rate and the execution efficiency of each construction node are recorded, automatic expansion is carried out on the construction nodes with longer task queuing time, labor cost is saved, and the efficiency of a continuous integration system is improved.
Fig. 2 shows a flow of an embodiment of the method for dynamically scheduling tasks of the persistent integration system of the present invention. Referring to fig. 2, the steps of the task dynamic scheduling method of the embodiment are detailed as follows.
Step 1: and triggering and starting the construction task to construct a task queue.
The cases where the build task is triggered include: the development tester is triggered when the continuous integrated system is manually constructed, and the developers PUSH and MR arrive at a git warehouse, and the timed task is triggered.
Step 2: and distributing the construction tasks to corresponding construction nodes meeting the conditions according to the environment resource labels of the construction tasks.
This step is accomplished by the build node assignment module.
And step 3: grouping according to the construction nodes, and calculating the basic value and the expected execution time of each construction task according to the task attributes.
This step is performed by the task basic priority assignment module, and the specific calculation process of the basic value and the expected execution time of the task is as follows.
1. The process of calculating the base value of the build task:
the following A-D are parameters of the construction task:
A. task purpose, importance (emergency release construction > batch test > general construction);
B. task type (value: timing trigger < MR trigger, PUSH trigger < manual trigger);
C. a task construction state (the last construction condition: construction after failure > first construction > construction after success);
D. the number of the depended tasks (the more the depended tasks are, the higher the importance of the task is);
the base value of the task is a × a + B × B + C × C + D × D, where a, B, C, D respectively represent the value scores of the task on the above four parameters, and a, B, C, D respectively represent the influence weights of the above four parameters on the total base value.
In the process, the construction tasks in the continuous integration system are considered from four different angles, and different weight parameters are set, so that the technical effect of constructing the basic value of the task in a multi-azimuth quantitative manner can be realized, and the problem that most of the conventional continuous integration systems are not comprehensive in task value calculation at present is solved
2. Calculating the predicted execution time T of each construction task: and if the construction is carried out before, the average value of the historical construction time of the construction task is taken, and if the construction is not carried out, the average value of the predicted execution time of the task of the same construction node is taken.
And 4, step 4: and calculating the dynamic priority of the tasks, and dynamically scheduling the constructed tasks according to the priority.
The step is completed by a task dynamic priority scheduling module, and the specific process of calculating the task dynamic priority is as follows:
step 3, the construction task with the highest basic value first obtains the execution right, and then the priority of each construction task in the system changes dynamically along with the time of the system. The execution urgency of the executing task is unchanged, but the residual value density is increased along with the increase of the executed time, and the execution urgency of the waiting task is unchanged, but the residual value density is increased along with the increase of the waiting time.
And secondly, sequencing according to the dynamic priorities of the tasks calculated in the previous step, comparing the product of the current execution task priority multiplied by a preemption threshold coefficient k with the task of the current highest dynamic priority, entering step 5 if the product is greater than the highest dynamic priority, starting execution by the task preemption system resource if the product is less than the highest dynamic priority, and stopping execution possibly by preemption after the task preemption system resource starts execution, so that the dynamic priority needs to be calculated simultaneously with the waiting task in the executing state of the task to perform dynamic scheduling.
In the step, the tasks are dynamically scheduled by using two attributes of constructing the residual value density and the execution urgency, and the influence weight is adjusted by using two adjusting parameters, so that the balance of the dynamic scheduling of the high-value tasks and the high-urgency tasks is ensured, and meanwhile, the problem of repeated preemption among the tasks caused by the alternate rise of the priority of two or more tasks is solved by introducing a preemption threshold.
And 5: entering a waiting queue, judging whether the number of the unexecuted tasks is larger than a set capacity expansion threshold (the capacity expansion threshold represents the number of the tasks of the waiting queue of the construction nodes), if so, automatically expanding the capacity of the construction resources, adding the nodes, reallocating the construction nodes to the unexecuted construction tasks, returning to the step 3, and if not, returning to the step 4.
In the step, the load condition of the construction node is analyzed, the capacity is automatically expanded to reduce the load pressure, the waiting construction task is enabled to be executed as soon as possible, the technical effect of flexibly stretching and retracting construction resources is achieved, and the problem of efficiency caused by the fact that a continuous integrated system needs to manually manage the resources is solved.
While, for purposes of simplicity of explanation, the methodologies are shown and described as a series of acts, it is to be understood and appreciated that the methodologies are not limited by the order of acts, as some acts may, in accordance with one or more embodiments, occur in different orders and/or concurrently with other acts from that shown and described herein or not shown and described herein, as would be understood by one skilled in the art.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.
In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a web site, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk (disk) and disc (disc), as used herein, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk and blu-ray disc where disks (disks) usually reproduce data magnetically, while discs (discs) reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (11)
1. A task dynamic scheduling system of a continuous integrated system is characterized by comprising a construction node distribution module, a task basic priority distribution module, a task dynamic priority scheduling module and a construction resource load analysis module, wherein:
the construction node distribution module is used for distributing the construction tasks to corresponding construction nodes according to the required resource labels of the construction tasks;
the task basic priority distribution module is used for quantifying the basic value of the construction task according to the parameters of the construction task and distributing different weight parameters to each construction task so as to adjust the influence of each task attribute on the total basic value;
the task dynamic priority scheduling module is used for dynamically scheduling the constructed tasks according to the residual value density and execution urgency of the tasks;
and the construction resource load analysis module is used for automatically realizing the adjustment scheme including capacity expansion and expansion of the construction resources according to the task execution load condition of each construction node.
2. The system for dynamically scheduling task for the persistent integration system as recited in claim 1, wherein the build node assignment module is further configured to: the method comprises the steps of creating a construction task through template steps, setting a required environment resource label for each template step to identify a corresponding environment on which each step depends, distributing the construction task to the corresponding construction node to execute when the construction node has all resource label sets of the construction task, and distributing the construction node with the lowest load to execute the construction task when a plurality of construction nodes meet the construction task.
3. The task dynamic scheduling system of the persistent integration system according to claim 1, wherein the parameters of the build task in the task basic priority assignment module include: task use, importance degree, task type, task construction state and the number of depended tasks.
4. The system according to claim 1, wherein the residual value density of the tasks in the task dynamic priority scheduling module is calculated by the following formula:
wherein, T represents the predicted execution time of the task, T represents the executed time, V represents the basic value of the task, and m represents the variation of the instant value generation speed of the task.
5. The task dynamic scheduling system of the persistent integration system as claimed in claim 4, wherein the urgency of execution of the task in the task dynamic priority scheduling module is calculated by the following formula:
wherein T represents the predicted execution time of the task, T represents the executed time, n is used for adjusting the influence of the task urgency on the dynamic priority of the task, d represents the task ending time, and tau represents the current time.
6. The task dynamic scheduling system of the persistent integration system as claimed in claim 5, wherein the influence weight of the task residual value density and the execution urgency on the task dynamic priority is adjusted by adjusting the size of the m and n parameters.
7. The system of claim 6, wherein the task dynamic priority scheduling module sets a preemption threshold coefficient k during the dynamic scheduling of the constructed tasks, and only when the dynamic priority of the waiting task is greater than the product of the dynamic priority of the currently executing task multiplied by the preemption threshold coefficient k, the resource execution can be preempted, otherwise, the currently executing task is continuously executed.
8. The system according to claim 1, wherein the build resource load analysis module is further configured to: and setting a background data acquisition program at each construction node, recording the load condition of the construction node, pushing the load condition of the construction node to an administrator interface, simultaneously recording the task execution success rate and the execution efficiency of each construction node by the background data acquisition program, and automatically expanding the construction nodes with the task queuing time length higher than a set value.
9. A method for dynamically scheduling tasks of a persistent integration system is characterized by comprising the following steps:
step 1: the construction task is triggered and started, and a task queue is constructed;
step 2: distributing the construction tasks to corresponding construction nodes meeting the conditions according to the environment resource labels of the construction tasks;
and step 3: grouping according to the construction nodes, and calculating the basic value and the expected execution time of each construction task according to the task attributes;
and 4, step 4: calculating the dynamic priority of the tasks, and dynamically scheduling the construction tasks according to the priority, wherein the calculation of the dynamic priority of the tasks is obtained by the residual value density and execution urgency of the construction tasks;
and 5: entering a waiting queue, judging whether the number of the tasks which are not executed is larger than a set capacity expansion threshold, if so, automatically expanding the capacity of the resources and adding nodes, reallocating the tasks which are not executed to the nodes and returning to the step 3, and if not, returning to the step 4.
10. The method for dynamically scheduling task of persistent integration system according to claim 9, wherein step 3 further comprises:
the basic value of the construction task is calculated by summing up the products of each parameter of the construction task and the corresponding influence weight;
calculating the predicted execution time of the construction task by adopting the following steps: and if the construction is carried out before, the average value of the historical construction time of the construction task is taken, and if the construction is not carried out before, the average value of the predicted execution time of the task of the same construction node is taken.
11. The method for dynamically scheduling task of persistent integration system according to claim 9, wherein step 4 further comprises:
step 3, the construction task with the highest basic value first obtains the execution right, and then the dynamic priority of each construction task in the real-time calculation system is calculated, wherein the dynamic priority of the task is
The method comprises the following steps that T represents predicted execution time of a task, T represents executed time, V represents basic value of the task, m represents change size of instant value generation speed of the task, n is used for adjusting influence of task urgency on dynamic priority of the task, d represents task ending time, and tau represents current time;
and secondly, sequencing the dynamic priorities of the tasks calculated in the previous step, comparing the current execution task priority multiplied by a preemption threshold coefficient k with the task of the current highest dynamic priority, entering step 5 if the product is greater than the highest dynamic priority, starting execution by the task preemption system resource if the product is less than the highest dynamic priority, and calculating the dynamic priority simultaneously with the waiting task after the task preemption system resource starts execution to perform dynamic scheduling.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210054155.8A CN114416315A (en) | 2022-01-18 | 2022-01-18 | Dynamic task scheduling method and system in continuous integrated system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210054155.8A CN114416315A (en) | 2022-01-18 | 2022-01-18 | Dynamic task scheduling method and system in continuous integrated system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114416315A true CN114416315A (en) | 2022-04-29 |
Family
ID=81274071
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210054155.8A Pending CN114416315A (en) | 2022-01-18 | 2022-01-18 | Dynamic task scheduling method and system in continuous integrated system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114416315A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116225666A (en) * | 2023-05-06 | 2023-06-06 | 中国第一汽车股份有限公司 | Task scheduling method and device for operating system, electronic equipment and storage medium |
| CN116414464A (en) * | 2023-05-26 | 2023-07-11 | 摩尔线程智能科技(北京)有限责任公司 | Method and device for scheduling tasks, electronic equipment and computer readable medium |
| CN116991374A (en) * | 2023-09-27 | 2023-11-03 | 北京凌云雀科技有限公司 | Control method, device, electronic equipment and medium for constructing continuous integration task |
-
2022
- 2022-01-18 CN CN202210054155.8A patent/CN114416315A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116225666A (en) * | 2023-05-06 | 2023-06-06 | 中国第一汽车股份有限公司 | Task scheduling method and device for operating system, electronic equipment and storage medium |
| CN116225666B (en) * | 2023-05-06 | 2023-08-15 | 中国第一汽车股份有限公司 | Task scheduling method and device for operating system, electronic equipment and storage medium |
| CN116414464A (en) * | 2023-05-26 | 2023-07-11 | 摩尔线程智能科技(北京)有限责任公司 | Method and device for scheduling tasks, electronic equipment and computer readable medium |
| CN116414464B (en) * | 2023-05-26 | 2023-09-22 | 摩尔线程智能科技(北京)有限责任公司 | Method and device for scheduling tasks, electronic equipment and computer readable medium |
| CN116991374A (en) * | 2023-09-27 | 2023-11-03 | 北京凌云雀科技有限公司 | Control method, device, electronic equipment and medium for constructing continuous integration task |
| CN116991374B (en) * | 2023-09-27 | 2024-01-19 | 北京凌云雀科技有限公司 | Control method, device, electronic equipment and medium for constructing continuous integration task |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114416315A (en) | Dynamic task scheduling method and system in continuous integrated system | |
| US20190034223A1 (en) | Task resource scheduling method and apparatus | |
| US8473956B2 (en) | Priority based scheduling system for server | |
| WO2016078008A1 (en) | Method and apparatus for scheduling data flow task | |
| US9513835B2 (en) | Impact-based migration scheduling from a first tier at a source to a second tier at a destination | |
| JP2015133112A (en) | Job scheduling method, data analyzer, data analysis apparatus, computer system and computer readable medium | |
| CN109144716A (en) | Operating system dispatching method and device, equipment based on machine learning | |
| US20190171479A1 (en) | Method and system for scheduling transactions in a data system | |
| WO2020172852A1 (en) | Computing resource scheduling method, scheduler, internet of things system, and computer readable medium | |
| WO2020238989A1 (en) | Method and apparatus for scheduling task processing entity | |
| US11868808B2 (en) | Automatic driving simulation task scheduling method and apparatus, device, and readable medium | |
| CN108595251B (en) | Dynamic graph updating method, device, storage engine interface and program medium | |
| CN112068943B (en) | Micro-service scheduling method based on complex heterogeneous environment and implementation system thereof | |
| CN108055161B (en) | Networking optimization method and device | |
| WO2024032781A1 (en) | Algorithm testing method and apparatus, and storage medium | |
| CN117149388A (en) | Batch task scheduling method and system, electronic equipment and storage medium | |
| CN116483546A (en) | Distributed training task scheduling method, device, equipment and storage medium | |
| US20230105953A1 (en) | Systems and methods for object migration in storage devices | |
| CN115576924A (en) | Data migration method | |
| US20220229692A1 (en) | Method and device for data task scheduling, storage medium, and scheduling tool | |
| CN115061811A (en) | Resource scheduling method, device, equipment and storage medium | |
| CN114418282A (en) | Station scene management method, device, equipment and computer program product | |
| CN114461356A (en) | Control method for number of processes of scheduler and IaaS cloud platform scheduling system | |
| CN110780993A (en) | Kubernetes-based resource scheduling optimization method, equipment and medium | |
| CN111813527B (en) | Data-aware task scheduling method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |