CN113849160A - Dynamic migration method and device for compiling task, electronic equipment and storage medium - Google Patents
Dynamic migration method and device for compiling task, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113849160A CN113849160A CN202111122247.7A CN202111122247A CN113849160A CN 113849160 A CN113849160 A CN 113849160A CN 202111122247 A CN202111122247 A CN 202111122247A CN 113849160 A CN113849160 A CN 113849160A
- Authority
- CN
- China
- Prior art keywords
- compiler
- task
- target
- cluster
- candidate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 78
- 230000005012 migration Effects 0.000 title claims abstract description 42
- 238000013508 migration Methods 0.000 title claims abstract description 42
- 238000012544 monitoring process Methods 0.000 claims abstract description 16
- 230000008569 process Effects 0.000 claims description 34
- 238000012545 processing Methods 0.000 claims description 20
- 238000004891 communication Methods 0.000 claims description 19
- 238000001514 detection method Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 8
- 230000002159 abnormal effect Effects 0.000 claims description 4
- 238000012806 monitoring device Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000004806 packaging method and process Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 230000006378 damage Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/37—Compiler construction; Parser generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/76—Adapting program code to run in a different environment; Porting
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
- Devices For Executing Special Programs (AREA)
Abstract
The application discloses a dynamic migration method and device of a compiling task, electronic equipment and a storage medium. The method comprises the following steps: monitoring current first resource use data of each compiler in the compiler cluster; determining an original compiler by the compiler with first resource use data in the compiler cluster being greater than or equal to a first preset threshold; determining a target compiling task which meets a migration condition in an original compiler; and acquiring a target compiler in the compiler cluster, and migrating a task file corresponding to the target compilation task from the original compiler to the target compiler. According to the method and the device, the original compiler needing task migration is determined by monitoring the resource use data of each compiler, then the target compilation task meeting the migration condition on the compiler and the target compiler capable of receiving the target compilation task are screened out according to the resource use data, and finally the target compilation task is migrated, so that the usable resources of the compilation task are improved under the existing environment, and the compilation working efficiency is improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a dynamic migration method and apparatus for a compilation task, an electronic device, and a storage medium.
Background
Koji is a software system for compiling RPM packages, and is generally composed of a Koji Hub and a plurality of kojids, and the Koji Hub is randomly allocated to each Kojid (RPM package compiler) according to the submitted compiling task, and the following problems are existed in the actual compiling process: hardware resources used during compiling of different RPM (revolution speed) packets are different, so that although the compiling task numbers on different Kojids (RPM packet compilers) are the same, the resource utilization rates are different greatly, and sometimes, the resource utilization rate of a certain Kojid (RPM packet compiler) is too high, so that the certain Kojid (RPM packet compiler) is blocked, and the compiling efficiency is influenced.
Disclosure of Invention
In order to solve the technical problem or at least partially solve the technical problem, the present application provides a dynamic migration method, apparatus, electronic device and storage medium for a compilation task.
According to an aspect of an embodiment of the present application, there is provided a dynamic migration method for a compilation task, including:
monitoring current first resource usage data of each compiler in a compiler cluster, wherein at least one compilation task is deployed in each compiler;
determining an original compiler by the compiler with the first resource use data in the compiler cluster being greater than or equal to a first preset threshold;
determining a target compiling task which meets a migration condition in the original compiler;
and acquiring a target compiler in the compiler cluster, and migrating a task file corresponding to the target compilation task from the original compiler to the target compiler, wherein the target compiler is the compiler of which the first resource usage data in the compiler cluster is smaller than a first preset threshold value.
Further, the first resource usage data includes: a first central processing unit utilization rate and a first memory utilization rate;
the determining, by the compiler that uses the first resource usage data in the compiler cluster to be greater than or equal to the first preset threshold, an original compiler includes:
and determining the compiler with the first central processing unit utilization rate greater than or equal to the preset central processing unit utilization rate and/or the first memory utilization rate greater than or equal to the preset memory utilization rate in the compiler cluster as the original compiler.
Further, the determining that the original compiler meets the target compilation task of the migration condition includes:
querying second resource use data corresponding to each process in the original compiler;
determining the process of which the second resource usage data is greater than or equal to a second preset threshold value as a target process;
and determining the compiling task corresponding to the target process as the target compiling task.
Further, the obtaining a target compiler in the compiler cluster includes:
acquiring a first candidate compiler with first resource usage data smaller than a first preset threshold value in the compiler cluster and third resource usage data corresponding to the first candidate compiler;
determining the first candidate compiler as the target compiler if the sum of the third resource usage data and the second resource usage data is less than the first preset threshold.
Further, the obtaining a target compiler in the compiler cluster includes:
generating a compiler creating task when a compiler with first resource usage data smaller than a preset threshold value does not exist in the compiler cluster, or when the sum of the third resource usage data and the second resource usage data is larger than the first preset threshold value;
acquiring an environment configuration script based on the compiler creating task;
and creating the target compiler according to the environment configuration script.
Further, the method further comprises:
acquiring a second candidate compiler with the first resource use data smaller than or equal to a third preset threshold in the compiler cluster;
querying a compilation task deployed on the second candidate compiler;
releasing resources used by the second candidate compiler and retiring the second candidate compiler if there is no compilation task on the second candidate compiler.
Further, the method further comprises:
under the condition that the second candidate compiler does not have the compiling task, acquiring a compiling state corresponding to the compiling task in the second candidate compiler;
detecting the second candidate compiler to obtain a detection result under the condition that the compiling state belongs to an abnormal state;
and executing migration operation on the compiling task in the second candidate compiler under the condition that the detection result is used for indicating the second candidate compiler to be in fault.
According to another aspect of the embodiments of the present application, there is also provided a dynamic migration apparatus for compiling tasks, including:
the monitoring module is used for monitoring the current first resource use data of each compiler in the compiler cluster, wherein at least one compiling task is stored in each compiler;
the determining module is used for determining the compiler with the first resource use data larger than or equal to a first preset threshold value in the compiler cluster as an original compiler;
the query module is used for determining a target compiling task which meets the migration condition in the original compiler;
and the processing module is used for acquiring a target compiler in the compiler cluster and transferring a task file corresponding to the target compilation task from the original compiler to the target compiler, wherein the target compiler is a compiler of which first resource use data in the compiler cluster is smaller than a first preset threshold value.
According to another aspect of the embodiments of the present application, there is also provided a storage medium including a stored program that executes the above steps when the program is executed.
According to another aspect of the embodiments of the present application, there is also provided an electronic apparatus, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus; wherein: a memory for storing a computer program; a processor for executing the steps of the method by running the program stored in the memory.
Embodiments of the present application also provide a computer program product containing instructions, which when run on a computer, cause the computer to perform the steps of the above method.
Compared with the prior art, the technical scheme provided by the embodiment of the application has the following advantages: according to the method and the device, the original compiler needing task migration is determined by monitoring the resource use data of each compiler, then the target compilation task meeting the migration condition on the compiler and the target compiler capable of receiving the target compilation task are screened out according to the resource use data, and finally the target compilation task is migrated, so that the usable resources of the compilation task are improved under the existing environment, and the compilation working efficiency is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.
Fig. 1 is a flowchart of a dynamic migration method for a compilation task according to an embodiment of the present application;
FIG. 2 is a flowchart of a method for live migration of a compilation task according to another embodiment of the present application;
FIG. 3 is a flowchart of a method for live migration of a compilation task according to another embodiment of the present application;
FIG. 4 is a block diagram of a dynamic migration apparatus for compiling tasks according to an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application, it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments, and the illustrative embodiments and descriptions thereof of the present application are used for explaining the present application and do not constitute a limitation to the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another similar entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The embodiment of the application provides a dynamic migration method and device for a compilation task, electronic equipment and a storage medium. The method provided by the embodiment of the invention can be applied to any required electronic equipment, for example, the electronic equipment can be electronic equipment such as a server and a terminal, and the method is not particularly limited herein, and is hereinafter simply referred to as electronic equipment for convenience in description.
According to an aspect of embodiments of the present application, there is provided a method embodiment of a live migration method of a compilation task. Fig. 1 is a flowchart of a method for dynamically migrating a compilation task according to an embodiment of the present application, where as shown in fig. 1, the method includes:
step S11, monitoring current first resource usage data of each compiler in the compiler cluster, wherein at least one compilation task is deployed in each compiler.
The method provided by the embodiment of the application is suitable for monitoring equipment in the Koji system, the monitoring equipment is mainly used for monitoring the compiling condition of each compiler in a compiler cluster, and the compiler cluster is deployed in the Koji system.
In the embodiment of the present application, Kojihub (task distribution device) in the Koji system randomly allocates the compiling tasks uploaded by the users, so that at least one compiling task is deployed in each compiler in the Koji system.
In this embodiment of the present application, a monitoring device in a Koji system mainly collects first resource usage data of each compiler in a current cycle, where the first resource usage data includes: a first cpu utilization n1 and a first memory utilization m 1. The first central processor usage rate is (CPU usage rate).
In step S12, the compiler with the first resource usage data in the compiler cluster being greater than or equal to the first preset threshold is determined as the original compiler.
In the embodiment of the application, the monitoring device counts first resource usage data of each compiler in the compiler cluster, sorts the compilers according to the first resource usage data, obtains the compiler of which the first resource usage data is greater than or equal to a first preset threshold, and determines the compiler as an original compiler.
In this embodiment of the present application, determining, by a compiler in a compiler cluster, an original compiler, where first resource usage data is greater than or equal to a first preset threshold, includes: and determining the original compiler by the compiler with the first central processing unit utilization rate greater than the first preset central processing unit utilization rate and/or the first memory utilization rate greater than or equal to the first preset memory utilization rate.
It should be noted that, in the embodiment of the present application, by sequencing the compilers according to the first resource usage data, the task pressure of each compiler can be determined in time, and corresponding compilation task scheduling is executed according to the task pressure of the compiler. The compiling efficiency of the whole compiler cluster is improved.
Step S13, determining a target compilation task satisfying the migration condition in the original compiler.
In this embodiment of the present application, the step S13 of determining that the original compiler satisfies the target compilation task of the migration condition includes the following steps a 1-A3:
step A1, querying second resource usage data corresponding to each process in the original compiler.
In this embodiment of the present application, the monitoring apparatus retrieves a currently running process and second resource usage data corresponding to each process from the original compiler, where the second resource usage data includes: a second central processing unit utilization rate and a second memory utilization rate.
Step a2, determining the process with the second resource usage data being greater than or equal to the second preset threshold as the target process.
In the embodiment of the application, the processes are sorted by using the second resource usage data, the process with the second resource usage data being greater than or equal to the second preset threshold is determined, and the process is determined as the target process.
In this embodiment of the application, the second resource data being greater than or equal to the second preset threshold includes: the second central processing unit utilization rate is greater than a second preset central processing unit utilization rate, and/or the second memory utilization rate is greater than or equal to a second preset memory utilization rate.
It should be noted that, in the embodiment of the present application, by collecting the resource usage data of each process, which compilation tasks currently occupy more physical resources can be determined, so that the compilation tasks occupying more material resources are migrated in the following.
Step A3, determining the compiling task corresponding to the target process as the target compiling task.
In the embodiment of the present application, since the target process corresponding to the compilation task occupies too many physical resources of the original compiler, the compilation task needs to be determined as the target compilation task, and the task needs to be migrated, so as to ensure the compilation efficiency of the target compiler.
Step S14, obtaining a target compiler in the compiler cluster, and migrating a task file corresponding to a target compilation task from the original compiler to the target compiler, where the target compiler is a compiler whose first resource usage data in the compiler cluster is smaller than a first preset threshold.
In this embodiment of the present application, in step S14, acquiring a target compiler in a compiler cluster includes the following steps B1-B2:
step B1, obtaining a first candidate compiler in the compiler cluster, where the first resource usage data is smaller than a first preset threshold, and third resource usage data corresponding to the first candidate compiler.
In the embodiment of the present application, a first candidate compiler that uses the first resource in the cluster of compilers in the current period and is smaller than a first preset threshold is obtained, and it is determined that the resource usage data of the first candidate compiler in the current period is determined as third resource data, where the third resource usage data includes a third cpu usage rate and a third memory usage rate.
And step B2, determining the first candidate compiler as the target compiler under the condition that the sum of the third resource usage data and the second resource usage data is less than the first preset threshold value.
In the embodiment of the present application, after the third resource usage data is determined, the third resource usage data and the second resource usage data are used for calculation, and a sum of the third resource usage data and the second resource usage data is determined.
As an example, the second resource usage data includes a second cpu usage n2 and a second memory usage m2, and the third resource usage data includes a third cpu usage n3 and a third memory usage m 3. Then, the sum of the second central processor usage n2 and the third central processor usage n3 is calculated as n4, and the sum of the second memory usage m2 and the third memory usage m3 is calculated as m 4. If n4 is less than the first predetermined CPU usage and m4 is less than the first predetermined memory usage, then the first candidate compiler is determined to be the target compiler.
It should be noted that, by calculating the sum of the third resource usage data and the second resource usage data, it can be determined whether the first candidate compiler will affect the efficiency of the target compiler after receiving the target compilation task. Therefore, in the embodiment of the present application, when the sum of the third resource usage data and the second resource usage data is smaller than the first preset threshold, the first candidate compiler is determined as the target compiler. And migrating the task file corresponding to the target compiling task from the original compiler to the target compiler.
In the embodiment of the present application, the migration process of the target migration task is as follows: and inquiring a storage address of the target compiling task from the original compiler, calling a compiling environment configuration file, a source code file and a mock root directory corresponding to the target compiling task based on the storage address, packaging the compiling environment configuration file, the source code file and the mock root directory, sending the packed compiling environment configuration file, the source code file and the mock root directory to the target compiler, and finishing the scheduling of the compiling task.
In this embodiment of the present application, in step S14, acquiring a target compiler in a compiler cluster includes the following steps E1-E3:
and E1, generating a compiler creating task when the compiler cluster does not have a compiler with the first resource usage data smaller than the preset threshold, or when the sum of the third resource usage data and the second resource usage data is larger than the first preset threshold.
Step E2, the environment configuration script is obtained based on the compiler creating task.
Step E3, create the target compiler according to the environment configuration script.
In the embodiment of the application, if a compiler with first resource usage data smaller than a preset threshold value does not exist in the compiler cluster, a compiler creating task is generated, at this time, an environment configuration script uploaded by a user is obtained according to the compiler creating task, a new compiler is generated based on the configuration script, and the newly created compiler is determined as a target compiler.
In another embodiment of the present application, if a first candidate compiler with first resource usage data smaller than a preset threshold exists in the compiler cluster, and a sum of third resource usage data and second resource usage data of the first candidate compiler is greater than the first preset threshold, it is determined that the compilation efficiency of the first candidate compiler after receiving the target compilation task will be affected, and therefore in this case, a compiler creation task is also generated, a new compiler is generated based on the configuration script, and the newly created compiler is determined as the target compiler.
According to the method and the device, the original compiler needing task migration is determined by monitoring the resource use data of each compiler, then the target compilation task meeting the migration condition on the compiler and the target compiler capable of receiving the target compilation task are screened out according to the resource use data, and finally the target compilation task is migrated, so that the usable resources of the compilation task are improved under the existing environment, and the compilation working efficiency is improved.
In an embodiment of the present application, fig. 2 is a flowchart of a dynamic migration method of a compilation task provided in the embodiment of the present application, and as shown in fig. 2, the method further includes:
in step S21, a second candidate compiler with the first resource usage data less than or equal to a third preset threshold in the compiler cluster is obtained.
And step S22, inquiring the compiling task deployed on the second candidate compiler.
In step S23, when the compilation task does not exist on the second candidate compiler, the resource used by the second candidate compiler is released, and the second candidate compiler is deactivated.
In this embodiment of the application, the monitoring device is further configured to periodically obtain a second candidate compiler that uses the first resource usage data less than or equal to a third threshold in the compiler cluster, where as may be understood, the second candidate compiler may be a compiler that uses a smaller resource in the compiler cluster, query a compilation task deployed on the second candidate compiler at this time, and if there is no compilation task on the second candidate compiler, determine that compilation of the compilation task of the second candidate compiler is completed, at this time, release the resource used by the second candidate compiler, and cancel the second candidate compiler.
In the embodiment of the application, the compiler with the first resource usage data smaller than or equal to the third threshold in the compiler cluster is periodically queried, and the compiler is cancelled under the condition that the compiler does not have a compiling task, so that the compiling resources available for other compilers are increased, and the overall compiling efficiency of the compiler cluster is effectively ensured.
In this embodiment of the present application, fig. 3 is a flowchart of a dynamic migration method of a compilation task provided in this embodiment of the present application, and as shown in fig. 3, the method further includes:
step S31, under the condition that the second candidate compiler has no compilation task, acquiring a compilation state corresponding to the compilation task in the second candidate compiler;
step S32, detecting a second candidate compiler to obtain a detection result under the condition that the compiling state belongs to the abnormal state;
and step S33, in the case that the detection result is used for indicating the second candidate compiler fault, executing the migration operation on the compiling task in the second candidate compiler.
The following method for dynamically migrating a compilation task provided by the application embodiment includes the following specific processes:
and (1) the task distribution equipment (Kojihub) is responsible for sending the compiling tasks to each RPM packet compiling machine (Kojid), and when the compiling tasks exist on the Kojid, the monitoring device is started and is responsible for inquiring the first resource use data on each Kojid. Polling every 30s, and sequencing the first resource use data of each Kojid;
step (2), if the first resource usage data on a certain Kojid is inquired to be larger than or equal to a first preset threshold value, determining the Kojid as an original compiler;
step (3), inquiring the second resource utilization rate of each process on the Kojid, and inquiring the disk file read and written by the process so as to obtain a compiling task corresponding to the process;
step (4), inquiring the third resource utilization rates of other current Kojids, and if the sum of the third resource utilization rate and the second resource utilization rate of a certain Kojid is smaller than a first preset threshold value, determining the Kojid as a target compiler;
step (5), packaging the environment configuration file, the source code file and the mock root directory on the original compiler, transferring the environment configuration file, the source code file and the mock root directory to a target compiler, decompressing and recovering the packaged file by the target compiler, executing a compiling command, and recovering a compiling task;
step (6), if the sum of the third resource utilization rate and the second resource utilization rate of a certain Kojid is greater than a first preset threshold device, a new Kojid is created;
and (7) packaging the environment configuration file, the source code file and the mock root directory on the original compiler, and migrating the environment configuration file, the source code file and the mock root directory to a new compiler.
Polling until the current use data of a plurality of kojid first resources are lower than a second preset threshold value, inquiring whether a compiling task exists on the kojid, and counting the no _ built by +1 if the compiling task does not exist;
and (9) when the no _ built count is larger than 1, executing a kojid destruction task, and releasing the kojid resource to the host for use as other business requirements.
Fig. 4 is a block diagram of a dynamic migration apparatus for compiling tasks, which may be implemented as part of or all of an electronic device by software, hardware, or a combination of the two, according to an embodiment of the present application. As shown in fig. 4, the apparatus includes:
a monitoring module 41, configured to monitor current first resource usage data of each compiler in a compiler cluster, where at least one compilation task is stored in each compiler;
a determining module 41, configured to determine, as an original compiler, a compiler in the compiler cluster whose first resource usage data is greater than or equal to a first preset threshold;
a query module 42, configured to determine a target compilation task that satisfies a migration condition in the original compiler;
the processing module 43 is configured to acquire a target compiler in the compiler cluster, and migrate a task file corresponding to a target compilation task from the original compiler to the target compiler, where the target compiler is a compiler in which first resource usage data in the compiler cluster is smaller than a first preset threshold.
In an embodiment of the present application, the first resource usage data includes: a first central processing unit utilization rate and a first memory utilization rate;
in an embodiment of the present application, the determining module is configured to determine, as the original compiler, a compiler with a first central processing unit usage rate greater than or equal to a preset central processing unit usage rate and a first memory usage rate greater than or equal to a preset memory usage rate in the compiler cluster.
In the embodiment of the application, the query module is used for querying second resource usage data corresponding to each process in the original compiler; determining the process with the second resource use data being larger than or equal to a second preset threshold value as a target process; and determining the compiling task corresponding to the target process as a target compiling task.
In an embodiment of the present application, the processing module is configured to obtain a first candidate compiler and third resource usage data corresponding to the first candidate compiler, where first resource usage data in a compiler cluster is smaller than a first preset threshold; and determining the first candidate compiler as the target compiler under the condition that the sum of the third resource usage data and the second resource usage data is less than a first preset threshold value.
In this embodiment of the present application, the processing module is configured to generate a compiler creating task when there is no compiler with first resource usage data smaller than a preset threshold in the compiler cluster, or when a sum of third resource usage data and second resource usage data is larger than a first preset threshold; creating a task based on a compiler to obtain an environment configuration script; and creating a target compiler according to the environment configuration script.
In this embodiment of the present application, the dynamic migration apparatus for compiling tasks further includes: the revocation module is used for acquiring a second candidate compiler with the first resource use data smaller than or equal to a third preset threshold value in the compiler cluster; querying a compilation task deployed on a second candidate compiler; and under the condition that the compiling task does not exist on the second candidate compiler, releasing the resources used by the second candidate compiler and undoing the second candidate compiler.
In this embodiment of the present application, the dynamic migration apparatus for compiling tasks further includes: the detection module is used for acquiring the compiling state corresponding to the compiling task in the second candidate compiler under the condition that the second candidate compiler does not have the compiling task; detecting the second candidate compiler to obtain a detection result under the condition that the compiling state belongs to the abnormal state; and in the case that the detection result is used for indicating the second candidate compiler to be in fault, executing the migration operation on the compiling task in the second candidate compiler.
An embodiment of the present application further provides an electronic device, as shown in fig. 5, the electronic device may include: the system comprises a processor 1501, a communication interface 1502, a memory 1503 and a communication bus 1504, wherein the processor 1501, the communication interface 1502 and the memory 1503 complete communication with each other through the communication bus 1504.
A memory 1503 for storing a computer program;
the processor 1501 is configured to implement the steps of the above embodiments when executing the computer program stored in the memory 1503.
The communication bus mentioned in the above terminal may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the terminal and other equipment.
The Memory may include a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
In yet another embodiment provided by the present application, a computer-readable storage medium is further provided, which has instructions stored therein, and when the instructions are executed on a computer, the instructions cause the computer to execute the dynamic migration method of the compiling task in any one of the above embodiments.
In yet another embodiment provided by the present application, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the method for live migration of a compilation task as described in any of the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital subscriber line) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk), among others.
The above description is only for the preferred embodiment of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application are included in the protection scope of the present application.
The above description is merely exemplary of the present application and is presented to enable those skilled in the art to understand and practice the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method for dynamically migrating a compilation task, comprising:
monitoring current first resource usage data of each compiler in a compiler cluster, wherein at least one compilation task is deployed in each compiler;
determining an original compiler by the compiler with the first resource use data in the compiler cluster being greater than or equal to a first preset threshold;
determining a target compiling task which meets a migration condition in the original compiler;
and acquiring a target compiler in the compiler cluster, and migrating a task file corresponding to the target compilation task from the original compiler to the target compiler, wherein the target compiler is the compiler of which the first resource usage data in the compiler cluster is smaller than a first preset threshold value.
2. The method of claim 1, wherein the first resource usage data comprises: a first central processing unit utilization rate and a first memory utilization rate;
the determining, by the compiler that uses the first resource usage data in the compiler cluster to be greater than or equal to the first preset threshold, an original compiler includes:
and determining the compiler with the first central processing unit utilization rate greater than or equal to the preset central processing unit utilization rate and/or the first memory utilization rate greater than or equal to the preset memory utilization rate in the compiler cluster as the original compiler.
3. The method of claim 1, wherein the determining that the original compiler satisfies the target compilation task of the migration condition comprises:
querying second resource use data corresponding to each process in the original compiler;
determining the process of which the second resource usage data is greater than or equal to a second preset threshold value as a target process;
and determining the compiling task corresponding to the target process as the target compiling task.
4. The method of claim 3, wherein the obtaining a target compiler in the compiler cluster comprises:
acquiring a first candidate compiler with first resource usage data smaller than a first preset threshold value in the compiler cluster and third resource usage data corresponding to the first candidate compiler;
determining the first candidate compiler as the target compiler if the sum of the third resource usage data and the second resource usage data is less than the first preset threshold.
5. The method of claim 4, wherein the obtaining a target compiler in the compiler cluster comprises:
generating a compiler creating task when a compiler with first resource usage data smaller than a preset threshold value does not exist in the compiler cluster, or when the sum of the third resource usage data and the second resource usage data is larger than the first preset threshold value;
acquiring an environment configuration script based on the compiler creating task;
and creating the target compiler according to the environment configuration script.
6. The method of claim 1, further comprising:
acquiring a second candidate compiler with the first resource use data smaller than or equal to a third preset threshold in the compiler cluster;
querying a compilation task deployed on the second candidate compiler;
releasing resources used by the second candidate compiler and retiring the second candidate compiler if there is no compilation task on the second candidate compiler.
7. The method of claim 6, further comprising:
under the condition that the second candidate compiler does not have the compiling task, acquiring a compiling state corresponding to the compiling task in the second candidate compiler;
detecting the second candidate compiler to obtain a detection result under the condition that the compiling state belongs to an abnormal state;
and executing migration operation on the compiling task in the second candidate compiler under the condition that the detection result is used for indicating the second candidate compiler to be in fault.
8. An apparatus for dynamically migrating a compilation task, comprising:
the monitoring module is used for monitoring the current first resource use data of each compiler in the compiler cluster, wherein at least one compiling task is stored in each compiler;
the determining module is used for determining the compiler with the first resource use data larger than or equal to a first preset threshold value in the compiler cluster as an original compiler;
the query module is used for determining a target compiling task which meets the migration condition in the original compiler;
and the processing module is used for acquiring a target compiler in the compiler cluster and transferring a task file corresponding to the target compilation task from the original compiler to the target compiler, wherein the target compiler is a compiler of which first resource use data in the compiler cluster is smaller than a first preset threshold value.
9. A storage medium, characterized in that the storage medium comprises a stored program, wherein the program is operative to perform the method steps of any of the preceding claims 1 to 7.
10. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus; wherein:
a memory for storing a computer program;
a processor for performing the method steps of any of claims 1-7 by executing a program stored on a memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111122247.7A CN113849160B (en) | 2021-09-24 | 2021-09-24 | Dynamic migration method and device for compiling tasks, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111122247.7A CN113849160B (en) | 2021-09-24 | 2021-09-24 | Dynamic migration method and device for compiling tasks, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113849160A true CN113849160A (en) | 2021-12-28 |
| CN113849160B CN113849160B (en) | 2023-11-14 |
Family
ID=78979239
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111122247.7A Active CN113849160B (en) | 2021-09-24 | 2021-09-24 | Dynamic migration method and device for compiling tasks, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113849160B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116684468A (en) * | 2023-08-02 | 2023-09-01 | 腾讯科技（深圳）有限公司 | Data processing method, device, equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111694517A (en) * | 2020-05-28 | 2020-09-22 | 新浪网技术(中国)有限公司 | Distributed data migration method and system and electronic equipment |
| CN112764928A (en) * | 2021-01-21 | 2021-05-07 | 苏州浪潮智能科技有限公司 | Method, device and medium for dynamic management of compiling resources |
-
2021
- 2021-09-24 CN CN202111122247.7A patent/CN113849160B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111694517A (en) * | 2020-05-28 | 2020-09-22 | 新浪网技术(中国)有限公司 | Distributed data migration method and system and electronic equipment |
| CN112764928A (en) * | 2021-01-21 | 2021-05-07 | 苏州浪潮智能科技有限公司 | Method, device and medium for dynamic management of compiling resources |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116684468A (en) * | 2023-08-02 | 2023-09-01 | 腾讯科技（深圳）有限公司 | Data processing method, device, equipment and storage medium |
| CN116684468B (en) * | 2023-08-02 | 2023-10-20 | 腾讯科技（深圳）有限公司 | Data processing method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113849160B (en) | 2023-11-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108376118B (en) | Service distribution system, method, device and storage medium | |
| CN110768912B (en) | API gateway current limiting method and device | |
| WO2019227689A1 (en) | Data monitoring method and apparatus, and computer device and storage medium | |
| WO2019019400A1 (en) | Task distributed processing method, device, storage medium and server | |
| WO2017177806A1 (en) | Method and apparatus for managing resources | |
| CN103763346A (en) | Distributed resource scheduling method and device | |
| CN107832151B (en) | CPU resource allocation method, device and equipment | |
| CN110109748A (en) | A kind of hybrid language task executing method, device and cluster | |
| CN115113997A (en) | Task scheduling method and device, electronic equipment and storage medium | |
| CN113849160A (en) | Dynamic migration method and device for compiling task, electronic equipment and storage medium | |
| CN114629960B (en) | Resource scheduling method, device, system, equipment, medium and program product | |
| CN109002348B (en) | Load balancing method and device in virtualization system | |
| CN111124382A (en) | Attribute assignment method and device in Java and server | |
| CN110209548B (en) | Service control method, system, electronic device and computer readable storage medium | |
| CN114090623A (en) | Method and device for creating cache resources, electronic equipment and storage medium | |
| CN114297630A (en) | Malicious data detection method and device, storage medium and processor | |
| CN112559565A (en) | Abnormity detection method, system and device | |
| CN114070755A (en) | Virtual machine network flow determining method and device, electronic equipment and storage medium | |
| CN111162947A (en) | PCRE hot switching method, network device and storage medium | |
| CN111352710B (en) | Process management method and device, computing equipment and storage medium | |
| CN113806050A (en) | Processing method and device of computing resources, electronic equipment and storage medium | |
| CN111090627B (en) | Log storage method and device based on pooling, computer equipment and storage medium | |
| CN113064765A (en) | Node exception handling method and device, electronic equipment and machine-readable storage medium | |
| CN111754117A (en) | Task issuing method and device, electronic equipment and storage medium | |
| CN110888741A (en) | Resource scheduling method and device for application container, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |