CN110633135A - Asynchronous task allocation method and device, computer equipment and storage medium - Google Patents
Asynchronous task allocation method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110633135A CN110633135A CN201910752334.7A CN201910752334A CN110633135A CN 110633135 A CN110633135 A CN 110633135A CN 201910752334 A CN201910752334 A CN 201910752334A CN 110633135 A CN110633135 A CN 110633135A
- Authority
- CN
- China
- Prior art keywords
- task
- executed
- tasks
- available node
- equipment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000013507 mapping Methods 0.000 claims abstract description 35
- 230000008569 process Effects 0.000 claims abstract description 17
- 238000004590 computer program Methods 0.000 claims description 8
- 230000006870 function Effects 0.000 description 8
- 239000010453 quartz Substances 0.000 description 7
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N silicon dioxide Inorganic materials O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 description 7
- 238000010586 diagram Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
Abstract
The invention discloses an asynchronous task allocation method, an asynchronous task allocation device, computer equipment and a storage medium, which are used for allocating timed tasks to node equipment in a cluster, and are characterized in that the method comprises the following steps: acquiring task information to be executed, the number of available node equipment and a batch execution upper limit, wherein the batch execution upper limit represents the maximum task number processed by all the available node equipment in a cluster at a time; determining a task to be executed corresponding to each available node device according to task information to be executed, the number of available node devices and a batch execution upper limit, so that the difference between the number of the tasks to be executed distributed on each available node device and the number of the tasks to be executed distributed on any other available node device is not greater than a first threshold value; establishing a mapping relation between a task to be executed and available node equipment; and storing the mapping relation into a memory of the available node equipment so that the available node equipment can process the task to be executed according to the mapping relation.
Description
Technical Field
The present invention relates to the field of computer task scheduling technologies, and in particular, to an asynchronous task allocation method, an asynchronous task allocation device, a computer device, and a storage medium.
Background
The need for task scheduling is almost always encountered by public enterprise applications, and the following timed tasks are often required as exemplified by forums: and generating an RSS file of the essence article every half hour, counting the point ranking of the forum user every morning, and executing a locking user unlocking task every 30 minutes. Quartz has been a solution for task scheduling by a number of projects since its release in 2001, providing great flexibility without sacrificing simplicity, and its powerful functions to cope with the vast majority of scheduling requirements.
In the quartz cluster environment, multiple node devices are allowed to be deployed, but only one node device can execute tasks, and if one node device is down, another node device can be started to execute the tasks. For the quartz cluster environment, only one node device processes tasks when the task processing time is reached, so the task processing capacity is low, and the increasing task processing requirements cannot be met.
Therefore, how to improve the task processing capability in the cluster environment becomes a technical problem to be solved urgently by those skilled in the art.
Disclosure of Invention
The present invention provides an asynchronous task allocation method, an asynchronous task allocation device, a computer device, and a storage medium, so as to solve the above problems in the prior art.
In order to achieve the above object, the present invention provides an asynchronous task allocation method, which comprises the following steps:
acquiring task information to be executed, the number of available node equipment and a batch execution upper limit, wherein the batch execution upper limit represents the maximum task number processed by all the available node equipment in a cluster at a time;
determining a task to be executed corresponding to each available node device according to task information to be executed, the number of available node devices and a batch execution upper limit, so that the difference between the number of the tasks to be executed distributed on each available node device and the number of the tasks to be executed distributed on any other available node device is not greater than a first threshold value;
establishing a mapping relation between a task to be executed and available node equipment;
and storing the mapping relation into a memory of the available node equipment so that the available node equipment can process the task to be executed according to the mapping relation.
According to the asynchronous task allocation method provided by the present invention, the step of allocating all the tasks to be executed to the available node devices such that the difference between the number of tasks allocated to each available node device and the number of tasks allocated to any one of the other available node devices is not greater than the first threshold value includes:
judging whether the number of all the tasks to be executed is greater than or equal to the batch execution upper limit;
if yes, distributing tasks for the available node equipment according to a first distribution rule;
and if not, distributing the tasks to the available node equipment according to a second distribution rule.
According to the asynchronous task allocation method provided by the invention, the step of allocating tasks to the available node equipment according to the first allocation rule comprises the following steps:
obtaining M to-be-executed tasks, wherein M is equal to the upper limit of batch execution;
averagely distributing the M to-be-executed tasks to the available node equipment;
and after the available node equipment finishes processing the M to-be-executed tasks, modifying the states of the M to-be-executed tasks into processed states in a task state table.
According to the asynchronous task allocation method provided by the present invention, the step of allocating a task to the available node device according to the second allocation rule includes:
respectively setting a unique task number for all the tasks to be executed, and respectively setting a unique equipment number for the available node equipment, wherein each task number corresponds to one equipment number;
and distributing the task to be executed corresponding to the task number to the available node equipment corresponding to the equipment number corresponding to the task number.
According to the asynchronous task allocation method provided by the present invention, the step of allocating a task to the available node device according to the second allocation rule includes:
respectively setting unique and continuous task numbers for all the tasks to be executed;
dividing the task number of each task to be executed by the number of the available node devices, and then performing remainder operation to obtain a task remainder;
corresponding different task remainders to different available node equipment;
and distributing the task to be executed corresponding to each task remainder to corresponding available node equipment.
To achieve the above object, the present invention provides an asynchronous task assigning apparatus, including:
the parameter acquisition module is suitable for acquiring information of tasks to be executed, the number of available node equipment and a batch execution upper limit, wherein the batch execution upper limit represents the maximum number of tasks processed by all the available node equipment in the cluster at one time;
the task allocation module is suitable for determining the tasks to be executed corresponding to each available node device according to the information of the tasks to be executed, the number of the available node devices and the upper limit of batch execution, so that the difference between the number of the tasks to be executed allocated on each available node device and the number of the tasks to be executed allocated on any one of the other available node devices is not greater than a first threshold value;
the mapping establishing module is suitable for establishing a mapping relation between the task to be executed and the available node equipment;
and the mapping storage module is suitable for storing the mapping relation into a memory of the available node equipment so that the available node equipment can process the task to be executed according to the mapping relation.
According to the asynchronous task allocation device provided by the invention, the task allocation module comprises:
the quantity judgment submodule is suitable for judging whether the quantity of all the tasks to be executed is greater than or equal to the batch execution upper limit;
the first allocation submodule is suitable for allocating tasks to the available node equipment according to a first allocation rule when the number of all the tasks to be executed is greater than or equal to the batch execution upper limit;
and the second distribution submodule is suitable for distributing the tasks to the available node equipment according to a second distribution rule when the number of all the tasks to be executed is smaller than the upper limit of batch execution.
According to the asynchronous task allocation device provided by the present invention, the second allocation submodule includes:
the task numbering unit is suitable for respectively setting unique task numbers for all the tasks to be executed;
the equipment numbering unit is suitable for respectively setting unique equipment numbers for the available node equipment;
a matching unit adapted to correspond each task number to one of the device numbers;
and the task allocation unit is suitable for allocating the task to be executed corresponding to the task number to the available node equipment corresponding to the equipment number corresponding to the task number.
To achieve the above object, the present invention further provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the above method when executing the computer program.
To achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the above method.
The asynchronous task allocation method, the asynchronous task allocation device, the computer equipment and the computer readable storage medium provided by the invention provide a task balanced allocation scheme for effectively improving task processing capacity for task scheduling in a cluster environment. The invention determines different task allocation schemes according to the relationship between the number of the tasks to be executed currently and the batch execution upper limit which can be processed by all available node equipment in the cluster each time: if the number of the tasks to be executed currently is larger than or equal to the upper limit of batch execution, distributing the tasks to be executed containing the upper limit of batch execution to the available node equipment averagely; and if the number of the current tasks to be executed is less than the upper limit of batch execution, corresponding each task to be executed to one of the node devices according to a certain allocation strategy, so that the difference between the number of the tasks allocated on each node device and the number of the tasks allocated on any other node device is not greater than a first threshold value. By the asynchronous task allocation scheme, the task processing load of each node device can be balanced, so that the processing efficiency of asynchronous tasks in a cluster environment is effectively improved.
Drawings
FIG. 1 is a flowchart of a first embodiment of an asynchronous task assignment method of the present invention;
FIG. 2 is a block diagram of a first exemplary embodiment of an asynchronous task assignment device;
FIG. 3 is a diagram illustrating a hardware structure of an asynchronous task assignment device according to a first embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The asynchronous task allocation method, the asynchronous task allocation device, the computer equipment and the computer readable storage medium provided by the invention provide a task balanced allocation scheme for effectively improving task processing capacity for task scheduling in a cluster environment. The invention determines different task allocation schemes according to the relationship between the number of the tasks to be executed currently and the batch execution upper limit which can be processed by all available node equipment in the cluster each time: if the number of the tasks to be executed currently is larger than or equal to the upper limit of batch execution, distributing the tasks to be executed containing the upper limit of batch execution to the available node equipment averagely; and if the number of the current tasks to be executed is less than the upper limit of batch execution, corresponding each task to be executed to one of the node devices according to a certain allocation strategy, so that the difference between the number of the tasks allocated on each node device and the number of the tasks allocated on any other node device is not greater than a first threshold value. By the asynchronous task allocation scheme, the task processing load of each node device can be balanced, so that the processing efficiency of asynchronous tasks in a cluster environment is effectively improved.
Example one
The embodiment provides an asynchronous task allocation method, which specifically includes the following steps:
and S1, acquiring the task information to be executed, the number of the available node devices and the upper limit of batch execution, wherein the upper limit of batch execution represents the maximum task number which can be processed by all the available node devices in the cluster at a single time.
The method comprises the steps of obtaining relevant parameter information from a configuration file of a database, wherein the relevant parameter information comprises the name of a task to be executed and trigger time information, the number of available node equipment and a batch execution upper limit. The default file name of the configuration file in Quartz is Quartz.
The information of the tasks to be executed comprises names and the number of all the tasks needing to be processed in the quartz cluster environment and relevant information of triggering time of the tasks. The task triggering time refers to the starting time of executing the timing task each time, and when the task triggering time is reached, the node equipment reads and processes the task to be executed. Simple timed trigger times can be set in the Quartz system, such as 22:00 pm on 30 days in 1 month, or more complex task trigger times can be set using cron expressions, such as setting 10 am triggers per day or 12:00 pm on wednesday.
The number of available node devices refers to the number of node devices currently available in the cluster at the time of task triggering. The number of node devices can be realized by searching for instance.name in the configuration file, where the instance.name represents an instance name, and each node device corresponds to a unique instance name, so that the number of currently available node devices can be known by inquiring how many instance names exist.
The batch execution upper limit refers to the maximum number of tasks that can be processed by all available node devices in the cluster at a single time. Generally, the maximum number of tasks that can be processed by each node device at a time is fixed, for example, 100 tasks can be processed at a time, 1000 tasks can be processed at a time, and the maximum number of tasks that can be processed by each node device at a time multiplied by the number of currently available node devices is the upper limit of batch execution.
And S2, determining the tasks to be executed corresponding to each available node device according to the task information to be executed, the number of the available node devices and the upper limit of batch execution, so that the difference between the number of the tasks to be executed distributed on each available node device and the number of the tasks to be executed distributed on any one of the other available node devices is not more than a first threshold value.
The method comprises the steps of determining tasks to be executed to be distributed on each node device according to the number of tasks to be executed, the number of available node devices and a batch execution upper limit, and ensuring that the difference between the number of tasks distributed on each node device and the number of tasks distributed on any other node device is not larger than a first threshold value through a specific distribution strategy. In other words, by the allocation policy of this step, the tasks to be executed allocated to each node device are distributed substantially in a balanced manner, and even if there is an unbalanced condition, the difference value of the tasks to be executed allocated to each node device is not greater than the first threshold value, so that all the tasks to be executed are guaranteed to be distributed to less than all the node devices in a balanced manner, and a situation where one or more node devices process tasks at full capacity and the remaining node devices are completely idle does not occur. And the method is favorable for reasonably utilizing all node equipment resources and improving the task processing capacity in the cluster environment.
The following step introduces the task allocation process in detail from two cases, one case is that the number of all the tasks to be executed currently is greater than or equal to the upper limit of batch execution, and the other case is that the number of all the tasks to be executed currently is less than the upper limit of batch execution.
And S21, the number of all the tasks to be executed is larger than or equal to the upper limit of batch execution.
The number of all the tasks to be executed currently is greater than or equal to the upper limit of batch execution, that is, the number of the tasks to be executed currently in the task pool exceeds or is just equal to the maximum number of people that all the available node devices can handle currently. In this case, all available node devices may be fully loaded with processing tasks. For example, 10 node devices are included in the cluster, and each node device can process 100 tasks at a time, so the upper limit of batch execution is 1000. Assuming that the number of the tasks to be executed currently in the task pool is 1011 and is greater than the upper limit of batch execution by 1000, the 1000 tasks to be executed can be averagely allocated to 10 available node devices at one time at the moment of task triggering, and each node device allocates 100 tasks for processing. At this time, the difference value of the tasks to be processed distributed on each node device is zero, so that the task processing capacity of each node device can be utilized to the greatest extent, and the processing efficiency is improved.
Further, each available node device obtains the task to be executed from the database on the basis that it has been determined that each node device allocates 100 tasks to be executed.
By the allocation method, on one hand, all tasks to be executed are relatively evenly allocated to the available node equipment, the computing resources of each node equipment are fully utilized, the condition that the task of one node equipment is far larger than that of other node equipment does not exist, and the task processing capacity is favorably improved. On the other hand, the allocation method of the invention can ensure that each task to be executed is allocated to a unique node device, and the situation of repeated allocation of the tasks can be avoided without adopting a locking mechanism in the prior art.
And S22, the number of all the tasks to be executed is smaller than the upper limit of batch execution.
If all the tasks to be executed currently in the task pool are smaller than the upper limit of batch execution that all the available node devices can handle, a second allocation strategy is adopted for task allocation, so that the difference between the number of tasks allocated on each node device and the number of tasks allocated on any other node device is not greater than a first threshold, and the first threshold can be a fixed natural number such as 3, 5, 10 and the like.
Further, the second allocation policy in the present proposal may be simply understood as that the second allocation policy is used to classify each element according to the key and then store the element where the corresponding class is located. The elements in the above description correspond to the tasks to be performed in the present proposal, and the places where the corresponding classes described in the above description are located correspond to the available node devices in the present proposal. Therefore, the second allocation policy in the present proposal is equivalent to classify each task to be executed according to the keywords, and then allocate the task to be executed to the available node device corresponding to the classification.
The keyword in this proposal refers to a field associated with each task to be executed, and may be, for example, a task number of each task to be executed, some bits in the task number, a numerical value obtained by performing a function operation on the task number, or the like. The step of classifying each to-be-executed task according to the keywords refers to classifying each to-be-executed task according to the task number of each to-be-executed task, some digits in the task number, a numerical value obtained by performing function operation on the task number, or any other parameter associated with the to-be-executed task. The invention classifies the tasks to be executed, namely, determines a specific certain node device to which the tasks to be executed are to be distributed.
The specific classification rule in this proposal may be that different device numbers are set for each available node device, so that the keyword of each task to be executed corresponds to one of the device numbers, and each task to be executed is allocated to the available node device corresponding to the keyword. It should be noted that, the allocation rule in this proposal ensures that the difference between the number of tasks allocated on each node device and the number of tasks allocated on any other node device is not greater than the first threshold.
For example, consecutive task numbers may be set for all currently-executed tasks, and consecutive device numbers may be set for currently-available node devices, and since the task numbers are different from each other and have continuity, and the device numbers are also different from each other and have continuity, classification may be performed according to the numerical relationship between the task numbers and the device numbers. For example, the last digit of the device number and the task number are compared, and the task number with the same digit is allocated to the available node device corresponding to the device number with the same last digit.
In addition to the above-mentioned method for comparing the last digit, the method can also be classified by a remainder method, and the specific steps include: dividing each task number by the number of the current available node equipment to obtain a remainder; and corresponding the different remainders to different equipment numbers, so as to allocate corresponding available node equipment to each task to be executed.
For example, suppose that there are 11 tasks to be executed currently, and each task to be executed is numbered consecutively, resulting in 11 task numbers: 1001,1002,1003,1004, …,1010,1011;
assuming that there are 10 currently available node devices, each available node device is numbered consecutively, resulting in 10 device numbers: 01, 02, 03, …, 09, 10;
dividing each task number by the number of available node devices by 10, and then taking the remainder to obtain the following remainder values respectively: 1, 2,3, …,9, 0, 11;
according to the classification rule, the last digit of each remainder is corresponding to the last digit of the equipment number, namely: the remainder of the task number 1001 is 1, and the node device with the number 01 corresponds to the remainder; the remainder of the task number 1002 is 2, and corresponds to the node equipment with the number 02; …, the remainder of the task number 1010 is 0, corresponding to the node device with number 10; the remainder of task number 1011 is 11, corresponding to node device number 01.
Through the classification, two tasks to be executed are distributed on the node device 01, one task to be executed is distributed on each of the node devices 02 to 10, and the difference value of the tasks distributed on every two node devices is at most 1, so that in general, all available node devices in the cluster basically achieve balance in processing tasks, and the improvement of task processing efficiency under the cluster environment is facilitated. Meanwhile, each task to be executed is distributed to the only node equipment, and the situation of repeated distribution of the tasks can be avoided without adopting a locking mechanism in the prior art.
S3: and establishing a mapping relation between the task to be executed and the available node equipment.
The mapping relationship stated in this proposal refers to that different tasks to be executed correspond to different available node devices. The mapping relation in the invention represents the allocation strategy of the tasks to be executed, namely which tasks are allocated to which node equipment, and the allocation strategy needs to ensure that each task to be executed is already allocated and is only allocated to one node equipment, thereby avoiding the situation of task repeated allocation.
The mapping relationship in the invention can be embodied by the device number and the task number. For example, a unique task number is set for each task to be executed, a unique device number is set for each available node device, and the tasks to be executed distributed to different node devices are represented by the corresponding relationship between the task numbers and the device numbers.
S4: and storing the mapping relation into a memory of the available node equipment so that the available node equipment can process the task to be executed according to the mapping relation when reaching the task triggering time.
The invention stores the mapping relationship in the memory of each available node device. Therefore, when the task triggering time is reached, the available node equipment checks the mapping relation from the local memory to determine whether the task to be executed exists or not. And if so, acquiring corresponding task data from the database according to the task number for processing. By storing the mapping relation in the memory of the available node equipment, the invention avoids reading a uniform task configuration table from a remote database, and can know the task to be executed which needs to be processed by a certain available node equipment by only calling the mapping relation stored in the local memory, thereby saving the time for reading the configuration table from the remote database and improving the task processing efficiency.
Referring to fig. 2, an asynchronous task assigning apparatus is shown, in this embodiment, the asynchronous task assigning apparatus 10 may include or be divided into one or more program modules, and the one or more program modules are stored in a storage medium and executed by one or more processors to implement the asynchronous task assigning method according to the present invention. The program module referred to in the present invention refers to a series of computer program instruction segments capable of performing specific functions, and is more suitable than the program itself for describing the execution process of the asynchronous task assigning apparatus 10 in the storage medium. The following description will specifically describe the functions of the program modules of the present embodiment:
the parameter obtaining module 11 is adapted to obtain task information to be executed, the number of available node devices, and a batch execution upper limit, where the batch execution upper limit represents the maximum number of tasks processed by all available node devices in a cluster at a single time;
the task allocation module 12 is adapted to determine a task to be executed corresponding to each available node device according to the information of the task to be executed, the number of available node devices, and the upper limit of batch execution, so that a difference between the number of the tasks to be executed allocated to each available node device and the number of the tasks to be executed allocated to any one of the other available node devices is not greater than a first threshold;
the mapping establishing module 13 is adapted to establish a mapping relationship between the task to be executed and the available node device;
and the mapping storage module 14 is adapted to store the mapping relationship in a memory of the available node device, so that the available node device processes the task to be executed according to the mapping relationship.
Wherein the task allocation module 12 includes:
the quantity judgment submodule 121 is adapted to judge whether the quantity of all the tasks to be executed is greater than or equal to the batch execution upper limit;
a first allocating sub-module 122, adapted to allocate the tasks to the available node devices according to a first allocation rule when the number of all the tasks to be executed is greater than or equal to the upper limit of batch execution;
and the second allocating submodule 123 is adapted to allocate the tasks to the available node device according to a second allocating rule when the number of all the tasks to be executed is smaller than the upper limit of batch execution.
The second allocating sub-module 123 includes:
a task numbering unit 1231, adapted to set unique task numbers for all the tasks to be executed, respectively;
an equipment numbering unit 1232, adapted to set unique equipment numbers for the available node devices, respectively;
a matching unit 1233 adapted to correspond each task number to one of the device numbers;
the task allocating unit 1234 is adapted to allocate the task to be executed corresponding to the task number to the available node device corresponding to the device number corresponding to the task number.
The embodiment also provides a computer device, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server or a rack server (including an independent server or a server cluster composed of a plurality of servers) capable of executing programs, and the like. The computer device 20 of the present embodiment includes at least, but is not limited to: a memory 21, a processor 22, which may be communicatively coupled to each other via a system bus, as shown in FIG. 3. It is noted that fig. 3 only shows the computer device 20 with components 21-22, but it is to be understood that not all shown components are required to be implemented, and that more or fewer components may be implemented instead.
In the present embodiment, the memory 21 (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the storage 21 may be an internal storage unit of the computer device 20, such as a hard disk or a memory of the computer device 20. In other embodiments, the memory 21 may also be an external storage device of the computer device 20, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, provided on the computer device 20. Of course, the memory 21 may also include both internal and external storage devices of the computer device 20. In this embodiment, the memory 21 is generally used for storing an operating system and various application software installed on the computer device 20, such as the program code of the asynchronous task assigning apparatus 10 in the first embodiment. Further, the memory 21 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 22 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 22 is typically used to control the overall operation of the computer device 20. In this embodiment, the processor 22 is configured to execute the program code stored in the memory 21 or process data, for example, execute the asynchronous task assigning apparatus 10, so as to implement the asynchronous task assigning method according to the first embodiment.
The present embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application mall, etc., on which a computer program is stored, which when executed by a processor implements corresponding functions. The computer-readable storage medium of this embodiment is used for storing the asynchronous task assigning apparatus 10, and when being executed by a processor, the asynchronous task assigning method of the first embodiment is implemented.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example" or "some examples" or the like are intended to mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. An asynchronous task allocation method for allocating timed tasks to node devices in a cluster, the method comprising the steps of:
acquiring task information to be executed, the number of available node equipment and a batch execution upper limit, wherein the batch execution upper limit represents the maximum task number processed by all the available node equipment in a cluster at a time;
determining a task to be executed corresponding to each available node device according to task information to be executed, the number of available node devices and a batch execution upper limit, so that the difference between the number of the tasks to be executed distributed on each available node device and the number of the tasks to be executed distributed on any other available node device is not greater than a first threshold value;
establishing a mapping relation between a task to be executed and available node equipment;
and storing the mapping relation into a memory of the available node equipment so that the available node equipment can process the task to be executed according to the mapping relation.
2. The asynchronous task distribution method according to claim 1, wherein the step of establishing a mapping relationship between the task to be executed and the available node devices comprises:
judging whether the number of all the tasks to be executed is greater than or equal to the batch execution upper limit;
if yes, distributing tasks for the available node equipment according to a first distribution rule;
and if not, distributing the tasks to the available node equipment according to a second distribution rule.
3. The asynchronous task assignment method of claim 2, wherein the step of assigning tasks to the available node devices according to a first assignment rule comprises:
obtaining M to-be-executed tasks, wherein M is equal to the upper limit of batch execution;
averagely distributing the M to-be-executed tasks to the available node equipment;
and after the available node equipment finishes processing the M to-be-executed tasks, modifying the states of the M to-be-executed tasks into processed states in a task state table.
4. The asynchronous task assignment method of claim 2, wherein said step of assigning tasks to said available node devices according to a second assignment rule comprises:
respectively setting a unique task number for all the tasks to be executed, and respectively setting a unique equipment number for the available node equipment, wherein each task number corresponds to one equipment number;
and distributing the task to be executed corresponding to the task number to the available node equipment corresponding to the equipment number corresponding to the task number.
5. The asynchronous task assignment method of claim 2, wherein said step of assigning tasks to said available node devices according to a second assignment rule comprises:
respectively setting unique and continuous task numbers for all the tasks to be executed;
dividing the task number of each task to be executed by the number of the available node devices, and then performing remainder operation to obtain a task remainder;
corresponding different task remainders to different available node equipment;
and distributing the task to be executed corresponding to each task remainder to corresponding available node equipment.
6. An asynchronous task allocation apparatus for allocating timed tasks to node devices in a cluster, the apparatus comprising:
the parameter acquisition module is suitable for acquiring information of tasks to be executed, the number of available node equipment and a batch execution upper limit, wherein the batch execution upper limit represents the maximum number of tasks processed by all the available node equipment in the cluster at one time;
the task allocation module is suitable for determining the tasks to be executed corresponding to each available node device according to the information of the tasks to be executed, the number of the available node devices and the upper limit of batch execution, so that the difference between the number of the tasks to be executed allocated on each available node device and the number of the tasks to be executed allocated on any one of the other available node devices is not greater than a first threshold value;
the mapping establishing module is suitable for establishing a mapping relation between the task to be executed and the available node equipment;
and the mapping storage module is suitable for storing the mapping relation into a memory of the available node equipment so that the available node equipment can process the task to be executed according to the mapping relation.
7. The asynchronous task assignment device of claim 6, wherein the task assignment module comprises:
the quantity judgment submodule is suitable for judging whether the quantity of all the tasks to be executed is greater than or equal to the batch execution upper limit;
the first allocation submodule is suitable for allocating tasks to the available node equipment according to a first allocation rule when the number of all the tasks to be executed is greater than or equal to the batch execution upper limit;
and the second distribution submodule is suitable for distributing the tasks to the available node equipment according to a second distribution rule when the number of all the tasks to be executed is smaller than the upper limit of batch execution.
8. The asynchronous task assignment device of claim 7, wherein the second assignment sub-module comprises:
the task numbering unit is suitable for respectively setting unique task numbers for all the tasks to be executed;
the equipment numbering unit is suitable for respectively setting unique equipment numbers for the available node equipment;
a matching unit adapted to correspond each task number to one of the device numbers;
and the task allocation unit is suitable for allocating the task to be executed corresponding to the task number to the available node equipment corresponding to the equipment number corresponding to the task number.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 5 are implemented by the processor when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910752334.7A CN110633135A (en) | 2019-08-15 | 2019-08-15 | Asynchronous task allocation method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910752334.7A CN110633135A (en) | 2019-08-15 | 2019-08-15 | Asynchronous task allocation method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110633135A true CN110633135A (en) | 2019-12-31 |
Family
ID=68969707
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910752334.7A Pending CN110633135A (en) | 2019-08-15 | 2019-08-15 | Asynchronous task allocation method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110633135A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111966502A (en) * | 2020-09-21 | 2020-11-20 | 北京百度网讯科技有限公司 | Method and device for adjusting number of instances, electronic equipment and readable storage medium |
| CN112650582A (en) * | 2020-12-21 | 2021-04-13 | 贝壳技术有限公司 | Distributed task processing method and system and processor |
| CN112965796A (en) * | 2021-03-01 | 2021-06-15 | 亿企赢网络科技有限公司 | Task scheduling system, method and device |
| CN113515357A (en) * | 2021-04-20 | 2021-10-19 | 建信金融科技有限责任公司 | Batch task execution method and device in cluster environment |
| CN113609168A (en) * | 2021-07-14 | 2021-11-05 | 远景智能国际私人投资有限公司 | Data export method, device, terminal and readable storage medium |
| CN113608847A (en) * | 2021-07-27 | 2021-11-05 | 远景智能国际私人投资有限公司 | Task processing method, device, equipment and storage medium |
| CN113626280A (en) * | 2021-06-30 | 2021-11-09 | 广东浪潮智慧计算技术有限公司 | Cluster state control method and device, electronic equipment and readable storage medium |
| CN114860505A (en) * | 2022-04-20 | 2022-08-05 | 中国科学院计算机网络信息中心 | Object storage data asynchronous backup method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103347055A (en) * | 2013-06-19 | 2013-10-09 | 北京奇虎科技有限公司 | System, device and method for processing tasks in cloud computing platform |
| CN105389212A (en) * | 2015-10-21 | 2016-03-09 | 浪潮(北京)电子信息产业有限公司 | Job assigning method and apparatus |
| CN107093046A (en) * | 2017-04-21 | 2017-08-25 | 北京京东尚科信息技术有限公司 | Unmanned dispensing vehicle method for allocating tasks, system and unmanned dispensing vehicle |
| CN107229511A (en) * | 2017-05-11 | 2017-10-03 | 东软集团股份有限公司 | Cluster task equalization scheduling method, device, storage medium and electronic equipment |
-
2019
- 2019-08-15 CN CN201910752334.7A patent/CN110633135A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103347055A (en) * | 2013-06-19 | 2013-10-09 | 北京奇虎科技有限公司 | System, device and method for processing tasks in cloud computing platform |
| CN105389212A (en) * | 2015-10-21 | 2016-03-09 | 浪潮(北京)电子信息产业有限公司 | Job assigning method and apparatus |
| CN107093046A (en) * | 2017-04-21 | 2017-08-25 | 北京京东尚科信息技术有限公司 | Unmanned dispensing vehicle method for allocating tasks, system and unmanned dispensing vehicle |
| CN107229511A (en) * | 2017-05-11 | 2017-10-03 | 东软集团股份有限公司 | Cluster task equalization scheduling method, device, storage medium and electronic equipment |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111966502A (en) * | 2020-09-21 | 2020-11-20 | 北京百度网讯科技有限公司 | Method and device for adjusting number of instances, electronic equipment and readable storage medium |
| CN112650582A (en) * | 2020-12-21 | 2021-04-13 | 贝壳技术有限公司 | Distributed task processing method and system and processor |
| CN112965796A (en) * | 2021-03-01 | 2021-06-15 | 亿企赢网络科技有限公司 | Task scheduling system, method and device |
| CN112965796B (en) * | 2021-03-01 | 2024-04-09 | 亿企赢网络科技有限公司 | Task scheduling system, method and device |
| CN113515357A (en) * | 2021-04-20 | 2021-10-19 | 建信金融科技有限责任公司 | Batch task execution method and device in cluster environment |
| CN113515357B (en) * | 2021-04-20 | 2024-03-08 | 建信金融科技有限责任公司 | Method and device for executing batch tasks in cluster environment |
| CN113626280A (en) * | 2021-06-30 | 2021-11-09 | 广东浪潮智慧计算技术有限公司 | Cluster state control method and device, electronic equipment and readable storage medium |
| CN113626280B (en) * | 2021-06-30 | 2024-02-09 | 广东浪潮智慧计算技术有限公司 | Cluster state control method and device, electronic equipment and readable storage medium |
| CN113609168B (en) * | 2021-07-14 | 2024-01-05 | 远景智能国际私人投资有限公司 | Data export method, device, terminal and readable storage medium |
| CN113609168A (en) * | 2021-07-14 | 2021-11-05 | 远景智能国际私人投资有限公司 | Data export method, device, terminal and readable storage medium |
| CN113608847A (en) * | 2021-07-27 | 2021-11-05 | 远景智能国际私人投资有限公司 | Task processing method, device, equipment and storage medium |
| CN114860505A (en) * | 2022-04-20 | 2022-08-05 | 中国科学院计算机网络信息中心 | Object storage data asynchronous backup method and system |
| CN114860505B (en) * | 2022-04-20 | 2022-12-23 | 中国科学院计算机网络信息中心 | Object storage data asynchronous backup method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110633135A (en) | Asynchronous task allocation method and device, computer equipment and storage medium | |
| CN111767143A (en) | Transaction data processing method, device, equipment and system | |
| CN108415925B (en) | Electronic device, data call log generation and query method and storage medium | |
| CN111225050B (en) | Cloud computing resource allocation method and device | |
| CN108446171B (en) | Electronic device, distributed system execution task allocation method and storage medium | |
| CN110162344B (en) | Isolation current limiting method and device, computer equipment and readable storage medium | |
| CN113886069A (en) | Resource allocation method and device, electronic equipment and storage medium | |
| CN110764913A (en) | Data calculation method based on rule calling, client and readable storage medium | |
| CN110751376B (en) | Work order distribution scheduling method and device, computer equipment and storage medium | |
| CN111580948A (en) | Task scheduling method and device and computer equipment | |
| CN109033184B (en) | Data processing method and device | |
| CN112698962A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN111988429A (en) | Algorithm scheduling method and system | |
| CN112364005A (en) | Data synchronization method and device, computer equipment and storage medium | |
| CN112860387A (en) | Distributed task scheduling method and device, computer equipment and storage medium | |
| CN109657317B (en) | CPLD pin allocation method, system and equipment | |
| CN115190010B (en) | Distributed recommendation method and device based on software service dependency relationship | |
| CN113342502B (en) | Performance diagnosis method and device for data lake, computer equipment and storage medium | |
| CN113127289B (en) | Resource management method, computer equipment and storage medium based on YARN cluster | |
| CN108363614B (en) | Application service module management method and device and server | |
| CN113010310A (en) | Job data processing method and device and server | |
| CN112506626A (en) | Application program starting method, computer equipment and storage medium | |
| CN111858542B (en) | Data processing method, device, equipment and computer readable storage medium | |
| CN110673931A (en) | Distributed calculation method for document synthesis, document synthesis system and control device thereof | |
| CN110674214A (en) | Big data synchronization method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |