CN111625364B

CN111625364B - Task allocation method, device, electronic equipment and computer readable medium

Info

Publication number: CN111625364B
Application number: CN202010480927.5A
Authority: CN
Inventors: 吕德超; 张相龙
Original assignee: Douyin Vision Co Ltd
Current assignee: Douyin Vision Co Ltd
Priority date: 2020-05-30
Filing date: 2020-05-30
Publication date: 2023-05-16
Anticipated expiration: 2040-05-30
Also published as: CN111625364A

Abstract

The disclosure provides a task allocation method, a task allocation device, electronic equipment and a computer readable medium, and relates to the technical field of data processing. The method comprises the following steps: receiving a task allocation request sent by one of a plurality of equipment nodes in a task cluster; distributing new tasks to a preset number of virtual nodes according to task distribution requests; and distributing the tasks to be distributed on the virtual nodes to the equipment nodes bound with the virtual nodes. After the new task is acquired, the task on the virtual node is distributed to the corresponding equipment nodes by distributing the task to the preset number of virtual nodes, the distribution mode does not need to be readjusted according to the number of the nodes each time, the task can be ensured to be acquired by one of the involved nodes in the cluster, the task is scheduled to be executed according to the set execution time, and failure processing work is done when the node fails, so that the task is ensured not to be lost due to the node failure.

Description

Task allocation method, device, electronic equipment and computer readable medium

Technical Field

The present disclosure relates to the technical field of data processing, and in particular, to a task allocation method, a device, an electronic apparatus, and a computer readable medium.

Background

The timing task, i.e. the task which needs to be scheduled to be executed according to the set time, such as a mobile phone alarm clock, is a typical timing task. The timing tasks are classified into a single timing task and a distributed timing task as a whole.

In the prior art, for distributed timing tasks, the solution mainly comprises a time-sharing scheme, a high-availability scheme, a multi-path heartbeat scheme, a task preemption scheme and a task polling scheme, and a single-machine-version Quartz (expanded tera formula) framework which extends based on the above schemes, wherein the single-machine-version Quartz framework has the problems of single-point faults, limited execution scale and the like, and the distributed scenes are gradually increased, so that the distributed Quartz is realized by means of a database on the basis of the single-machine-version Quartz, but for the scene with a large number of short tasks, each node can wait for a large number of threads due to preempting database locks, and resource waste is caused.

Therefore, the problems of single point failure, limited execution scale and the like exist in the prior art when the tasks are processed, and the technical problem of resource waste caused by a large number of threads waiting in each node in the distributed Quartz due to the preemption of the database lock is solved.

Disclosure of Invention

The purpose of the present disclosure is to solve at least one of the above technical drawbacks, and particularly, the prior art has the problems of single point failure, limited execution scale, etc. when processing a task, while each node in the distributed quantiz may wait for a large number of threads due to preempting the database lock, which causes a waste of resources.

In a first aspect, a task allocation method is provided, the method including:

receiving a task allocation request sent by one of a plurality of equipment nodes in a task cluster; the task cluster comprises a plurality of equipment nodes and virtual nodes, and each virtual node is bound with the equipment node; when the equipment node receives a new task, the task allocation request is sent out;

distributing the new tasks to a preset number of virtual nodes according to the task distribution requests;

and distributing the tasks to be distributed on the virtual node to the equipment node bound with the virtual node.

In a second aspect, there is provided a task allocation device, the device comprising:

the task allocation request acquisition module is used for receiving a task allocation request sent by one of a plurality of equipment nodes in the task cluster; the task cluster comprises a plurality of equipment nodes and virtual nodes, and each virtual node is bound with the equipment node; when the equipment node receives a new task, the task allocation request is sent out;

The first task allocation module is used for allocating the new tasks to a preset number of virtual nodes according to the task allocation request;

and the second task allocation module is used for allocating the task to be allocated on the virtual node to the equipment node bound with the virtual node.

In a third aspect, an electronic device is provided, the electronic device comprising:

one or more processors;

a memory;

one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications configured to: and executing the task allocation method.

In a fourth aspect, a computer readable medium is provided, where at least one instruction, at least one program, code set, or instruction set is stored, where at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the task allocation method described above.

After the new task is acquired, the task on the virtual node is distributed to the corresponding equipment nodes according to the number of the equipment nodes bound with the virtual node by firstly speaking the task and distributing the task to the preset number of the virtual nodes, the distribution mode does not need to be readjusted according to the number of the nodes each time, the task can be acquired by a pen-involved node in the cluster, the task is scheduled to be executed according to the set execution time, and when the node fails, failure processing work is done, so that the task is ensured not to be lost due to the node failure.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings that are required to be used in the description of the embodiments of the present disclosure will be briefly introduced below.

Fig. 1 is a schematic flow chart of a task allocation method according to an embodiment of the disclosure;

fig. 2 is a schematic structural diagram of a task cluster according to an embodiment of the present disclosure;

fig. 3 is a schematic flow chart of node registration according to an embodiment of the disclosure;

fig. 4 is a schematic diagram of a node registration step provided in an embodiment of the present disclosure;

fig. 5 is a schematic diagram of a task allocation method according to an embodiment of the present disclosure;

fig. 6 is a flow chart of a node change query method according to an embodiment of the disclosure;

fig. 7 is a schematic diagram of a node change query procedure according to an embodiment of the present disclosure;

fig. 8 is a schematic flow chart of node change synchronization according to an embodiment of the disclosure;

FIG. 9 is a schematic diagram illustrating steps for node change synchronization according to an embodiment of the present disclosure;

fig. 10 is a schematic flow chart of a node self-checking provided in an embodiment of the disclosure;

fig. 11 is a schematic structural diagram of a task allocation device according to an embodiment of the present disclosure;

fig. 12 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure.

The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure have been shown in the accompanying drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.

It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order and/or performed in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.

The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments. Related definitions of other terms will be given in the description below.

It should be noted that the terms "first," "second," and the like in this disclosure are used merely to distinguish one device, module, or unit from another device, module, or unit, and are not intended to limit the order or interdependence of the functions performed by the devices, modules, or units.

It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.

The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

The following describes the technical solutions of the present disclosure and how the technical solutions of the present disclosure solve the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present disclosure will be described below with reference to the accompanying drawings.

The embodiment of the disclosure provides a task allocation method, as shown in fig. 1, which includes:

step S101, receiving a task allocation request sent by one of a plurality of equipment nodes in a task cluster; the task cluster comprises a plurality of equipment nodes and virtual nodes, wherein each virtual node is bound with the equipment nodes; when the equipment node receives a new task, a task allocation request is sent out;

step S102, distributing new tasks to a preset number of virtual nodes according to task distribution requests;

and step S103, distributing the task to be distributed on the virtual node to the equipment node bound with the virtual node.

The task allocation method provided by the embodiment of the present disclosure is applied to a distributed task allocation cluster, as shown in fig. 2, where the distributed task allocation cluster includes a plurality of device points 201 and a database 202, and the task allocation method provided by the embodiment of the present disclosure is used for scheduling and managing the cluster, where the database uses a dis (Remote Dictionary Server, remote dictionary service) set database example, and negotiation between each node is completed by means of a special key value pair in the dis database, so as to perform task allocation.

In step S101, a task allocation request sent by one of a plurality of device nodes in a task cluster is received; the task cluster comprises a plurality of equipment nodes and virtual nodes, wherein each virtual node is bound with the equipment nodes; when the equipment node receives a new task, a task allocation request is sent out.

In the embodiment of the disclosure, the task allocation system includes a task cluster and a database 202, where the task cluster includes a plurality of device nodes 201, where a device node refers to a terminal device in the task cluster, such as a computer device, and the database may be a redis (Remote Dictionary Server, remote dictionary service) database, where a plurality of virtual nodes are further disposed in the task cluster, where the virtual nodes are bound to the device nodes.

For the disclosed embodiments, when any one or more device nodes of the task cluster receives a new task, a task allocation request is issued, and the task cluster receives the task allocation request. For convenience of explanation, an embodiment is described, where there are device nodes a1, a2, a3 in a task cluster, and when the node a1 receives a new task, a task allocation request is sent to the task cluster, and the task cluster receives the task allocation request and performs task allocation according to the task allocation request.

In step S102, a new task is allocated to a preset number of virtual nodes according to the task allocation request.

In the embodiment of the disclosure, a plurality of virtual nodes are arranged in a task cluster, and the number of the virtual nodes is fixed and is used for pre-distributing tasks.

For the embodiment of the disclosure, after receiving a task allocation request sent by a device node, a task cluster allocates a new task to the virtual node based on the task allocation request. For convenience of explanation, taking a specific embodiment as an example, n preset virtual nodes and m new tasks are taken, after a task cluster receives a task allocation request sent by an equipment node, the m new tasks are allocated to the n nodes, specifically, the n new tasks can be allocated in an average allocation manner or in a sequential allocation manner, which is not limited in the disclosure.

In step S103, the task to be allocated on the virtual node is allocated to the device node bound to the virtual node.

In the embodiment of the disclosure, a plurality of device nodes are in the task cluster, wherein one device node corresponds to one device terminal, and the device nodes are bound with the virtual nodes.

For the embodiment of the disclosure, after the task cluster distributes the new task to the virtual node, the task on the virtual node is distributed to the device nodes corresponding to the virtual node, where one device node may correspond to one or more virtual nodes. For convenience of explanation, taking a specific embodiment as an example, there are 6 virtual nodes and 2 device nodes, where one device node corresponds to 3 virtual nodes, after a task cluster allocates a new task to a virtual node, the virtual node allocates the new task to the corresponding device node.

After the new task is acquired, the task is distributed to the preset number of virtual nodes, the task on the virtual node is distributed to the corresponding equipment nodes according to the number of the equipment nodes bound with the virtual nodes, the distribution mode is not required to be readjusted according to the number of the nodes each time, the task can be acquired by a certain equipment node in the cluster, the task is scheduled to be executed according to the set execution time, and when the node fails, failure processing work is done, so that the task is ensured not to be lost due to the node failure.

The embodiment of the present disclosure provides a possible implementation manner, in this implementation manner, before receiving a task allocation request sent by one of a plurality of device nodes in a task cluster, as shown in fig. 3, further includes:

Step S301, a node registration request sent by a device node in a task cluster is received;

step S302, determining a first key value pair of a device node according to a node registration request, and synchronizing the first key value pair into a preset database; the first key value pair is used for representing the number information of the equipment node in the task cluster and the running state of the equipment node;

step S303, obtaining node information of a cluster, wherein the node information comprises the number of equipment nodes and the working state of the equipment nodes; and determining a second key value pair in the preset database based on the node information, wherein the value of the second key value pair comprises a version number for representing the version information of the task cluster and the total number of the device nodes of the number of the device nodes in the task cluster.

In the embodiment of the present disclosure, before the task is allocated, the device node in the task cluster needs to be registered in the task cluster, and the registration process of the device node is shown in fig. 4.

In step S301, a node registration request sent by a device node in a task cluster is received.

In the embodiment of the disclosure, when the equipment node performs registration, a node registration request is sent, and the task cluster receives the node registration request and starts node registration.

In step S302, determining a first key value pair of a device node according to a node registration request, and synchronizing the first key value pair to a preset database; the first key value pair is used for representing the number information of the equipment node in the task cluster and the running state of the equipment node.

In the embodiment of the present disclosure, for convenience of explanation, taking an embodiment as an example, each node in a task cluster corresponds to a first key value pair, where the first key value pair is used to represent number information of the device node in the task cluster and an operation state of the device node, and for convenience of explanation, a first key value pair that appears later in the disclosure is denoted by key1, and for a node, the key1 includes the number of nodes in the task cluster recorded by the node, that is, the number of local cache nodes.

For the embodiment of the disclosure, key1 is "{ psm }:index", wherein psm is a service identifier, and index is the number information of the device node and is used for indicating that the device node is a position in the task cluster; the value (key) of key1 is the running state of the cluster to be reported, such as the number of currently held tasks, the number of nodes of the cluster equipment, and the like. key1 is used to identify each device node in the node change query and the node change synchronization, each node corresponding to a unique index. The use of the hashTag mechanism for the psm ensures that key1 of the same service can be distributed to the same position in the cluster mode, reduces communication among nodes during inspection, and improves efficiency. As shown in fig. 4, when a key1 is registered, after a node starts up, taking registration of a device node as an example, the key1 of the node is determined, and index establishment in the key1 is initialized. With the setnx method of Redis, an attempt is made from index 1 until the setting is successful. After establishment, the value of the local version number version_local is initialized to-1, wherein the local version number is a field locally stored by the device node and is used for representing the version number of the task cluster. After the key1 is registered, the key1 is stored in a preset redis database.

In step S303, node information of the cluster is obtained, where the node information includes the number of device nodes and the working state of the device nodes; and determining a second key value pair in the preset database based on the node information, wherein the value of the second key value pair comprises a version number for representing the version information of the task cluster and the total number of the device nodes of the number of the device nodes in the task cluster.

In the embodiment of the present disclosure, one service corresponds to one second key value pair, and for convenience of explanation, a subsequent second key value pair in the present disclosure is denoted by key2, where a value of key2 includes a version number for indicating task cluster version information and a total number of device nodes of the number of device nodes in the task cluster.

For the embodiment of the disclosure, after registration of key1 is completed, the equipment node acquires node information of the acquired task cluster, and determines key2 according to the node information, wherein key2 is "psm: distribution: nodes: transform", psm is used for identifying a service, and suffix ": distribution: nodes: transform" is fixed and is only used for distinguishing different keys; the value of key2 is "version, node_sum_1, node_sum_2, node_sum_3", version corresponds to version, similar to the concept of any period in the raft protocol, node_sum_1 is the total number of current cluster nodes, node_sum_2 is used for identifying the number of nodes which finish node change query in the task cluster, node change query identification is initialized to be equal to node_sum_1, and when one node finishes node change query, node_sum_2 indicates that all equipment nodes in the task cluster finish node change query when node_sum_2 is reduced to 0; the node_sum_3 is used for identifying the number of the device nodes for completing the task return in the task cluster, initializing 0, adding 1 to the node_sum_3 every time one device node completes the task return, and when the node_sum_3 is consistent with the current node number, indicating that all the device nodes in the task cluster complete the task return, so that the task can be redistributed.

In the embodiment of the disclosure, when registering the equipment nodes, the number and the running state of the equipment nodes in the cluster are conveniently determined by the cluster through the key1 by constructing the key1 corresponding to the equipment nodes one by one and storing the key1 in the database, so that the subsequent task allocation is facilitated.

The embodiment of the disclosure provides a possible implementation method, in this implementation manner, when the second key value pair does not exist, determining the second key value pair in the preset database according to the node information, and further includes:

and creating and initializing a second key value pair according to the node information, and storing the second key value pair in a preset database.

In the embodiment of the present disclosure, when the device node that initiates the registration request is the first device node to register in the task cluster, and at this time, there is no key2 corresponding to the task cluster in the database, the device node is required to create a key2, and initialize the key2, as an embodiment of the present disclosure, the value of the initialized key2 is 0,1, 0.

In the embodiment of the disclosure, the registration condition of the first equipment node is considered, the first registered equipment node needs to create the key2, and the key2 is initialized, so that the normal operation of the cluster is ensured.

The embodiment of the present disclosure provides a possible implementation manner, in which, as shown in fig. 5, a new task is allocated to a preset number of virtual nodes according to a task allocation request, including:

obtaining the number of virtual nodes according to the task allocation request;

and distributing the new tasks to the virtual nodes through a hash algorithm according to the number of the virtual nodes.

Binding a preset number of virtual nodes with equipment nodes in a task cluster, wherein the binding comprises the following steps:

acquiring the number of equipment nodes in a task cluster;

and binding the virtual nodes distributed to the new tasks with the equipment nodes through a hash algorithm according to the number of the equipment nodes.

In the embodiment of the disclosure, when a new task is allocated to a node to be virtualized, the number of virtual nodes needs to be acquired first, as shown in fig. 5, after the number of virtual nodes is acquired, the new task is allocated to the virtual nodes through a hash algorithm, then the number of device nodes is acquired, and the task in the virtual nodes is allocated to the device nodes corresponding to the virtual nodes through the hash algorithm. For convenience of explanation, taking a specific embodiment as an example, as shown in fig. 5, n virtual nodes are provided, m device nodes are provided, after new tasks are allocated to the virtual nodes, the tasks in the virtual nodes are allocated to the device nodes according to the corresponding relationship between the virtual nodes and the device nodes.

The embodiment of the disclosure provides a possible implementation manner, in which the task allocation method further includes: monitoring whether node information changes; responding to the node information change, recovering the task to be completed of each equipment node, and reallocating the task to be completed based on the changed node information.

In the embodiment of the present disclosure, the task allocation method provided by the embodiment of the present disclosure is further used to monitor whether node information in a task cluster changes, and when the node information changes, recover a task to be completed by each device node, and redistribute the task to be completed based on the changed node information.

For the embodiment of the disclosure, for convenience of explanation, taking an embodiment as an example, each device node in the task cluster monitors node information of the task cluster, where the node information includes, but is not limited to, the number of device nodes in the task cluster, an operation state of each device node, and the like, and if a certain node monitors that the number of nodes in the task cluster changes, the task cluster redistributes the task according to the changed number of device nodes.

According to the embodiment of the disclosure, the node information of the task cluster is monitored, and when the node information in the task cluster is monitored to change, the tasks are redistributed according to the changed node information, so that the tasks can be executed.

The disclosed embodiments provide a possible implementation manner, in which, as shown in fig. 6, monitoring whether node information changes includes: each of the plurality of device nodes periodically triggers a node change query.

In the embodiment of the present disclosure, there are a plurality of device nodes in a task cluster, each device node performs a periodic node change query, where the period may be manually set as needed, and the node change query includes:

In step S601, the current query node acquires a first key value pair of each of the plurality of nodes, and determines whether the value of the first key value pair of each of the plurality of nodes is continuous.

In the embodiment of the disclosure, each device node in the task cluster performs node change query, and for convenience of description, a query process of one node is taken as an example to describe, as shown in fig. 7, when a current query node performs node change query, first, a key1 of the same psm in the cluster is checked, and a node state in the cluster is obtained. And judging whether the values of all the first key value pairs are continuous or not according to the searched cluster state, if the values of the first key value pairs are in the PSm 1 and the PSm 3 in the Redis, the PSm 2 is missing, and the fact that the values of the first key value pairs are discontinuous is indicated.

In step S602, in response to the discontinuity, a node information change is determined, a second key value pair is acquired, and the total number of nodes in the second key value pair is updated to be the current total number of nodes.

In the embodiment of the present disclosure, when the value of the first key value pair is found to be discontinuous, it is determined that node information in the cluster is changed, a corresponding second key value pair key2 in the cluster needs to be obtained, and the total number of nodes in the key2 is modified to be the current total number of nodes, where the current total number of nodes is the number of keys 1 queried by the current query node in the task cluster. For convenience of explanation, taking a specific embodiment as an example, the number of key1 queried by the current query node is 5, and the values of key1 are respectively psm 1, psm 2, psm 4, psm 5 and psm 6, i.e. the values of key1 are discontinuous, then the key2 needs to be updated; at this time, key2 is acquired, and after the last change, the values of key2 are version=4, node_sum_1=6, node_sum_2=6, and node_sum_3=0, and based on the query result of key1 above, the total number of nodes of key2 is updated to be the current total number of nodes, that is, node_sum_1=5 of key2 is updated.

In step S603, the total number of nodes stored in the current query node is updated.

In the embodiment of the present disclosure, the total number of nodes locally used to represent the number of device nodes in the cluster is stored in the current query node, and under the condition that the system operates normally, the value of the first key value pair is discontinuous, which indicates that the node is changed, and that the total number of nodes stored in the current query node is inconsistent with the total number of the current node may be caused, and the total number of nodes stored in the current query node needs to be modified to be consistent with the current node number.

According to the task cluster device node, the device nodes of the task cluster can find out in time and modify the second key value pair when the nodes in the task cluster change through periodical node change inquiry, and accuracy of the cluster on the cognition of the number of the device nodes is guaranteed.

The embodiments of the present disclosure provide a possible implementation manner, in which the node change query further includes:

in response to the continuation, determining whether a minimum value of the plurality of first key-value pairs median is a ranking start value;

in response to not being a ranking start value, changing a maximum value of the median of the plurality of first key-value pairs to the ranking start value;

and circularly executing node change inquiry until the values in the plurality of first key value pairs are continuous and the minimum value is the ordering starting value.

In the embodiment of the disclosure, as shown in fig. 7, when the ranking start value is 1 and the values of all keys 1 obtained by the current query node are continuous, it is determined whether the minimum value of the values of the keys 1 is 1, when the minimum value is not 1, the maximum value of the keys 1 is changed to 1, and whether the values of the keys 1 are continuous is repeatedly maintained, when the values of the keys 1 are discontinuous, the maximum value of the keys 1 is filled into the discontinuous place until the values of the keys 1 are continuous and the minimum value is 1. For convenience of explanation, taking a specific embodiment as an example, the values of key1 obtained by the current query node are respectively in the form of psm 3, psm 4, psm 5, psm 6 and psm 7, when the values of key1 are continuous but the minimum value is not in the form of 1, the values of key1 after modification are in the form of psm 1, psm 3, psm 4, psm 5 and psm 6, when the minimum value of key1 is in the form of 1 but the values of key1 after modification are not continuous are in the form of psm 2, and when the values of key1 after modification are in the form of psm 1, psm 2, psm 3, psm 4 and psm 5, the values of key1 are continuous and the minimum value is 1.

According to the method and the device for adjusting the value of the key1, the value of the key1 is guaranteed to be continuous, the minimum value of the key1 is 1, the maximum value of the key1 is guaranteed to be the number of nodes in the task cluster, and follow-up task allocation is facilitated.

in response to the values in the plurality of first key value pairs being consecutive and the minimum value being 1, determining whether the maximum value in the plurality of first key value pairs is equal to the total number of nodes stored in the current query node;

responding to the maximum value being greater than the total number of the stored nodes, determining node information change, acquiring a second key value pair, and updating the total number of the nodes in the second key value pair to be consistent with the maximum value;

and determining node information change in response to the maximum value being greater than the total number of the stored nodes, acquiring a second key value pair, and updating the total number of the nodes in the second key value pair to be consistent with the maximum value.

In the embodiment of the disclosure, when the key1 value is continuous and the minimum value is 1, the maximum value of the key1 value is designated as the current node number, the maximum value is judged to be equal to the total number of nodes stored in the current query node, if the maximum value is equal to the total number of nodes stored in the current query node, the node number in the task cluster is not changed, and adjustment of the key2 is not needed, if the node number is not equal, the change of the node number in the task cluster is indicated, the distribution of the task is influenced, and the key2 is needed to be modified.

In the embodiment of the present disclosure, as shown in fig. 7, when the maximum value of key1 is greater than the total number of nodes stored in the current query node, it is determined that node information in the task cluster changes, a second key value pair is obtained, and the total number of nodes in the second key value pair is updated to be consistent with the maximum value. When the maximum value of key1 is smaller than the total number of nodes stored in the current query node, determining that node information in the task cluster changes, acquiring a second key value pair, and updating that the total number of nodes in the second key value pair is consistent with the maximum value.

According to the method and the device for processing the task cluster, whether the maximum value in the key1 is consistent with the total number of the nodes stored in the current query node is compared, if the maximum value is inconsistent with the total number of the nodes stored in the current query node, the node information of the task cluster is changed, the total number of the nodes in the key2 is modified to be consistent with the maximum value, and the cognition accuracy of the task cluster on the number of the device nodes is guaranteed.

The embodiment of the disclosure provides a possible implementation manner, in which, as shown in fig. 8, in response to a node information change, recovering tasks to be completed by each device node includes:

in step S801, a node change synchronization request sent by a current query node is received in response to a node information change; the node change synchronization request includes a second key value pair version number change request, a node total number change request, a local cache node number change request, and a local version number change request.

In the embodiment of the disclosure, when the node information of the task cluster is determined to change, the task to be completed on the equipment node needs to be recovered, and before that, the cognition of all the equipment nodes in the task cluster on the node information of the cluster needs to be ensured to be synchronous, and the node change synchronization of the equipment nodes needs to be performed.

For the embodiment of the disclosure, in response to the change of the node information, a node change synchronization request sent by a current query node is received, wherein the node change synchronization request comprises a second key value pair version number change request, a node total number change request, a local cache node number change request and a local version number change request.

In step S802, a node change instruction is sent to the current query node according to the node change synchronization request, so that the current query node performs node change synchronization according to the node information change request.

In the embodiment of the disclosure, a node change instruction is sent to a current query node according to a node change synchronization request, so that the current query node performs node change synchronization according to the node information change request. For convenience of explanation, taking an embodiment as an example, after the total number of nodes in the second key pair is modified, the current query node needs to modify the version number in key2 to be consistent with the local version number, and modify the total number of nodes stored in the current query node to be consistent with the current node number.

According to the embodiment of the disclosure, before task redistribution, node change synchronization is needed, and the cognition consistency of all equipment nodes in a task cluster on cluster node information is ensured.

The disclosed embodiments provide a possible implementation manner, in which node change synchronization includes: receiving a second key value pair acquisition request sent by any equipment node; transmitting the second key value pair to any equipment node so as to enable any node to modify node information; receiving a node change synchronization identification change request sent by any equipment node; and changing the node change synchronization identifier according to the node change synchronization identifier change request. When the node change synchronization mark indicates that all the equipment nodes in the task cluster do not complete synchronization, repeating the node change synchronization until the node change synchronization mark indicates that all the equipment nodes in the task cluster complete the node change synchronization

In the embodiment of the present disclosure, the node change synchronization identifier for indicating the progress of node change synchronization is also included in the second key value pair key2, and is node_sum_2, as shown in fig. 9, as one embodiment of the present disclosure, first, the value of key2 is obtained from Redis, and whether node_sum_1 in key2 is consistent with the local node_sum_local is compared, and the next round of node change synchronization is not awaited.

Then compare if version in key2 and local version_local agree: if not, it means that a node change is required. First, the transform flag is turned on, the own version number is modified, and node_sum_2 in key2 is updated, the value of node_sum_2 is reduced by 1, and the version number in key2 is increased by 1. The transform flag here is used to identify that the current node needs to be changed. After the change, the value of key2 is checked in a loop until the query result is node_sum_2=0. Since all device nodes update node_sum_2 when synchronizing, it means that all nodes perceive a node change when node_sum_2=0.

According to the embodiment of the disclosure, before task redistribution, the total number and version number of the nodes locally cached by the equipment nodes and the total number and version number of the nodes in the key2 are synchronized, so that the cognition consistency of all the equipment nodes to the cluster is ensured.

The embodiment of the disclosure provides a possible implementation manner, in the implementation manner, when all equipment nodes in a task cluster complete synchronization, task redistribution is triggered, and the redistribution of the tasks to be completed based on the changed node information comprises the following steps: receiving an unexecuted task allocation request sent by a device node in a task cluster; distributing the unexecuted tasks to a preset number of virtual nodes according to unexecuted task distribution requests; binding a preset number of virtual nodes with equipment nodes in a task node cluster; and distributing the unexecuted tasks on the virtual nodes to the device nodes bound with the virtual nodes.

In the embodiment of the disclosure, before tasks are allocated, enough virtual nodes are constructed, by means of firstly hashing all tasks onto the virtual nodes and then unbinding different virtual nodes by equipment nodes, the tasks in the system have four different states of unallocated (undo), to-be-executed (todo), executing (doing) and executing completion (done), and operations need to be performed on the tasks in different task states, for example, periodic task acquisition threads can acquire tasks in all the undo states and update the tasks into todo states, the task states are put into keys, and task state change is realized by directly moving data between different keys. The task storage structure in the Redis cluster is designed as follows: as shown in table 1, the integrated task storage structure is an integrated task storage structure, and is stored in a Set mode, wherein "psm" in a key refers to a service; "n" refers to the number of virtual nodes, and when submitted, the virtual nodes will be hashed with tasks; "undo" is used to identify that the task state is unassigned, and value is the json string set of tasks that hash to the current key.

TABLE 1 Indo State task Key value vs. Specification Table

Similarly, the task storage structures of the todo state and the doing state are shown in table 2 and table 3 respectively, only the suffix of the key is changed, and the other is consistent with the undo, and the done type is not separately set for storage, and the execution is completed and the database is directly deleted.

TABLE 2 todo State task Key value vs. Specification Table

TABLE 3 DOING State task Key value vs. Specification Table

In the three types of keys, n is wrapped by using { } to ensure the tasks of the same hashIndex (index after hash according to a virtual node) in three states by using the hashTag mechanism provided by Redis, in the Redis cluster, the tasks can be distributed to the same slot, the state change is realized by moving among different keys, and the same slot can be ensured to be on the same Redis node, so that time delay caused by moving among different nodes can be avoided during moving, and the system performance is optimized. In summary, the task allocation flow provided by the embodiment of the present disclosure includes: the timing task hashes to the virtual node through a certain Hash algorithm according to the task ID, and stores the Hash into a psm_ { n } _ undo set corresponding to the virtual node; each node periodically acquires threads, and acquires unassigned tasks from the psm_ { n } _ undo set according to the corresponding rules of the equipment nodes and the virtual nodes and the node index determined in the registration stage and the total number of nodes acquired in the inspection stage; and moving the task from the psm_ { n } -undo set to the psm_ { n } -todo set, indicating that it has been allocated; after the task is acquired, adding the task into a local priority queue, and waiting for the execution of the execution process to schedule; if cluster change occurs, the task in execution is moved from the psm_ { n } _ todo set to the psm_ { n } _ doing set, and repeated allocation is avoided. And after all the nodes are negotiated, re-acquiring the nodes from the psm_ { n } _ todo set by using the new index and the total number of the nodes, thereby completing the re-allocation logic, adding the tasks into a local priority queue after re-executing the acquisition, waiting for the scheduling execution of the execution process, and deleting the tasks from the Redis set after the execution of the tasks is completed.

According to the embodiment of the disclosure, all tasks are hashed to the virtual node according to the task ID, and when the node needs to be changed each time, the task does not need to be hashed again, so that the system performance is improved.

The embodiment of the present disclosure provides a possible implementation manner, in this implementation manner, as shown in fig. 10, a first key value pair of one device node in a task cluster further includes a survival time of the one device node, and each device node in the task cluster performs periodic self-checking, where the periodic self-checking includes:

in step S1001, when heartbeat information sent by one device node is received in the survival time, the survival time is updated.

In the embodiment of the present disclosure, the key1 further includes an update time ttl of the key1, which is stored in the Redis database, and when heartbeat information of a node is received at least once in the ttl time period, the node is determined to be valid, and the ttl of the node in the Redis database is updated. If ttl is set to 2 seconds, the sending interval of the heartbeat information is 500ms, and for the node a, if the node a receives the heartbeat information of the node a at least once in the Redis database for 2 seconds, the ttl of key1 corresponding to the node a is updated when the heartbeat information is received, and the node a is determined to be in effect.

In step S1002, when heartbeat information sent by one device node is not received in the alive time, a first key value pair corresponding to the one device node is deleted in the database.

In the embodiment of the present disclosure, the key1 further includes an update time ttl of the key1, which is stored in the Redis database, and when no heartbeat information of a node is received in the ttl time period, the node is determined to be valid, and the ttl of the node in the Redis database is updated. If ttl is set to 2 seconds and the sending interval of the heartbeat information is 500ms, for the node a, if the heartbeat information of the node a is not received in 2 seconds in the Redis database, deleting key1 of the node a, and judging that the node a fails.

According to the embodiment of the disclosure, the node state is known in time by self-checking the nodes in the cluster, so that task allocation is convenient according to the number of the nodes.

In the embodiment of the disclosure, after a node acquires a task, the task is required to be executed, and after a certain node acquires the task, the task is added into a local task priority queue; the local task execution thread acquires the task from the priority queue according to the task execution time and submits the task to the cooperative pool for execution; the cooperative program pool work cooperative program obtains callback function execution timing tasks from the local cache; and deleting the timing task after the execution is successful. For periodic timing tasks, the next timing task is parsed and added to the database. The task execution module mainly comprises three parts, namely a main thread of the task execution module, a task priority queue and a cooperative pool. The main task execution thread is responsible for executing the unified scheduling task, and the main logic is to continuously acquire the task which needs to be executed recently from the task priority queue, and submit the task to the coroutine pool for execution when the execution time is reached. And if the submission is successful, acquiring the task from the task priority queue again, and repeating the steps. If the coroutine pool is full, then the block waits until the commit is successful. In addition, the main thread also provides interfaces to other modules. Initializing, namely initializing a main thread, a task priority queue and a coroutine pool; monitoring a newly added task request, namely receiving a task acquired by a periodic task acquisition thread, adding the task into a priority queue, and updating the next execution time; the method comprises the steps of monitoring a task returning request, namely receiving the task returning request triggered by a task distribution module, blocking a main thread scheduling task, marking the executing task in a Redis (normal logic is marking the unexecuted task in the Redis, and performing reassignment when the former task is distributed conveniently).

In the embodiment of the disclosure, at the beginning of triggering the task return, the task marked in redis and not yet executed in the last round needs to be removed, and the subsequent re-marking is waited, so that the task cannot be redistributed due to the fact that the node hangs up after the task is marked as being executed, but only needs to be executed once, and therefore key3 is introduced for distributed control. Only one key3 is used for controlling task cleaning. Wherein the key is "psm: reset: doing", wherein the psm identifies the service, and the suffix "reset: doing" is fixed; value is "version" to identify which version the now clear identification is executed. The clearing of the in-execution flag is only performed if version in key3 is less than version_local, otherwise, it is indicated that other nodes have already executed, and no further clearing is required. The priority queue is implemented by a minimum heap, and task execution time is used as a sorting standard. The most recently executed will be placed on top of the heap, and is scheduled for execution by the main thread preferentially.

In the embodiment of the disclosure, the coroutine pool is responsible for managing coroutines executed by all tasks, and internally realizes capacity expansion and capacity contraction mechanisms. When the amount of tasks executed at the same time is too large, the capacity can be automatically expanded; when no task is executed for a long time, the capacity is automatically contracted. Thus optimizing node memory and cpu usage and providing better service.

Embodiments of the present disclosure provide a task assigning apparatus, as shown in fig. 11, the task assigning apparatus 110 may include: a task allocation request acquisition module 1101, a first task allocation module 1102, and a second task allocation module 1103, wherein,

a task allocation request obtaining module 1101, configured to receive a task allocation request sent by one of a plurality of device nodes in a task cluster; the task cluster comprises a plurality of equipment nodes and virtual nodes, wherein each virtual node is bound with the equipment nodes; when the equipment node receives a new task, a task allocation request is sent out;

A first task allocation module 1102, configured to allocate a new task to a preset number of virtual nodes according to a task allocation request;

the second task allocation module 1103 is configured to allocate a task to be allocated on the virtual node to a device node bound to the virtual node.

Optionally, the task allocation request obtaining module 1101 provided in the embodiment of the present disclosure may be further configured, before receiving a task allocation request sent by one of a plurality of device nodes in a task cluster, to: receiving a node registration request sent by a device node in a task cluster;

determining a first key value pair of the equipment node according to the node registration request, and synchronizing the first key value pair into a preset database; the first key value pair is used for representing the number information of the equipment node in the task cluster and the running state of the equipment node;

acquiring node information of a cluster, wherein the node information comprises the number of equipment nodes and the working state of the equipment nodes; and determining a second key value pair in the preset database based on the node information, wherein the value of the second key value pair comprises a version number for representing the version information of the task cluster and the total number of the device nodes of the number of the device nodes in the task cluster. Optionally, when the second key value pair does not exist, the task allocation request obtaining module 1101 provided in the embodiment of the present disclosure may be configured to:

Optionally, when the first task allocation module 1102 provided in the embodiments of the present disclosure allocates a new task to a preset number of virtual nodes according to a task allocation request, the first task allocation module may be configured to:

acquiring the number of equipment nodes in a task cluster;

binding the virtual nodes distributed to the new tasks with the equipment nodes through a hash algorithm according to the number of the equipment nodes, and distributing the tasks on the virtual nodes to the equipment nodes corresponding to the virtual nodes.

Optionally, the task allocation device provided by the embodiment of the present disclosure may further be used to:

monitoring whether node information changes; responding to the node information change, recovering the task to be completed of each equipment node, and reallocating the task to be completed based on the changed node information.

Optionally, when the information of the listening node changes, the task allocation device provided in the embodiment of the present disclosure may be used to:

the current query node acquires a first key value pair of each of the plurality of nodes and judges whether the value of the first key value pair of each of the plurality of nodes is continuous or not;

responding to the discontinuity, determining node information change, acquiring a second key value pair, and updating the total number of nodes in the second key value pair to be the current total number of nodes;

Updating the total number of nodes stored in the current query node.

Optionally, when performing the task allocation query, the task allocation device provided by the embodiment of the present disclosure may be used to:

in response to the succession, determining whether a minimum value of the plurality of first key-value pairs median is 1;

changing a maximum value of the median of the plurality of first key values to 1 in response to not being 1;

the node change query is circularly executed until the values in the plurality of first key value pairs are continuous and the minimum value is 1.

Optionally, when responding to the change of the node information, the task allocation device provided by the embodiment of the present disclosure may be used to recycle the task to be completed for each device node:

Responding to the node information change, and recovering tasks to be completed of each equipment node comprises the following steps:

responding to the node information change, and receiving a node change synchronization request sent by a current query node; the node change synchronization request comprises a second key value pair version number change request, a node total number change request, a local cache node number change request and a local version number change request;

and sending a node change instruction to the current query node according to the node change synchronization request so as to enable the current query node to perform node change synchronization according to the node information change request.

Optionally, when performing node change synchronization, the task allocation device provided by the embodiment of the present disclosure may be used to:

receiving a second key value pair acquisition request sent by any equipment node;

transmitting the second key value pair to any equipment node so as to enable any node to modify node information; receiving a node change synchronization identification change request sent by any equipment node;

and changing the node change synchronization identifier according to the node change synchronization identifier change request.

and repeating the node change synchronization when the node change synchronization mark indicates that all the equipment nodes in the task cluster do not complete the synchronization, until the node change synchronization mark indicates that all the equipment nodes in the task cluster complete the node change synchronization.

Optionally, the task allocation device provided in the embodiment of the present disclosure reallocates the task to be completed based on the changed node information. Can be used for:

receiving an unexecuted task allocation request sent by a device node in a task cluster;

distributing the unexecuted tasks to a preset number of virtual nodes according to unexecuted task distribution requests;

binding a preset number of virtual nodes with equipment nodes in a task node cluster;

and distributing the unexecuted tasks on the virtual nodes to the device nodes bound with the virtual nodes.

updating the survival time when the heartbeat information sent by one equipment node is received in the survival time;

and deleting a first key value pair corresponding to one equipment node in the database when heartbeat information sent by the one equipment node is not received in the survival time.

The task allocation device according to the embodiments of the present disclosure may perform the task allocation method shown in the foregoing embodiments of the present disclosure, and the implementation principle is similar, and will not be described herein.

Referring now to fig. 12, a schematic diagram of an electronic device 1200 suitable for use in implementing embodiments of the present disclosure is shown. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 12 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.

An electronic device includes: a memory and a processor, where the processor may be referred to as a processing device 1201, and the memory may include at least one of a Read Only Memory (ROM) 1202, a Random Access Memory (RAM) 1203, and a storage device 1208, as follows:

as shown in fig. 12, the electronic apparatus 1200 may include a processing device (e.g., a central processor, a graphics processor, etc.) 1201, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 1202 or a program loaded from a storage device 1208 into a Random Access Memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the electronic apparatus 1200 are also stored. The processing device 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input/output (I/O) interface 1205 is also connected to the bus 1204.

In general, the following devices may be connected to the I/O interface 1205: input devices 1206 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, and the like; an output device 1207 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 1208 including, for example, magnetic tape, hard disk, etc.; and a communication device 1209. The communication means 1209 may allow the electronic device 1200 to communicate wirelessly or by wire with other devices to exchange data. While fig. 12 shows an electronic device 1200 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.

In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a non-transitory computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication device 1209, or installed from the storage device 1208, or installed from the ROM 1202. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by the processing device 1201.

It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable medium, or any combination of the two. The computer readable medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may be any computer readable medium that is not a computer readable medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.

In some implementations, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: receiving a task allocation request sent by one of a plurality of equipment nodes in a task cluster; the task cluster comprises a plurality of equipment nodes and virtual nodes, wherein each virtual node is bound with the equipment nodes; when the equipment node receives a new task, a task allocation request is sent out; distributing new tasks to a preset number of virtual nodes according to task distribution requests; and distributing the tasks to be distributed on the virtual nodes to the equipment nodes bound with the virtual nodes.

Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

According to one or more embodiments provided by the present disclosure, there is provided a task allocation method including:

receiving a task allocation request sent by one of a plurality of equipment nodes in a task cluster; the task cluster comprises a plurality of equipment nodes and virtual nodes, wherein each virtual node is bound with the equipment nodes; when the equipment node receives a new task, a task allocation request is sent out;

distributing new tasks to a preset number of virtual nodes according to task distribution requests;

and distributing the tasks to be distributed on the virtual nodes to the equipment nodes bound with the virtual nodes.

Further, before receiving the task allocation request sent by one of the plurality of device nodes in the task cluster, the method further includes: receiving a node registration request sent by a device node in a task cluster;

acquiring node information of a cluster, wherein the node information comprises the number of equipment nodes and the working state of the equipment nodes; and determining a second key value pair in the preset database based on the node information, wherein the value of the second key value pair comprises a version number for representing the version information of the task cluster and the total number of the device nodes of the number of the device nodes in the task cluster. Further, when the second key value pair does not exist, determining the second key value pair in the preset database according to the node information, and further including:

Further, the task allocation request is used for allocating the new task to the virtual nodes with the preset number, which comprises the following steps:

obtaining the number of virtual nodes according to the task allocation request;

Further, the task to be allocated on the virtual node is allocated to the equipment node bound with the virtual node, which comprises the following steps:

acquiring the number of equipment nodes in a task cluster;

Further, the method further comprises:

Further, monitoring whether the node information changes includes:

each of the plurality of device nodes periodically triggers a node change query, the node change query comprising:

updating the total number of nodes stored in the current query node.

Further, the node change query further includes:

Further, responding to the node information change, recovering tasks to be completed of each equipment node comprises:

Further, the node change synchronization includes:

Further, the method further comprises:

Further, when all the equipment nodes in the task cluster complete synchronization, triggering task redistribution, and redistributing the tasks to be completed based on the changed node information comprises the following steps:

Further, the first key value pair of one device node in the task cluster further includes survival time of the one device node, and each device node in the task cluster performs periodic self-checking, where the periodic self-checking includes:

According to one or more embodiments provided by the present disclosure, there is provided a task allocation device including:

The task allocation request acquisition module is used for receiving a task allocation request sent by one of a plurality of equipment nodes in the task cluster; the task cluster comprises a plurality of equipment nodes and virtual nodes, wherein each virtual node is bound with the equipment nodes; when the equipment node receives a new task, a task allocation request is sent out;

the first task allocation module is used for allocating new tasks to a preset number of virtual nodes according to task allocation requests;

Optionally, the task allocation request obtaining module 1201 provided in the embodiment of the present disclosure may be further configured, before receiving a task allocation request sent by one of a plurality of device nodes in a task cluster, to: receiving a node registration request sent by a device node in a task cluster;

acquiring node information of a cluster, wherein the node information comprises the number of equipment nodes and the working state of the equipment nodes; and determining a second key value pair in the preset database based on the node information, wherein the value of the second key value pair comprises a version number for representing the version information of the task cluster and the total number of the device nodes of the number of the device nodes in the task cluster. Optionally, when the second key value pair does not exist, the task allocation request obtaining module 1201 provided in the embodiment of the present disclosure may be configured to:

Optionally, when the first task allocation module 1202 provided in the embodiments of the present disclosure allocates a new task to a preset number of virtual nodes according to a task allocation request, the first task allocation module may be configured to:

acquiring the number of equipment nodes in a task cluster;

Updating the total number of nodes stored in the current query node.

According to one or more embodiments provided by the present disclosure, there is provided an electronic device including:

one or more processors;

a memory;

According to one or more embodiments provided by the present disclosure, there is provided a computer readable medium storing at least one instruction, at least one program, code set, or instruction set, the at least one instruction, at least one program, code set, or instruction set being loaded and executed by a processor to implement the task allocation method described above.

The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by persons skilled in the art that the scope of the disclosure referred to in this disclosure is not limited to the specific combinations of features described above, but also covers other embodiments which may be formed by any combination of features described above or equivalents thereof without departing from the spirit of the disclosure. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).

Moreover, although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are example forms of implementing the claims.

Claims

1. A method of task allocation, comprising:

distributing the new task on the virtual node to the equipment node bound with the virtual node;

before receiving the task allocation request sent by one of the plurality of device nodes in the task cluster, the method further includes: receiving a node registration request sent by a device node in the task cluster;

Determining a first key value pair of the equipment node according to the node registration request, and synchronizing the first key value pair into a preset database;

and acquiring node information of the cluster, and determining a second key value pair in the preset database based on the node information.

2. The task allocation method according to claim 1, wherein the first key value pair is used to represent number information of the device node in the task cluster and an operation state of the device node; the node information comprises the number of the equipment nodes and the working state of the equipment nodes; the value of the second key value pair includes a version number indicating version information of the task cluster and a total number of device nodes of the number of device nodes in the task cluster.

3. The method of claim 2, wherein when the second key-value pair does not exist, the determining the second key-value pair in the preset database according to the node information further comprises:

and creating and initializing the second key value pair according to the node information, and storing the second key value pair in the preset database.

4. The method of claim 1, wherein the assigning the new task to a predetermined number of virtual nodes according to the task assignment request comprises:

Acquiring the number of the virtual nodes according to the task allocation request;

5. The method of claim 1, wherein the assigning the task to be assigned on the virtual node to the device node bound to the virtual node comprises:

acquiring the number of equipment nodes in a task cluster;

6. The task allocation method according to claim 2, further comprising:

monitoring whether the node information changes; and responding to the node information change, recovering the task to be completed of each equipment node, and reallocating the task to be completed based on the changed node information.

7. The task allocation method according to claim 6, wherein said monitoring whether the node information is changed comprises:

The current query node acquires the first key value pair of each of the plurality of nodes and judges whether the value of the first key value pair of each of the plurality of nodes is continuous or not;

responding to discontinuity, determining the node information change, acquiring the second key value pair, and updating the total number of nodes in the second key value pair to be the current total number of nodes;

updating the total number of nodes stored in the current query node.

8. The task allocation method according to claim 7, wherein the node change query further comprises:

in response to the continuation, determining whether a minimum value of the median of the plurality of the first key-value pairs is a ranking start value;

in response to not being a ranking start value, changing a maximum value of a median of the plurality of first key-value pairs to a ranking start value;

and circularly executing the node change query until the values in the plurality of first key value pairs are continuous and the minimum value is a sequencing starting value.

9. The task allocation method according to claim 8, wherein the node change query further comprises:

And in response to the maximum value being greater than the total number of stored nodes, determining the node information change, acquiring the second key value pair, and updating that the total number of nodes in the second key value pair is consistent with the maximum value.

10. The task allocation method according to claim 7, wherein reclaiming the device nodes to complete tasks in response to the node information change comprises:

responding to the node information change, and receiving a node change synchronization request sent by the current query node; the node change synchronization request comprises a second key value pair version number change request, a node total number change request, a local cache node number change request and a local version number change request;

11. The task allocation method according to claim 10, wherein the node change synchronization includes:

receiving a second key value pair acquisition request;

transmitting the second key value pair to a current node change synchronization equipment node so that the current node change synchronization equipment node modifies the node information; receiving a node change synchronization identification change request sent by any equipment node;

And changing the node change synchronous identification according to the node change synchronous identification change request.

12. The method as recited in claim 11, further comprising:

judging whether the node change synchronization mark is that each equipment node in the cluster has completed synchronization or not;

and in response to the incompletion, repeating the node change synchronization until the node change synchronization identifier indicates that all the device nodes in the task cluster complete the node change synchronization.

13. The method of claim 12, wherein triggering a task reassignment in response to having completed, the reassigning the task to be completed based on the changed node information comprises:

receiving an unexecuted task allocation request sent by a device node in the task cluster;

distributing the unexecuted tasks to a preset number of virtual nodes according to the unexecuted task distribution request;

binding the preset number of virtual nodes with equipment nodes in the task node cluster;

and distributing the unexecuted task on the virtual node to the equipment node bound with the virtual node.

14. The method of claim 2, wherein the first key-value pair of one device node in the task cluster further comprises a time-to-live of the one device node, each device node in the task cluster performing a periodic self-test comprising:

Updating the survival time when heartbeat information sent by the equipment node is received in the survival time;

and deleting a first key value pair corresponding to the equipment node in the database when the heartbeat information sent by the equipment node is not received in the survival time.

15. A task assigning apparatus, comprising:

the second task allocation module is used for allocating the new task on the virtual node to the equipment node bound with the virtual node;

16. An electronic device, comprising:

one or more processors;

a memory;

one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications configured to: the task allocation method according to any one of claims 1 to 14 is performed.

17. A computer readable medium having stored thereon at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by a processor to implement a task allocation method according to any of claims 1 to 14.