CN111835809A - Work order message distribution method, work order message distribution device, server and storage medium - Google Patents
Work order message distribution method, work order message distribution device, server and storage medium Download PDFInfo
- Publication number
- CN111835809A CN111835809A CN201910902122.2A CN201910902122A CN111835809A CN 111835809 A CN111835809 A CN 111835809A CN 201910902122 A CN201910902122 A CN 201910902122A CN 111835809 A CN111835809 A CN 111835809A
- Authority
- CN
- China
- Prior art keywords
- work order
- node
- consumption
- consuming
- order message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1034—Reaction to server failures by a load balancer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Hardware Redundancy (AREA)
Abstract
The application provides a work order message distribution method, a work order message distribution device, a server and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: after a first work order message is sent to a first consumption node, if the consumption service of the first consumption node is determined to be unavailable, a second consumption node is determined, wherein the session subscribed by the first consumption node and the session subscribed by the second consumption node are the same; and sending a second work order message to the second consumption node. The working mode that multiple consumption nodes are used for message consumption is adopted, the first work order message is sent to the first consumption node, when the consumption service of the first consumption node is unavailable, the second consumption node is determined, the second work order message is sent to the second consumption node, the sequence of work order task execution is guaranteed, the situation that when the first consumption node breaks down, the whole system is broken down and stops working is avoided, and therefore the processing efficiency of the work order message is improved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a work order message distribution method, a work order message distribution device, a work order message distribution server and a storage medium.
Background
The work order system is also called a work order management system, and is a network software system which is used for managing, maintaining and tracking a series of problems and requests in a targeted manner according to the requirements of different organizations, departments and external clients. How to realize the high availability and the sequence of the work order system and improve the processing efficiency of the work order information becomes important.
In the prior art, a single-consumer work order processing mode is adopted, that is, based on one session (topic), one isolation unit (partition) and one consumer (consumer) realize work order processing, when work order message processing is performed, only one work order is executed at the same time, and consumption displacement (offset) of the consumer is maintained in the execution process to record consumption condition of the consumer, if offset is not maintained, after a work order system crashes, department work order messages are lost, so that unavailability of the work order system is increased.
However, according to the prior art, when the work order system using the single customer work order processing mode has a system failure due to external factors, for example: frequent operation or sudden system crash will cause the whole work order message processing system to crash, resulting in poor system availability and low work order message processing efficiency.
Disclosure of Invention
The present invention provides a work order message distribution method, device, server and storage medium, so as to solve the problems of poor work order system availability and low work order message processing efficiency in the prior art.
In order to achieve the above purpose, the technical solutions adopted in the embodiments of the present application are as follows:
in a first aspect, an embodiment of the present application provides a work order message allocation method, including:
after a first work order message is sent to a first consumption node, if the consumption service of the first consumption node is determined to be unavailable, a second consumption node is determined, wherein the session subscribed by the first consumption node and the session subscribed by the second consumption node are the same;
and sending a second work order message to the second consumption node.
Optionally, after sending the second work order message to the second consuming node, the method further includes:
receiving a lock contention request sent by the second consumption node;
after acquiring the lock released by the first consuming node, allocating the lock to the second consuming node according to the lock competition request.
Optionally, the method further comprises:
and if a consuming node adding instruction is received, selecting a balanced consuming node from the added consuming nodes and sending a new work order message to the balanced consuming node, wherein the first consuming node and the balanced consuming node subscribe to the same session.
Optionally, if it is determined that the consumption service of the first consumption node is not available, determining a second consumption node includes:
if the consumption service of the first consumption node is determined to be unavailable, determining whether the consumption service of the first consumption node is available or not after waiting for a preset time;
and if the consumption service of the first consumption node is still unavailable, determining a second consumption node.
In a second aspect, an embodiment of the present application further provides a work order message distribution method, where the method is applied to a work order message distribution system, and the system includes: a first consuming node and a second consuming node; the method comprises the following steps:
a second consumption node receives a work order message sent by a distributed server after the consumption service of the first consumption node is unavailable;
and the second consumption node enters a waiting state according to the work order message.
Optionally, the second consuming node enters a waiting state according to the work order message, including:
and sending a lock contention request to the distributed server.
Optionally, after the second consuming node enters a waiting state according to the work order message, the method further includes:
receiving a lock allocated by the distributed server after the first consuming node releases the lock;
and executing the work order corresponding to the work order message based on the lock.
Optionally, the executing the work order corresponding to the work order message based on the lock includes:
and executing the work order corresponding to the work order message based on the lock and a pre-configured idempotent strategy.
In a third aspect, an embodiment of the present application further provides a work order message allocation apparatus, including: the device comprises a determining module and a first sending module;
the determining module is configured to determine a second consuming node if it is determined that the consuming service of the first consuming node is unavailable after sending the first work order message to the first consuming node, where sessions subscribed by the first consuming node and the second consuming node are the same;
the first sending module is configured to send a second work order message to the second consuming node.
Optionally, the method further comprises: the system comprises a receiving module and a distribution module;
the receiving module is configured to receive a lock contention request sent by the second consuming node;
the allocating module is configured to allocate the lock to the second consuming node according to the lock contention request after acquiring the lock released by the first consuming node.
Optionally, the apparatus further comprises: a second sending module;
and the second sending module is used for selecting a balanced consuming node from the added consuming nodes and sending a new work order message to the balanced consuming node if a consuming node adding instruction is received, wherein the first consuming node and the balanced consuming node subscribe to the same session.
Optionally, the determining module is specifically configured to determine, if it is determined that the consuming service of the first consuming node is unavailable, whether the consuming service of the first consuming node is recovered to be available after waiting for a preset time period; and if the consumption service of the first consumption node is still unavailable, determining a second consumption node.
In a fourth aspect, an embodiment of the present application further provides a work order message distribution apparatus, including: the device comprises a receiving module and an undetermined module;
the receiving module is used for the second consumption node to receive the work order message sent by the distributed server after the consumption service of the first consumption node is unavailable;
and the undetermined module is used for enabling the second consumption node to enter a waiting state according to the work order message.
Optionally, the method further comprises: a sending module;
the sending module is configured to send a lock contention request to the distributed server.
Optionally, the system further comprises an execution module;
the receiving module is further configured to receive a lock allocated by the distributed server after the first consuming node releases the lock;
and the execution module is used for executing the work order corresponding to the work order message based on the lock.
Optionally, the execution module is specifically configured to execute the work order corresponding to the work order message based on the lock and a preconfigured idempotent policy.
In a fifth aspect, an embodiment of the present application further provides a server, including: a processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, the processor and the storage medium communicate with each other through the bus when the server runs, and the processor executes the program instructions to execute the steps of the work order message distribution method according to the first and second aspects.
In a sixth aspect, the present application further provides a storage medium, where a computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the work order message distribution method according to the first aspect and the second aspect are performed.
The beneficial effect of this application is: according to the work order message distribution method, the work order message distribution device, the server and the storage medium, through the working mode that multiple consumption nodes are adopted for message consumption, the first work order message is sent to the first consumption node, when the consumption service of the first consumption node is unavailable, the second consumption node is determined, the second work order message is sent to the second consumption node, the sequence of work order task execution is guaranteed, the situation that when the first consumption node breaks down, the whole system is broken down and stops working is avoided, and therefore the work order message processing efficiency is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 is a block diagram of a work order message distribution system according to an embodiment of the present disclosure;
fig. 2 is a schematic flow chart of a work order message distribution method according to an embodiment of the present disclosure;
fig. 3 is a schematic flow chart of another work order message distribution method according to an embodiment of the present application;
fig. 4 is a schematic flowchart of another work order message distribution method provided in the embodiment of the present application;
fig. 5 is a schematic flow chart of a work order message distribution method according to an embodiment of the present application;
fig. 6 is a schematic flow chart of another work order message distribution method according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of a work order message distribution apparatus according to an embodiment of the present disclosure;
fig. 8 is a schematic structural diagram of another work order message distribution apparatus according to an embodiment of the present application;
fig. 9 is a schematic structural diagram of another work order message distribution apparatus according to an embodiment of the present application;
fig. 10 is a schematic structural diagram of a work order message distribution apparatus according to an embodiment of the present application;
fig. 11 is a schematic structural diagram of another work order message distribution apparatus according to an embodiment of the present application;
fig. 12 is a schematic structural diagram of a server according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention.
Fig. 1 is a block diagram of a work order message distribution system according to an embodiment of the present disclosure; as shown in fig. 1, the work order message distribution system may be a distributed message distribution system, which may include: a subscription message queue, a plurality of work order systems, an etcd (distributed key value storage device) cluster and a GW (GateWay) cluster. The subscription message queue may be a kafka subscription message queue, the kafka subscription message queue is a component for providing message storage, and may be equivalent to a cloud server, the work order systems may be message consumption nodes, that is, corresponding to the message consumption node servers, the message consumption nodes belong to the same consumption group, and the message session types subscribed by the message consumption nodes are the same. The information can be distributed to a plurality of work order systems through the kafka message queue, different work order systems can correspondingly consume different information, the work order systems perform lock competition (request lock) after consuming the information, execute corresponding work order tasks after competing the lock, only one work order system executes the work order tasks at the same time, release the lock after the task execution is finished, submit offset message offset to the kafka message queue, and simultaneously send an execution result to a process in the GW cluster for storage.
Fig. 2 is a schematic flow chart of a work order message distribution method provided in an embodiment of the present application, where an execution subject of the work order message distribution method may be a kafka cluster server, and as shown in fig. 2, the method includes:
s101, after a first work order message is sent to a first consumption node, if the consumption service of the first consumption node is determined to be unavailable, a second consumption node is determined, wherein the first consumption node and the second consumption node subscribe to the same session.
It should be noted that the work order message may be understood as a work order task, that is, a to-be-processed item, and the consumption node may correspond to the work order system in fig. 1, that is, a system for executing the to-be-processed item, in other words, the work order message may correspond to a work task, and the consumption node may correspond to an employee, and after the leader allocates the work task to the employee, the employee completes the task according to the task requirement. The work order message may include a plurality of, not limited to the two, and correspondingly, the number of the consumption nodes may correspond to the number of the work order messages.
Optionally, the consumption nodes accessing the kafka cluster server may configure an order in advance, the first consumption node may be determined according to preset configuration information of the system, and the kafka cluster server sends a first work order message to the preset first consumption node, where the first work order message may be work order task data corresponding to the first consumption node. The work order messages sent by the kafka cluster server to different consuming nodes may be the same, or may be work order messages determined according to preset configuration information, that is, the second work order message sent to the second consuming node may be the same as, or different from, the first work order message, or may be a second work order message corresponding to the second consuming node, which is obtained according to the preset configuration information. When the second work order message sent to the second consumption node is the same as the first work order message sent to the first consumption node, in order to avoid the repeatability of the work order task execution, the idempotent guarantee can be added, that is, even if the second work order message sent to the second consumption node is the same as the first work order message, the second consumption node cannot execute the work order task corresponding to the second work order message after consuming the second work order message. Thereby improving the work order task execution efficiency.
After the first work order message is sent to the first consumption node, the first consumption node may cause that the consumption service is unavailable due to a fault, that is, the first consumption node is disconnected from the kafka cluster server in communication, and the execution condition of the work order message cannot be fed back to the kafka cluster server. Similarly to the determination method of the first consumption node, the second consumption node can be determined according to the preset configuration information, and the work order message corresponding to the second consumption node is sent to the second consumption node.
In addition, it should be further noted that sessions subscribed by a plurality of different consuming nodes are the same, that is, the types of work order messages consumed by different consuming nodes are the same, for example: the type of the work order message is to download a file from the database, so that the first work order message corresponding to the first consumption node can be to download a file 1 from the database, and the second work order message corresponding to the second consumption node can be to download a file 2 from the database, which all belong to the same work order message type, that is, the subscribed sessions are the same.
It should be noted that, when performing work order processing based on the kafka subscription message queue, in order to ensure high availability of the work order processing system and sequentiality of message processing, the kafka service may be based on one session (topic), one partition (minimum storage unit), one consumer group (consumer-group), and a single GW work order system, and one consumer group consumer-group may include multiple consumer providers, where a consumer provider: kafka message consumer, client that reads kafka messages, the contermer subscriptions in the same consumer-group are the same topic.
Wherein each message sent to kafka has a category, which is called topic; the partitions are the smallest units of kafka message storage, and each topic may contain multiple partitions.
In order to ensure the high efficiency of work order task processing, the work order message distribution method provided by the application adopts the kafka service based on a session (topic), a partition (minimum storage unit), a consumer-group (consumer-group) and a plurality of GW work order systems.
For the sequentiality of the execution of the work order tasks, the kafka message takes the partition as the minimum storage unit, and the data storage in the same partition can ensure the sequentiality. When the work order message is sent to the kafka, the work order message can be set to be sent to the same topic, and the topic only has one partition, so that the storage sequence of the work order is guaranteed. Meanwhile, the kafka consumer belongs to a consumer-group, namely the first consumer node and the second consumer node belong to a consumer group, the consumers in the group subscribe to the same topic, namely the sessions subscribed by the first consumer node and the second consumer node are the same, each consumer correspondingly consumes one partition message, namely the first consumer node correspondingly consumes the first work order message, and the second consumer node correspondingly consumes the second work order message. When kafka consumes the same topic and the same partition in a group, the number of consumers is larger than that of the partition, and redundant consumers are idle only, so that the sequential consumption of the work order is ensured.
In addition, in order to ensure the uniqueness of work order task execution, only one consumer consumes data at the same time in one consumer-group, and after the consumer consumes the data, the etcd distributed lock needs to be acquired to execute the work order. For the case where only one consumer is consuming data at the same time in a single consumer-group. From the work order data kafka, there is only one partition under the same topic, whereas the same partition under the same consumer-group can be guaranteed for consumption by only one consumer of the consumer.
And S102, sending a second work order message to a second consumption node.
And after the second consumption node is determined according to the method, sending a second work order message to the determined second consumption node. It should be noted that, when the consumption service of the first consumption node is unavailable, the kafka cluster server does not repeatedly send the first work order message corresponding to the first consumption node to the second consumption node, but sends the second work order message corresponding to the second consumption node according to the preset configuration information, that is, when any consumption node fails, the work order message distribution method adopted in the present application does not repeatedly consume the messages, but sequentially executes the work order messages corresponding to the consumption nodes, that is, the entire work order message distribution system is not crashed and stops working, so that the work order message processing efficiency is effectively improved.
To sum up, the work order message distribution method provided by the embodiment of the application sends the first work order message to the first consumption node through the working mode of message consumption by adopting multiple consumption nodes, determines the second consumption node when the consumption service of the first consumption node is unavailable, and sends the second work order message to the second consumption node, so that the execution sequence of work order tasks is ensured, and the problem that the whole system is crashed and stops working when the first consumption node fails is avoided, thereby improving the processing efficiency of the work order messages.
Fig. 3 is a schematic flow chart of another work order message distribution method provided in the embodiment of the present application, and further, as shown in fig. 3, after the step S102 sends the second work order message to the second consuming node, the method further includes:
s201, receiving a lock competition request sent by a second consumption node.
In some embodiments, in order to ensure the uniqueness of the work order message execution, that is, only one consuming node executes the work order task at the same time, in this embodiment, an etcd distributed locking mechanism may be adopted.
The maintenance principle of the etcd distributed lock mechanism can be explained in that all consumption nodes maintain a unique key value in the etcd cluster, when a first consumption node acquires the unique key value, the value is locked, when a second consumption node wants to acquire a lock, the key value is found to be locked, the release of the lock is waited, and when the second consumption node competes for the released lock, the corresponding work order task can be executed.
Optionally, after sending the second work order message to the second consuming node, the kafka cluster server receives a lock contention request sent by the second consuming node, and if there is a lock in an idle state at this time, the kafka cluster server may directly allocate the lock to the second consuming node.
S202, after the lock released by the first consuming node is obtained, the lock is distributed to the second consuming node according to the lock competition request.
When the kafka cluster server receives the lock contention request sent by the second consuming node, if the lock is currently in a use state, that is, the first consuming node has not released the lock, the lock is still waited to be released by the first consuming node, and after the lock is released by the first consuming node, the lock is allocated to the second consuming node according to the lock contention request sent by the second consuming node.
Optionally, when only a second consuming node competes for the lock, the kafka cluster server directly allocates the lock to the second consuming node according to a lock competition request sent by the second consuming node, and when there are other consuming nodes that compete for the lock together with the second consuming node, the kafka cluster server may sort the plurality of consuming nodes according to a preset lock competition rule, so as to select a successful lock competition from the plurality of consuming nodes according to a sorting result, and allocate the lock to the corresponding consuming node. Optionally, other competition rules may also be adopted, and the preset lock competition rule is not limited in the present application.
Further, the work order message distribution method further comprises the following steps:
and if a consuming node adding instruction is received, selecting a balanced consuming node from the added consuming nodes, and sending a new work order message to the balanced consuming node, wherein the first consuming node and the balanced consuming node subscribe the same session.
In some embodiments, in order to improve the availability of the work order message distribution system and improve the efficiency of work order task processing, in the process of consuming a work order message by any consuming node, more consuming nodes may be added, and after receiving a consuming node adding instruction, the kafka cluster server triggers consumption rebalancing, that is, a balanced consuming node is redetermined from the consuming nodes, and a new work order message is sent to the determined balanced consuming node, where the new work order message and the consumed work order message both belong to the same session.
For example: after the kafka cluster server sends the first work order message to the first consumption node, the first consumption node is in the process of executing the work order, at the moment, the kafka cluster server receives a third consumption node adding instruction, the third consumption node does not exist in a consumer group before, the kafka cluster server can immediately trigger consumption rebalancing, a balanced consumption node is determined from the second consumption node and the third consumption node, and a new work order message is sent to the determined balanced consumption node. Optionally, the new work order message is different from other work order messages that have already been consumed, i.e., not previously consumed by other consuming nodes. The new work order message is of the same category as the first work order message corresponding to the first consumption node, for example, the new work order message may be file 3 downloaded from a database. By adding the new consumption node, the work order message distribution system has stronger usability, and simultaneously, a plurality of work order messages can be consumed, thereby improving the work order consumption efficiency.
Fig. 4 is a flowchart illustrating a further work order message distribution method provided in an embodiment of the present application, and further, as shown in fig. 4, in step S101, if it is determined that the consumption service of the first consumption node is not available, determining the second consumption node includes:
s301, if the consumption service of the first consumption node is determined to be unavailable, whether the consumption service of the first consumption node is available or not is determined after waiting for a preset time.
Optionally, in order to avoid the idle consumption nodes and waste of resources, when the first consumption node fails and consumption services are unavailable, a preset time duration is waited to determine whether the consumption services of the first consumption node are restored to normal (that is, the available state is restored), if the consumption services of the first consumption node are restored to normal, the first consumption node continues to execute the work order task, and if the consumption services of the first consumption node are not restored to normal, the lock is released, and the next consumption node continues to execute the work order task.
In some embodiments, when the first consuming node fails, the first consuming node server may detect failure information in real time and report the failure information to the kafka cluster server, if all failure information is received by the kafka cluster server within a preset time period, it may be determined that the consuming service of the first consuming node is not recoverable, and if the failure recovery information is received by the kafka cluster server within the preset time period, it may be determined that the consuming service of the first consuming node is recovered.
S302, if the consumption service of the first consumption node is still unavailable, determining a second consumption node.
Optionally, if the first consuming node is still in the failure and the consuming service is still unavailable after the preset duration, the second consuming node may be determined according to the enumerated method.
To sum up, the work order message distribution method provided by the embodiment of the application sends the first work order message to the first consumption node through the working mode of message consumption by adopting the multiple consumption nodes, determines the second consumption node when the consumption service of the first consumption node is unavailable, and sends the second work order message to the second consumption node, so that the execution sequence of the work order tasks is ensured, and the phenomenon that the whole system is crashed and stops working when the first consumption node breaks down is avoided, thereby improving the work order task processing efficiency. In addition, by combining with an etcd distributed lock mechanism, only one consumption node executes the work order task at the same time, thereby effectively ensuring the uniqueness of work order message execution.
Fig. 5 is a schematic flow chart of a work order message distribution method provided in an embodiment of the present application, where the method is applied to a work order message distribution system, and the system includes: a first consuming node and a second consuming node; the work order message distribution method comprises the following steps:
s401, the second consumption node receives a work order message sent by the distributed server after the consumption service of the first consumption node is unavailable.
Optionally, the consumption nodes included in the work order message distribution system are not limited to the first consumption node and the second consumption node, the number of the consumption nodes is not limited to two, and may be multiple, and in this embodiment, the two consumption nodes are used for auxiliary explanation.
In some embodiments, after determining that the consumption service of the first consumption node is unavailable, the kafka cluster server sends a second work order message to the second consumption node, and the second consumption node server receives the second work order message and performs message consumption, where the message consumption may also be understood as obtaining a work order execution task from the second work order message, that is, an employee receives a job task assigned by the leader.
S402, the second consumption node enters a waiting state according to the work order message.
Optionally, in order to ensure uniqueness of task execution, that is, only one task is executed at the same time, the second consuming node does not execute the task immediately after acquiring the work order to execute the task, but enters a waiting state, that is, waits for issuing a password for starting execution of the task. For example: in the hectometer race, an athlete consumes information and knows that the task of the athlete is to run one hectometer, and after consuming the information, the athlete can be in a waiting state, namely a ready state, and also needs to wait for a starting whistle of a judge, and the whistle sounds to start running to finish the task.
Fig. 6 is a schematic flow chart of another work order message distribution method provided in the embodiment of the present application, and further, as shown in fig. 6, in the step S402, the entering of the second consuming node into a waiting state according to the work order message includes:
s501, a lock competition request is sent to the distributed server.
After the second consumption node enters the waiting state, the lock competition request needs to be sent to the kafka cluster server, and after the lock competition is successful, the second consumption node can execute the corresponding work order task according to the received second work order message.
Optionally, in step S402, after the second consuming node enters the waiting state according to the work order message, the method further includes:
s502, receiving the distributed lock of the distributed server after the first consumption node releases the lock.
Optionally, after the lock contention of the second consuming node is successful, the lock sent by the kafka cluster server may be received, where the lock may be a lock released by the first consuming node after executing the first work order task, and the kafka cluster server acquires the lock released by the first consuming node and allocates the lock to the second consuming node according to a lock contention request sent by the second consuming node.
And S503, executing the work order corresponding to the work order message based on the lock.
And after the second consumption node acquires the lock, executing a corresponding work order task according to the second work order message of consumption. After the second consuming node completes the work order task, correspondingly, the lock also needs to be released to the kafka cluster server, so that other consuming nodes can continue to acquire the lock, execute the task, and ensure the continuity of task execution.
Further, the work order corresponding to the work order message is executed based on the lock, and the method comprises the following steps:
and executing the work order corresponding to the work order message based on the lock and the pre-configured idempotent strategy.
It should be noted that the consumption node consumption message is maintained by offset, where offset is a message offset, and a position of the message is located, that is, the executed work order task message and a message of the work order task to be currently executed can be obtained through offset, if the first consumption node is always in a normal working state and does not have a fault, the first consumption node submits the offset to the kafka cluster server after executing the finished work order task, and the kafka cluster server can send the work order message to the second consumption node according to the located work order task to be currently executed. If the first consuming node fails after consuming the first work order message, the first consuming node disconnects communication with the kafka cluster server, so that although the first consuming node continues to execute the work order task corresponding to the first work order message, the first consuming node does not submit offset to the kafka cluster server after the execution is completed, and thus the kafka cluster server may continue to send the first work order message corresponding to the first consuming node to the second consuming node, and after the second consuming node consumes the work order message, the second consuming node continues to execute the work order task corresponding to the work order message, thereby causing repeated execution of the work order task.
In order to effectively avoid the repeated execution phenomenon of the work order tasks, the execution of the work order tasks is ensured according to the combination of idempotent. It can be understood that each work order message corresponds to an identification ID, once any work order message is consumed by one consumption node, the identification ID of the work order message is recorded by the kafka cluster server, and when other consumption nodes consume the work order message repeatedly, the server may determine that the work order task corresponding to the work order message is not executed again according to the recorded work order message ID, thereby avoiding the repeated execution of the work order task. That is, even if the same work order message is acquired by multiple consuming nodes, only one consuming node can execute the work order task corresponding to the work order message according to the work order message.
For example: after the first consumption node consumes the first work order message, the first consumption node has a fault, the first consumption node is disconnected from the kafka cluster server in communication, the first consumption node still executes the work order task corresponding to the first work order message, after the execution is finished, the offset is not submitted to the kafka cluster server, the offset maintained by the kafka cluster server is not updated offset, namely the offset of the first work order message which is not consumed, so that the kafka cluster server sends the first work order message to the second consumption node again, the second consumption node continues to consume the first work order message, but due to the addition of the power equality guarantee, the ID of the first work order message is recorded by the kafka cluster server after the first work order message is consumed by the first consumption node, so that the second consumption node can consume the first work order message again, or is not limited to the second consumption node, and more consumption nodes, for example, the third consuming node and the fourth consuming node both consume the first work order message, but the work order task corresponding to the true first work order message is executed only once, that is, the first consuming node that consumes the work order message executes the work order task, thereby effectively avoiding the work order task from being repeatedly executed.
To sum up, the work order message distribution method provided in the embodiment of the present application sends a first work order message to a first consumption node through a work mode in which multiple consumption nodes perform message consumption, determines a second consumption node when the consumption service of the first consumption node is unavailable, and sends a second work order message to the second consumption node, thereby ensuring the order of work order task execution, and avoiding that the whole system crashes and stops working when the first consumption node fails. And secondly, combining an etcd distributed locking mechanism, only one consumption node executes the work order task at the same time, thereby effectively ensuring the uniqueness of work order message execution. In addition, in the process of executing the work order task, the idempotent guarantee is added, so that the work order task corresponding to the work order message can be executed only once no matter the same work order message is consumed for several times, and the repeated execution of the work order task is effectively avoided.
Fig. 7 is a schematic structural diagram of a work order message distribution apparatus according to an embodiment of the present application, where the apparatus includes: a determining module 601, a first sending module 602;
a determining module 601, configured to determine a second consuming node if it is determined that the consuming service of the first consuming node is unavailable after sending the first work order message to the first consuming node, where sessions subscribed by the first consuming node and the second consuming node are the same;
a first sending module 602, configured to send a second work order message to a second consuming node.
Further, as shown in fig. 8, the apparatus further includes: a receiving module 603 and a distributing module 604;
a receiving module 603, configured to receive a lock contention request sent by a second consuming node;
the allocating module 604 is configured to, after acquiring the lock released by the first consuming node, allocate the lock to the second consuming node according to the lock contention request.
Further, as shown in fig. 9, the apparatus further includes: a second sending module 605;
a second sending module 605, configured to select a balanced consuming node from the added consuming nodes and send a new work order message to the balanced consuming node if a consuming node adding instruction is received, where sessions subscribed by the first consuming node and the balanced consuming node are the same.
Further, the determining module 601 is specifically configured to determine whether the consuming service of the first consuming node is available after waiting for a preset time period if it is determined that the consuming service of the first consuming node is not available; if the consuming service of the first consuming node is still not available, a second consuming node is determined.
Fig. 10 is a schematic structural diagram of a work order message distribution apparatus according to an embodiment of the present application, and as shown in fig. 10, the apparatus includes: a receiving module 701 and an undetermined module 702;
a receiving module 701, configured to receive, by the second consuming node, a work order message sent by the distributed server after the consuming service of the first consuming node is unavailable;
and the pending module 702 is configured to enable the second consuming node to enter a waiting state according to the work order message.
Further, as shown in fig. 11, the apparatus further includes: a sending module 703 and an executing module 704;
a sending module 703, configured to send a lock contention request to a distributed server;
a receiving module 701, further configured to receive a lock allocated by the distributed server after the first consuming node releases the lock;
and an executing module 704, configured to execute the work order corresponding to the work order message based on the lock.
Further, the executing module 704 is specifically configured to execute the work order corresponding to the work order message based on the lock and the preconfigured idempotent policy.
The above-mentioned apparatus is used for executing the method provided by the foregoing embodiment, and the implementation principle and technical effect are similar, which are not described herein again.
These above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), or one or more microprocessors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), among others. For another example, when one of the above modules is implemented in the form of a processing element scheduler code, the processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
Fig. 12 is a schematic structural diagram of a server according to an embodiment of the present disclosure, where the server may be integrated in a work order processing device or a chip of the work order processing device, and the work order processing device may be a computing device with a work order message distribution function.
The server includes: a processor 801 and a memory 802.
The memory 802 is used for storing programs, and the processor 801 calls the programs stored in the memory 802 to execute the above-mentioned method embodiments. The specific implementation and technical effects are similar, and are not described herein again.
Optionally, the invention also provides a program product, for example a computer-readable storage medium, comprising a program which, when being executed by a processor, is adapted to carry out the above-mentioned method embodiments.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.
The integrated unit implemented in the form of a software functional unit may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute some steps of the methods according to the embodiments of the present invention. And the aforementioned storage medium includes: a U disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Claims (18)
1. A work order message distribution method is characterized by comprising the following steps:
after a first work order message is sent to a first consumption node, if the consumption service of the first consumption node is determined to be unavailable, a second consumption node is determined, wherein the session subscribed by the first consumption node and the session subscribed by the second consumption node are the same;
and sending a second work order message to the second consumption node.
2. The method of claim 1, wherein after sending the second work order message to the second consuming node, further comprising:
receiving a lock contention request sent by the second consumption node;
after acquiring the lock released by the first consuming node, allocating the lock to the second consuming node according to the lock competition request.
3. The method of claim 1, further comprising:
and if a consuming node adding instruction is received, selecting a balanced consuming node from the added consuming nodes and sending a new work order message to the balanced consuming node, wherein the first consuming node and the balanced consuming node subscribe to the same session.
4. The method of claim 1, wherein determining a second consuming node if it is determined that the consuming service of the first consuming node is not available comprises:
if the consumption service of the first consumption node is determined to be unavailable, determining whether the consumption service of the first consumption node is available or not after waiting for a preset time;
and if the consumption service of the first consumption node is still unavailable, determining a second consumption node.
5. A work order message distribution method is applied to a work order message distribution system, and the system comprises: a first consuming node and a second consuming node, the method comprising:
a second consumption node receives a work order message sent by a distributed server after the consumption service of the first consumption node is unavailable;
and the second consumption node enters a waiting state according to the work order message.
6. The method of claim 5, wherein the second consuming node enters a wait state based on the work order message, comprising:
and sending a lock contention request to the distributed server.
7. The method of claim 6, wherein after the second consuming node enters a wait state according to the work order message, further comprising:
receiving a lock allocated by the distributed server after the first consuming node releases the lock;
and executing the work order corresponding to the work order message based on the lock.
8. The method of claim 7, wherein executing the work order corresponding to the work order message based on the lock comprises:
and executing the work order corresponding to the work order message based on the lock and a pre-configured idempotent strategy.
9. A work order message distribution apparatus, comprising: the device comprises a determining module and a first sending module;
the determining module is configured to determine a second consuming node if it is determined that the consuming service of the first consuming node is unavailable after sending the first work order message to the first consuming node, where sessions subscribed by the first consuming node and the second consuming node are the same;
the first sending module is configured to send a second work order message to the second consuming node.
10. The apparatus of claim 9, further comprising: the system comprises a receiving module and a distribution module;
the receiving module is configured to receive a lock contention request sent by the second consuming node;
the allocating module is configured to allocate the lock to the second consuming node according to the lock contention request after acquiring the lock released by the first consuming node.
11. The apparatus of claim 9, further comprising: a second sending module;
and the second sending module is used for selecting a balanced consuming node from the added consuming nodes and sending a new work order message to the balanced consuming node if a consuming node adding instruction is received, wherein the first consuming node and the balanced consuming node subscribe to the same session.
12. The apparatus according to claim 9, wherein the determining module is specifically configured to determine whether the consuming service of the first consuming node is available after waiting for a preset time period if it is determined that the consuming service of the first consuming node is not available; and if the consumption service of the first consumption node is still unavailable, determining a second consumption node.
13. A work order message distribution apparatus, comprising: the device comprises a receiving module and an undetermined module;
the receiving module is used for the second consumption node to receive the work order message sent by the distributed server after the consumption service of the first consumption node is unavailable;
and the undetermined module is used for enabling the second consumption node to enter a waiting state according to the work order message.
14. The apparatus of claim 13, further comprising: a sending module;
the sending module is configured to send a lock contention request to the distributed server.
15. The apparatus of claim 14, further comprising: an execution module;
the receiving module is further configured to receive a lock allocated by the distributed server after the first consuming node releases the lock;
and the execution module is used for executing the work order corresponding to the work order message based on the lock.
16. The apparatus of claim 15, wherein the execution module is specifically configured to execute the work order corresponding to the work order message based on the lock and a preconfigured idempotent policy.
17. A server, comprising: a processor, a storage medium and a bus, the storage medium storing program instructions executable by the processor, the processor and the storage medium communicating via the bus when the server is running, the processor executing the program instructions to perform the steps of the work order message distribution method as claimed in any one of claims 1 to 8 when executed.
18. A storage medium having stored thereon a computer program for performing the steps of the work order message distribution method of any of claims 1 to 8 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910902122.2A CN111835809B (en) | 2019-09-23 | 2019-09-23 | Work order message distribution method, work order message distribution device, server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910902122.2A CN111835809B (en) | 2019-09-23 | 2019-09-23 | Work order message distribution method, work order message distribution device, server and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111835809A true CN111835809A (en) | 2020-10-27 |
| CN111835809B CN111835809B (en) | 2023-04-18 |
Family
ID=72912020
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910902122.2A Active CN111835809B (en) | 2019-09-23 | 2019-09-23 | Work order message distribution method, work order message distribution device, server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111835809B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113590314A (en) * | 2021-07-13 | 2021-11-02 | 上海一谈网络科技有限公司 | Network request data processing method and system |
| CN117114366A (en) * | 2023-10-23 | 2023-11-24 | 天津中新智冠信息技术有限公司 | Work order distribution method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104092767A (en) * | 2014-07-21 | 2014-10-08 | 北京邮电大学 | Posting/subscribing system for adding message queue models and working method thereof |
| WO2016206281A1 (en) * | 2015-06-23 | 2016-12-29 | 中兴通讯股份有限公司 | Consumption quota monitoring method and device, storage medium |
| CN109190025A (en) * | 2018-08-21 | 2019-01-11 | 北京京东尚科信息技术有限公司 | information monitoring method, device, system and computer readable storage medium |
| CN109656725A (en) * | 2018-11-09 | 2019-04-19 | 北京字节跳动网络技术有限公司 | Message consumer's switching method, device, storage medium and electronic equipment |
-
2019
- 2019-09-23 CN CN201910902122.2A patent/CN111835809B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104092767A (en) * | 2014-07-21 | 2014-10-08 | 北京邮电大学 | Posting/subscribing system for adding message queue models and working method thereof |
| WO2016206281A1 (en) * | 2015-06-23 | 2016-12-29 | 中兴通讯股份有限公司 | Consumption quota monitoring method and device, storage medium |
| CN109190025A (en) * | 2018-08-21 | 2019-01-11 | 北京京东尚科信息技术有限公司 | information monitoring method, device, system and computer readable storage medium |
| CN109656725A (en) * | 2018-11-09 | 2019-04-19 | 北京字节跳动网络技术有限公司 | Message consumer's switching method, device, storage medium and electronic equipment |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113590314A (en) * | 2021-07-13 | 2021-11-02 | 上海一谈网络科技有限公司 | Network request data processing method and system |
| CN117114366A (en) * | 2023-10-23 | 2023-11-24 | 天津中新智冠信息技术有限公司 | Work order distribution method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111835809B (en) | 2023-04-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108737270B (en) | Resource management method and device for server cluster | |
| US9736034B2 (en) | System and method for small batching processing of usage requests | |
| CN109117252B (en) | Method and system for task processing based on container and container cluster management system | |
| EP3489825A1 (en) | Method, apparatus and computer readable storage medium for processing service | |
| CN109766172B (en) | Asynchronous task scheduling method and device | |
| US20090276519A1 (en) | Method and System for Achieving Better Efficiency in a Client Grid Using Node Resource Usage and Tracking | |
| US10884880B2 (en) | Method for transmitting request message and apparatus | |
| CN111835809B (en) | Work order message distribution method, work order message distribution device, server and storage medium | |
| CN113886089A (en) | Task processing method, device, system, equipment and medium | |
| CN108718327B (en) | License resource scheduling method and system | |
| CN111427551A (en) | User code operation method of programming platform, equipment and storage medium | |
| CN105373563B (en) | Database switching method and device | |
| CN111163140A (en) | Method, apparatus and computer readable storage medium for resource acquisition and allocation | |
| CN111835797A (en) | Data processing method, device and equipment | |
| CN113434591B (en) | Data processing method and device | |
| CN113965563B (en) | Service processing method and device based on model and server | |
| CN115329005A (en) | Multi-cluster cooperation method, system, device and computer readable storage medium | |
| CN117009063A (en) | Task execution method and device | |
| CN114489978A (en) | Resource scheduling method, device, equipment and storage medium | |
| CN113703930A (en) | Task scheduling method, device and system and computer readable storage medium | |
| EP2118747B1 (en) | Interface module | |
| CN114710485B (en) | Processing method and processing device | |
| CN111726418B (en) | Cloud resource allocation method and device, terminal equipment and storage medium | |
| CN116010126B (en) | Service aggregation method, device and system | |
| CN113973049B (en) | Method for managing and deploying bit stream of FPGA (field programmable Gate array) cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |