CN109656725B - Message consumer switching method and device, storage medium and electronic equipment - Google Patents
Message consumer switching method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN109656725B CN109656725B CN201811333129.9A CN201811333129A CN109656725B CN 109656725 B CN109656725 B CN 109656725B CN 201811333129 A CN201811333129 A CN 201811333129A CN 109656725 B CN109656725 B CN 109656725B
- Authority
- CN
- China
- Prior art keywords
- message consumer
- data
- lock state
- data queue
- modification instruction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Mobile Radio Communication Systems (AREA)
Abstract
The disclosure relates to a message consumer switching method, a device, a storage medium and an electronic device, which comprises receiving a data reading request sent by a first message consumer; determining a second message consumer consuming the data queue according to the path of the data queue, and sending a first notification instruction to the second message consumer; receiving a first lock state modification instruction sent by the second message consumer and executing the instruction; sending a second notification instruction to the first message consumer, wherein the second notification instruction is used for indicating that the first message consumer can start reading the data queue. Through the technical scheme, when a new message consumer needs to read an occupied data queue, the message consumer of the data queue can be quickly and accurately switched to the new message consumer, and the trouble of reconfiguring a new group for the new message consumer can be avoided.
Description
Technical Field
The present disclosure relates to the field of data processing, and in particular, to a message consumer switching method, apparatus, storage medium, and electronic device.
Background
Flink is a new generation of data stream processing computing engine. Flink needs to guarantee exact-once (i.e., just-once messaging) at high throughput, supporting both streaming and batch application types. Flink treats the batch task as a finite stream processing task and then executes the batch task using a streaming execution model. Kafka, a very popular message middleware at present, can not only provide great throughput, but also cooperate with Flink to achieve exact-once at the consuming side. It has the characteristics of rapidness, expandability and persistence. The message is organized in a topic form, and the topic is divided into a plurality of partitions (data queues) so as to achieve the purposes of distributed processing and parallel processing. The producer writes data into the topic, and the consumer (message consumer) subscribes to the topic, obtains the data from the topic and processes the data. Several consumers are combined together to form a group. A partition can only be consumed by one consumer in the same group, and different groups are not affected. Kafka is widely used in streaming as a source of data consumption for streaming.
For Kafka itself, two types of api (Application Programming Interface) for acquiring data, i.e., high-level-api and low-level-api, are provided. Under the condition of using high-level-api, a message Consumer (Consumer) for reading the message in kafka can ensure the problem of exact-once without considering how the message is distributed, and the message Consumer can directly read the message; in the case of low-level-api, the message consumer can specify a message consuming a specific data queue (partition) and manage itself an offset, which is a sequential sequence number for locating each message appended to a partition, i.e., an offset, to flexibly acquire the message.
Disclosure of Invention
The purpose of the present disclosure is to provide a message consumer switching method, apparatus, storage medium and electronic device, which can quickly and accurately switch a message consumer of a data queue to a new consumer when the new message consumer needs to read the already occupied data queue.
In order to achieve the above object, the present disclosure provides a message consumer switching method, the method comprising:
receiving a data reading request sent by a first message consumer, wherein the data reading request comprises a path of a data queue requested to be read by the first message consumer;
determining a second message consumer consuming the data queue according to the path of the data queue, and sending a first notification instruction to the second message consumer, wherein the first notification instruction is used for instructing the second message consumer to stop reading the data queue;
receiving a first lock state modification instruction sent by the second message consumer and executing the instruction, wherein the first lock state modification instruction is used for modifying the lock state corresponding to the data queue into a release state;
sending a second notification instruction to the first message consumer, wherein the second notification instruction comprises lock state information of the data queue, and the second notification instruction is used for indicating that the first message consumer can start reading the data queue.
Optionally, after receiving and executing the first lock state modification instruction sent by the second message consumer, the method further comprises:
and receiving a second lock state modification instruction sent by the first message consumer and executing the instruction, wherein the second lock state modification instruction is used for modifying the lock state into a locking state.
Optionally, the data read request further includes a first data request state modification instruction, where the first data request state modification instruction is used to modify a data request state corresponding to the data queue into a request state, and trigger the step of determining, according to a path of the data queue, a second message consumer consuming the data queue, and sending a first notification instruction to the second message consumer, where the data request state is used to characterize whether the data queue has a message consumer requesting to read;
the method further comprises the following steps: when the first message consumer sending the first data request state modification instruction is a preset consumer, executing the first data request state modification instruction.
Optionally, the method further includes receiving a second data request state modification instruction sent by the second message consumer, and executing the second data request state modification instruction, where the second data request state modification instruction is used to modify a data request state corresponding to the data queue into a no-request state.
Optionally, the method further comprises:
receiving a target offset corresponding to the data queue sent by one of the first message consumer and the second message consumer, and updating a current offset corresponding to the data queue to the target offset;
sending the target offset to the other of the first message consumer and the second message consumer.
Optionally, after the sending of the second notification instruction to the first message consumer, the method further comprises:
receiving a data reading stop request sent by the first message consumer;
sending a third notification instruction to the second message consumer, the third notification instruction being used for indicating that the second message consumer can resume reading the data queue.
Optionally, the data reading stop request further includes a third lock state modification instruction, configured to modify the lock state into a release state, and trigger the step of sending a third notification instruction to the second message consumer;
the method further comprises receiving the third lock state modification instruction sent by the first message consumer and executing the instruction; and/or
And receiving a fourth lock state modification instruction sent by the second message consumer and executing the fourth lock state modification instruction, wherein the fourth lock state modification instruction is used for modifying the lock state into a locking state.
The present disclosure also provides a message consumer switching apparatus, the apparatus comprising:
a first receiving module, configured to receive a data reading request sent by a first message consumer, where the data reading request includes a path of a data queue requested to be read by the first message consumer;
a first sending module, configured to determine, according to a path of the data queue, a second message consumer who is consuming the data queue, and send a first notification instruction to the second message consumer, where the first notification instruction is used to instruct the second message consumer to stop reading the data queue;
a second receiving module, configured to receive a first lock state modification instruction sent by the second message consumer and execute the first lock state modification instruction, where the first lock state modification instruction is used to modify a lock state corresponding to the data queue into a release state;
a second sending module, configured to send a second notification instruction to the first message consumer, where the second notification instruction includes lock state information of the data queue, and the second notification instruction is used to instruct the first message consumer to start reading the data queue.
Optionally, the apparatus further comprises:
and a third receiving module, configured to receive a second lock state modification instruction sent by the first message consumer and execute the second lock state modification instruction after the second receiving module receives and executes the first lock state modification instruction sent by the second message consumer, where the second lock state modification instruction is used to modify the lock state into the lock state.
Optionally, the data request further includes a first data request state modification instruction, where the first data request state modification instruction is used to modify a data request state corresponding to the data queue into a request state, and trigger the step of determining, according to a path of the data queue, a second message consumer consuming the data queue, and sending a first notification instruction to the second message consumer, where the data request state is used to characterize whether the data queue has a message consumer requesting to read;
the device further comprises: the first execution module is configured to execute the first data request state modification instruction when the first message consumer sending the first data request state modification instruction is a preset consumer.
Optionally, the apparatus further comprises:
a fourth receiving module, configured to receive a second data request state modification instruction sent by the second message consumer, and execute the second data request state modification instruction, where the second data request state modification instruction is used to modify a data request state corresponding to the data queue in the preset storage space into a no-request state.
Optionally, the apparatus further comprises:
a fifth receiving module, configured to receive a target offset corresponding to the data queue sent by one of the first message consumer and the second message consumer, and update a current offset corresponding to the data queue to the target offset;
a third sending module to send the target offset to the other of the first message consumer and the second message consumer.
Optionally, the apparatus further comprises:
a sixth receiving module, configured to receive a data reading stop request sent by the first message consumer after the second sending module sends the second notification instruction to the first message consumer;
a fourth sending module, configured to send a third notification instruction to the second message consumer, where the third notification instruction is used to instruct the second message consumer to resume reading the data queue.
Optionally, the data reading stop request further includes a third lock state modification instruction, configured to modify the lock state into a release state, and trigger the step of sending a third notification instruction to the second message consumer;
the device further comprises:
a seventh receiving module, configured to receive the third lock state modification instruction sent by the first message consumer and execute the third lock state modification instruction; and/or
An eighth receiving module, configured to receive a fourth lock state modification instruction sent by the second message consumer and execute the fourth lock state modification instruction, where the fourth lock state modification instruction is used to modify the lock state into a lock state.
The present disclosure also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method described above.
The present disclosure also provides an electronic device, comprising:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to implement the steps of the method described above.
Through the technical scheme, when a new message consumer needs to read an occupied data queue, the message consumer of the data queue can be quickly and accurately switched to the new message consumer, and the new message consumer can belong to the same group with the original message consumer of the data queue, so that the trouble of reconfiguring a new group for the new message consumer can be avoided.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure without limiting the disclosure. In the drawings:
fig. 1 is a flowchart illustrating a message consumer switching method according to an exemplary embodiment of the present disclosure.
Fig. 2 is a flowchart illustrating yet another message consumer switching method according to an exemplary embodiment of the present disclosure.
Fig. 3 is an interaction diagram illustrating a message consumer switching method according to an exemplary embodiment of the present disclosure.
Fig. 4 is a schematic block diagram illustrating a message consumer switching apparatus according to an exemplary embodiment of the present disclosure.
Fig. 5 is a schematic block diagram illustrating yet another message consumer switching apparatus according to an exemplary embodiment of the present disclosure.
FIG. 6 is a block diagram illustrating an electronic device in accordance with an example embodiment.
Detailed Description
The following detailed description of specific embodiments of the present disclosure is provided in connection with the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present disclosure, are given by way of illustration and explanation only, not limitation.
In the actual production process of data streams, after a new data stream processing code is completed, how to test the functionality of the completed code is a problem that often occurs in the process. The commonly adopted scheme is to acquire historical dump file data, and compare the processing effect obtained by processing the historical dump file data by using the completed data stream processing code with the actual effect to judge the effectiveness of the function of the code. However, this conventional method has a disadvantage of a large amount of data, and the actual effect of the history dump file data is not easily obtained.
The applicant therefore thought that it is possible to process with new data streaming code by tapping off a small portion of the traffic without affecting the majority of the old tasks, i.e. by processing with a new task, and verify the functionality of the code by processing the result of this small portion of the data processing, i.e. by verifying the small traffic online, i.e. by letting the new task run simultaneously with the old tasks. The new task only acquires a small part of data in the data source, and the old task acquires the remaining large part of data in the data source. The outputs of the new and old tasks are all provided for downstream consumption, and the change situation of various indexes of the downstream is observed. If the index change accords with the expectation, the small flow verification is successful, the new task can be used for replacing the old task to perform full flow operation, namely, the new data flow type processing code can be used for replacing the original code to perform data processing; and if the index change does not meet the expectation, the small flow task needs to be cancelled, the old task is rolled back to the full flow, and the original code still processes the data.
In this case, how to allow the old task and the new task to run simultaneously becomes the problem to be solved most. Since the above-mentioned Flink generally adopts the low-level-api of Kafka in practical application, the Flink itself manages the correspondence between the partition (data queue) of Kafka and the consumer (message consumer), the offset of each partition in topic, and the read status of each partition of Kafka. Thus, when switching a portion of the data queue of an old task to a new task, i.e., a new message consumer, is subject to certain rules. For example, if a message consumer in the same group cannot read the same data queue (partition), that is, if a new task and an old task belong to two message consumers in the same group, the old task needs to stop reading the specified data queue before the new task reads the data in the specified data queue, and here, a switching process of the new and old tasks is involved, and a new mechanism for establishing communication between the new and old tasks is needed. Alternatively, if the new task and the old task belong to different groups, the new task does not need to wait for the old task to release the data queue before reading, but this approach causes the following two problems:
1. after a new group id is set for a new task, when the new task reads the data of the specified data queue, the old task can also read the data of the data queue read by the new task at the same time, so that the problem of data repeated consumption is caused, and the subsequent judgment on index change is not facilitated;
2. each time the small flow verification is performed, a group id needs to be set, which results in a cumbersome configuration.
The applicant has therefore proposed a message consumer switching method that can be applied in this case to switch between a new task and an old task, as shown in fig. 1. Fig. 1 is a flowchart illustrating a message consumer switching method according to an exemplary embodiment of the present disclosure, including steps 101 to 104.
In step 101, a data reading request sent by a first message consumer is received, where the data reading request includes a path of a data queue requested to be read by the first message consumer. Wherein the first message consumer can be the new task for small-flow verification described above. When the low-traffic verification is started, the first message consumer as a new task sends a data reading request including a specified data queue path to indicate that the first message consumer needs to read data from the specified data queue, and triggers step 102 to notify the second message consumer, that is, an old task, to stop reading the specified data queue.
In a possible implementation manner, the data read request may further include a first data request state modification instruction, where the first data request state modification instruction is used to modify a data request state corresponding to the data queue into a request state, and trigger the step 102 to notify the second message consumer. The data request state is used for characterizing whether the data queue has a message consumer request to read, and may be defined as "isrequireLock", and the state may be, for example, "false" for characterizing a no-request state, that is, the current data queue has no other message consumer request to read, and "true" for characterizing a state in the request, that is, the current data queue has other message consumer request to read. When the first message consumer sends the first data request state modification instruction and wants to modify the value of "isrequireLock" corresponding to the data queue to "true", thereby triggering step 102 to notify the second message consumer who is consuming the data queue to stop reading the data queue. In addition, the method may further include: when the first message consumer sending the first data request state modification instruction is a preset consumer, executing the first data request state modification instruction. That is, when a first message consumer sending the first data request state modification instruction needs to satisfy a condition of a preset consumer, the modification instruction can be executed, and step 102 can be triggered, otherwise, if the first message consumer does not belong to a task in the low-traffic verification, the first message consumer does not satisfy the condition of the preset consumer, and therefore, the modification response cannot be performed on the modification instruction, and step 102 can not be triggered. The default consumer may be a message consumer with a small-flow verification flag, so as to ensure that only a new message consumer belonging to the small-flow verification can request to modify the value of "isRequireLock" of the data queue from "false" to "true", so as to trigger step 102 to notify the second message consumer that the old task stops reading the data queue.
In step 102, a second message consumer consuming the data queue is determined according to the path of the data queue, and a first notification instruction is sent to the second message consumer, wherein the first notification instruction is used for instructing the second message consumer to stop reading the data queue. After receiving the data reading request sent by the first message consumer, responding to the data reading request, determining a second message consumer, namely an old task, which is reading the data queue according to the path of the data queue contained in the request, and then sending a first notification instruction to the second message consumer to inform the second message consumer to stop reading the data queue. The first notification instruction may further include a path of the data queue to notify the second message consumer that the second message consumer can accurately stop reading the data queue that the first message consumer needs to read.
In one possible implementation, step 102 may be triggered after the received first data request state modification instruction modifies the value of "isrequireLock" to "true".
In step 103, a first lock state modification instruction sent by the second message consumer is received and executed, where the first lock state modification instruction is used to modify the lock state corresponding to the data queue into a release state. The lock state may be defined as "isLocked" and may be "true" to indicate that the data queue is in a locked state or "false" to indicate that the data queue is in a released state. After receiving the first notification instruction, the second message consumer stops reading the data queue, and after stopping reading the data queue, in order to notify the first message consumer, the second message consumer sends a first lock state modification instruction for modifying the lock state corresponding to the data queue to a release state, i.e., "false", so as to trigger step 104, thereby notifying the first message consumer that the data queue can start to be consumed.
In step 104, a second notification instruction is sent to the first message consumer, where the second notification instruction includes lock status information of the data queue, and the second notification instruction is used to indicate that the first message consumer can start reading the data queue. The lock state information may include, for example, "isLocked" which takes the value "false". After the second notification instruction is sent to the first message consumer, the first message consumer can start to read the data queue appointed by the first message consumer, so that switching between new and old message consumers, namely new and old tasks, is completed, the message consumer of the appointed data queue is switched from the original second message consumer to the new first message consumer, and subsequent judgment on index change is completed.
Through the technical scheme, after a data reading request sent by a newly added first message consumer in small-flow verification is received, the second message consumer of the data queue appointed by the first message consumer stops reading the data queue, so that the first message consumer can start consuming the data queue, and switching between new and old message consumers is completed.
In a possible embodiment, after performing step 103 as shown in fig. 1, the method further comprises: and receiving a second lock state modification instruction sent by the first message consumer and executing the instruction, wherein the second lock state modification instruction is used for modifying the lock state into a locking state. After receiving the first lock state modification instruction sent by the second message consumer and executing the instruction, the lock state of the designated data queue is the release state, and at this time, a second lock state modification instruction sent by the first message consumer after starting reading the data queue may be received, and is used to modify the lock state into the lock state. In this way, the stability of the data read after the first message consumer starts reading the data queue can be further ensured.
In a possible implementation method, the method further includes receiving a second data request state modification instruction sent by the second message consumer, and executing the second data request state modification instruction, where the second data request state modification instruction is used to modify the data request state corresponding to the data queue to a no-request state, that is, after receiving another instruction for stopping reading the specified data queue, the second message consumer, in addition to sending the first lock state modification instruction to modify the lock state corresponding to the data queue to a release state, also sends the second data request state modification instruction to modify the data request state corresponding to the data queue to a no-request state, that is, to change the value of "isRequireLock" corresponding to the data queue to "false". Therefore, various state information corresponding to the data queue can be modified in time, and various state information of the data queue can be updated in time.
In one possible embodiment, the method further comprises: receiving a target offset corresponding to the data queue sent by one of the first message consumer and the second message consumer, and updating a current offset corresponding to the data queue to the target offset; sending the target offset to the other of the first message consumer and the second message consumer.
The offset is the offset. In Kafka's low-level-api, a user needs to determine an offset management method and an offset reporting timing, and there are usually three types of offset reporting options: reporting offset to Kafka at the time of checkpoint, reporting offset by the consumer (message consumer) on its own at regular intervals, and disabling reporting of offset. No matter which reporting mode is selected, when the old and new message consumers switch, the offset of different message consumers may not be reported to the kafka in time, so that in order to ensure the accuracy of the offset in the switching process of the old and new message consumers, the first message consumer and the second message consumer should report the offset to each other when finishing reading the data queue.
Fig. 2 is a diagram illustrating yet another message consumer switching method according to an exemplary embodiment of the present disclosure. As shown in fig. 2, the method includes step 201 and step 202 in addition to step 101 to step 104 shown in fig. 1.
In step 201, a data read stop request sent by the first message consumer is received. The data read request is used to indicate that the first message consumer has stopped reading the data queue and triggers step 202 to notify the second message consumer. In a possible implementation manner, the data reading stop request may further include a third lock state modification instruction, configured to modify the lock state into a release state, and trigger the step of sending a third notification instruction to the second message consumer.
In step 202, a third notification instruction is sent to the second message consumer, where the third notification instruction is used to indicate that the second message consumer can resume reading the data queue.
After switching the message consumers of the data queue from the second message consumer to the new first message consumer through steps 101 to 104 shown in fig. 1, a judgment is made on the index change of the next data processing, if the index change is expected, it is indicated that the new streaming data processing code can be pushed to the full flow, and if the index change fails to reach the expectation, a rollback is required, that is, the consumer just switched to the data queue of the new first message consumer is switched back to the old second message consumer. Therefore, by performing step 201 and step 202, the above-described requirements can be achieved.
In one possible embodiment, the method further comprises: receiving the third lock state modification instruction sent by the first message consumer and executing the instruction; and/or receiving a fourth lock state modification instruction sent by the second message consumer and executing the fourth lock state modification instruction, wherein the fourth lock state modification instruction is used for modifying the lock state into a locked state. The third lock state modification instruction is used for modifying the lock state into a release state and triggering the step of sending a third notification instruction to the second message consumer.
In one possible embodiment, the mechanism for communicating between the first message consumer and the second message consumer may be, for example, Zookeeper. The Zookeeper has a watch mechanism, and can actively push messages such as node modification and the like by the Zookeeper so as to reduce overhead and delay, while Kafka also depends on Zookeeper service. Therefore, the Zookeeper is adopted as a communication frame and the lock state corresponding to the data storage queue, so that the introduction of unnecessary external frames can be reduced as much as possible.
Information transmitted between the first message consumer, the Zookeeper, and the second message consumer, such as the first lock state modification instruction, the first data request state modification instruction, and the like, may be transmitted using an overall lock message, which may be, for example, as follows:
as shown above, a lock message may include addresses ("topic" and "partition") for defining a data queue, and may further include an offset ("offset"), and a lock status ("isLocked") and a data request status ("isRequireLock") of the data queue, and the like, and by integrating the various instructions or information into a lock message, the communication efficiency between the three can be made more efficient and faster. The lock message may be transmitted in json, for example.
A message consumer switching method shown according to an exemplary embodiment of the present disclosure is described below according to an interaction diagram as shown in fig. 3.
As shown in fig. 3, when a first message consumer needs to request data of a data queue, first, step 1 sends a data request instruction to a Zookeeper, where the data request instruction may include a path of the data queue to be read and a first data request state modification instruction, so as to modify a value of "isRequireLock" in a lock message of the data queue in the Zookeeper to "true", so as to trigger the Zookeeper to execute the next step 2;
in the step 2, the Zookeeper sends a first notification instruction to a second message consumer, namely an old task, so as to notify the second message consumer to stop reading the data queue, and the second message consumer stops reading the data queue after receiving the first notification instruction and then executes the step 3;
in step 3, the second message consumer sends a first lock state modification instruction, a second data request state modification instruction and a first offset to the Zookeeper, wherein the first lock state modification instruction is used for modifying the value of "isLocked" in the lock message of the data queue in the Zookeeper to "false" to indicate that the lock of the data queue is currently in a released state and can be read by other message consumers, the second data request state modification instruction is used for modifying the value of "isRequireLock" in the lock message of the data queue in the Zookeeper to "false", and the first offset is the current actual offset of the second message consumer;
in the step 4, the Zookeeper sends a second notification instruction to the first message consumer to inform the first message consumer that the first message consumer can start reading the data queue, and updates a first offset sent by the second message consumer to the first message consumer so as to avoid the problem of repeated data reading;
in step 5, after the first message consumer successfully starts to read data from the data queue, a second lock state modification instruction is sent to the Zookeeper to modify the lock state of the data queue to a lock state, that is, to modify the value of "isLocked" to "true".
By this time, a process of switching the message consumer from the old task to the new task is completed.
If the above handover procedure needs to be rolled back after a period of time, step 6 may be continued:
in step 6, the first message consumer sends a data reading stop request to the Zookeeper, where the data reading stop request may include a third lock state modification instruction and a second offset, where the third lock state modification instruction modifies a value of a lock state "isLocked" of a data queue in the Zookeeper to "false" so as to trigger the Zookeeper to execute step 7, where the second offset is a current actual offset of the first message consumer;
in step 7, the Zookeeper is triggered to send a third notification instruction to the second message consumer to inform the second message consumer that the second message consumer can read the data of the released data queue, and simultaneously send the second offset to the second message consumer to avoid repeated consumption of the data;
in step 8, after receiving the third notification instruction, the second message consumer starts to resume reading data from the released data queue, and sends a fourth lock state modification instruction to the Zookeeper, and modifies the value of the lock state "isLocked" of the data queue to "true", thereby locking the data queue.
At this point, the rollback of the message consumer switch is complete.
Fig. 4 is a diagram illustrating a message consumer switching apparatus according to an exemplary embodiment of the present disclosure, the apparatus, as shown in fig. 4, including:
a first receiving module 10, configured to receive a data reading request sent by a first message consumer, where the data reading request includes a path of a data queue requested to be read by the first message consumer;
a first sending module 20, configured to determine, according to a path of the data queue, a second message consumer who is consuming the data queue, and send a first notification instruction to the second message consumer, where the first notification instruction is used to instruct the second message consumer to stop reading the data queue;
a second receiving module 30, configured to receive a first lock state modification instruction sent by the second message consumer and execute the first lock state modification instruction, where the first lock state modification instruction is used to modify a lock state corresponding to the data queue into a release state;
a second sending module 40, configured to send a second notification instruction to the first message consumer, where the second notification instruction includes lock state information of the data queue, and the second notification instruction is used to instruct the first message consumer to start reading the data queue.
Through the technical scheme, after a data reading request sent by a newly added first message consumer in small-flow verification is received, the second message consumer of the data queue appointed by the first message consumer stops reading the data queue, so that the first message consumer can start consuming the data queue, and switching between new and old message consumers is completed.
Fig. 5 is a further message consumer switching apparatus, shown in fig. 5, according to an exemplary embodiment of the present disclosure, the apparatus comprising:
a third receiving module 50, configured to receive and execute a second lock state modification instruction sent by the first message consumer after the second receiving module 30 receives and executes the first lock state modification instruction sent by the second message consumer, where the second lock state modification instruction is used to modify the lock state into the lock state.
In a possible implementation manner, the data request further includes a first data request state modification instruction, where the first data request state modification instruction is used to modify a data request state corresponding to the data queue into a request state, and trigger the step of determining, according to a path of the data queue, a second message consumer consuming the data queue, and sending a first notification instruction to the second message consumer, where the data request state is used to characterize whether the data queue has a message consumer requesting to read;
as shown in fig. 5, the apparatus further includes: a first executing module 60, configured to execute the first data request state modification instruction when the first message consumer sending the first data request state modification instruction is a preset consumer.
In a possible embodiment, as shown in fig. 5, the apparatus further comprises: a fourth receiving module 70, configured to receive a second data request state modification instruction sent by the second message consumer, and execute the second data request state modification instruction, where the second data request state modification instruction is used to modify a data request state corresponding to the data queue in the preset storage space into a no-request state.
In a possible embodiment, as shown in fig. 5, the apparatus further comprises: a fifth receiving module 80, configured to receive a target offset corresponding to the data queue sent by one of the first message consumer and the second message consumer, and update a current offset corresponding to the data queue to the target offset; a third sending module 90 for sending the target offset to the other of the first message consumer and the second message consumer.
In a possible embodiment, as shown in fig. 5, the apparatus further comprises: a sixth receiving module 100, configured to receive a data reading stop request sent by the first message consumer after the second sending module 40 sends the second notification instruction to the first message consumer; a fourth sending module 110, configured to send a third notification instruction to the second message consumer, where the third notification instruction is used to indicate that the second message consumer can resume reading the data queue.
In a possible implementation manner, the data reading stop request further includes a third lock state modification instruction, configured to modify the lock state into a release state, and trigger the step of sending a third notification instruction to the second message consumer; as shown in fig. 5, the apparatus further includes: a seventh receiving module 120, configured to receive the third lock state modification instruction sent by the first message consumer and execute the third lock state modification instruction; and/or an eighth receiving module 130, configured to receive a fourth lock state modification instruction sent by the second message consumer and execute the fourth lock state modification instruction, where the fourth lock state modification instruction is used to modify the lock state into a locked state.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Fig. 6 is a block diagram illustrating an electronic device 600 according to an example embodiment. For example, the electronic device 600 may be provided as a server. Referring to fig. 6, the electronic device 600 includes a processor 622, which may be one or more in number, and a memory 632 for storing computer programs executable by the processor 622. The computer program stored in memory 632 may include one or more modules that each correspond to a set of instructions. Further, the processor 622 may be configured to execute the computer program to perform the message consumer switching method described above.
Additionally, electronic device 600 may also include a power component 626 that may be configured to perform power management of electronic device 600 and a communication component 650 that may be configured to enable communication, e.g., wired or wireless communication, of electronic device 600. The electronic device 600 may also include input/output (I/O) interfaces 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server, Mac OSXTM, UnixTM, LinuxTM, and the like.
In another exemplary embodiment, there is also provided a computer readable storage medium including program instructions which, when executed by a processor, implement the steps of the message consumer switching method described above. For example, the computer readable storage medium may be the memory 632 described above that includes program instructions executable by the processor 622 of the electronic device 600 to perform the message consumer switching method described above.
The preferred embodiments of the present disclosure are described in detail with reference to the accompanying drawings, however, the present disclosure is not limited to the specific details of the above embodiments, and various simple modifications may be made to the technical solution of the present disclosure within the technical idea of the present disclosure, and these simple modifications all belong to the protection scope of the present disclosure.
It should be noted that the various features described in the above embodiments may be combined in any suitable manner without departing from the scope of the invention. In order to avoid unnecessary repetition, various possible combinations will not be separately described in this disclosure.
In addition, any combination of various embodiments of the present disclosure may be made, and the same should be considered as the disclosure of the present disclosure, as long as it does not depart from the spirit of the present disclosure.
Claims (16)
1. A message consumer switching method, the method comprising:
receiving a data reading request sent by a first message consumer, wherein the data reading request comprises a path of a data queue requested to be read by the first message consumer;
determining a second message consumer consuming the data queue according to the path of the data queue, and sending a first notification instruction to the second message consumer, wherein the first notification instruction is used for instructing the second message consumer to stop reading the data queue;
receiving a first lock state modification instruction sent by the second message consumer and executing the instruction, wherein the first lock state modification instruction is used for modifying the lock state corresponding to the data queue into a release state;
sending a second notification instruction to the first message consumer, wherein the second notification instruction comprises lock state information of the data queue, and the second notification instruction is used for indicating that the first message consumer can start reading the data queue.
2. The method of claim 1, wherein after receiving and executing a first lock state modification instruction sent by the second message consumer, the method further comprises:
and receiving a second lock state modification instruction sent by the first message consumer and executing the instruction, wherein the second lock state modification instruction is used for modifying the lock state into a locking state.
3. The method according to claim 1, wherein the data reading request further includes a first data request state modification instruction, the first data request state modification instruction is used to modify a data request state corresponding to the data queue into a request-in state, and trigger the step of determining a second message consumer consuming the data queue according to a path of the data queue and sending a first notification instruction to the second message consumer, wherein the data request state is used to characterize whether the data queue has a message consumer requesting to read;
the method further comprises the following steps: when the first message consumer sending the first data request state modification instruction is a preset consumer, executing the first data request state modification instruction.
4. The method of claim 3, further comprising receiving a second data request state modification instruction sent by the second message consumer and executing the second data request state modification instruction, wherein the second data request state modification instruction is configured to modify a data request state corresponding to the data queue to a no request state.
5. The method of claim 1, further comprising:
receiving a target offset corresponding to the data queue sent by one of the first message consumer and the second message consumer, and updating a current offset corresponding to the data queue to the target offset;
sending the target offset to the other of the first message consumer and the second message consumer.
6. The method of claim 1, wherein after said sending a second notification instruction to said first message consumer, said method further comprises:
receiving a data reading stop request sent by the first message consumer;
sending a third notification instruction to the second message consumer, the third notification instruction being used for indicating that the second message consumer can resume reading the data queue.
7. The method according to claim 6, wherein the data reading stop request further comprises a third lock state modification instruction for modifying the lock state to a release state and triggering the step of sending a third notification instruction to the second message consumer;
the method further comprises receiving the third lock state modification instruction sent by the first message consumer and executing the instruction; and/or
And receiving a fourth lock state modification instruction sent by the second message consumer and executing the fourth lock state modification instruction, wherein the fourth lock state modification instruction is used for modifying the lock state into a locking state.
8. A message consumer switching apparatus, the apparatus comprising:
a first receiving module, configured to receive a data reading request sent by a first message consumer, where the data reading request includes a path of a data queue requested to be read by the first message consumer;
a first sending module, configured to determine, according to a path of the data queue, a second message consumer who is consuming the data queue, and send a first notification instruction to the second message consumer, where the first notification instruction is used to instruct the second message consumer to stop reading the data queue;
a second receiving module, configured to receive a first lock state modification instruction sent by the second message consumer and execute the first lock state modification instruction, where the first lock state modification instruction is used to modify a lock state corresponding to the data queue into a release state;
a second sending module, configured to send a second notification instruction to the first message consumer, where the second notification instruction includes lock state information of the data queue, and the second notification instruction is used to instruct the first message consumer to start reading the data queue.
9. The apparatus of claim 8, further comprising:
and a third receiving module, configured to receive a second lock state modification instruction sent by the first message consumer and execute the second lock state modification instruction after the second receiving module receives and executes the first lock state modification instruction sent by the second message consumer, where the second lock state modification instruction is used to modify the lock state into the lock state.
10. The apparatus according to claim 8, wherein the data read request further includes a first data request state modification instruction, the first data request state modification instruction is configured to modify a data request state corresponding to the data queue into a request-in state, and trigger the step of determining, according to a path of the data queue, a second message consumer consuming the data queue, and sending a first notification instruction to the second message consumer, where the data request state is used to characterize whether the data queue has a message consumer requesting to read;
the device further comprises: the first execution module is configured to execute the first data request state modification instruction when the first message consumer sending the first data request state modification instruction is a preset consumer.
11. The apparatus of claim 10, further comprising:
a fourth receiving module, configured to receive a second data request state modification instruction sent by the second message consumer, and execute the second data request state modification instruction, where the second data request state modification instruction is used to modify a data request state corresponding to the data queue into a no-request state.
12. The apparatus of claim 8, further comprising:
a fifth receiving module, configured to receive a target offset corresponding to the data queue sent by one of the first message consumer and the second message consumer, and update a current offset corresponding to the data queue to the target offset;
a third sending module to send the target offset to the other of the first message consumer and the second message consumer.
13. The apparatus of claim 8, further comprising:
a sixth receiving module, configured to receive a data reading stop request sent by the first message consumer after the second sending module sends the second notification instruction to the first message consumer;
a fourth sending module, configured to send a third notification instruction to the second message consumer, where the third notification instruction is used to instruct the second message consumer to resume reading the data queue.
14. The apparatus according to claim 13, wherein the data reading stop request further comprises a third lock state modification instruction, configured to modify the lock state into a release state, and trigger the step of sending a third notification instruction to the second message consumer;
the device further comprises:
a seventh receiving module, configured to receive the third lock state modification instruction sent by the first message consumer and execute the third lock state modification instruction; and/or
An eighth receiving module, configured to receive a fourth lock state modification instruction sent by the second message consumer and execute the fourth lock state modification instruction, where the fourth lock state modification instruction is used to modify the lock state into a lock state.
15. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
16. An electronic device, comprising:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to carry out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811333129.9A CN109656725B (en) | 2018-11-09 | 2018-11-09 | Message consumer switching method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811333129.9A CN109656725B (en) | 2018-11-09 | 2018-11-09 | Message consumer switching method and device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109656725A CN109656725A (en) | 2019-04-19 |
| CN109656725B true CN109656725B (en) | 2019-12-27 |
Family
ID=66110791
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811333129.9A Active CN109656725B (en) | 2018-11-09 | 2018-11-09 | Message consumer switching method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109656725B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111835809B (en) * | 2019-09-23 | 2023-04-18 | 北京嘀嘀无限科技发展有限公司 | Work order message distribution method, work order message distribution device, server and storage medium |
| CN112882839A (en) * | 2019-11-29 | 2021-06-01 | 中国移动通信集团设计院有限公司 | Message processing method and device based on kafka |
| CN111049915B (en) * | 2019-12-17 | 2023-04-07 | 书行科技(北京)有限公司 | Container under-cloud message queue agent system and method |
| CN111784329B (en) * | 2020-06-30 | 2024-04-05 | 京东科技控股股份有限公司 | Service data processing method and device, storage medium and electronic device |
| CN112148504A (en) * | 2020-09-15 | 2020-12-29 | 海尔优家智能科技（北京）有限公司 | Target message processing method and device, storage medium and electronic device |
| CN114520832B (en) * | 2020-11-19 | 2023-08-25 | 腾讯科技（深圳）有限公司 | Data stream processing method, device, equipment and computer storage medium |
| CN112506960B (en) * | 2020-12-17 | 2024-03-19 | 青岛以萨数据技术有限公司 | Multi-model data storage method and system based on ArangoDB engine |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6167423A (en) * | 1997-04-03 | 2000-12-26 | Microsoft Corporation | Concurrency control of state machines in a computer system using cliques |
| CN103645942B (en) * | 2013-12-12 | 2017-02-01 | 北京奇安信科技有限公司 | Message queue based write and read method and system of shared memory |
| CN103888441B (en) * | 2014-01-03 | 2017-03-08 | 北京太一星晨信息技术有限公司 | Information transferring method and processing meanss between a kind of application program and protocol stack |
| JP2017004050A (en) * | 2015-06-04 | 2017-01-05 | 富士通株式会社 | Queue management method, queue management program, and queue management device |
| CN107786623B (en) * | 2016-09-20 | 2021-03-30 | 平安科技（深圳）有限公司 | Message asynchronous processing method and device |
-
2018
- 2018-11-09 CN CN201811333129.9A patent/CN109656725B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109656725A (en) | 2019-04-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109656725B (en) | Message consumer switching method and device, storage medium and electronic equipment | |
| CN108566290B (en) | Service configuration management method, system, storage medium and server | |
| US11748154B2 (en) | Computing node job assignment using multiple schedulers | |
| US10977070B2 (en) | Control system for microkernel architecture of industrial server and industrial server comprising the same | |
| US10417062B2 (en) | Method and apparatus of unloading out of memory processing flow to user space | |
| CN110651256B (en) | System and method for preventing service interruption during software update | |
| CN111950988A (en) | Distributed workflow scheduling method and device, storage medium and electronic equipment | |
| US20220083375A1 (en) | Method and apparatus for scheduling task processing entity | |
| CN112667383B (en) | Task execution and scheduling method, system, device, computing equipment and medium | |
| US20180242177A1 (en) | Monitoring management method and apparatus | |
| US8843545B2 (en) | Supervision timer control mechanisms | |
| CN111190732A (en) | Timed task processing system and method, storage medium and electronic device | |
| US10216593B2 (en) | Distributed processing system for use in application migration | |
| US10740161B2 (en) | Broadcast message sending method and apparatus, and mobile terminal | |
| CN107634868B (en) | Method and system for managing network service | |
| CN109144676A (en) | A kind of self-starting detection method, device and the server of application program | |
| CN111741041A (en) | Message processing method and device, electronic equipment and computer readable medium | |
| CN116150273A (en) | Data processing method, device, computer equipment and storage medium | |
| CN112737896B (en) | Bandwidth data checking method, device, medium and electronic equipment | |
| US8151274B2 (en) | Task management system | |
| CN114816866A (en) | Fault processing method and device, electronic equipment and storage medium | |
| CN111080250B (en) | Flow backspacing compensation method and device, storage medium and electronic equipment | |
| JP2018538632A (en) | Method and device for processing data after node restart | |
| US20080247308A1 (en) | Providing session redundancy using priority during a syncing process | |
| CN111258728A (en) | Task execution method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |