CN109656725A - Message consumer's switching method, device, storage medium and electronic equipment - Google Patents
Message consumer's switching method, device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN109656725A CN109656725A CN201811333129.9A CN201811333129A CN109656725A CN 109656725 A CN109656725 A CN 109656725A CN 201811333129 A CN201811333129 A CN 201811333129A CN 109656725 A CN109656725 A CN 109656725A
- Authority
- CN
- China
- Prior art keywords
- message consumer
- data
- instruction
- request
- data queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Mobile Radio Communication Systems (AREA)
Abstract
This disclosure relates to a kind of message consumer's switching method, device, storage medium and electronic equipment, the data read request sent including receiving first message consumer；The second message consumer for consuming the data queue is determined according to the path of the data queue, and sends the first notification instruction to the second message consumer；The first lock status modification that the second message consumer sends is received to instruct and execute the instruction；The second notification instruction is sent to the first message consumer, second notification instruction, which is used to indicate the first message consumer, can start to be read out the data queue.Through the above technical solution, it can be when there is new message consumer needs reading occupied data queue, the message consumer of the data queue is quickly and accurately switched into the new message consumer, and can be avoided in order to which to reconfigure new group other cumbersome by the new message consumer.
Description
Technical field
This disclosure relates to data processing field, and in particular, to a kind of message consumer's switching method, device, storage are situated between
Matter and electronic equipment.
Background technique
Flink is the Data Stream Processing computing engines of a new generation.Flink needs guarantee in the case where high-throughput
Exactly-once (i.e. just primary message transmission), can support two kinds of application types of stream process and batch processing.Flink will
Batch processing task regards limited stream process task as, then executes model with streaming and executes batch processing task.Kafka is as current
Popular message-oriented middleware, it can not only provide great handling capacity, additionally it is possible to Flink be cooperated to reach in consumption terminal
exactly-once.It has it is quick, expansible, can persistence the characteristics of.Message tissue in the form of topic is divided into topic
Several partition (data queue) are to achieve the purpose that distributed treatment and parallel processing.Producer writes into topic
Enter data, consumer (message consumer) subscribes to topic and obtains data from topic and handle.Several consumer
It is grouped together into group.One partition can only be consumed by a consumer in the same group, different
It will not be impacted between group.Kafka is widely used in Stream Processing, the data consumption source as Stream Processing.
For Kafka itself, api (the Application Programming of two kinds of acquisition data is provided
Interface, application programming interface), i.e. high-level-api and low-level-api.Using high-level-
In the case where api, how the message consumer (Consumer) for reading message in kafka distributes just without the concern for message
The problem of can guarantee exactly-once directly reads；Using low-level-api, message consumer
Can specify consumption particular data queue (partition) message, and voluntarily manage offset flexibly to obtain message,
In, offset is one continuous for positioning the sequence number for each message for being appended to subregion, i.e. offset.
Summary of the invention
Purpose of this disclosure is to provide a kind of message consumer's switching method, device, storage medium and electronic equipments, can
When there is new message consumer needs reading occupied data queue, quickly and accurately by the data queue
Message consumer switches to the new consumer.
To achieve the goals above, the disclosure provides a kind of message consumer's switching method, which comprises
The data read request that first message consumer sends is received, is disappeared in the data read request including described first
Breath consumer requests the path of the data queue read；
It is determined according to the path of the data queue and is consuming the second message consumer of the data queue, and to institute
It states second message consumer and sends the first notification instruction, first notification instruction is used to indicate the second message consumer and stops
Only to the reading of the data queue；
It receives the first lock status modification that the second message consumer sends and instructs and execute the instruction, first lock
Status modifier is instructed for will be revised as release conditions with lock status corresponding to the data queue；
The second notification instruction is sent to the first message consumer, second notification instruction includes the data queue
Lock status information, second notification instruction, which is used to indicate the first message consumer, can start to the data queue
It is read out.
Optionally, it instructs in the first lock status modification for receiving the second message consumer transmission and executes this and refer to
After order, the method also includes:
It receives the second lock status modification that the first message consumer sends and instructs and execute the instruction, second lock
Status modifier is instructed for the lock status to be revised as lock state.
It optionally, further include the instruction of the first request of data status modifier, first data in the data read request
Solicited status modification instruction is touched for that will be state in request with request of data status modifier corresponding to the data queue
It sends out described to be determined according to the path of the data queue and is consuming the second message consumer of the data queue, and to described
Second message consumer sends the step of the first notification instruction, wherein the request of data state is for characterizing the data team
Whether column have message consumer to request to read；
The method also includes: as the first message consumer for sending the first request of data status modifier instruction
When to preset consumer, the first request of data status modifier instruction is executed.
Optionally, the second request of data status modifier sent the method also includes receiving the second message consumer
Instruction, and execute the second request of data status modifier instruction, the second request of data status modifier instruction is for will be with
Request of data status modifier corresponding to the data queue is no solicited status.
Optionally, the method also includes:
Receive one of the first message consumer and the second message consumer transmission with the data team
The corresponding target offset amount of column, and current offset corresponding to the data queue is updated to the target offset amount；
The target offset amount is sent to the other of the first message consumer and the second message consumer.
Optionally, after second notification instruction of transmission to the first message consumer, the method also includes:
It receives the reading data that the first message consumer sends and stops request；
Third notice instruction is sent to the second message consumer, the third notice instruction is used to indicate described second
Message consumer can restore the reading to the data queue.
Optionally, the reading data stops further including the modification instruction of third lock status in request, is used for the lock shape
State is revised as release conditions, and triggers described the step of sending third notice instruction to the second message consumer；
The method also includes receiving the third lock status modification instruction of the first message consumer transmission and hold
The row instruction；And/or
It receives the 4th lock status modification that the second message consumer sends and instructs and execute the instruction, the 4th lock
Status modifier is instructed for the lock status to be revised as lock state.
The disclosure also provides a kind of message consumer's switching device, and described device includes:
First receiving module, for receiving the data read request of first message consumer transmission, the reading data is asked
It include the path that the first message consumer requests the data queue read in asking；
First sending module, for consuming the second of the data queue according to the determination of the path of the data queue
Message consumer, and the first notification instruction is sent to the second message consumer, first notification instruction is used to indicate institute
State reading of the second message consumer stopping to the data queue；
Second receiving module, the first lock status modification for receiving the second message consumer transmission are instructed and are executed
The instruction, the first lock status modification instruction is for will be revised as release shape with lock status corresponding to the data queue
State；
Second sending module, for sending the second notification instruction to the first message consumer, second notice refers to
The lock status information including the data queue is enabled, second notification instruction is used to indicate the first message consumer and starts
The data queue is read out.
Optionally, described device further include:
Third receiving module, for receiving the first lock that the second message consumer sends in second receiving module
After status modifier instructs and executes the instruction, receives the second lock status modification that the first message consumer sends and instruct simultaneously
The instruction is executed, the second lock status modification instruction is for being revised as lock state for the lock status.
It optionally, further include the instruction of the first request of data status modifier, first request of data in the request of data
Status modifier instruction triggers institute for that will be state in request with request of data status modifier corresponding to the data queue
It states and is determined according to the path of the data queue and consuming the second message consumer of the data queue, and to described second
Message consumer sends the step of the first notification instruction, wherein the request of data state, which is used to characterize the data queue, is
It is no to there is message consumer to request to read；
Described device further include: the first execution module, for when transmission the first request of data status modifier instruction
When the first message consumer is default consumer, the first request of data status modifier instruction is executed.
Optionally, described device further include:
4th receiving module refers to for receiving the second request of data status modifier that the second message consumer sends
It enables, and executes the second request of data status modifier instruction, the second request of data status modifier instruction is used for will be described
It with request of data status modifier corresponding to the data queue is no solicited status in default memory space.
Optionally, described device further include:
5th receiving module, for receiving one of the first message consumer and the second message consumer hair
Send with target offset amount corresponding to the data queue, and current offset corresponding to the data queue is updated to
The target offset amount；
Third sending module, for being sent out to the other of the first message consumer and the second message consumer
Send the target offset amount.
Optionally, described device further include:
6th receiving module refers to for sending the second notice to the first message consumer in second sending module
After order, receives the reading data that the first message consumer sends and stop request；
4th sending module, for sending third notice instruction to the second message consumer, the third notice refers to
It enables and is used to indicate the second message consumer and can restore reading to the data queue.
Optionally, the reading data stops further including the modification instruction of third lock status in request, is used for the lock shape
State is revised as release conditions, and triggers described the step of sending third notice instruction to the second message consumer；
Described device further include:
7th receiving module instructs simultaneously for receiving the third lock status modification that the first message consumer sends
Execute the instruction；And/or
8th receiving module, the 4th lock status modification for receiving the second message consumer transmission are instructed and are executed
The instruction, the 4th lock status modification instruction is for being revised as lock state for the lock status.
The disclosure also provides a kind of computer readable storage medium, is stored thereon with computer program, and the program is processed
The step of approach described above is realized when device executes.
The disclosure also provides a kind of electronic equipment, comprising:
Memory is stored thereon with computer program；
Processor, for executing the computer program in the memory, the step of to realize approach described above.
Through the above technical solutions, can need to read occupied data queue there is new message consumer
When, the message consumer of the data queue is quickly and accurately switched into the new message consumer, and new information consumption
Person can belong to same group with the former message consumer of data queue, be can be avoided in this way for the new message consumer again
It is other cumbersome to configure new group.
Other feature and advantage of the disclosure will the following detailed description will be given in the detailed implementation section.
Detailed description of the invention
Attached drawing is and to constitute part of specification for providing further understanding of the disclosure, with following tool
Body embodiment is used to explain the disclosure together, but does not constitute the limitation to the disclosure.In the accompanying drawings:
Fig. 1 is a kind of flow chart of message consumer's switching method shown according to one exemplary embodiment of the disclosure.
Fig. 2 is the flow chart of another message consumer's switching method shown according to one exemplary embodiment of the disclosure.
Fig. 3 is a kind of interaction figure of message consumer's switching method shown according to one exemplary embodiment of the disclosure.
Fig. 4 is a kind of schematic block diagram of message consumer's switching device shown according to one exemplary embodiment of the disclosure.
Fig. 5 is the schematic block diagram of another message consumer's switching device shown according to one exemplary embodiment of the disclosure.
Fig. 6 is the block diagram of a kind of electronic equipment shown according to an exemplary embodiment.
Specific embodiment
It is described in detail below in conjunction with specific embodiment of the attached drawing to the disclosure.It should be understood that this place is retouched
The specific embodiment stated is only used for describing and explaining the disclosure, is not limited to the disclosure.
In the actual production process of data flow, after new data stream type processing code is completed, how to the completion
It is the often problem in this process that the functionality of code, which carries out test,.The scheme generallyd use is to obtain history dump
File data handles the processing that code handles history dump file data by the data stream type using completion
Effect and actual effect are compared to judge the validity of the function of the code.But this common way has data volume
More disadvantage, also, the actual effect of history dump file data is not easy to obtain.
Therefore, expect can be by the case where not influencing most of old tasks, separating sub-fraction flow by applicant
It is handled, i.e., is handled by new task, and by this sub-fraction number to utilize new data stream type to handle code
The functionality of the code is verified according to the result of processing, i.e., small flow online verification, that is, new task is allowed to appoint with old
It is engaged in while running.New task only obtains the fraction data in data source, and old task obtains remaining most of data in data source.
The output of the old and new's task is both provided to downstream consumption, and observes the situation of change of the indices in downstream.If index variation meets
It is expected that then small flow verification success, can substitute old task with new task and carry out full flow operation, it can enough new data flows
Formula handles code and carries out data processing to replace original code；If index variation does not meet expection, need to cancel small traffic assignment,
It allows old task to be rolled back to full flow, still data is handled by original code.
In this case, how to allow old task and new task that can run simultaneously, just solve the problems, such as at needing most.
Since above-mentioned Flink would generally use the low-level-api of Kafka in practical applications, by Flink self-management Kafka
Partition (data queue) and consumer (message consumer) corresponding relationship, each partition in topic
The reading state of each partition of offset and Kafka.Therefore, it is switched to by the data queue of the old task of a part
New task, i.e. new information consumer when, just will receive the constraint of certain rule.Such as the message consumer in same group is not
The same data queue (partition) can be read, that is to say, that if new task and old task belong in same group
Two message consumers, then needing old task first to stop to this before the data that new task reads in specified data queue
The reading of specified data queue, relates to the handoff procedure to the old and new's task here, needs to establish between the old and new's task
The new mechanism of communication.Another way, if new task and old task belong to different group, new task is without waiting for old
Task will be read out again after data queue release, but this mode will cause following two problems:
1, it is arranged after new group id for new task, it is old to appoint when new task reads the data of specified data queue
The problem of business can also read the data of the read data queue of new task simultaneously, will cause Data duplication consumption in this way,
It is unfavorable for the subsequent judgement to index variation；
2, it carries out being required to one group id of setting when small flow verification each time, will lead in this way numerous in configuration
It is trivial.
Therefore, applicant proposed what one kind can be applicable in this case to switch between new task and old task
Message consumer switching method, as shown in Figure 1.Fig. 1 is a kind of message shown according to one exemplary embodiment of the disclosure
The flow chart of consumer's switching method, including step 101 is to step 104.
In a step 101, the data read request that first message consumer sends is received, is wrapped in the data read request
Include the path that the first message consumer requests the data queue read.Wherein, first message consumer can with for
The above-mentioned new task for small flow verification.First message consumer when starting to carry out small flow verification, as new task
The data read request comprising specified data queue path can be sent, is needed to characterize first message consumer to this
Specified data queue carries out reading data, and triggers step 102 to notify second message consumer, i.e., old task stops to this
The reading of specified data queue.
It can also include the first request of data status modifier in the data read request in a kind of possible embodiment
Instruction, the first request of data status modifier instruction for by with request of data status modifier corresponding to the data queue
For state in request, and the step 102 is triggered to notify second message consumer.Wherein, the request of data state is used for table
Levy whether the data queue has message consumer to request to read, which can be defined as
" isRequireLock ", state can be such as " false " for characterizing without solicited status, i.e. current data queue does not have
Other message consumers request to read, and " true " has other information consumptions for characterizing state in request, i.e. current data queue
Person requests to read.It is instructed when first message consumer has sent the first request of data status modifier, it will be with the data queue
When the value of corresponding " isRequireLock " is revised as " true ", so that triggering step 102 notice is consuming the data team
The second message consumer of column stops the reading to the data queue.In addition, the method can also accordingly include: when transmission institute
When the first message consumer for stating the instruction of the first request of data status modifier is default consumer, first data are executed
Solicited status modification instruction.That is, the first message consumer needs satisfaction for sending the first request of data status modifier instruction is
When the condition of default consumer, modification instruction can be performed, and can trigger step 102, otherwise, if this first disappears
Breath consumer is not belonging to the task in small flow verification, then first message consumer is centainly unsatisfactory for presetting consumer's
Condition, therefore modify response and triggering step 102 can not be also instructed to the modification.The default consumer can for
The message consumer of small flow verification label guarantees that the new information consumer only belonged in small flow verification can with this
The value of " isRequireLock " of the data queue is revised as " true " from " false " by request, is led to triggering step 102
Know that the i.e. old task of second message consumer stops the reading of data queue.
In a step 102, the second message for consuming the data queue is determined according to the path of the data queue
Consumer, and the first notification instruction is sent to the second message consumer, first notification instruction is used to indicate described the
Two message consumers stop the reading to the data queue.It is asked in the reading data for receiving first message consumer transmission
It after asking, responds thereto, is determined according to the path for the data queue for including in the request and the data queue is carried out
Then the second message consumer of reading, i.e., old task send the first notification instruction to second message consumer, be somebody's turn to do with informing
Second message consumer stops the reading to the data queue.It can also include the road of the data queue in first notification instruction
Diameter, to inform that second message consumer can accurately stop the reading for the data queue for needing to read to first message consumer
It takes.
In a kind of possible embodiment, step 102 be can be in the above-mentioned first request of data status maintenance received
Change instruction the value of " isRequireLock " is revised as " true " and is triggered later.
In step 103, it receives the first lock status modification that the second message consumer sends and instructs and execute this and refer to
It enables, the first lock status modification instruction is for will be revised as release conditions with lock status corresponding to the data queue.Lock
State can be defined as " isLocked ", and state can be in the lock state for " true " with characterizing the data queue, or
Release conditions are in for " false " to characterize the data queue.Second message consumer receive first notification instruction it
It can stop the reading to the data queue afterwards, after stopping to the reading of the data queue, in order to notify first message to consume
Person can send the first lock status modification instruction, for lock status corresponding to the data queue to be revised as release conditions,
That is " false " triggers step 104 with this, thus to notify first message consumer that can start to consume the data queue.
At step 104, Xiang Suoshu first message consumer sends the second notification instruction, and second notification instruction includes
The lock status information of the data queue, second notification instruction, which is used to indicate the first message consumer, can start pair
The data queue is read out.The lock status information may include " isLocked " that such as value is " false ".It should
Second notification instruction is sent to after first message consumer, and first message consumer can start to its specified data queue
It is read out, thus complete the old and new message consumer, i.e. switching between the old and new's task, by the information consumption of specified data queue
Person has switched to new first message consumer from original second message consumer, to complete subsequent to sentence index variation
It is disconnected.
Through the above technical solutions, the data sent in the newly-increased first message consumer received in small flow verification
After read requests, the second message consumer of first message consumer's specified data queue is enabled to stop to the data queue
Reading, enable first message consumer to start to consume the data queue, to complete between the old and new message consumer
Switching, so that without newly configuring group ID for new information consumer when small flow verification, cumbersome, the Neng Gouzhi of the configuration avoided
It connects and increases message consumer newly at the same group ID, and new information consumer needs to read occupied data queue
When, the message consumer of the data queue is quickly and accurately switched into the new message consumer.
In a kind of possible embodiment, after executing step 103 as shown in Figure 1, the method also includes: it connects
It receives the second lock status modification that the first message consumer sends and instructs and execute the instruction, the second lock status modification refers to
It enables for the lock status to be revised as lock state.It is modified receiving the first lock status that the second message consumer sends
After instructing and executing the instruction, the lock status of appointed data queue is release conditions, at this moment can receive first message
The second lock status that consumer sends after starting to be read out the data queue modifies instruction, for repairing the lock status
It is changed to lock state.In first message consumer starting that the data queue is started to read it in such manner, it is possible to further ensure
The stability of reading data afterwards.
In a kind of possible implementation method, the method also includes receiving the second of the second message consumer transmission
The instruction of request of data status modifier, and execute the second request of data status modifier instruction, the second request of data state
For that will be no solicited status with request of data status modifier corresponding to the data queue, i.e. second message disappears for modification instruction
Expense person is after receiving its another stopping and continuing the instruction read to specified data queue, in addition to that can send the first lock status
The corresponding lock status of the data queue is revised as release conditions by modification instruction, can also be sent the second request of data status modifier and be referred to
Enabling the corresponding request of data status modifier of the data queue is no solicited status, i.e., the data queue is corresponding
The value of " isRequireLock " is changed to " false ".In this way can the corresponding various status informations of time update data queue, make
The various status informations for obtaining data queue, which can access, to timely update.
In a kind of possible embodiment, the method also includes: receive the first message consumer and described
One of two message consumers send with target offset amount corresponding to the data queue, and by the institute, data queue
Corresponding current offset is updated to the target offset amount；To the first message consumer and the second message consumer
The other of send the target offset amount.
Above-mentioned offset is offset.In the low-level-api of Kafka, user oneself is needed to determine offset
Way to manage and report opportunity of offset, it will usually report selection there are three types of offset: when checkepoint
It reports offset, consumer (message consumer) periodically voluntarily to report offset to Kafka and forbids reporting for offset.
Any reporting schemes no matter are selected, when the old and new message consumer switches over, are likely to will appear different messages consumption
The offset of person is not reported to the possibility of kafka in time, therefore, in order to guarantee during the old and new message consumer switches
Guarantee that the accurate of offset, first message consumer and second message consumer also answer when terminating the reading to the data queue
Offset is also reported to other side by this, and therefore, the method also includes receiving first message consumer and second message consumer
One of the target offset amount that sends, the corresponding current offset of the data queue is updated to the target offset amount, then will
The target offset amount is sent to the other of first message consumer and second message consumer.
Fig. 2 is according to another message consumer's switching method shown in one exemplary embodiment of the disclosure.Such as Fig. 2 institute
Show, the method in addition to include the steps that as shown in Figure 1 101 to step 104 other than, further include step 201 and step 202.
In step 201, it receives the reading data that the first message consumer sends and stops request.The reading data
Reading to the data queue is stopped for characterizing first message consumer in request, and has triggered step 202 and notify the
The step of two message consumers.Wherein, in a kind of possible embodiment, the reading data stops being also possible to wrap in request
The modification instruction of third lock status is included, for the lock status to be revised as release conditions, and is triggered described to the second message
Consumer sends the step of third notice instruction.
In step 202, Xiang Suoshu second message consumer sends third notice instruction, and the third notice instruction is used for
Indicate that the second message consumer can restore the reading to the data queue.
The message consumer of data queue is being consumed from second message by step 101 shown in Fig. 1 to step 104
Person switches to after new first message consumer, judges the index variation of next data processing, if this refers to
Mark variation meets expection, then illustrates that new stream data processing code can push full flow to, if index variation could not reach
It is expected that then needing to carry out rollback, i.e., consumer's switchback again of the data queue of new first message consumer will be switched to just now
Old second message consumer.Therefore, by executing step 201 and step 202, it will be able to realize above-mentioned demand.
In a kind of possible embodiment, the method also includes: receive the institute of the first message consumer transmission
The modification of third lock status is stated to instruct and execute the instruction；And/or receive the 4th lock status that the second message consumer sends
Modification instructs and executes the instruction, and the 4th lock status modification instruction is for being revised as lock state for the lock status.Institute
The modification instruction of third lock status is stated for the lock status to be revised as release conditions, and triggers and described disappears to the second message
The person of expense sends the step of third notice instruction.
In a kind of possible embodiment, for what is communicated in the first message consumer and second message consumer
Mechanism can be such as Zookeeper.Zookeeper itself has watch mechanism, can be by Zookeeper active push node
The message such as modification, to reduce expense and delay, and Kafka itself also relies on Zookeeper service.Therefore it uses
Zookeeper can reduce unnecessary external frame to the greatest extent as communications framework and the corresponding lock status of storing data queue
Introducing.
The information transmitted between first message consumer, Zookeeper, second message consumer, such as the first lock shape
The lock message an of entirety can be used to be passed in state modification instruction, the instruction of the first request of data status modifier etc. information
Defeated, which can be for example shown once:
As it appears from the above, one lock message in may include formulate data queue address (" topic " and "
Partition "), it can also include the lock status (" isLocked ") and data of offset (" offset ") and data queue
Solicited status (" isRequireLock ") etc. can be made by the way that above-mentioned a variety of instructions or information are integrated into a lock message
The communication efficiency obtained between above-mentioned three is more efficiently and quick.Wherein, above-mentioned lock message can use such as transmission side json
Formula.
Disappear below according to interaction figure as shown in Figure 3 to according to a kind of message shown in one exemplary embodiment of the disclosure
Expense person's switching method is described.
As shown in figure 3, when first message consumer needs to request the data of a data queue, firstly, step 1 first to
Zookeeper sends a data requesting instructions, wherein may include path and the first number for needing the data queue read
It modifies and instructs according to solicited status, by the value of " isRequireLock " in the lock message of the data queue in Zookeeper
It is revised as " true ", executes next step 2 to trigger Zookeeper；
Zookeeper is to second message consumer in step 2, i.e., old task sends first notification instruction, with notice
Second message consumer stop the data queue being read out, second message consumer receive first notification instruction it
Stop the reading to the data queue afterwards, and executes step 3 later；
In step 3, second message consumer is to Zookeeper sends the first lock status modification instruction, the second data are asked
Status modifier is asked to instruct and the first offset, wherein the modification instruction of the first lock status is for by the data team in Zookeeper
The value of " isLocked " in the lock message of column is revised as " false " to indicate that the lock of the data queue is currently in release shape
State can be read by other message consumers, and the instruction of the second request of data status modifier is for by the data in Zookeeper
The value of " isRequireLock " in the lock message of queue is revised as " false ", and the first offset is second message consumer
Currently practical offset；
In step 4, Zookeeper sends the second notification instruction to first message consumer to notify first message consumer
It has been able to start to be read out the data queue, and the first offset that second message consumer is sent is updated to first
Message consumer, to avoid there is the problem of Data duplication reading；
It, can be to after first message consumer successfully starts to read data to data queue in step 5
Zookeeper sends the modification instruction of the second lock status, and the lock status of data queue is revised as lock state, i.e., will
The value of " isLocked " is revised as " true ".
So far, the handoff procedure of the primary message consumer from old task to new task would have been completed.
If over time, become need to above-mentioned handoff procedure carry out rollback, step 6 can be continued to execute:
In step 6, first message consumer sends reading data to Zookeeper and stops request, wherein may include
Third lock status modification instruction and the second offset, the modification of third lock status are instructed the lock of data queue in Zookeeper
The value of state " isLocked " is revised as " false ", so that triggering Zookeeper executes step 7, which is the
The current real offset of one message consumer；
In step 7, Zookeeper, which is triggered, sends third notice instruction to second message consumer, with inform this
Two message consumers can carry out reading data to the data queue being released, and second offset is sent to second simultaneously
Message consumer to avoid data Double Spending；
In step 8, second message consumer starts to restore the number to being released after receiving third notice instruction
The modification instruction of the 4th lock status is sent according to the reading data of queue, and to Zookeeper, by the lock status of the data queue
The value of " isLocked " is revised as " true ", to lock the data queue.
So far, the rollback of message consumer's switching just completes.
Fig. 4 is a kind of message consumer's switching device shown according to a kind of exemplary embodiment of the disclosure, such as Fig. 4 institute
Show, described device includes:
First receiving module 10, for receiving the data read request of first message consumer transmission, the reading data
It include the path that the first message consumer requests the data queue read in request；
First sending module 20 is consuming the of the data queue for determining according to the path of the data queue
Two message consumers, and the first notification instruction is sent to the second message consumer, first notification instruction is used to indicate
The second message consumer stops the reading to the data queue；
Second receiving module 30, the first lock status modification for receiving the second message consumer transmission are instructed and are held
The row instruction, the first lock status modification instruction is for will be revised as release shape with lock status corresponding to the data queue
State；
Second sending module 40, for sending the second notification instruction, second notice to the first message consumer
Instruction includes the lock status information of the data queue, and second notification instruction is used to indicate the first message consumer and opens
Beginning is read out the data queue.
Through the above technical solutions, the data sent in the newly-increased first message consumer received in small flow verification
After read requests, the second message consumer of first message consumer's specified data queue is enabled to stop to the data queue
Reading, enable first message consumer to start to consume the data queue, to complete between the old and new message consumer
Switching, so that without newly configuring group ID for new information consumer when small flow verification, cumbersome, the Neng Gouzhi of the configuration avoided
It connects and increases message consumer newly at the same group ID, and new information consumer needs to read occupied data queue
When, the message consumer of the data queue is quickly and accurately switched into the new message consumer.
Fig. 5 is another message consumer's switching device shown according to a kind of exemplary embodiment of the disclosure, such as Fig. 5 institute
Show, described device includes:
Third receiving module 50, for receiving that the second message consumer sends in second receiving module 30
After the modification of one lock status instructs and executes the instruction, receives the second lock status modification that the first message consumer sends and refer to
The instruction is enabled and executes, the second lock status modification instruction is for being revised as lock state for the lock status.
It further include the instruction of the first request of data status modifier in a kind of possible embodiment, in the request of data,
The first request of data status modifier instruction is for that will be to ask with request of data status modifier corresponding to the data queue
Middle state is sought, and triggers the second message that the data queue is being consumed according to the determination of the path of the data queue and disappears
Fei Zhe, and the step of sending the first notification instruction to the second message consumer, wherein the request of data state is used for table
Levy whether the data queue has message consumer to request to read；
As shown in figure 5, described device further include: the first execution module 60, for when transmission the first request of data shape
When the first message consumer of state modification instruction is default consumer, executes the first request of data status modifier and refer to
It enables.
In a kind of possible embodiment, as shown in figure 5, described device further include: the 4th receiving module 70, for connecing
The second request of data status modifier instruction that the second message consumer sends is received, and executes the second request of data state
Modification instruction, the second request of data status modifier instruction for by the default memory space with the institute, data queue
Corresponding request of data status modifier is no solicited status.
In a kind of possible embodiment, as shown in figure 5, described device further include: the 5th receiving module 80, for connecing
Receive one of the first message consumer and the second message consumer transmission with corresponding to the data queue
Target offset amount, and current offset corresponding to the data queue is updated to the target offset amount；Third sends mould
Block 90, for sending the target offset to the other of the first message consumer and the second message consumer
Amount.
In a kind of possible embodiment, as shown in figure 5, described device further include: the 6th receiving module 100 is used for
After second sending module 40 sends the second notification instruction to the first message consumer, the first message is received
The reading data that consumer sends stops request；4th sending module 110, for sending third to the second message consumer
Notification instruction, third notice instruction, which is used to indicate the second message consumer, can restore reading to the data queue
It takes.
In a kind of possible embodiment, the reading data stops further including that the modification of third lock status refers in request
It enables, for the lock status to be revised as release conditions, and triggers described to second message consumer transmission third notice
The step of instruction；As shown in figure 5, described device further include: the 7th receiving module 120, for receiving the first message consumption
The third lock status modification that person sends instructs and executes the instruction；And/or the 8th receiving module 130, it is described for receiving
The 4th lock status modification that second message consumer sends instructs and executes the instruction, and the 4th lock status modification instruction is used for
The lock status is revised as lock state.
About the device in above-described embodiment, wherein modules execute the concrete mode of operation in related this method
Embodiment in be described in detail, no detailed explanation will be given here.
Fig. 6 is the block diagram of a kind of electronic equipment 600 shown according to an exemplary embodiment.For example, electronic equipment 600 can
To be provided as a server.Referring to Fig. 6, electronic equipment 600 includes processor 622, and quantity can be one or more, with
And memory 632, for storing the computer program that can be executed by processor 622.The computer program stored in memory 632
May include it is one or more each correspond to one group of instruction module.In addition, processor 622 can be configured as
The computer program is executed, to execute above-mentioned message consumer's switching method.
In addition, electronic equipment 600 can also include power supply module 626 and communication component 650, which can be with
It is configured as executing the power management of electronic equipment 600, which, which can be configured as, realizes electronic equipment 600
Communication, for example, wired or wireless communication.In addition, the electronic equipment 600 can also include input/output (I/O) interface 658.Electricity
Sub- equipment 600 can be operated based on the operating system for being stored in memory 632, such as Windows ServerTM, Mac OS
XTM, UnixTM, LinuxTM etc..
In a further exemplary embodiment, a kind of computer readable storage medium including program instruction is additionally provided, it should
The step of above-mentioned message consumer's switching method is realized when program instruction is executed by processor.For example, this computer-readable is deposited
Storage media can be the above-mentioned memory 632 including program instruction, and above procedure instruction can be by the processor of electronic equipment 600
622 execute to complete above-mentioned message consumer's switching method.
The preferred embodiment of the disclosure is described in detail in conjunction with attached drawing above, still, the disclosure is not limited to above-mentioned reality
The detail in mode is applied, in the range of the technology design of the disclosure, a variety of letters can be carried out to the technical solution of the disclosure
Monotropic type, these simple variants belong to the protection scope of the disclosure.
It is further to note that specific technical features described in the above specific embodiments, in not lance
In the case where shield, it can be combined in any appropriate way.In order to avoid unnecessary repetition, the disclosure to it is various can
No further explanation will be given for the combination of energy.
In addition, any combination can also be carried out between a variety of different embodiments of the disclosure, as long as it is without prejudice to originally
Disclosed thought equally should be considered as disclosure disclosure of that.
Claims (16)
1. a kind of message consumer's switching method, which is characterized in that the described method includes:
The data read request that first message consumer sends is received, includes that the first message disappears in the data read request
Expense person requests the path of the data queue read；
It is determined according to the path of the data queue and is consuming the second message consumer of the data queue, and to described the
Two message consumers send the first notification instruction, and first notification instruction is used to indicate the second message consumer stopping pair
The reading of the data queue；
The first lock status modification that the second message consumer sends is received to instruct and execute the instruction, first lock status
Modification instruction is for will be revised as release conditions with lock status corresponding to the data queue；
The second notification instruction is sent to the first message consumer, second notification instruction includes the lock of the data queue
Status information, second notification instruction, which is used to indicate the first message consumer, can start to carry out the data queue
It reads.
2. the method according to claim 1, wherein in receive that the second message consumer sends the
After the modification of one lock status instructs and executes the instruction, the method also includes:
The second lock status modification that the first message consumer sends is received to instruct and execute the instruction, second lock status
Modification instruction is for being revised as lock state for the lock status.
3. the method according to claim 1, wherein further including the first request of data in the data read request
Status modifier instruction, the first request of data status modifier instruction for by with request of data corresponding to the data queue
Status modifier is state in request, and triggers described determine according to the path of the data queue and consuming the data queue
Second message consumer, and to the second message consumer send the first notification instruction the step of, wherein the data are asked
Ask state for characterizing whether the data queue has message consumer to request to read；
The method also includes: when the first message consumer for sending the first request of data status modifier instruction is pre-
If when consumer, executing the first request of data status modifier instruction.
4. according to the method described in claim 3, it is characterized in that, the method also includes receiving the second message consumer
The the second request of data status modifier instruction sent, and the second request of data status modifier instruction is executed, second number
According to solicited status modification instruction for that will be no solicited status with request of data status modifier corresponding to the data queue.
5. the method according to claim 1, wherein the method also includes:
Receive one of the first message consumer and the second message consumer transmission with the institute, data queue
Corresponding target offset amount, and current offset corresponding to the data queue is updated to the target offset amount；
The target offset amount is sent to the other of the first message consumer and the second message consumer.
6. the method according to claim 1, wherein described logical to first message consumer transmission second
After knowing instruction, the method also includes:
It receives the reading data that the first message consumer sends and stops request；
Third notice instruction is sent to the second message consumer, the third notice instruction is used to indicate the second message
Consumer can restore the reading to the data queue.
7. according to the method described in claim 6, it is characterized in that, the reading data stops further including third lock shape in request
State modification instruction for the lock status to be revised as release conditions, and triggers described to second message consumer transmission
The step of third notice instructs；
The method also includes receiving the third lock status modification instruction of the first message consumer transmission and execute this
Instruction；And/or
The 4th lock status modification that the second message consumer sends is received to instruct and execute the instruction, the 4th lock status
Modification instruction is for being revised as lock state for the lock status.
8. a kind of message consumer's switching device, which is characterized in that described device includes:
First receiving module, for receiving the data read request of first message consumer transmission, in the data read request
Path including the data queue that the first message consumer requests to read；
First sending module, for determining the second message for consuming the data queue according to the path of the data queue
Consumer, and the first notification instruction is sent to the second message consumer, first notification instruction is used to indicate described the
Two message consumers stop the reading to the data queue；
Second receiving module refers to for receiving the first lock status modification that the second message consumer sends and instructing and execute this
It enables, the first lock status modification instruction is for will be revised as release conditions with lock status corresponding to the data queue；
Second sending module, for sending the second notification instruction, the second notification instruction packet to the first message consumer
The lock status information of the data queue is included, second notification instruction is used to indicate the first message consumer and starts to institute
Data queue is stated to be read out.
9. device according to claim 8, which is characterized in that described device further include:
Third receiving module, for receiving the first lock status that the second message consumer sends in second receiving module
After modification instructs and executes the instruction, receives the second lock status modification that the first message consumer sends and instruct and execute
The instruction, the second lock status modification instruction is for being revised as lock state for the lock status.
10. device according to claim 8, which is characterized in that further include the first request of data shape in the request of data
State modification instruction, the first request of data status modifier instruction for by with request of data shape corresponding to the data queue
State is revised as state in request, and triggers described determine according to the path of the data queue and consuming the data queue
Second message consumer, and the step of sending the first notification instruction to the second message consumer, wherein the request of data
State is for characterizing whether the data queue has message consumer to request to read；
Described device further include: the first execution module, described in being instructed when transmission the first request of data status modifier
When first message consumer is default consumer, the first request of data status modifier instruction is executed.
11. device according to claim 10, which is characterized in that described device further include:
4th receiving module, the second request of data status modifier instruction sent for receiving the second message consumer, and
The second request of data status modifier instruction is executed, the second request of data status modifier instruction is for depositing described preset
Storing up in space with request of data status modifier corresponding to the data queue is no solicited status.
12. device according to claim 8, which is characterized in that described device further include:
5th receiving module, for receiving one of the first message consumer and the second message consumer transmission
With target offset amount corresponding to the data queue, and current offset corresponding to the data queue is updated to described
Target offset amount；
Third sending module, for sending institute to the other of the first message consumer and the second message consumer
State target offset amount.
13. device according to claim 8, which is characterized in that described device further include:
6th receiving module, for second sending module to the first message consumer send the second notification instruction it
Afterwards, it receives the reading data that the first message consumer sends and stops request；
4th sending module, for sending third notice instruction to the second message consumer, the third notice instruction is used
In instruction, the second message consumer can restore the reading to the data queue.
14. device according to claim 13, which is characterized in that the reading data stops further including third lock in request
Status modifier instruction for the lock status to be revised as release conditions, and triggers described to second message consumer hair
The step of sending third notice to instruct；
Described device further include:
7th receiving module, the third lock status modification for receiving the first message consumer transmission are instructed and are executed
The instruction；And/or
8th receiving module refers to for receiving the 4th lock status modification that the second message consumer sends and instructing and execute this
It enables, the 4th lock status modification instruction is for being revised as lock state for the lock status.
15. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the program is by processor
The step of any one of claim 1-7 the method is realized when execution.
16. a kind of electronic equipment characterized by comprising
Memory is stored thereon with computer program；
Processor, for executing the computer program in the memory, to realize described in any one of claim 1-7
The step of method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811333129.9A CN109656725B (en) | 2018-11-09 | 2018-11-09 | Message consumer switching method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811333129.9A CN109656725B (en) | 2018-11-09 | 2018-11-09 | Message consumer switching method and device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109656725A true CN109656725A (en) | 2019-04-19 |
| CN109656725B CN109656725B (en) | 2019-12-27 |
Family
ID=66110791
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811333129.9A Active CN109656725B (en) | 2018-11-09 | 2018-11-09 | Message consumer switching method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109656725B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111049915A (en) * | 2019-12-17 | 2020-04-21 | 书行科技(北京)有限公司 | Message queue agent grid under container cloud and method |
| CN111784329A (en) * | 2020-06-30 | 2020-10-16 | 京东数字科技控股有限公司 | Service data processing method and device, storage medium and electronic device |
| CN111835809A (en) * | 2019-09-23 | 2020-10-27 | 北京嘀嘀无限科技发展有限公司 | Work order message distribution method, work order message distribution device, server and storage medium |
| CN112148504A (en) * | 2020-09-15 | 2020-12-29 | 海尔优家智能科技（北京）有限公司 | Target message processing method and device, storage medium and electronic device |
| CN112882839A (en) * | 2019-11-29 | 2021-06-01 | 中国移动通信集团设计院有限公司 | Message processing method and device based on kafka |
| CN114520832A (en) * | 2020-11-19 | 2022-05-20 | 腾讯科技（深圳）有限公司 | Data stream processing method, device, equipment and computer storage medium |
| CN112506960B (en) * | 2020-12-17 | 2024-03-19 | 青岛以萨数据技术有限公司 | Multi-model data storage method and system based on ArangoDB engine |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6167423A (en) * | 1997-04-03 | 2000-12-26 | Microsoft Corporation | Concurrency control of state machines in a computer system using cliques |
| CN103645942A (en) * | 2013-12-12 | 2014-03-19 | 北京奇虎科技有限公司 | Message queue based write and read method and system of shared memory |
| CN103888441A (en) * | 2014-01-03 | 2014-06-25 | 北京太一星晨信息技术有限公司 | Information transmitting method between application and protocol stack and processing device |
| US20160357588A1 (en) * | 2015-06-04 | 2016-12-08 | Fujitsu Limited | Queue management method, non-transitory computer-readable recording medium and queue management device |
| CN107786623A (en) * | 2016-09-20 | 2018-03-09 | 平安科技（深圳）有限公司 | Message asynchronous processing method and device |
-
2018
- 2018-11-09 CN CN201811333129.9A patent/CN109656725B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6167423A (en) * | 1997-04-03 | 2000-12-26 | Microsoft Corporation | Concurrency control of state machines in a computer system using cliques |
| CN103645942A (en) * | 2013-12-12 | 2014-03-19 | 北京奇虎科技有限公司 | Message queue based write and read method and system of shared memory |
| CN103888441A (en) * | 2014-01-03 | 2014-06-25 | 北京太一星晨信息技术有限公司 | Information transmitting method between application and protocol stack and processing device |
| US20160357588A1 (en) * | 2015-06-04 | 2016-12-08 | Fujitsu Limited | Queue management method, non-transitory computer-readable recording medium and queue management device |
| CN107786623A (en) * | 2016-09-20 | 2018-03-09 | 平安科技（深圳）有限公司 | Message asynchronous processing method and device |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111835809A (en) * | 2019-09-23 | 2020-10-27 | 北京嘀嘀无限科技发展有限公司 | Work order message distribution method, work order message distribution device, server and storage medium |
| CN111835809B (en) * | 2019-09-23 | 2023-04-18 | 北京嘀嘀无限科技发展有限公司 | Work order message distribution method, work order message distribution device, server and storage medium |
| CN112882839A (en) * | 2019-11-29 | 2021-06-01 | 中国移动通信集团设计院有限公司 | Message processing method and device based on kafka |
| CN111049915A (en) * | 2019-12-17 | 2020-04-21 | 书行科技(北京)有限公司 | Message queue agent grid under container cloud and method |
| CN111049915B (en) * | 2019-12-17 | 2023-04-07 | 书行科技(北京)有限公司 | Container under-cloud message queue agent system and method |
| CN111784329A (en) * | 2020-06-30 | 2020-10-16 | 京东数字科技控股有限公司 | Service data processing method and device, storage medium and electronic device |
| CN111784329B (en) * | 2020-06-30 | 2024-04-05 | 京东科技控股股份有限公司 | Service data processing method and device, storage medium and electronic device |
| CN112148504A (en) * | 2020-09-15 | 2020-12-29 | 海尔优家智能科技（北京）有限公司 | Target message processing method and device, storage medium and electronic device |
| CN114520832A (en) * | 2020-11-19 | 2022-05-20 | 腾讯科技（深圳）有限公司 | Data stream processing method, device, equipment and computer storage medium |
| CN114520832B (en) * | 2020-11-19 | 2023-08-25 | 腾讯科技（深圳）有限公司 | Data stream processing method, device, equipment and computer storage medium |
| CN112506960B (en) * | 2020-12-17 | 2024-03-19 | 青岛以萨数据技术有限公司 | Multi-model data storage method and system based on ArangoDB engine |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109656725B (en) | 2019-12-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109656725A (en) | Message consumer's switching method, device, storage medium and electronic equipment | |
| CN105824664B (en) | Application program launching monitoring method, device and terminal device | |
| CN104156263A (en) | Interruption of chip component managing tasks, chip, and assembly comprising chip | |
| CN109154890A (en) | Software renewing apparatus and software update system | |
| CN108960629A (en) | Distribution method, device and the computer readable storage medium of maintenance task | |
| CN104346198B (en) | Information processing unit, server unit, information processing method and storage equipment | |
| CN109391505A (en) | Network example management method and relevant device | |
| JP2011028752A5 (en) | ||
| CN110166524B (en) | Data center switching method, device, equipment and storage medium | |
| CN104793975B (en) | The update method and device of application program in electronic control unit | |
| CN110149231A (en) | Update method, apparatus, storage medium and the equipment of virtual switch | |
| CN104408110B (en) | The method, apparatus and system of request of data | |
| CN106412078B (en) | Accurate information method for pushing, device and system | |
| CN109286957A (en) | Switching method, device, electronic equipment and the storage medium of a kind of time source link | |
| US20230029198A1 (en) | Scheduling complex jobs in a distributed network | |
| CN111953785B (en) | Data transmission method, system and server | |
| CN109257255A (en) | Traffic monitoring method, telephone watch, server and mobile terminal | |
| CN108445788A (en) | A kind of hotel's scenery control method and relevant apparatus | |
| CN106327143B (en) | Method and apparatus for nodes in a flow | |
| US20170123941A1 (en) | Distributed processing system | |
| CN106161086B (en) | The control method and device that master control borad is restarted | |
| JP2006229634A (en) | Presence management server, and system | |
| CN113515366A (en) | Bus load balancing processing method and device and storage medium | |
| CN110851186A (en) | Network equipment restarting method and device, electronic equipment and readable storage medium | |
| CN104951325B (en) | A kind of information display method and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |