CN106933672A - A kind of distributed environment coordinates consumption queuing method and device - Google Patents
A kind of distributed environment coordinates consumption queuing method and device Download PDFInfo
- Publication number
- CN106933672A CN106933672A CN201511021136.1A CN201511021136A CN106933672A CN 106933672 A CN106933672 A CN 106933672A CN 201511021136 A CN201511021136 A CN 201511021136A CN 106933672 A CN106933672 A CN 106933672A
- Authority
- CN
- China
- Prior art keywords
- queue
- burst
- client
- time
- queue burst
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000003860 storage Methods 0.000 claims description 45
- 230000002085 persistent effect Effects 0.000 claims description 34
- 238000012544 monitoring process Methods 0.000 claims description 31
- 230000002688 persistence Effects 0.000 claims description 8
- 230000004048 modification Effects 0.000 claims description 3
- 238000012986 modification Methods 0.000 claims description 3
- 239000000178 monomer Substances 0.000 claims description 2
- 230000008569 process Effects 0.000 description 20
- 238000004364 calculation method Methods 0.000 description 10
- 238000005192 partition Methods 0.000 description 10
- 230000008859 change Effects 0.000 description 8
- 230000032297 kinesis Effects 0.000 description 7
- 238000004590 computer program Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000009826 distribution Methods 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000003111 delayed effect Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000002045 lasting effect Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000010187 selection method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
Abstract
The embodiment of the present application provides a kind of distributed environment and coordinates consumption queuing method and device, is related to distributed technical field.Methods described includes：For a queue burst to be consumed, the status data of the queue burst obtained based on Tenancy Agreement；According to the status data, determine whether that other clients are consuming the queue burst；If it is determined that consuming the queue burst without other clients, then the status data of the queue burst is updated, and obtain the current consumption progress of the queue burst；Continued to consume the queue burst according to current consumption progress, and the new consumption progress of the queue burst is recorded.The application can be when load balancing, certain client break down, when active client A seizes the queue burst that other clients are being consumed, the consumption progress of queue burst can be with seamless delivery to active client, it is to avoid the Double Spending of partial data, makes consumption result more accurate.
Description
Technical field
The application is related to distributed technical field, and more particularly to a kind of distributed environment coordinates consumption team
Row method and a kind of distributed environment coordinate consumption array device.
Background technology
As cloud computing, big data epoch arrive, the source that data are produced is more and more wider, and speed is more next
Faster, quantity is also increasing.Such as in web server, various clients, sensor, in real time
The data of magnanimity are generated, various user access requests, monitoring data, running state of programs is have recorded
Etc. information.For the value of more preferable mining data, often there are multiple system costs these data, it is such as each
Class is real-time, off-line system is carried out to data user behavior analysis, monitoring alarm, program intrusion detection
Deng.
In order to reduce the degree of coupling of data production and consumption, usually using queue system caching, collect it is many
The data of individual generator's production, then number is consumed from queue as the client of calculate node by multiple
According to.General queue system can all use multiple queue burst (such as shard in order to be able to support mass data
Or partition) support the horizontal extension of mass data.In order to increase the handling capacity of write-in, it is only necessary to
Increase the number of shard (or partition).
When the client as calculate node needs the consumption data from multiple shard (or partition)
When, it is necessary to identical program is run in multiple client and consumes these data to cooperate with.And it is objective
Family end is also required to specify the crawl from which shard (or partition) when consumption data, equally
Data, complete consumption, generally for a data queue being made up of N number of shard (or partition),
Can be cooperateed with by M (span of M be 1~N) individual customer end consumption these shard (or
Partition), average every client consumes N/M shard (or partition).In order to correctly disappear
Take the data in all shard (or partition), cooperative cooperating is needed between this M client.
In first technology, for each client under distributed environment, there is following scheme and cooperate with each
Client is consumed to each queue burst：Such as use Kinesis Client Library.
Kinesis is the data real-time queue service that AWS is provided, and Kinesis Client Library are to use
In the collaboration lib of consumption Kinesis data.Kinesis Client Library depend on DynamoDB to come
Complete the collaboration between client.Although Kinesis Client Library support newly-increased client consumption number
According to, but when carrying out the operation such as load balancing, it is directly heavy in certain client to seizing queue burst
Newly consumed, therefore the Double Spending of partial data can be caused, and cause consumption result inaccurate.
The content of the invention
In view of the above problems, it is proposed that the embodiment of the present application so as to provide one kind overcome above mentioned problem or
A kind of distributed environment for solving the above problems at least in part coordinates consumption queuing method and corresponding one
Plant distributed environment and coordinate consumption array device.
In order to solve the above problems, this application discloses the side that a kind of distributed environment coordinates consumption queue
Method, including：
For a queue burst to be consumed, the status number of the queue burst obtained based on Tenancy Agreement
According to；
According to the status data, determine whether that other clients are consuming the queue burst；
If it is determined that consuming the queue burst without other clients, then the queue burst is updated
Status data, and obtain the current consumption progress of the queue burst；
Continue to consume the queue burst according to current consumption progress, and disappear the queue burst is new
Expense progress is recorded.
Disclosed herein as well is the device that a kind of distributed environment coordinates consumption queue, including：
State data acquisition module, for for a queue burst to be consumed, being obtained based on Tenancy Agreement
The queue burst status data；
Consumption judge module, for according to the status data, determining whether other clients in consumption
The queue burst；
Progress acquisition module, for if it is determined that consuming the queue burst without other clients, then
The status data of the queue burst is updated, and obtains the current consumption progress of the queue burst；
Consumption module, for being continued to consume the queue burst according to current consumption progress, and will be described
The new consumption progress of queue burst is recorded.
The embodiment of the present application includes advantages below：
Under distributed environment, for a client, it accounts for queue point robbing to the embodiment of the present application
After piece, the queue burst is queue burst to be consumed for the client, then the client
Need to be primarily based on the status data of the queue burst of Tenancy Agreement acquisition, then according to the state
Data, determine whether that other clients are consuming the queue burst, it is determined that without other clients
After the queue burst is consumed, the client can just obtain the current consumption progress of the queue burst, and
The status data of the queue burst is updated, then the client continues to consume and is somebody's turn to do with current consumption progress again
Queue burst simultaneously records new consumption progress.The embodiment of the present invention passes through said process, in existing customer
End A is when seizing the queue burst that other clients are being consumed, and the consumption progress of the queue burst can be with
Seamless delivery makes when queue burst load balancing is carried out in active client, or certain is
The client for consuming queue burst breaks down, after the queue burst of certain client is seized by active client,
Active client A is robbed after accounting for above-mentioned queue burst, can be according to the consumption consumed of the queue burst
Progress, continues to consume the queue burst, it is to avoid the Double Spending of partial data, makes consumption result more smart
Really.
Brief description of the drawings
Fig. 1 is that a kind of distributed environment of the application coordinates the step of consuming queuing method embodiment flow
Figure；
Fig. 2 is that a kind of distributed environment of the application is coordinated to consume the structured flowchart of array device embodiment；
Fig. 2A is a kind of example architecture of distributed environment of the application.
Specific embodiment
To enable above-mentioned purpose, the feature and advantage of the application more obvious understandable, with reference to attached
Figure and specific embodiment are described in further detail to the application.
One of core idea of the embodiment of the present application is, for distributed environment under, when generator writes
When entering data, different queues can be write data into by way of Hash or polling dispatching
Burst (shard or partition) completes load balancing.And as the calculate node under distributed environment
Client, its be also required to when consumption data, equally specify capture number from which queue burst
According to, consumption is completed, generally for a data queue being made up of N number of queue burst, can be by M (M
Span be 1~N) individual customer end consumes these queue bursts to cooperate with, average each client
N/M queue burst of end consumption.In the embodiment of the present application, describe for convenience, queue burst with
Shard is represented.
In order to correctly consume the data in all shard, cooperative cooperating is needed between this M client,
It needs to consider following aspects：First, how each client correctly selects the shard of consumption, with
Ensure that any time any shard has and only one of which customer end；Second, while how to process certain
Individual or some clients are delayed in the case of machine, when carrying out load balancing again, the number on all of shard
According to correct treatment is still able to, consumption is not repeated；3rd, when processing pressure increases, it is necessary to new
Increase the client for the treatment of, how to accomplish automatic load balancing, and can guarantee that arbitrary data is not repeated to disappear
Take.
Based on ensureing that data are not repeated the consideration of consumption, the embodiment of the present application under distributed environment,
For a customer end A, after robbing and accounting for queue burst, the queue burst is for the customer end A for it
It is queue burst to be consumed, then the customer end A needs to be primarily based on the institute of Tenancy Agreement acquisition
The status data of queue burst is stated, then according to the status data, determines whether that other clients exist
Consume the queue burst, it is determined that without other clients consume the queue burst after, the client
A can just obtain the current consumption progress of the queue burst, and update the status data of the queue burst,
Then the customer end A continues to consume the queue burst and record new consumption with current consumption progress again
Progress.So circulation, when another customer end B seizes the queue burst of client, then according to identical
Logic perform.The embodiment of the present invention seizes other clients by said process in active client A
Rectify in the queue burst of consumption, the consumption progress of the queue burst can be with seamless delivery to current visitor
In the end of family, make when queue burst load balancing is carried out, or certain is consuming the visitor of queue burst
Family end is broken down, and after the queue burst of certain client is seized by active client, active client A is seized
To after above-mentioned queue burst, can continue to consume and be somebody's turn to do according to the consumption progress consumed of the queue burst
Queue burst, it is to avoid the Double Spending of partial data, makes consumption result more accurate.
For clearer description the embodiment of the present application, the application defines following several terms：
Shard：Queue burst, it is the container that data are actually preserved in data queue, a data team
Row are made up of multiple shard.Need to select specific shard during each Worker consumption data.When
So, it is also possible to which partition represents that the application is not any limitation as to it.
Worker：Can be understood as a client progress of work in other words, one calculating of correspondence
Node, worker can consume the data in one or more shard.
Worker_name:The name of each worker, for distinguishing correspondence different clients.
Lease：Each shard has a lock (lease), only robs the worker for accounting for the lease
The data in the shard could be consumed.
Check Point：Consumption progress, the current position letters consumed of one shard of its record
Breath, represents which data has been consumed.
Embodiment one
Reference picture 1, shows that a kind of distributed environment of the application coordinates consumption queuing method embodiment
Flow chart of steps, specifically may include steps of：
Step 110, for a queue burst to be consumed, based on the queue point that Tenancy Agreement is obtained
The status data of piece；
The embodiment of the present application is there is provided each in many consumers' collaboration consumption queues under a set of distributed environment
The Tenancy Agreement of shard, realizes the process of the data in many calculate node collaboration consumption multiple shard.
According to the Tenancy Agreement, during certain client consumption shard, progress record to can be consumed
In the memory space of individual persistence, and the status data to the shard can be updated.The shape of the shard
State data are used to judge whether the shard has client to consume.
So for active client worker A, if it is robbed account for a shard, the shard
It is exactly a shard to be consumed for the client, then active client worker A need
The status data of the queue burst obtained based on Tenancy Agreement, into step 120.
In the preferred embodiment of the application one, also include before step 110：
Step 100, the queue burst to be consumed of active client demand is determined based on Tenancy Agreement.
In actual applications, firstly the need of determination according to Tenancy Agreement, is gone, it can be consumed active client
Several shard, then seize above-mentioned shard from where, used as the to be consumed of active client worker A
Shard.
In the preferred embodiment of the application one, the step 100 includes sub-step M1：
Sub-step M1, based on Tenancy Agreement every the cycle very first time, determines active client demand
Queue burst to be consumed.
In the embodiment of the present application, Tenancy Agreement defines active client worker A every the very first time
In the cycle, go fixed its to consume several shard, then above-mentioned shard is seized from where, as current visitor
The shard to be consumed at family end.In this way, can be born to each shard under distributed environment in real time
Carry balanced.
In the preferred embodiment of the application one, the step 100 includes sub-step 101-102：
Sub-step 101, obtains active client sum U, queue burst sum P and existing customer
The queue burst sum Q that end has been consumed, the queue burst quantity N seized with calculating active client to need.
In actual applications, active client represents that the client is normal, can share treatment
shard.The shard quantity N seized to calculate active client A to need, quantity N need to ensure
The load balancing of distributed environment, then need to obtain active client sum U, queue burst sum
The queue burst sum Q that P and active client have been consumed, then calculates quantity N.
In another preferred embodiment of the application, sub-step 101 includes：Sub-step A11-A14：
Sub-step A11, in the client instance table and queue rental period table that are stored from persistent storage space
Obtain active client sum U.
In the embodiment of the present application, can in advance in persistent storage space build client instance table and
Queue rental period table.The persistent storage space can be database, or such as distributed caching, certainly
Can be other kinds of persistent storage space, the embodiment of the present application is not any limitation as to it.In this Shen
Please it is preferred in embodiment shown in persistent storage space be database, certain database can be
Mysql databases, or other kinds of database, the embodiment of the present application is not any limitation as to it.
Wherein, queue rental period table client_shard_lease, can be in the form of table one：
| consume_group | shard_id | lease_id | lease_owner | consumer_owner | check_point | update_time |
Table one
In table one：
consume_group：Consumption group field, data type be Char (64), be table major key it
One, represent the consumption group name for some queue.In actual applications, shard is drawn with consuming group
Point, client be also to consume group what is created, therefore, the application can use consume_group
Distinguish different consumption groups.
shard_id：Queue identity field, data type is Char (64), is one of major key of table, team
The mark of each shard in row；
lease_id：Identification field is rented, data type is int (20), and worker rents what shard was used
id.In the embodiment of the present application, operated using the test and set of atom, it is ensured that any time, only
Having an owner can change the value of lease, that is, can get the shard.Wherein, " test " and
" set " operation is completed in an atomic operation that cannot split, and synchronization there was only 1
Client can seize the shard, to ensure the correctness of data manipulation.
lease_owner：Permit holder field, data type is Char (64), seizes the lease of the shard
The owner, i.e. certain worker_name；
consumer_owner：Consumer's field, data type Char (64), currently consumes this
The owner of shard, i.e. certain worker_name；
check_point：Progress field, data type is Text, records what the shard had currently been consumed
Consumption progress.
update_time：Renewal time field, data type is DateTime, when being updated for recording
Between, used for monitoring.
Certainly, in actual applications, the client_shard_lease tables of the embodiment of the present application can include
Shard_id, lease_owner, consumer_owner, check_point, with the client seized
And accounting between client of being robbed, what transmission was robbed account for queue burst is by disappearing of being robbed that client consumed
Take progress.Other several fields can be preferred field.
In the embodiment of the present application, when each client consumes certain shard, the shard in table one can be updated
Status data.
Wherein, client instance table client_worker_instance, can be in the form of table two：
| consume_group | worker_name | create_time |
Table two
consume_group：Be consumer's field, data type is Char (128), be table major key it
One, represent the consumption group name for some queue；
worker_name：It is client name, data type is Char (64), is one of major key of table.
create_time：Data type is DateTime, is the time of client terminal start-up.
Table two can record each client created to a consumption group.
In actual applications, after for a client terminal start-up for consumption group, the client
Worker_name can be written in the client instance table, and records the time of its startup.
Specifically, including client lib and data process method in the embodiment of the present application, in client.
Lient lib perform seizing and the logic such as re-renting for the application, and data process method is for normal to number in shard
According to the logic consumed.Wherein, the logic of above-mentioned consumption, such as user behavior analysis, monitoring report
The specifically analyzing and processing operation such as police, program intrusion detection.Wherein, client lib can be understood as client
One thread at end, client lib implementation procedures are substantially：
1st, when worker starts, client lib are by the title worker_name and current system of worker
Time is written in the client_worker_instance tables shown in table two.
2nd, client lib obtain the title of all shard in queue, i.e. shard_id.
3rd, Client lib obtain the information of all shard_id from client_shard_lease, including every
The corresponding lease_id and lease_owner of individual shard；
4th, Client lib judge whether the shard_id obtained in 2 occurs in client_shard_lease tables；
If do not occurred, the record of the shard_id is added in client_shard_lease tables, set
Lease_id is 0, and lease_owner, consumer_owner, check_point, update_time set
It is set to sky.
Certainly in actual applications, client can be created and seize thread and rental thread.Seizing thread is used for
Shard is seized, thread shard for rental is rented and is consumed.
Due in practical application, then the embodiment of the present application can obtain active visitor from table one and table two
Family end, so as to calculate active client sum U.
Further, the sub-step A11 includes：Sub-step A111-A113：
Sub-step A111, obtains and is opened in the nearest cycle very first time recorded in client instance table
The quantity live1 of dynamic client；
In actual applications, if after a client terminal start-up, run time is short, then may the client
Any one shard is not seized, then the just not no note of the client of permit holder field in Table 1
Record.
So, client_worker_instance tables as the aforementioned, after each client terminal start-up, base
In the consumption group name of its queue for processing, its worker_name and the time write-in of startup can be all somebody's turn to do
In client_worker_instance tables.
The embodiment of the present application can subtract the t that the startup time obtains using the system time of client, and judging should
Whether t is less than very first time cycle T, if it is less, the client is active, then statistics is obtained
The quantity of such active client obtains live1.
Sub-step A112, obtains in queue rental period table, is not timed-out the corresponding client of queue burst
Quantity live2；The client being not timed-out is the client recorded under the permit holder field of queue burst
End；
The embodiment of the present application obtains the record of all of shard also from client_shard_lease tables,
The shard records that all lease are not timed-out are preserved, is then therefrom extracted under lease_owner
Worker_name, its corresponding client be it is active, then statistics obtain such active client
Quantity obtain live2.
Sub-step A113, total active client terminal quantity U is obtained by live1 plus live2.
U=live1+live2, the U are active client sum.
Sub-step A12, obtains the total number P of queue burst from queue system；
In the embodiment of the present application, for a consumption group, obtained in queue system under the consumption group
Shard sums P.
Sub-step A13, active client is obtained in the queue rental period table stored from persistent storage space
The queue burst sum Q for having consumed.
In for client_shard_lease tables any shard record, lease_owner and
When the worker_name of consumer_owner is identical, and the consumer_owner is current
The consumer_owner of the progress of work seized, then illustrate that the worker_name occupies this
shard.If the worker_name of lease_owner and consumer_owner is differed, say
Worker_name in bright consumer_owner no longer allows to take the shard, and the shard is by it
His worker is seized.
Therefore, the embodiment of the present application counts client_shard_lease then for a worker_name
Table, lease_owner and consumer_owner all in the case of the worker_name, shard's
Quantity P, the shard quantity that as worker has taken.
Assuming that the form for being recorded as table described as follows three of table one：
Table three
Such as active client worker A, in client_shard_lease tables, have
That the lease_owner and consumer_owner of shard_1, shard_2 is all worker A, and
The lease_owner of shard_3 is workerB, and consumer_owner is worker B.So then
Worker A take shard_1, shard_2, and are not take up shard_3.Therefore worker A take
Shard quantity Q is 2.
Order the embodiment of the present application between certain sub-step A11-A13 is not any limitation as to it.
Sub-step A14, by N=[P/U]-Q, calculates the queue burst number that active client needs to seize
Amount N.
The embodiment of the present application calculates the queue that active client needs to seize in being calculated using N=[P/U]-Q
Burst quantity N.
Wherein [P/U], the shard numbers M that the worker should take is calculated.Wherein [P/U] represents right
The result of calculation of P/U rounds up.
Such as before worker A are specifically seized, worker F occupy 11 shard,
Worker B occupy 10 shard, and worker C occupy 10 shard, and worker D are occupied
10 shard, worker E occupy 11 shard.The worker A are newly-built worker, are not accounted for
Use shard.And total shard is 54, then the quantity that worker A should take is 54/6=9,
It is 9-0=9 that worker A need the shard for taking.Other situations are by that analogy.
Sub-step 101, from the queue burst that the queue burst and/or other clients of time-out are being consumed
In, seize to be consumed queue burst of N number of queue burst as active client.
In the embodiment of the present application, because for all of shard, it is understood that there may be following several situations：
1st, there is the queue burst of time-out, and the shard quantity L of time-out is more than or equal to N.
2nd, there is the queue burst of time-out, and the shard quantity L of time-out is less than N.
3rd, in the absence of the queue burst of time-out.
For the 1st kind of situation, worker A directly seize N number of queue burst i.e. from the shard of time-out
Can.
For the 2nd kind of situation, worker A can seize the no more than team of L from the queue burst of time-out
Row burst, remaining N-L shard is seized from other worker.
For the 3rd kind of situation, then each shard of N are directly seized from other worker.
In another preferred embodiment of the application, the sub-step 101 includes sub-step A21-A24：
Whether sub-step A21, judge the N more than 0；
It is understood that the N obtained for abovementioned steps, if N=0, for the worker,
Without seizing.If N>0, then just seize.
Sub-step A22, if the N is more than 0, judges whether the queue burst quantity L of time-out is small
In N；
Sub-step A23, if the queue burst quantity L of time-out is more than or equal to N, from the queue of time-out
N number of queue burst is seized in burst；
In the embodiment of the present application, if L >=N, N is seized at random from L overtime queue burst
It is individual.
Sub-step A24, if it is determined that the queue burst quantity L of time-out is less than N, then from the queue of time-out
L queue burst is seized in burst, and in the queue burst consumed from other clients, is seized
N-L queue burst.
If L<N, then seize the L queue burst, and then remaining N-L is individual from other worker
Seized in the shard for consuming.
If it is appreciated that L=0, worker A need what is consumed from other worker
N number of shard is seized in shard.
Further, in another preferred embodiment of the application, from other visitors described in sub-step A24
Family is rectified in the queue burst of consumption, the step of seize N-L queue burst, including：Sub-step
A241：
Sub-step A241, seizes N-L each in the queue burst that active client takes from other clients
Queue burst, and each client is taken the quantity of most queue bursts and is taken minimum queue burst
The difference of quantity is no more than specified quantity.
In the embodiment of the present application, each worker seizes thread, and when seizing, it ensures each
The quantity of the shard that worker takes meets following condition：Take the quantity and occupancy of most queue bursts
The difference of the quantity of minimum queue burst is no more than specified quantity.The specified quantity is such as 1.
So active client worker A also make each client when shard is seized from other worker
The difference of the quantity for taking most queue bursts and the quantity for taking minimum queue burst is no more than specified quantity.
Such as before worker A are specifically seized, worker B occupy 11 shard,
Worker C occupy 10 shard, and workerD occupies 10 shard, and worker E are occupied
10 shard, worker F occupy 11 shard.And total shard is 54, the queue of time-out
Burst is 2.Worker A are the new worker for starting.
So need to seize 2 shard from worker B for worker A, 1 seized from worker C,
1 shard is seized from worker D, 1 shard is seized from worker E, 2 are seized from worker F
Individual shard, 2 are seized from the shard of time-out.Ensure the shard quantity of each worker occupancy not
More than 1.Other situations are by that analogy.
In another preferred embodiment of the application, the sub-step A241 includes：Sub-step A2411-
Sub-step A2412：
Sub-step A2411, based on the queue rental period table, the queue that each client is taken according to it
Burst quantity, from being more to ranked up less；
In the embodiment of the present application, before worker A seize shard, it is primarily based on
Client_shard_lease tables, count the quantity of the shard_id of each worker_name occupancy.
Such as previous example, is ranked up by quantity and obtained such as table four：
| worker B | 11 |
| worker C | 11 |
| worker D | 10 |
| worker E | 10 |
| worker F | 10 |
| worker A | 0 |
Table four
Wherein, overtime shard has 2, and worker A need to seize 9.
Sub-step A2412, seizes one or more and seizes queue burst in K client of each the past
Afterwards, the preceding K client is made to be preempted after J, the preceding remaining queue burst of K client is put down
Mean makes preceding K client more than the queue burst quantity that the K+1 client currently takes
In, the remaining queue burst number of each client is differed no more than specified quantity with the average, until
Successfully rob and account for N-L queue burst.
It is understood that worker A are when the shard of other worker is seized, it is latter for seizing
Quantity of the shard quantity of individual worker no more than previous shard.
It is 1 with specified quantity, foregoing table four is seized, due to 2 can be seized for worker A
Overtime queue.So worker A need to seize 7 shard from other worker.
K=1 is such as first set, is seized from the 1st, be then unsatisfactory for remaining shard quantity more than the 2nd
Individual shard quantity 11.
K=2 is reset, due to making each remaining queue burst number of the preceding K progress of work flat with described
Mean difference is no more than specified quantity, and seizes the average of rear remaining queue burst more than K+1
The queue burst quantity that the individual progress of work currently takes, two limitations of condition, it is impossible to 2 in the past
The shard more than 2 is seized in work.1 can be so seized from the 1st, the 2nd is not seized.
Now worker F also need to seize 6 shard, it is necessary to continue to seize.
Now table four is changed to table five：
| worker B | 10 |
| worker C | 11 |
| worker D | 10 |
| worker E | 10 |
| worker F | 10 |
| worker A | 3 |
Table five
K=3 is reset, now because if after seizing, the average of remaining queue burst is more than the
The K+1 condition limitation of the current queue burst quantity for taking of the progress of work, it is impossible to robbed in the past 3 and appointed
One shard.K=4 is similar to.
So set K be equal to 5, then from worker C seize 2 shard, from worker B,
After seizing 1 shard in worker D, worke E, worker F respectively, both of the aforesaid bar is just met
Part, other preempt-modes are unsatisfactory for both of the aforesaid condition.
Based on foregoing queue rental period table, in another preferred embodiment of the application, the sub-step 101
Including：
Sub-step 1011, it is described to be preempted queue by queue rental period table when a queue burst is seized
Consumer's field of burst is revised as current client.
In such as table three, after worker A are robbed from worker B account for shard_3, then by lease_owner
Worker A are revised as, then the change of table three turns to following table six：
| consume_group | shard_id | lease_id | lease_owner | consumer_owner | check_point | update_time |
| 1 | shard_1 | 1 | worker A | worker A | 1/3 | …… |
| 1 | shard_2 | 2 | worker A | worker A | 1/4 | …… |
| 1 | shard_3 | 3 | worker A | worker B | 2/5 | …… |
Table six
It is also understood that client is after robbing and account for a queue burst, also can be according to Tenancy Agreement more
The status data of the new shard.
It is that client worker A are robbed and be account for N shard in this way, by above-mentioned steps, N number of shard
It is exactly queue burst to be consumed for worker A.
In actual applications, needed to judge the queue burst of time-out before sub-step 101, it is preferred that
Also included before sub-step 101：Sub-step A31-A34：
Sub-step A31, every the cycle very first time, obtains queue rental period table, and judge that the queue is rented
In phase table under individual queue burst permit holder field rental ID, if on the map tables being locally stored
The rental ID of the respective queue burst of secondary record is identical；
In the embodiment of the present application, the client lib of each worker create a map table, the map
Table includes queue segmental identification field shard_id, rents identification field lease_id, last time monitoring period
Field last_update_time.
Wherein, the map tables are in the internal memory of calculation server where worker.
After the establishment of map tables, the relevant information of all shard is first obtained from client_shard_lease tables
Write-in map tables.The relevant information, for each shard, is recorded one by one such as with shard_id as major key
Shard_id, lease_id and present system time.Then time-out can be carried out into step B11 to monitor
Journey.
In the embodiment of the present application, a client lib of worker is every very first time cycle T, from number
According to the client_shard_lease tables in storehouse, obtain each shard_id's and shard_id
lease_id。
Then, by the shard_id+lease_id of client_shard_lease tables, with map tables
Shard_id+lease_id is matched.
If shard_id is not matched, illustrate that the shard is emerging, then there is no phase in map
Should record, the shard_id+lease_id is added in map tables, and corresponding
Current system time is recorded under last_update_time fields.
In addition, the application is for emerging shard.If the lease_id of shard_id is not 0,
Illustrate that the shard is seized or taken by certain worker, then can be by shard_id, the lease_id
Map tables are recorded, and current time is recorded in last_update_time.If shard_id's
Lease_id is 0, illustrates that no worker seizes or take the shard, is recorded in map tables
The lease_id=0 of shard_id, last_update_time are set to 0, should be it is fair to consider that the shard
Time-out.
If shard_id is matched, and lease_id is not matched, then into step A32.
If shard_id+lease_id is matched, into step A33.
Sub-step A32, if in queue rental period table a queue burst rental ID, with map tables on
The rental ID of the respective queue burst of secondary record is different, then update the rent of queue burst described in map tables
It is the rental ID in queue rental period table with ID, and updates the last time prison of queue burst described in map tables
Control time field is current system time；
In the embodiment of the present application, because the thread of re-renting of each worker is every the second time cycle
The worker re-rents itself lease_owner and consumer_owner identical shard.Wherein this second
Time cycle is less than the cycle very first time, the second time cycle such as T1/2.Re-renting a shard
When, client_shard_lease tables lease_id can be changed for lease_id+1.If so worker
Shard is re-rented, then lease_id changes, illustrate that worker is normally consuming the shard.If
Lease_owner differences identical with consumer_owner, then do not allow the worker to re-rent the shard.
So, the active client workerA of the embodiment of the present application, for each shard, judges
Map tables shard_id is matched with the shard_id of queue rental period table, and lease_id is not matched, then
It is determined that the shard of response has not timed out, the last_update_time fields after the shard_id of map tables
Under, the system time recorded in its last_update_time is updated to current system time.
Sub-step A33, if in queue rental period table a queue burst rental ID, with map tables on
The rental ID of the respective queue burst of secondary record is identical, then maintain the system under last time monitoring period field
Time, and judge that whether present system time subtracts the system time of last time monitoring more than the week very first time
Phase；
As it was previously stated, if that be not the shard that worker is re-rented, then lease_id does not change,
Illustrate that worker does not consume the shard normally.
So, worker A maintain the system time under the last_update_time fields in its map table
It is constant, then judge that whether present system time sys_time subtracts last_update_time more than first
Period of time T.
Sub-step A34, if the system time that present system time subtracts last time monitoring is more than the very first time
Cycle, it is determined that respective queue burst time-out.
If present system time sys_time subtracts last_update_time more than very first time cycle T,
Then illustrate shard time-out.The shard then can be by as being preempted object.Step A11 can be entered.
So after active client is robbed and account for individual queue burst, the step 110 includes sub-step 111：
Sub-step 111, for a queue burst to be consumed, the team stored from persistent storage space
The status data of the queue burst is obtained in row rental period table.
In the embodiment of the present application, when being consumed by any client due to individual queue burst, or the team
When row burst is newly-generated, its status data is written in foregoing queue rental period table, then active client
Worker A can then obtain the queue burst after robbing and account for N number of queue burst from queue rental period table
Status data.Into step 120.
Step 120, according to the status data, determines whether that other clients are consuming the queue
Burst；
In the embodiment of the present application, in the N number of shard for being seized for active client worker A
Each shard, the status data of each shard is obtained due to abovementioned steps, then can be according to described
Status data, analyzes whether these shard also have other worker to be currently in use the shard.If not yet
Have using the shard, then into step 130.If also used, it is new that continuation obtains the shard
Status data, carries out above-mentioned judgement.
Preferably, based on foregoing queue rental period table, the status data includes team described in queue rental period table
The consumption time of the value of consumer's field, the value of permit holder field and renewal time field under row burst；Institute
Consumption time is stated for active client seizes the value that the renewal time field is changed after the queue burst
Obtain.
Based on foregoing queue rental period table, for each shard that worker A are seized, according to shard_id
Obtain the status data of the shard from queue rental period table, such as lease_owner, consumer_owner,
The respective values of update_time.
When the value of the update_time is that worker A seize the shard, consumption time value is based on
The shard_id generation times update request and are sent to database, by being somebody's turn to do for database update queue rental period table
The update_time of shard_id is worth to.
Certainly, in actual applications, consumption group is may also be combined with to go to obtain the status data of shard, and
Sent to database and update request, to update the status data in queue rental period table.
Further, the step 120 includes sub-step 121-123：
Sub-step 121, whether the value for judging the value of consumer's field of the queue burst is existing customer
End；
In actual applications, active client worker A, for its shard to be consumed, according to phase
The shard_id for answering, obtains to database status data and asks, after obtaining corresponding status data,
The value of lease_owner can be extracted from status data, if the value is worker A, is meaned
The client successfully seizes the shard.Into sub-step 121.
Sub-step 122, if the value of consumer's field of the queue burst is active client, sentences
Whether disconnected active client seizes the time more than the consumption time；
Sub-step 123, if active client seizes the time more than the consumption time, it is determined that do not have
There are other clients consuming the queue burst.
In being actually used in, after robbing and account for shard due to active client worker A, can update the data
Under queue rental period table in storehouse, the update_time values of corresponding shard_id are consumption time.
The time of seizing can be locally recorded in active client worker A.
So, it can be determined that whether active client seizes the time more than the consumption time.If worked as
The time of seizing of preceding client is more than the consumption time, then illustrate consuming the team without other clients
Row burst, the consumption progress of the queue burst updates without other clients.If active client is robbed
The time of accounting for is not more than the consumption time, then it represents that there may also be other clients and is consuming the queue point
Piece, the consumption progress of other clients may not be updated also in queue rental period table.
Preferably, described to seize the system time that the time is active client, the consumption time is described
System time and time period 1 sum when client seizes the queue burst.
In the embodiment of the present application, for convenience of calculation, shard is seized in active client workerA
When, will the lease_owner of queue rental period table when being updated to worker A, while can be by worker A
The consumption time that obtains plus the cycle very first time of present system time, update to queue rental period table this
Update_time under shard_id.
So after the status data that step 110 gets shard_id, sub-step 122 is from update_time
Consumption time is extracted, is then directly compared with the consumption time with worker A current system time, such as
Fruit present system time exceedes the consumption time, it is determined that consuming the queue point without other clients
Piece.
In actual applications, because each worker is in different calculation servers, and due to calculating
The system of server is different, so the system time of each worker also likely to be present difference.So originally
Application is then for the system time of each worker record their own, and unified very first time cycle T.
By taking foregoing worker A as an example, can be readily available since worker A are seized by abovementioned steps
Afterwards, by after T, the system time that system should be reached, the system time that this should be reached is i.e. foregoing
Consumption time.When the consumption time of the system time more than shard_3 of worker A, it is determined that without it
He is consuming the queue burst at client.System time when such as worker A seize shard_3 is
12:00:00:000, very first time cycle T is 50ms, then consumption time is 12:00:00:050, this
When worker A the status data of the shard_3 of queue rental period table can be obtained from database every T/2,
The status data includes update_time.If so system time goes to 12:00:00:051, then more than disappearing
It is time-consuming, then mean other clients in very first time cycle T, newest consumption progress is write
Enter in the check_point of shard_3, step 130 can have been entered.As system time is not more than consumption
Time, then explanation there may be other clients and consume the queue burst.In this way, avoiding each
The difference of the system time of client, causes active client pair to determine without other clients in consumption institute
State the judgement error of queue burst.
Step 130, if it is determined that consuming the queue burst without other clients, then updating described
The status data of queue burst, and obtain the current consumption progress of the queue burst；
In the embodiment of the present application, if seizing the time more than consumption time, it is determined that without other clients
Hold and consuming the queue burst, can be sent to database and update request, should in renewal queue rental period table
The status data of queue burst, while from the queue rental period table of database, obtaining entering for the queue burst
Spend the consumption progress of field.
If as it was previously stated, seizing time no more than consumption time, may have other clients in consumption
The shard, can continue executing with step 110, obtain status data and judged.
Based on foregoing queue rental period table, in another preferred embodiment of the application, step 130 includes son
Step 131-132：
Sub-step 131, rents the queue burst, and by the storage queue rental period in persistent storage space
Consumer's field under queue burst described in table is updated to active client；
Sub-step 132, in the queue rental period table stored from persistent storage space, obtains the queue
Consumption progress under the progress field of burst.
In actual applications, the shard any to be consumed for being seized for worker A, when judgement is seized
Time, the consumption progress for then illustrating the shard recorded in queue rental period table was most more than after consumption time
New shard, and consuming the shard without other worker.
So, the rental thread of worker A can rent the shard, then sent to database and asked,
It is worker A with the consumer_owner for updating the data the corresponding shard_id of queue rental period table in storehouse.
Certainly, in actual applications, the lease_id also in renewal queue rental period table under the shard_id is
Lease_id+1, represents that worker A occupy the queue burst.
Meanwhile, its note can be obtained under the update_time of the corresponding shard_id of queue rental period table
Record consumption progress.
If seizing the time more than consumption time, then can only update the shard_id in queue rental period table
Under lease_id be lease_id+1, do not allow other clients to seize the shard.
In this way, can be by the aforesaid way, for the shard of other client post-consumers, its is last
Consumption progress can be with seamless branches to active client in, in the absence of the problem of Double Spending.
In another preferred embodiment of the application, after sub-step 132, also including sub-step
133-134：
Sub-step 133, judges in queue rental period table, the person's of the seizing field under the queue burst and disappears
Whether expense person's field is active client；
Sub-step 134, if whether the person's of the seizing field and consumer's field stated under queue burst are current
Client, then for active client re-rents the queue burst.
It is appreciated that the thread of re-renting of each worker judges that the worker takes each shard,
Whether consumer_onwer and lease_owner under client_shard_lease tables each shard_id
It is identical.If identical, for the worker re-rents the shard.If it is different, then not being the worker
Re-rent the shard.
In actual applications, the thread of re-renting of each worker performs step A41 to A42 every T1/2
Process.
During each progress of work is consumed, also include：
Step A31, for each progress of work, is consuming the queue shared by the progress of work point
During piece, every the cycle very first time, in internal memory where the progress of work, based on last rental ID
The new rental ID of generation；
In the embodiment of the present application, during consuming any shard for each worker, Mei Ge
A period of time T.
Step A32, judges the last rental ID recorded in the progress of work internal memory, is rented with queue
Whether the rental ID of respective queue burst is identical in phase table；
Step A33, if identical, it is described new that identification field is rented in queue rental period table is updated
Rental ID；
Step A34, if it is different, then rental identification field is described during refusal updates queue rental period table
New rental ID.
In the embodiment of the present application, the corresponding clientlib of each worker are with each shard in current memory
Lease_id go to update the data the lease_id of the shard in storehouse.Client lib are every T time in internal memory
The middle lease_id+1 by the shard, while with the lease_id of last time and client_shard_lease tables
The lease_id of the shard compares, if identical, allows to update client_shard_lease tables
Lease_id is lease_id+1, does not allow to carry out above-mentioned renewal if difference.Due to the original of the operation
Sub- property, ensure that any time only one of which worker can be updated successfully, i.e. only one of which worker
The lease of the shard can be got.
Step 140, continues to consume the queue burst according to current consumption progress, and by the queue
The new consumption progress of burst is recorded.
When the consumption progress for getting above-mentioned shard, then worker A can enter from the consumption of the shard
Continue to consume the shard at degree.Meanwhile, during the shard is consumed, by new consumption progress more
It is new in the queue rental period table of database.
Preferably, the step of new consumption progress of the queue burst being recorded in step 140, bag
Include：Sub-step A41-A42：
Sub-step A41, every the second time cycle, in judging queue rental period table, the queue burst
Whether permit holder field is active client；
Sub-step A42, if the permit holder field of the queue burst is active client, will be current
Client updates entering to queue burst described in queue rental period table to the consumption progress of the queue burst
Under degree field.
Such as foregoing table six, active client worker A determine to consume shard_3
Afterwards, then consumption progress 2/5 is read under check_point fields under the shard_3 of table six, so
Afterwards at the 2/5 of shard_3 read data and consumed.
If it should be noted that the information of the check_point of the database shard is sky, basis
Advance configuration, at the begin or end of shard read data.Preferred disposition is from begin
Place starts to read data, it is to avoid data are omitted.
Meanwhile, during worker A consumption shard_3, new consumption progress is updated to data
Under the check_point fields of the shard_3 of the queue rental period table in storehouse.
It should be noted that the client lib of the embodiment of the present application provide check point interfaces, pass through
The check point interfaces complete to preserve the information of check point, it is ensured that carry out fail in worker
Over, or shard, when seized by different worker, worker correctly consumes data in shard.
The use of Check point is made up of following two parts：
(1) for the part of worker initialization check point：
(1.1) when a shard is determined consumable, client is loaded from database automatically
Check point information.Perform the correlation technique of step 120.
If the information of the check_point of the database shard is sky, according to advance configuration, from
Start to read data at the begin or end of shard.
(2) for the part of worker persistence check point, that is, step 140 related side is performed
Method：
(2.1) Clientlib provides interface and carries out the operation saveCheckPoint (Bool of check point
persistent).Wherein, parameter persistent is used to control whether it is understood that being persisted to external data
In storehouse, if persistent is true, persistence is to database immediately, otherwise, every certain
Time, persistence is once.The value of persistent is controlled by client lib in the embodiment of the present application.
(2.11) if persistent is true, consumption progress is persisted in database corresponding
Under the check_point fields of shard.In the embodiment of the present application, in the queue rental period table of queue burst
In permit holder field be modified to the progress of work seized after, then in the very first time being presently in
At the end of cycle T, persistent is revised as true, such that it is able to immediately be persisted to consumption progress
In database under the check_point fields of corresponding shard.
(2.12) if persistent is not true, consumption progress is stored in internal memory, meanwhile,
Backstage will periodically can exceed specifies time span also to consume progress without persistence for a long time, is persisted to
In database under the check_point fields of corresponding shard.
In the embodiment of the present application, it is identical with consumer_owner in the lease_owner of queue burst
When, can keep persistent for true, such that it is able to periodically will be more than specified time span for a long time
Progress also is consumed without persistence, the check_point fields of corresponding shard in database are persisted to
Under.This is periodically such as 2T.
(2.2) for a shard, under normal circumstances, only when in database,
When consumer_onwer is identical with worker_name, it is lasting that clientlib will just consume progress
Change in database under the check_point fields of corresponding shard.
(2.3) after a shard is seized or broken down by other worker, clientlib is called
Shutdown interfaces, notify upper layer application, and consumption progress is persisted into corresponding shard in database
Under check_point fields.
After the shard of worker is preempted, due in very first time cycle T, being somebody's turn to do in database
The consumer_owner information of shard is not yet updated, therefore the worker being preempted being capable of correct holding
Change the check point information of each shard of its consumption long.
Under distributed environment, for a client, it accounts for queue burst robbing to the embodiment of the present application
Afterwards, the queue burst is queue burst to be consumed for the client, then the client is needed
The status data of the queue burst of Tenancy Agreement acquisition is primarily based on, then according to the status number
According to determining whether that other clients are consuming the queue burst, it is determined that existing without other clients
After consuming the queue burst, the client can just obtain the current consumption progress of the queue burst, and more
The status data of the new queue burst, then the client is again with current consumption progress continuation consumption team
Row burst simultaneously records new consumption progress.The embodiment of the present invention passes through said process, in active client
When seizing the queue burst that other clients are being consumed, the consumption progress of the queue burst can be with seamless
It is delivered in active client, makes when queue burst load balancing is carried out, or certain is consumed
The client of queue burst breaks down, after the queue burst of certain client is seized by active client, currently
Client can continue to consume the queue burst according to the consumption progress consumed, it is to avoid partial data
Double Spending, makes consumption result more accurate.In addition, for the consideration of foregoing three aspects：First,
How each calculation server correctly selects the shard of consumption, to ensure that any time any shard has and only
There is a consumption calculation server；Second, while how to process certain or some calculation servers delaying
In the case of machine, when carrying out load balancing again, the data on all of shard are still able to correct place
Reason, is not repeated consumption；3rd, when processing pressure increase is, it is necessary to the calculating service of newly-increased treatment
How device, accomplish automatic load balancing, and can guarantee that arbitrary data is not repeated consumption.First technology
Also there is the scheme using Kafka Client, Kafka is that the distribution of a daily record for special disposal disappears
Breath queue, there is provided the Publish-subscribe function of message.Kakfa provides a set of senior consumption
API (Application Programming Interface, application programming interface) complete multiple work into
Synchronization between journey, the API relies on kafka backstages zookeeper systems.Due to relying on zookeeper
System, after progress of work A updates to the data that the key of certain queue burst A updates, it is impossible in time
Pass zookeeper back, then if again load balancing when, the queue burst A that other progresses of work are taken
Key still update before data.Therefore, in Kafka systems, when different operating process is seen
Zookeeper data are probably different versions, and load balancing operation fails when causing failover.
The said process of the embodiment of the present application, the progress of work of the application is right directly facing queue burst
Queue burst is processed, it also avoid Kafka systems again load balancing when, different operating process is seen
To the problem of different pieces of information version.
Furthermore, in first technology, Kinesis Client Library every time can only in load balancing again
Load balancing is carried out to a queue burst, such as progress of work A is consuming 100 queue bursts,
And queue burst is seized from progress of work A for progress of work B, during load balancing, its each load
When balanced, 1 queue burst can only be seized from progress of work A, if actually A will be seized from B
If 30 queue bursts, 30 load balancing operations are triggered, cause load balancing elapsed time
It is long, it is impossible to quickly to reach stable state
And the application, during a load balancing, can disposably seize the progress of work should
The N individual queue bursts that this is seized, the embodiment of the present application can need to seize shard's with rapid batch selection
Method, so as to be that each worker is rapidly achieved stable state, and makes what each worker was consumed
Shard numbers are balanced.
Especially based on storing in the client instance table and queue rental period table in persistent storage space,
The embodiment of the present application passes through queue rental period table, then the consumer's field in the queue rental period table
Consumer_owner, permit holder field lease_owner and progress field check_point, respectively
Record the consumption schedule parameter of the person of seizing, occupant and the occupant of each queue burst.Then one
When the progress of work seizes the queue burst that another progress of work takes, by the queue in queue rental period table
Consumer's field consumer_owner, the permit holder field lease_owner and progress field of burst
Check_point, will consume schedule parameter and passes to the progress of work seized, and make to enter in new work
Calculation server where journey, certain progress of work is delayed after machine or certain progress of work fail over, weight
When newly carrying out load balancing, when the queue burst that its original takes is seized by other progresses of work, seize
The position that can have been consumed according to it of queue burst proceed consumption, it is to avoid the repetition of partial data
Consumption, makes consumption result more accurate.
For clearer description the application, the present embodiment is with queue rental period table and client instance table as base
Plinth, is that new situation about starting is described with certain client worker A：
Step 201, creates client instance table and queue rental period table in database in advance.
The principle of similar embodiment one creates client_worker_instance tables and client_shard_lease
Table.
Then for the client executing step 202-227 of each new startup.
Step 202, starts client, and client name is write into client instance table.
When worker A are moved, client lib threads therein can be started, by the client lib threads of client
By worker_name, present system time is written to client_worker_instance tables.
Step 203, obtains the queue segmental identification of all of queue burst from queue system.
The shard_id of all shard in queue system is obtained by Client lib threads.
Step 204, obtains the rental id field of all queue bursts from the queue rental period table of database
With permit holder field.
Client lib threads obtain all shard according to shard_id from client_shard_lease tables
Lease_id and lease_owner
Step 205, in judging in queue system whether all queue bursts do not appear in queue rental period table；
If not appearing in the queue burst in queue rental period table, then the relevant information of the queue burst is write
In the queue rental period table of database.
In Client lib threads judge to have new shard_id not appear in client_shard_lease tables,
The corresponding records of the shard_id then are write to client_shard_lease tables, is by corresponding lease_id
0, other fields are set to sky.
Then Client lib thread creations seize thread and re-rent thread.Seize thread and perform queue burst
Preemption processes, re-rent thread perform queue burst rental process.
Step 206, obtains what is started in the nearest cycle very first time recorded in client instance table
The quantity live1. of client
Step 207, obtains in queue rental period table, is not timed-out the quantity of the corresponding client of queue burst
live2；The client being not timed-out is the client recorded under the permit holder field of queue burst.
Step 208, total active client terminal quantity U is obtained by live1 plus live2.
Step 209, obtains the total number P of queue burst from queue system；
Step 210, obtains the queue burst sum Q that active client has been consumed from queue rental period table；
Step 211, by N=[P/U]-Q, calculates the queue burst quantity that active client needs to seize
N。
Step 212, every the cycle very first time, obtains queue rental period table, and judge the queue rental period
In table under individual queue burst permit holder field rental ID, if with last time in the map tables being locally stored
The rental ID of the respective queue burst of record is identical；
Client seizes thread every the cycle very first time, obtains client_shard_lease in database
The lease_id of each shard in table, by its lease_id to the corresponding shard of last registration in map tables
Mutually it is compared, judges whether identical.
Step 213, if in queue rental period table a queue burst rental ID, remembered with last time in map tables
The rental ID of the respective queue burst of record is different, then update the rental ID of queue burst described in map tables
It is the rental ID in queue rental period table, and when updating the last time monitoring of queue burst described in map tables
Between field be current system time；
If last registration in the lease_id of a shard, with map tables in client_shard_lease tables
Corresponding shard lease_id it is different, then the lease_id for updating shard described in map tables is
Lease_id in client_shard_lease tables, and update the monitoring word of shard described in map tables
Section is current system time
Step 214, if in queue rental period table a queue burst rental ID, remembered with last time in map tables
The rental ID of the respective queue burst of record is identical, then when maintaining the system under last time monitoring period field
Between, and judge that whether present system time subtracts the system time of last time monitoring more than the week very first time
Phase；
If last registration in the lease_id of a shard, with map tables in client_shard_lease tables
Corresponding shard lease_id it is identical, then maintain the system time under last time monitoring field, and judge
Whether present system time subtracts the system time of last time monitoring more than very first time cycle T.
Step 215, if present system time subtracts the system time of last time monitoring more than the week very first time
Phase, it is determined that respective queue burst time-out.
Whether step 216, judge the N more than 0；
Step 217, if the N is more than 0, judges whether the queue burst quantity L of time-out is less than
N；
Step 218, if the queue burst quantity L of time-out is more than or equal to N, from the queue point of time-out
N number of queue burst is seized in piece；
Step 219, if it is determined that the queue burst quantity L of time-out is less than N, then from the queue point of time-out
L queue burst is seized in piece, and in the queue burst consumed from other clients, seizes N-L
Individual queue burst.
Wherein, for the queue burst being preempted, when a queue burst is seized, by the queue of database
In rental period table, the consumer's field for being preempted queue burst is revised as current client, and by team
Renewal time field in row rental period table is revised as consumption time；The consumption time is robbed for the client
System time and time period 1 sum when accounting for the queue burst.
It is described to be preempted by client_shard_lease tables for a shard being preempted
The lease_owner of shard is revised as the worker A, and consumption is recorded under update_time
Time.And the shard being preempted is put into re-rents thread.
For the shard seized from other clients, because this is robbed during client judges local map
The difference rented in ID and queue rental period table, now to be robbed client also identical with this for consumer's field,
Client time renewal consumption progress can be then robbed, and because the cycle re-rented is T/2, is robbed client
End can not re-rent the shard, it is impossible to re-rent, this is robbed client and cannot continue with the shard.Such as
This ensure that the consumption progress of queue burst is transmitted without being sewn in each client.
Above-mentioned steps 212- steps 219 seize thread execution for foregoing, seize each 2*T of thread execution one
It is secondary.When client worker A are newly added, its all queue burst is above-mentioned steps and seizes and obtains.
It is follow-up to perform every time, if determining that the client wants the team of other clients of load balancing by above-mentioned steps
Row burst, then perform preemption processes.
Step 220, for the queue burst seized, disappearing for the queue burst is obtained from queue rental period table
Take the value of this field, and judge whether the value of the value of consumer's field of the queue burst is existing customer
End.
Step 221, if the value of the value of consumer's field of the queue burst is active client,
Whether judge active client seizes the time more than the consumption time.
When this step judges whether the system time of the client seized is more than the consumption under update_time
Between.
Step 222, if the time of seizing of active client is more than the consumption time, rents described
Queue burst, and by the consumption in persistent storage space described in storage queue rental period table under queue burst
Person's field is updated to active client.
Whether the system time of the client that judgement is seized is more than the consumption time under update_time；Such as
The system time of really described client exceedes the consumption time, then be preempted shard's by described
Consumer_owner is revised as the client of worker A, and by the shard's being preempted
Lease_id is revised as lease_id+1, and such worker A have then rented the shard, and mean
Worker A seize successfully above-mentioned shard.
Step 223, for the queue burst that client takes, obtains from the queue rental period table of database
The person's of seizing field value and permit holder field value, and judge the person's of the seizing field under the queue burst and disappear
Whether expense person's field is active client；
Step 224, if whether the person's of the seizing field and consumer's field stated under queue burst are current visitor
Family end, then for active client re-rents the queue burst.
The step of above-mentioned steps 220-224 is performed to re-rent thread, re-rents now every T/2, for working as
The shard that preceding client takes, lease_owner in client_shard_lease tables are judged with
Consumer_owner is identical, then re-rented.
Step 225, successful queue burst is seized for client, from the queue rental period table of database
Consumption progress is obtained in progress field under the queue burst.
Client then extracts consumption progress from the check_point fields of client_shard_lease tables.When this
Consumption progress is that then the consumption progress of the shard is to be consumed since head for sky.
Step 226, client continues to consume the queue point from the consumption progress position of the queue burst
Piece.
Step 227, client is updated to database consumption progress during queue burst is consumed
In in progress field under the queue burst.
During worker consumption shard, the interface of Client lib thread dispatchings checkpoint,
Every T time, the progress of consumption is updated under the check_point of client_shard_lease tables；Or
Person when re-renting cycle arrival, calls checkpoint's after the shard lease_owner are preempted
Shutdown interfaces, notify upper layer application, and the successful check point information of current consumption is persisted to
Under the check_point of client_shard_lease tables；Or, when worker A break down, call
The shutdown interfaces of checkpoint, notify upper layer application, by the successful check point of current consumption
Information is persisted under the check_point of client_shard_lease tables.
Certainly, seized after queue burst for the first time for client executing, it can perform step 203
And later step.
It should be noted that for embodiment of the method, in order to be briefly described, therefore it is all expressed as into one
The combination of actions of series, but those skilled in the art should know, and the embodiment of the present application does not receive institute
The limitation of the sequence of movement of description, because according to the embodiment of the present application, some steps can use other
Order is carried out simultaneously.Secondly, those skilled in the art should also know, described in the specification
Embodiment belong to preferred embodiment, involved action not necessarily the embodiment of the present application is musted
Must.
Embodiment two
Reference picture 2, shows that a kind of distributed environment of the application coordinates the system embodiment of consumption queue
Structured flowchart, can specifically include such as lower module：
State data acquisition module 310, for for a queue burst to be consumed, based on Tenancy Agreement
The status data of the queue burst for obtaining；
Consumption judge module 320, for according to the status data, determining whether that other clients exist
Consume the queue burst；
Progress acquisition module 330, for if it is determined that without other clients consuming the queue burst,
The status data of the queue burst is then updated, and obtains the current consumption progress of the queue burst；
Consumption module 340, for being continued to consume the queue burst according to current consumption progress, and will
The new consumption progress of the queue burst is recorded.
In the preferred embodiment of the application one, before state data acquisition module 310, also include：
Queue burst determining module, for determining the to be consumed of active client demand based on Tenancy Agreement
Queue burst.
In the preferred embodiment of the application one, the queue burst determining module includes：
Quantity required determination sub-module, for obtaining active client sum U, queue burst sum P
And the queue burst sum Q that active client has been consumed, the team seized with calculating active client to need
Row burst quantity N；
Submodule is seized, for the queue consumed from the queue burst and/or other clients of time-out
In burst, to be consumed queue burst of N number of queue burst as active client is seized.
In the preferred embodiment of the application one, the quantity required determination sub-module includes：
Client sum acquisition submodule, for the client instance table stored from persistent storage space
With active client sum U is obtained in queue rental period table；
Overall queue burst sum acquisition submodule, total for obtaining queue burst from queue system
Number P；
Monomer queue burst sum acquisition submodule, the queue for being stored from persistent storage space is rented
The queue burst sum Q that active client has been consumed is obtained in phase table；
Quantity required calculating sub module, needs to seize for by N=[P/U]-Q, calculating active client
Queue burst quantity N.
In the preferred embodiment of the application one, the client sum acquisition submodule includes：
Newly-built client terminal quantity acquisition submodule, nearest one for obtaining record in client instance table
The quantity live1 of the client started in the cycle very first time；
Have not timed out client terminal quantity acquisition submodule, for obtaining queue rental period table in, be not timed-out queue
The quantity live2 of the corresponding client of burst；The client being not timed-out is the rental of queue burst
The client recorded under person's field；
Client terminal quantity adds up submodule, for live1 to be obtained into total active client plus live2
Quantity U.
In the preferred embodiment of the application one, the submodule of seizing includes：
First judging submodule, for judging the N whether more than 0；
Second judging submodule, if being more than 0 for the N, judges the queue burst quantity of time-out
Whether L is less than N；
Full time-out seizes submodule, if being more than or equal to N for the queue burst quantity L of time-out, from
N number of queue burst is seized in the queue burst of time-out；
Mixing seizes submodule, for if it is determined that the queue burst quantity L of time-out is less than N, then from super
When queue burst in seize L queue burst, and the queue burst consumed from other clients
In, seize N-L queue burst.
In the preferred embodiment of the application one, the mixing seizes module to be included：
First mixing seizes submodule, for the queue burst that active client takes from other clients
In seize N-L individual queue bursts, and each client is taken the quantity of most queue bursts and is taken most
The difference of the quantity of few queue burst is no more than specified quantity.
In the preferred embodiment of the application one, first mixing seizes submodule to be included：
Sorting sub-module, for based on the queue rental period table, the team that each client is taken according to it
Row burst quantity, from being more to ranked up less；
Second mixing seizes module, is seized for seizing one or more in K client of each the past
After queue burst, preceding K client is set to be preempted after J, the preceding remaining queue of K client
The average of burst makes preceding K more than the queue burst quantity that the K+1 client currently takes
In individual client, the remaining queue burst number of each client is differed no more than specified number with the average
Amount, until successfully rob accounting for N-L queue burst.
In the preferred embodiment of the application one, the submodule of seizing includes：
Consumer's field changes submodule, for when a queue burst is seized, by queue rental period table,
The consumer's field for being preempted queue burst is revised as current client.
In the preferred embodiment of the application one, it is described seize submodule before, also include：
Map table judging submodules, for every the cycle very first time, obtaining queue rental period table, and judge
In the queue rental period table under individual queue burst permit holder field rental ID, if be locally stored
The rental ID of the respective queue burst of last registration is identical in map tables；
Map monitoring periods update submodule, if for the rental of a queue burst in queue rental period table
ID, the rental ID from the respective queue burst of last registration in map tables is different, then in renewal map tables
The rental ID of the queue burst is the rental ID in queue rental period table, and is updated described in map tables
The last time monitoring period field of queue burst is current system time；
Map monitoring periods maintain submodule, if for the rental of a queue burst in queue rental period table
ID, the rental ID with the respective queue burst of last registration in map tables is identical, then maintain last time monitoring
System time under time field, and judge that the system time that present system time subtracts last time monitoring is
It is no more than the cycle very first time；
Overtime judging submodule, if for present system time subtract last time monitoring system time it is big
In the cycle very first time, it is determined that respective queue burst time-out.
In the preferred embodiment of the application one, the queue burst determining module includes：
First queue burst determination sub-module, for based on Tenancy Agreement every the cycle very first time, it is determined that
The queue burst to be consumed of active client demand.
In the preferred embodiment of the application one, the state data acquisition module 310 includes：
First state data acquisition submodule, for for a queue burst to be consumed, being deposited from persistence
The status data of the queue burst is obtained in the queue rental period table stored in storage space.
In the preferred embodiment of the application one, the status data includes queue described in queue rental period table
The consumption time of the value of consumer's field, the value of permit holder field and renewal time field under burst；It is described
Consumption time is changed the value of the renewal time field and is obtained for active client is seized after the queue burst
；
Then, the consumption judge module 320 includes：
Consumer's field judging submodule, the value of the value of the consumer's field for judging the queue burst
Whether it is active client；
Time judging submodule is seized, if being for the value of the value of consumer's field of the queue burst
Whether active client, then judge active client seizes the time more than the consumption time；
Determination sub-module, if being more than the consumption time for the time of seizing of active client, really
Surely the queue burst is being consumed without other clients.
It is described to seize the system time that the time is active client in the preferred embodiment of the application one,
The consumption time be system time when the client seizes the queue burst with the period 1 when
Between sum.
In the preferred embodiment of the application one, the progress acquisition module 330 includes：
Rent submodule, the queue burst for rental, and by storage queue in persistent storage space
Consumer's field described in rental period table under queue burst is updated to active client；
Progress field read module, for from persistent storage space store queue rental period table in, obtain
Take the consumption progress under the progress field of the queue burst.
In the preferred embodiment of the application one, the consumption module 320 includes：
Permit holder judging submodule, for every the second time cycle, judging queue rental period table in, it is described
Whether the permit holder field of queue burst is active client；
Consumption progress updates submodule, if being existing customer for the permit holder field of the queue burst
End, then update to described in queue rental period table active client to the consumption progress of the queue burst
Under the progress field of queue burst.
In the preferred embodiment of the application one, after submodule is rented, also include：
Judging submodule is re-rented, for judging in queue rental period table, seizing under the queue burst
Whether person's field and consumer's field are active client；
First re-rents submodule, if for stating the person's of the seizing field under queue burst and consumer's field
Whether it is active client, then for active client re-rents the queue burst.
With reference to Fig. 2A, it illustrates the example of the distributed environment of the application, wherein each calculate node
As one client.Dispatch server can be with distribution queue burst.Each client includes：Status number
According to acquisition module 310, consumption judge module 320, progress acquisition module 330, consumption module 340.When
Right each client may also comprise corresponding preferred module.
Under distributed environment, for a client, its after robbing and accounting for queue burst, the queue
Burst is queue burst to be consumed for the client, then the client needs to be primarily based on rent
The status data of the queue burst that agreement of renting is obtained, then according to the status data, judges whether
There are other clients consuming the queue burst, it is determined that consuming the queue point without other clients
After piece, the client can just obtain the current consumption progress of the queue burst, and update the queue burst
Status data, then the client continue to consume the queue burst and record with current consumption progress again
New consumption progress.The embodiment of the present invention seizes other clients by said process in active client
Rectify in the queue burst of consumption, the consumption progress of the queue burst can be with seamless delivery to current visitor
In the end of family, make when queue burst load balancing is carried out, or certain is consuming the visitor of queue burst
Family end breaks down, and after the queue burst of certain client is seized by active client, active client can be by
Continue to consume the queue burst according to the consumption progress consumed, it is to avoid the Double Spending of partial data, make
Consumption result is more accurate
For device embodiment, because it is substantially similar to embodiment of the method, so the ratio of description
Relatively simple, the relevent part can refer to the partial explaination of embodiments of method.
Each embodiment in this specification is described by the way of progressive, and each embodiment is stressed
Be all difference with other embodiment, between each embodiment identical similar part mutually referring to
.
It should be understood by those skilled in the art that, the embodiment of the embodiment of the present application can be provided as method, dress
Put or computer program product.Therefore, the embodiment of the present application can using complete hardware embodiment, completely
The form of the embodiment in terms of software implementation or combination software and hardware.And, the embodiment of the present application
Can use can be situated between in one or more computers for wherein including computer usable program code with storage
The computer journey implemented in matter (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.)
The form of sequence product.
In a typical configuration, the computer equipment includes one or more consumers
(CPU), input/output interface, network interface and internal memory.Internal memory potentially includes computer-readable medium
In volatile memory, the shape such as random access memory (RAM) and/or Nonvolatile memory
Formula, such as read-only storage (ROM) or flash memory (flash RAM).Internal memory is computer-readable medium
Example.Computer-readable medium includes permanent and non-permanent, removable and non-removable media
Information Store can be realized by any method or technique.Information can be computer-readable instruction,
Data structure, the module of program or other data.The example of the storage medium of computer includes, but
Phase transition internal memory (PRAM), static RAM (SRAM), dynamic random is not limited to deposit
Access to memory (DRAM), other kinds of random access memory (RAM), read-only storage
(ROM), Electrically Erasable Read Only Memory (EEPROM), fast flash memory bank or other in
Deposit technology, read-only optical disc read-only storage (CD-ROM), digital versatile disc (DVD) or other
Optical storage, magnetic cassette tape, tape magnetic rigid disk storage other magnetic storage apparatus or it is any its
His non-transmission medium, can be used to store the information that can be accessed by a computing device.According to herein
Define, computer-readable medium does not include the computer readable media (transitory media) of non-standing,
Such as the data-signal and carrier wave of modulation.
The embodiment of the present application is with reference to the method according to the embodiment of the present application, terminal device (system) and meter
The flow chart and/or block diagram of calculation machine program product is described.It should be understood that can be by computer program instructions
Realize each flow and/or square frame and flow chart and/or the square frame in flow chart and/or block diagram
The combination of flow and/or square frame in figure.Can provide these computer program instructions to all-purpose computer,
The consumer of special-purpose computer, embedded charge machine or other programmable data consumption terminal equipment is producing
One machine so that performed by the consumer of computer or other programmable data consumption terminal equipment
Instruction produce for realizing in one flow of flow chart or multiple one square frame of flow and/or block diagram or
The device of the function of being specified in multiple square frames.
These computer program instructions may be alternatively stored in can guide computer or other programmable datas to consume
In the computer-readable memory that terminal device works in a specific way so that storage is in the computer-readable
Instruction in memory is produced and includes the manufacture of command device, and command device realization is in flow chart one
The function of being specified in flow or multiple one square frame of flow and/or block diagram or multiple square frames.
These computer program instructions can also be loaded into computer or other programmable data consumption terminals set
It is standby upper so that execution series of operation steps is in terms of producing on computer or other programmable terminal equipments
The consumption that calculation machine is realized, so as to the instruction performed on computer or other programmable terminal equipments provides use
In realization in one flow of flow chart or multiple one square frame of flow and/or block diagram or multiple square frames
The step of function of specifying.
Although having been described for the preferred embodiment of the embodiment of the present application, those skilled in the art are once
Basic creative concept is known, then other change and modification can be made to these embodiments.So,
Appended claims are intended to be construed to include preferred embodiment and fall into the institute of the embodiment of the present application scope
Have altered and change.
Finally, in addition it is also necessary to explanation, herein, such as first and second or the like relation art
Language is used merely to make a distinction an entity or operation with another entity or operation, and not necessarily
It is required that or implying between these entities or operation there is any this actual relation or order.And
And, term " including ", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, from
And cause that the process, method, article or the terminal device that include a series of key elements not only will including those
Element, but also other key elements including being not expressly set out, or also include being this process, side
Method, article or the intrinsic key element of terminal device.In the absence of more restrictions, by sentence
The key element that "including a ..." is limited, it is not excluded that in the process including the key element, method, thing
Also there is other identical element in product or terminal device.
Coordinate consumption queuing method and a kind of distribution to a kind of distributed environment provided herein above
Formula environmental harmony consumes queue system, is described in detail, and specific case used herein is to this
The principle and implementation method of application are set forth, and the explanation of above example is only intended to help and understands
The present processes and its core concept；Simultaneously for those of ordinary skill in the art, according to this
The thought of application, will change in specific embodiments and applications, in sum,
This specification content should not be construed as the limitation to the application.
Claims (34)
1. a kind of method that distributed environment coordinates consumption queue, it is characterised in that including：
For a queue burst to be consumed, the status number of the queue burst obtained based on Tenancy Agreement
According to；
According to the status data, determine whether that other clients are consuming the queue burst；
If it is determined that consuming the queue burst without other clients, then the queue burst is updated
Status data, and obtain the current consumption progress of the queue burst；
Continue to consume the queue burst according to current consumption progress, and disappear the queue burst is new
Expense progress is recorded.
2. method according to claim 1, it is characterised in that for a queue to be consumed
Before the step of burst, status data of the queue burst obtained based on Tenancy Agreement, also include：
The queue burst to be consumed of active client demand is determined based on Tenancy Agreement.
3. method according to claim 2, it is characterised in that described to be determined based on Tenancy Agreement
The step of queue burst to be consumed of active client demand, including：
The team that the active client sum U of acquisition, queue burst sum P and active client have been consumed
Row burst sum Q, the queue burst quantity N seized with calculating active client to need；
In the queue burst consumed from the queue burst and/or other clients of time-out, N is seized
Individual queue burst as active client queue burst to be consumed.
4. method according to claim 3, it is characterised in that the active client of the acquisition
The step of queue burst sum M that total U, queue burst sum P and active client have been consumed,
Including：
Obtain active in the client instance table and queue rental period table that are stored from persistent storage space
Client sum U；
The total number P of queue burst is obtained from queue system；
The team that active client has been consumed is obtained in the queue rental period table stored from persistent storage space
Row burst sum Q；
By N=[P/U]-Q, the queue burst quantity N that active client needs to seize is calculated.
5. method according to claim 4, it is characterised in that described from persistent storage space
Active client sum U steps are obtained in the client instance table and queue rental period table of middle storage, is wrapped
Include：
Obtain the client of startup in the nearest cycle very first time recorded in client instance table
Quantity live1；
Obtain in queue rental period table, be not timed-out the quantity live2 of the corresponding client of queue burst；Institute
It is the client recorded under the permit holder field of queue burst to state the client being not timed-out；
Live1 is obtained into total active client terminal quantity U plus live2.
6. method according to claim 3, it is characterised in that the queue burst from time-out
And/or in the queue burst consumed of other clients, N number of queue burst is seized as current visitor
The step of queue burst to be consumed at family end, including：
Judge the N whether more than 0；
If the N is more than 0, judge the queue burst quantity L of time-out whether less than N；
If the queue burst quantity L of time-out is more than or equal to N, N is seized from the queue burst of time-out
Individual queue burst；
If it is determined that the queue burst quantity L of time-out is less than N, then L is seized from the queue burst of time-out
In individual queue burst, and the queue burst consumed from other clients, N-L queue point is seized
Piece.
7. method according to claim 6, it is characterised in that it is described from other clients
In the queue burst of consumption, the step of seize N-L queue burst, including：
N-L individual queue bursts are seized in the queue burst that active client takes from other clients, and
The quantity of the most queue bursts of each client occupancy and the difference of the quantity for taking minimum queue burst is set not to surpass
Cross specified quantity.
8. method according to claim 7, it is characterised in that the active client from other
N-L individual queue bursts are seized in the queue burst that client takes, and each client is taken most teams
The step of difference of the quantity of row burst and the quantity for taking minimum queue burst is no more than specified quantity, bag
Include：
Based on the queue rental period table, the queue burst quantity that each client is taken according to it, from many
To being ranked up less；
Seized in each K client of the past after one or more seize queue burst, make first K it is objective
Family end is preempted after J, and the average of the preceding remaining queue burst of K client is more than K+1
The queue burst quantity that individual client currently takes, and make preceding K client, each client is surplus
Remaining queue burst number is differed no more than specified quantity with the average, until successfully rob accounting for N-L
Queue burst.
9. method according to claim 3, it is characterised in that the queue burst from time-out
And/or in the queue burst consumed of other clients, N number of queue burst is seized as current visitor
The step of queue burst to be consumed at family end, includes：
When a queue burst is seized, by queue rental period table, the consumer for being preempted queue burst
Field is revised as current client.
10. method according to claim 3, it is characterised in that the queue burst from time-out
And/or in the queue burst consumed of other clients, N number of queue burst is seized as current visitor
Before the step of queue burst to be consumed at family end, also include：
Every the cycle very first time, queue rental period table is obtained, and judge individual queue in the queue rental period table
The rental ID of permit holder field under burst, if in the map tables being locally stored last registration it is corresponding
The rental ID of queue burst is identical；
If the rental ID of a queue burst in queue rental period table, in map tables last registration it is corresponding
The rental ID of queue burst is different, then the rental ID for updating queue burst described in map tables is queue
Rental ID in rental period table, and update the last time monitoring period field of queue burst described in map tables
It is current system time；
If the rental ID of a queue burst in queue rental period table, in map tables last registration it is corresponding
The rental ID of queue burst is identical, then maintain the system time under last time monitoring period field, and judge
Whether present system time subtracts the system time of last time monitoring more than the cycle very first time；
If the system time that present system time subtracts last time monitoring is more than the cycle very first time, really
Determine respective queue burst time-out.
11. methods according to claim 2, it is characterised in that described to be determined based on Tenancy Agreement
The step of queue burst to be consumed of active client demand, includes：
Based on Tenancy Agreement every the cycle very first time, the queue to be consumed of active client demand is determined
Burst.
12. according to one of claim 1-11 described method, it is characterised in that described to be directed to one
The step of queue burst to be consumed, status data of the queue burst of acquisition, including：
For a queue burst to be consumed, obtained in the queue rental period table stored from persistent storage space
Take the status data of the queue burst.
13. methods according to claim 12, it is characterised in that the status data includes team
The value of consumer's field, the value of permit holder field and time word is updated under queue burst described in row rental period table
The consumption time of section；The consumption time is seized after the queue burst described in modification more for active client
The value of new time field is obtained；
Then, it is described according to the status data, determine whether that other clients are consuming the queue point
The step of piece, including：
Whether the value for judging the value of consumer's field of the queue burst is active client；
If the value of the value of consumer's field of the queue burst is active client, current visitor is judged
Whether the time is seized more than the consumption time in family end；
If the time of seizing of active client is more than the consumption time, it is determined that without other clients
Consuming the queue burst.
14. methods according to claim 13, it is characterised in that the time of seizing is current
The system time of client, the consumption time is the system when client seizes the queue burst
Time and time period 1 sum.
15. methods according to claim 12, it is characterised in that the renewal queue point
The status data of piece, and the step of obtain the queue burst current consumption progress, including：
Rent the queue burst, and by queue described in storage queue rental period table in persistent storage space
Consumer's field under burst is updated to active client；
In the queue rental period table stored from persistent storage space, the progress word of the queue burst is obtained
Consumption progress under section.
16. methods according to claim 15, it is characterised in that described by the queue burst
The step of new consumption progress is recorded, including：
Every the second time cycle, judge in queue rental period table, the permit holder field of the queue burst is
No is active client；
If the permit holder field of the queue burst is active client, by active client to described
The consumption progress of queue burst, updates under the progress field of queue burst described in queue rental period table.
17. methods according to claim 15, it is characterised in that from persistent storage space
In the queue rental period table of middle storage, the step of obtain the consumption progress under the progress field of the queue burst
Afterwards, also include：
Judge in queue rental period table, the person's of the seizing field and consumer's field under the queue burst are
No is active client；
If whether the person's of the seizing field and consumer's field stated under queue burst are active client,
For active client re-rents the queue burst.
A kind of 18. distributed environments coordinate the device of consumption queue, it is characterised in that including：
State data acquisition module, for for a queue burst to be consumed, being obtained based on Tenancy Agreement
The queue burst status data；
Consumption judge module, for according to the status data, determining whether other clients in consumption
The queue burst；
Progress acquisition module, for if it is determined that consuming the queue burst without other clients, then
The status data of the queue burst is updated, and obtains the current consumption progress of the queue burst；
Consumption module, for being continued to consume the queue burst according to current consumption progress, and will be described
The new consumption progress of queue burst is recorded.
19. devices according to claim 18, it is characterised in that in state data acquisition module
Before, also include：
Queue burst determining module, for determining the to be consumed of active client demand based on Tenancy Agreement
Queue burst.
20. devices according to claim 19, it is characterised in that the queue burst determines mould
Block includes：
Quantity required determination sub-module, for obtaining active client sum U, queue burst sum P
And the queue burst sum Q that active client has been consumed, the team seized with calculating active client to need
Row burst quantity N；
Submodule is seized, for the queue consumed from the queue burst and/or other clients of time-out
In burst, to be consumed queue burst of N number of queue burst as active client is seized.
21. devices according to claim 19, it is characterised in that the quantity required determines son
Module includes：
Client sum acquisition submodule, for the client instance table stored from persistent storage space
With active client sum U is obtained in queue rental period table；
Overall queue burst sum acquisition submodule, total for obtaining queue burst from queue system
Number P；
Monomer queue burst sum acquisition submodule, the queue for being stored from persistent storage space is rented
The queue burst sum Q that active client has been consumed is obtained in phase table；
Quantity required calculating sub module, needs to seize for by N=[P/U]-Q, calculating active client
Queue burst quantity N.
22. devices according to claim 21, it is characterised in that the client sum is obtained
Submodule includes：
Newly-built client terminal quantity acquisition submodule, nearest one for obtaining record in client instance table
The quantity live1 of the client started in the cycle very first time；
Have not timed out client terminal quantity acquisition submodule, for obtaining queue rental period table in, be not timed-out queue
The quantity live2 of the corresponding client of burst；The client being not timed-out is the rental of queue burst
The client recorded under person's field；
Client terminal quantity adds up submodule, for live1 to be obtained into total active client plus live2
Quantity U.
23. devices according to claim 20, it is characterised in that the submodule of seizing includes：
First judging submodule, for judging the N whether more than 0；
Second judging submodule, if being more than 0 for the N, judges the queue burst quantity of time-out
Whether L is less than N；
Full time-out seizes submodule, if being more than or equal to N for the queue burst quantity L of time-out, from
N number of queue burst is seized in the queue burst of time-out；
Mixing seizes submodule, for if it is determined that the queue burst quantity L of time-out is less than N, then from super
When queue burst in seize L queue burst, and the queue burst consumed from other clients
In, seize N-L queue burst.
24. devices according to claim 23, it is characterised in that the mixing seizes module bag
Include：
First mixing seizes submodule, for the queue burst that active client takes from other clients
In seize N-L individual queue bursts, and each client is taken the quantity of most queue bursts and is taken most
The difference of the quantity of few queue burst is no more than specified quantity.
25. devices according to claim 24, it is characterised in that first mixing seizes son
Module includes：
Sorting sub-module, for based on the queue rental period table, the team that each client is taken according to it
Row burst quantity, from being more to ranked up less；
Second mixing seizes module, is seized for seizing one or more in K client of each the past
After queue burst, preceding K client is set to be preempted after J, the preceding remaining queue of K client
The average of burst makes preceding K more than the queue burst quantity that the K+1 client currently takes
In individual client, the remaining queue burst number of each client is differed no more than specified number with the average
Amount, until successfully rob accounting for N-L queue burst.
26. devices according to claim 20, it is characterised in that the submodule of seizing includes：
Consumer's field changes submodule, for when a queue burst is seized, by queue rental period table,
The consumer's field for being preempted queue burst is revised as current client.
27. devices according to claim 20, it is characterised in that it is described seize submodule before,
Also include：
Map table judging submodules, for every the cycle very first time, obtaining queue rental period table, and judge
In the queue rental period table under individual queue burst permit holder field rental ID, if be locally stored
The rental ID of the respective queue burst of last registration is identical in map tables；
Map monitoring periods update submodule, if for the rental of a queue burst in queue rental period table
ID, the rental ID from the respective queue burst of last registration in map tables is different, then in renewal map tables
The rental ID of the queue burst is the rental ID in queue rental period table, and is updated described in map tables
The last time monitoring period field of queue burst is current system time；
Map monitoring periods maintain submodule, if for the rental of a queue burst in queue rental period table
ID, the rental ID with the respective queue burst of last registration in map tables is identical, then maintain last time monitoring
System time under time field, and judge that the system time that present system time subtracts last time monitoring is
It is no more than the cycle very first time；
Overtime judging submodule, if for present system time subtract last time monitoring system time it is big
In the cycle very first time, it is determined that respective queue burst time-out.
28. devices according to claim 19, it is characterised in that the queue burst determines mould
Block includes：
First queue burst determination sub-module, for based on Tenancy Agreement every the cycle very first time, it is determined that
The queue burst to be consumed of active client demand.
29. according to one of claim 18-28 described device, it is characterised in that the state
Data acquisition module includes：
First state data acquisition submodule, for for a queue burst to be consumed, being deposited from persistence
The status data of the queue burst is obtained in the queue rental period table stored in storage space.
30. devices according to claim 29, it is characterised in that the status data includes team
The value of consumer's field, the value of permit holder field and time word is updated under queue burst described in row rental period table
The consumption time of section；The consumption time is seized after the queue burst described in modification more for active client
The value of new time field is obtained；
Then, the consumption judge module includes：
Consumer's field judging submodule, the value of the value of the consumer's field for judging the queue burst
Whether it is active client；
Time judging submodule is seized, if being for the value of the value of consumer's field of the queue burst
Whether active client, then judge active client seizes the time more than the consumption time；
Determination sub-module, if being more than the consumption time for the time of seizing of active client, really
Surely the queue burst is being consumed without other clients.
31. devices according to claim 30, it is characterised in that the time of seizing is current
The system time of client, the consumption time is the system when client seizes the queue burst
Time and time period 1 sum.
32. devices according to claim 29, it is characterised in that the progress acquisition module bag
Include：
Rent submodule, the queue burst for rental, and by storage queue in persistent storage space
Consumer's field described in rental period table under queue burst is updated to active client；
Progress field read module, for from persistent storage space store queue rental period table in, obtain
Take the consumption progress under the progress field of the queue burst.
33. devices according to claim 32, it is characterised in that the consumption module includes：
Permit holder judging submodule, for every the second time cycle, judging queue rental period table in, it is described
Whether the permit holder field of queue burst is active client；
Consumption progress updates submodule, if being existing customer for the permit holder field of the queue burst
End, then update to described in queue rental period table active client to the consumption progress of the queue burst
Under the progress field of queue burst.
34. devices according to claim 32, it is characterised in that after submodule is rented,
Also include：
Judging submodule is re-rented, for judging in queue rental period table, seizing under the queue burst
Whether person's field and consumer's field are active client；
First re-rents submodule, if for stating the person's of the seizing field under queue burst and consumer's field
Whether it is active client, then for active client re-rents the queue burst.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511021136.1A CN106933672B (en) | 2015-12-30 | 2015-12-30 | Distributed environment coordinated consumption queue method and device |
| PCT/CN2016/110230 WO2017114176A1 (en) | 2015-12-30 | 2016-12-16 | Method and apparatus for coordinating consumption queue in distributed environment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511021136.1A CN106933672B (en) | 2015-12-30 | 2015-12-30 | Distributed environment coordinated consumption queue method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106933672A true CN106933672A (en) | 2017-07-07 |
| CN106933672B CN106933672B (en) | 2021-04-13 |
Family
ID=59224458
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201511021136.1A Active CN106933672B (en) | 2015-12-30 | 2015-12-30 | Distributed environment coordinated consumption queue method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN106933672B (en) |
| WO (1) | WO2017114176A1 (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108199912A (en) * | 2017-12-15 | 2018-06-22 | 北京奇艺世纪科技有限公司 | Management, consuming method and the device of a kind of strange land distributed message mostly living |
| CN111309700A (en) * | 2020-02-14 | 2020-06-19 | 苏州浪潮智能科技有限公司 | Control method and system for multi-sharing directory tree |
| CN112527527A (en) * | 2020-12-16 | 2021-03-19 | 深圳市分期乐网络科技有限公司 | Consumption speed control method and device of message queue, electronic equipment and medium |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111913909B (en) * | 2019-05-08 | 2024-02-23 | 厦门网宿有限公司 | Repartitioning method and system in distributed storage system |
| CN111343252A (en) * | 2020-02-13 | 2020-06-26 | 深圳壹账通智能科技有限公司 | High-concurrency data transmission method based on http2 protocol and related equipment |
| CN111988359B (en) * | 2020-07-15 | 2023-08-15 | 中国科学院计算技术研究所数字经济产业研究院 | Data fragment synchronization method and system based on message queue |
| CN114448989B (en) * | 2022-01-26 | 2024-04-05 | 北京百度网讯科技有限公司 | Method, device, electronic equipment, storage medium and product for adjusting message distribution |
| CN117251508A (en) * | 2023-09-22 | 2023-12-19 | 湖南长银五八消费金融股份有限公司 | Borrowing batch accounting method, device, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103379021A (en) * | 2012-04-24 | 2013-10-30 | 中兴通讯股份有限公司 | Method and system for achieving distributed message queue |
| CN104468330A (en) * | 2014-12-03 | 2015-03-25 | 北京国双科技有限公司 | Data processing method and device of distributed information queue |
| US20150180792A1 (en) * | 2010-06-23 | 2015-06-25 | Amazon Technologies, Inc. | Balancing a load on a multiple consumer queue |
| CN104754036A (en) * | 2015-03-06 | 2015-07-01 | 合一信息技术（北京）有限公司 | Message processing system and processing method based on kafka |
| US20150200886A1 (en) * | 2014-01-14 | 2015-07-16 | International Business Machines Corporation | Message switch file sharing |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102289392A (en) * | 2011-09-08 | 2011-12-21 | 曙光信息产业股份有限公司 | Operation scheduling method and system based on check point |
-
2015
- 2015-12-30 CN CN201511021136.1A patent/CN106933672B/en active Active
-
2016
- 2016-12-16 WO PCT/CN2016/110230 patent/WO2017114176A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150180792A1 (en) * | 2010-06-23 | 2015-06-25 | Amazon Technologies, Inc. | Balancing a load on a multiple consumer queue |
| CN103379021A (en) * | 2012-04-24 | 2013-10-30 | 中兴通讯股份有限公司 | Method and system for achieving distributed message queue |
| US20150200886A1 (en) * | 2014-01-14 | 2015-07-16 | International Business Machines Corporation | Message switch file sharing |
| CN104468330A (en) * | 2014-12-03 | 2015-03-25 | 北京国双科技有限公司 | Data processing method and device of distributed information queue |
| CN104754036A (en) * | 2015-03-06 | 2015-07-01 | 合一信息技术（北京）有限公司 | Message processing system and processing method based on kafka |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108199912A (en) * | 2017-12-15 | 2018-06-22 | 北京奇艺世纪科技有限公司 | Management, consuming method and the device of a kind of strange land distributed message mostly living |
| CN111309700A (en) * | 2020-02-14 | 2020-06-19 | 苏州浪潮智能科技有限公司 | Control method and system for multi-sharing directory tree |
| CN111309700B (en) * | 2020-02-14 | 2022-11-29 | 苏州浪潮智能科技有限公司 | Control method and system for multi-sharing directory tree |
| CN112527527A (en) * | 2020-12-16 | 2021-03-19 | 深圳市分期乐网络科技有限公司 | Consumption speed control method and device of message queue, electronic equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106933672B (en) | 2021-04-13 |
| WO2017114176A1 (en) | 2017-07-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106933672A (en) | A kind of distributed environment coordinates consumption queuing method and device | |
| WO2018103590A1 (en) | Method, device, and system for assigning pickup tasks | |
| KR101959153B1 (en) | System for efficient processing of transaction requests related to an account in a database | |
| CN106406983A (en) | Task scheduling method and device in cluster | |
| CN109144699A (en) | Distributed task dispatching method, apparatus and system | |
| JPH11120106A (en) | Display system of server resource use state and storage medium for it | |
| CN106383764B (en) | A kind of collecting method and equipment | |
| CN110806996B (en) | System and method for collecting OPC data | |
| CN110163473A (en) | A kind of monitoring method and equipment of Task Progress | |
| CN109408286A (en) | Data processing method, device, system, computer readable storage medium | |
| CN106843945B (en) | PaaS-based GIS application deployment method and system | |
| CN108319495A (en) | Task processing method and device | |
| CN107451147A (en) | A kind of method and apparatus of kafka clusters switching at runtime | |
| CN110928655A (en) | Task processing method and device | |
| CN108337275A (en) | Task distribution method, device and equipment for Distributor | |
| CN105262835A (en) | Data storage method and device of multiple machine rooms | |
| CN109062697A (en) | It is a kind of that the method and apparatus of spatial analysis service are provided | |
| CN108492005A (en) | Project data processing method, device, computer equipment and storage medium | |
| CN108228709A (en) | Date storage method and system, electronic equipment, program and medium | |
| CN109614270A (en) | Data read-write method, device, equipment and storage medium based on Hbase | |
| CN115114359A (en) | User data processing method and device | |
| CN109558239A (en) | A kind of method for scheduling task, device, system, computer equipment and storage medium | |
| CN111625418A (en) | Process monitoring method and device | |
| CN107766306B (en) | Data display method and equipment | |
| CN115858499A (en) | Database partition processing method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |