CN109656684A - A kind of partition method of Kafka, partition system and relevant apparatus - Google Patents
A kind of partition method of Kafka, partition system and relevant apparatus Download PDFInfo
- Publication number
- CN109656684A CN109656684A CN201811512234.9A CN201811512234A CN109656684A CN 109656684 A CN109656684 A CN 109656684A CN 201811512234 A CN201811512234 A CN 201811512234A CN 109656684 A CN109656684 A CN 109656684A
- Authority
- CN
- China
- Prior art keywords
- subregion
- consumer
- container
- partitions
- difference
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application provides the partition method of Kafka a kind of, comprising: S1: obtaining the corresponding practical number of partitions of consumer's container of the practical total number of partitions, the consumer's container number currently survived and each survival；S2: the average deserved number of partitions is determined；S3: according to practical number of partitions peace should all scoring area number determine subregion difference；S4: a subregion is removed from the maximum first consumer container of subregion difference, distributes second consumer's container to subregion difference less than 0, and update the corresponding subregion difference of consumer's container；S5: repeating S4, until the absolute value of the difference of the corresponding practical number of partitions of any two consumer container is less than or equal to 1.The application greatly reduces the time spent by subregion, improves the efficiency of subregion, realizes dynamic partition, can satisfy high real-time requirements.The application also provides partition system, a kind of computer readable storage medium and a kind of partitioned server of a kind of Kafka, has above-mentioned beneficial effect.
Description
Technical field
This application involves field of cloud calculation, in particular to a kind of partition method of Kafka, partition system and relevant apparatus.
Background technique
Kafka is the open source stream process platform developed by Apache Software Foundation, by Scala and written in Java.
Its target be provided for processing real time data a unification, it is high handle up, the platform of low latency.Kafka cluster, which provides, to be similar to
The characteristic of JMS is sorted out when saving to message according to Topic, and sending message person becomes Producer, and message recipient becomes
Consumer, furthermore kafka cluster is made of multiple kafka examples, and each example (server) becomes broker.Either
Kafka cluster or producer (producer) and consumer (consumer) all rely on zookeeper to guarantee that system can
With property cluster and save some meta information.
In the prior art, Kafka cluster is when consumer and partition mapping relations change (as consumption is answered
Situations such as being changed with the offline, number of partitions), it will do it Rebalance operation, suspend all of corresponding consumer group
Consumer pulling data.When on every machine (example) of application just may require that of answering of offline phase for example allocation subregion or
Subregion is recycled, this will be made to execute a Rebalance using all consumer group, when using machine (example)
When number is more, Rebalance times number is more, and the general window phase is longer, i.e. the time of distribution subregion is longer, more it cannot be guaranteed that in real time
Property require.Specifically, when example issues, restarts, increases newly, reduce when be required to do Rebalance operation and easily cause consumption
Stop, all consumers is easy to cause to stop consumption, is unable to satisfy high real-time consumption.
Summary of the invention
The purpose of the application is to provide a kind of partition method, partition system, the computer readable storage medium of a kind of Kafka
With a kind of partitioned server, solve each consumer and subregion mapping relations in the prior art change be required to carry out it is global
The problem of the difference of real-time brought by Rebalance.
In order to solve the above technical problems, the application provides the partition method of Kafka a kind of, specific technical solution is as follows:
S1: when detecting subregion variation, obtaining the reality in partition queue, always the number of partitions, the consumer currently survived hold
The corresponding practical number of partitions of consumer's container of device number and each survival；
S2: determine that each consumer's container is corresponding flat according to consumer's container number of the total number of partitions of the reality and survival
It should all scoring area number；
S3: determine that each consumer's container is corresponding according to the practical number of partitions and the average deserved number of partitions
Subregion difference；
S4: a subregion, distribution to the subregion difference are removed from the maximum first consumer container of the subregion difference
Second consumer's container less than 0, and update the first consumer container and the corresponding subregion of the second consumer container
Difference；
S5: repeating S4, until the absolute value of the difference of the corresponding practical number of partitions of any two consumer container is less than
When equal to 1, terminate subregion.
Wherein, when consumer's container death, further includes:
The dead corresponding useless subregion of consumer's container is recycled, the useless subregion is put into subregion queue to be allocated.
Wherein, when consumer's container is in down status, further includes:
Stop in down status consumer's container allocation subregion or from be in down status the consumption
Person's container removes subregion.
Wherein, S4 includes:
S401: a subregion counter is removed from the maximum first consumer container of the subregion difference, described first disappears
The corresponding first subregion difference of expense person's container subtracts one, and the subregion counter is distributed to second of subregion difference less than 0
Consumer's container, the corresponding second subregion difference of the second consumer container add one；
Then correspondingly, S5 includes:
S501: repeating S401, until the absolute value of the difference of the corresponding practical number of partitions of any two consumer container
When less than or equal to 1, subregion is carried out according to the corresponding subregion counter number of each consumer's container.
Wherein, further includes:
The corresponding mirror image of all subregions in the partition queue is saved to duplication partition queue, whenever subregion is assigned
When, the subregion being assigned is locked in the duplication partition queue；Wherein, it is locked in the duplication partition queue
Subregion is forbidden being reallocated in the partition queue.
The application also provides the partition system of Kafka a kind of, comprising:
Module is obtained, for when detecting subregion variation, obtaining practical total number of partitions in partition queue, current survival
Consumer's container number and each survival the corresponding practical number of partitions of consumer's container；
First determining module, for determining each consumption according to the total number of partitions of the reality and consumer's container number of survival
The corresponding average deserved number of partitions of person's container；
Second determining module, for determining each described disappear according to the practical number of partitions and the average deserved number of partitions
The corresponding subregion difference of the person's of expense container；
Division module, for removing a subregion, distribution to institute from the maximum first consumer container of the subregion difference
Second consumer container of the subregion difference less than 0 is stated, and updates the first consumer container and the second consumer container
Corresponding subregion difference；
Loop module enters the division module for repeating, until the corresponding reality point of any two consumer container
When the absolute value of the difference of area's number is less than or equal to 1, terminate subregion.
Wherein, the division module includes:
Subregion counting unit is counted for removing a subregion from the maximum first consumer container of the subregion difference
Device, the corresponding first subregion difference of the first consumer container subtract one, and it is poor to the subregion that the subregion counter is distributed
It is worth second consumer's container less than 0, the corresponding second subregion difference of the second consumer container adds one；
Then correspondingly, the loop module includes:
Cycling element enters the subregion counting unit for repeating, until the corresponding reality of any two consumer container
When the absolute value of the difference of the border number of partitions is less than or equal to 1, according to the corresponding subregion counter number of each consumer's container into
Row subregion.
Wherein, the partition system further include:
Module is locked, for saving the corresponding mirror image of all subregions in the partition queue to duplication partition queue,
When being assigned subregion, the subregion being assigned is locked in the duplication partition queue；Wherein, in the duplication subregion
The subregion being locked in queue is forbidden being reallocated in the partition queue.
The application also provides a kind of computer readable storage medium, is stored thereon with computer program, the computer journey
The step of partition method as described above is realized when sequence is executed by processor.
The application also provides a kind of partitioned server, including memory and processor, has computer in the memory
The step of program, the processor realizes partition method as described above when calling the computer program in the memory.
A kind of partition method of Kafka provided herein, comprising: S1: when detecting subregion variation, subregion is obtained
Consumer's container of the practical total number of partitions, the consumer's container number currently survived and each survival in queue is corresponding
The practical number of partitions；S2: determine that each consumer's container is corresponding according to consumer's container number of the total number of partitions of the reality and survival
The average deserved number of partitions；S3: each consumer is determined according to the practical number of partitions and the average deserved number of partitions
The corresponding subregion difference of container；S4: a subregion, distribution to institute are removed from the maximum first consumer container of the subregion difference
Second consumer container of the subregion difference less than 0 is stated, and updates the first consumer container and the second consumer container
Corresponding subregion difference；S5: repeating S4, until the difference of the corresponding practical number of partitions of any two consumer container is absolute
When value is less than or equal to 1, terminate subregion.
Partition method provided by the present application, it is offline on there are example, i.e., it, only need to be according to meter when subregion variation occurring every time
The corresponding average number of partitions of each consumer's container at this time is calculated, the subregion of variation is divided again according to the average number of partitions later
Match, subregion is obtained from the consumer's container for currently occupying most subregions, point for occupying all consumer's containers
Area is almost equal, i.e., the difference of the number of partitions that any two consumer container is occupied is no more than one.Even if there are in multiple example
Offline, the variation of every subzone is only stripped from occupying in the most consumer's container of the number of partitions, other consumer's containers are had no
It influences, without being reset as prior art needs to the corresponding subregion of all consumer's containers, greatly reduces subregion and consumed
The time taken improves the efficiency of subregion, realizes dynamic partition, can satisfy high real-time requirements.The application also provides one
A kind of a kind of partition system, computer readable storage medium and the partitioned server of kind Kafka, has above-mentioned beneficial effect, this
Place repeats no more.
Detailed description of the invention
In order to illustrate the technical solutions in the embodiments of the present application or in the prior art more clearly, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The embodiment of application for those of ordinary skill in the art without creative efforts, can also basis
The attached drawing of offer obtains other attached drawings.
Fig. 1 is a kind of flow chart of the partition method of Kafka provided by the embodiment of the present application；
Fig. 2 is a kind of partition system structural schematic diagram of Kafka provided by the embodiment of the present application.
Specific embodiment
To keep the purposes, technical schemes and advantages of the embodiment of the present application clearer, below in conjunction with the embodiment of the present application
In attached drawing, the technical scheme in the embodiment of the application is clearly and completely described, it is clear that described embodiment is
Some embodiments of the present application, instead of all the embodiments.Based on the embodiment in the application, those of ordinary skill in the art
Every other embodiment obtained without making creative work, shall fall in the protection scope of this application.
Referring to FIG. 1, Fig. 1 is a kind of flow chart of the partition method of Kafka provided by the embodiment of the present application, the subregion
Method includes:
S1: when detecting subregion variation, obtaining the reality in partition queue, always the number of partitions, the consumer currently survived hold
The corresponding practical number of partitions of consumer's container of device number and each survival；
In kafka cluster, a topic may be considered a kind of message, and each topic will be divided into multiple
Partition (subregion), each partition are append log files in storage level.One Topic's is multiple
Partitions is distributed on multiple server in kafka cluster.Each server (namely kafka example, using machine
Device) it is responsible for the read-write operation of message in partitions；Furthermore kafka can also configure the number that partitions needs to back up
(replicas)；Each partition will be backed up on more machines, to improve availability.One consumer's container
(consumer_container) multiple threads be will start, the corresponding consumer of per thread, each consumer can be same
When consume multiple partition, the number of partitions that also therefore different consumer's container occupies is different.Each partition has
One server is " leader ".Leader is responsible for all read-write operations, if leader fails, will have other
Follower takes over (as new leader).
In this step, subregion usually is monitored using leader and is changed.Specifically, can be by monitoring partition queue to be allocated
The queue variation of (i.e. " partitions_waiting_to_be_assigned " queue) knows that subregion changes.When
Containers_waiting_for_partitions " queue has data entrance to indicate new consumption when that is, data increase
Person's container needs to obtain subregion consumption, and subregion needs change, and obtains the practical total number of partitions in partition queue, current at this time
The corresponding practical number of partitions of consumer's container of the consumer's container number and each survival of survival.It should be noted that
In kafka cluster, corresponding subregion, consumer or consumer's container etc. can be stored with queue or joint form.Example
" all_partitions " queue (i.e. partition queue) can be such as established under topic node, obtained for storing from kafka cluster
All newest partition of the topic taken, each partition is as a child node.Practical total score is then obtained at this time
Area's number can be obtained directly from partition queue.Likewise, obtain consumer's container number for currently surviving also can use it is pre-
" containers_alive_flag " queue first established, the queue are used to store the consumer's container currently stored (i.e.
consumer_container).And the corresponding practical number of partitions of consumer's container of each survival then can directly by
The attribute of each consumer's container is learnt in " containers_alive_flag " queue.
It should be noted that need to obtain in this step be survival consumer's container quantity, when data be deleted,
Then indicate that corresponding consumer's container is dead, it may be possible to cause due to machine delay machine, using offline or network disconnection.Specifically
, the survival and death condition of consumer's container can be determined by monitoring " containers_alive_flag " queue.This
When the dead corresponding useless subregion of consumer's container of recycling is responsible for by leader, it is to be allocated that the useless subregion is put into subregion
Queue (i.e. " partitions_waiting_to_be_assigned " queue).
In addition, consumer's container further includes down status, it is also not belonging to existing state described above at this time.At this time
Leader stop in down status consumer's container allocation subregion or from be in down status consumer's container
Remove subregion.
S2: determine that each consumer's container is corresponding flat according to consumer's container number of the total number of partitions of the reality and survival
It should all scoring area number；
After getting practical total number of partitions and the consumer's container number actually survived in S1, that is, it can determine and averagely each deposit
The number of partitions that consumer's container living should occupy.It should be noted that cause variation at first in S101 is consumer's container hair
Changing, specifically, leader monitors that " containers_waiting_for_partitions " queue occurs data and increases
Add, it is meant that there is new consumer's container to need to obtain subregion to consume, carry out the distribution for causing subregion.Therefore, this step and
The consumer's container survived in S101 refers to consumer's container of all survivals including new consumer's container.
It is easily understood that practical total quotient namely each survival of the number of partitions divided by the consumer's container number currently survived
The average deserved number of partitions of consumer's container be not necessarily integer.And at this point, the average deserved number of partitions reply is subsequent
Practical subregion process plays directive function, specifically, the number of partitions that consumer's container of any one survival is actually distributed
It should be one in two adjacent positive integers of the averagely deserved number of partitions.
S3: determine that each consumer's container is corresponding according to the practical number of partitions and the average deserved number of partitions
Subregion difference；
The practical number of partitions refers to the number of partitions that each consumer's container is actually occupied, this step, which is intended to be directed to, each to disappear
Expense person's container calculates its corresponding subregion difference.It should be noted that subregion difference should be the practical number of partitions subtract it is averagely deserved
The difference of the number of partitions.Such as consumer's container A occupies 5 subregions, consumer's container B occupies 2 subregions, and is computed averagely deserved
The number of partitions is 3.1, then the corresponding subregion difference of consumer's container A is 1.9, and the corresponding subregion difference of consumer's container B is ﹣
1.1。
S4: a subregion, distribution to the subregion difference are removed from the maximum first consumer container of the subregion difference
Second consumer's container less than 0, and update the first consumer container and the corresponding subregion of the second consumer container
Difference；
This step is intended to obtain a subregion from the largest number of first consumer containers of subregion are occupied according to subregion difference.
It is because first consumer's container may after obtaining a subregion it should be noted why only obtaining a subregion
Occupy most that of subregion in consumer's container of not all survival.In other words, every time obtain subregion Shi Douying from
Currently occupy one subregion of removing in the largest number of consumer's containers of subregion, i.e. first consumer's container is worked as when recycling every time
Occupy that most consumer's container of number of partitions in consumer's container of preceding all survivals.
And second consumer's container refers to consumer container of the subregion difference less than 0, it is each in cluster in order to make at this time
The number of partitions that consumer's container occupies is almost average, needs " taking mend less " more.It is easily understood that in second consumer's container,
Emphasis is less than -1.0 consumer's container for subregion difference, therefore this kind of consumer's containers centainly need to increase subregion.When
Subregion difference is less than 0, when being more than or equal to -1.0, it may not be necessary to increase subregion.Along illustrated content in S3 is drawn, when consumer holds
After a subregion is assigned in device B, subregion difference becomes -0.1, means that consumer's container B is detected next time at this time
The removing or distribution of subregion can not be made before subregion variation.
S5: judge whether that the absolute value of the difference of the corresponding practical number of partitions of any two consumer's container is less than or equal to 1；If
It is no, return to S4；If so, terminating subregion.
This step, which is intended to recycle, executes step in S4, along example is drawn, since the averagely deserved number of partitions is 3.1, exists at this time
A part of consumer's container respectively occupies 4 subregions, and another part consumer container respectively occupies 3 subregions, but centainly will not
In the presence of the consumer's container for occupying other quantity subregion.At this point, realizing the equilibrium that current all consumer's containers occupy subregion
Property.
The embodiment of the present application provides the partition method of Kafka a kind of, and disappearing for each survival can be realized by the above method
The harmony for the subregion that expense person's container occupies.In the prior art, whenever offline on example (sever), in each consumer's container
The corresponding subregion of all consumers be required to Rebalance, when multiple examples occur upper offline, multiple Rebalance is needed
The subregion repeatedly to occupy to all consumer's containers is redistributed.And the embodiment of the present application is then just for occupying subregion
Biggish consumer's container is differed with the average deserved number of partitions of each consumer's container, when subregion variation occurs, it is not necessary to right
All consumer's containers carry out Rebalance, and after offline in multiple examples, due to consumer's container each in cluster
In equilibrium state, only need to carry out subregion removing and subregion to small part consumer's container when handling offline in example each time
Distribution.
For example, it is assumed that there are 20 consumer's containers in cluster, wherein 5 consumer's containers occupy 4 subregions, remain
15 consumer's containers of remaininging occupy 3 subregions, and after increasing consumer's container newly, the average deserved number of partitions is become from 3.250
3.095, then only need to from 5 consumer's containers optional three respectively one subregion of removing and distributing to newly-increased consumer hold
Device ultimately forms 2 consumer's containers and occupies 4 subregions, and 16 consumer's containers occupy 3 subregions.Only to 3 during this
Consumer's container has carried out practical operation, is different from the prior art and needs to carry out entirety Rebalance to 20 consumer's containers,
The time spent by subregion is greatly reduced, the efficiency of subregion is improved, realizes dynamic partition, high real-time is can satisfy and wants
It asks.
Based on the above embodiment, as preferred embodiment, step S4 may include:
S401: a subregion counter is removed from the maximum first consumer container of the subregion difference, described first disappears
The corresponding first subregion difference of expense person's container subtracts one, and the subregion counter is distributed to second of subregion difference less than 0
Consumer's container, the corresponding second subregion difference of the second consumer container add one；
Then correspondingly, step S5 includes:
S501: repeating S401, until the absolute value of the difference of the corresponding practical number of partitions of any two consumer container
When less than or equal to 1, subregion is carried out according to the corresponding subregion counter number of each consumer's container.
The present embodiment is intended to finally be divided according to subregion counter number using subregion counter records subregion process
Area, it is not necessary to which subregion is removed and distributed one by one.The content of subregion counter records includes but is not limited to: being stripped subregion
Consumer's container (namely first consumer's container), be assigned the subregion target consumer's container (namely the second consumer hold
Device) and number of partitions.It is easily understood that consumer's container its subregion being stripped occupied more than number of partitions may divide
It is assigned to different consumer's containers, it is therefore desirable to subregion source, zoning objectives and number of partitions in clear subregion assigning process.
Number of partitions can determine by the number of subregion counter.
It certainly, can direct subregion counter records subregion number on the basis of this embodiment, it is not necessary to utilize subregion meter
The number of number device determines subregion number to be allocated.
It should be noted that the various embodiments described above are usually executed by leader.Since any one consumer's container is taken
Band leader functions of modules, Yao Chengwei leader need to carry out leader election by zookeeper, elect successfully the then consumption
Person's container opens leader function simultaneously, is uniformly coordinated all follower to consume all partition datas under the topic.When
When leader fails, need to select new leader, therefore above-mentioned each implementation in many follower by zookeeper
Example can also be executed by the follower of alternative leader.
Based on the above embodiment, as preferred embodiment, the distribution method can also include:
The corresponding mirror image of all subregions in the partition queue is saved to duplication partition queue, whenever subregion is assigned
When, the subregion being assigned is locked in the duplication partition queue；Wherein, it is locked in the duplication partition queue
Subregion is forbidden being reallocated in the partition queue.
In order to guarantee the quilt of a subregion in topic node and only by a consumer spending.It needs to divide subregion
It is controlled with distributed lock is carried out.
The present embodiment usually execution when listening to subregion variation.Specifically, " partitions_ can listened to
Waiting_to_be_consumed " queue executes when changing.It should be noted that usually this step can be born by follower
Duty is monitored.Duplication partition queue can use zookeeper, i.e., partition queue deposit zookeeper is formed duplication subregion team
Column.
When subregion variation occurs, subregion is usually transferred to another consumer by a consumer, or is returned to point
Area's queue to be consumed (no longer being occupied by consumer), or newly increased the subregion that will be occupied by consumer.At this time multiple
In partition queue processed, since subregion has been occupied before subregion variation occurs, then first stop consumption, and to duplication partition queue
In the subregion be unlocked, then the subregion that is occupied and the subregion newly increased are locked, will finally be changed
Subregion be allocated.
If the transfer between consumer occurs for subregion, then still need first to unlock to lock again in duplication partition queue；
If subregion is recovered, then unlocked in duplication partition queue；If newly-increased subregion will be assigned, since newly-increased subregion occurs
In partition queue, then the mirror image for replicating partition queue as partition queue can also have the newly-increased subregion, increase this newly subregion
It locks.
A kind of partition system of Kafka provided by the embodiments of the present application is introduced below, subregion system described below
System can correspond to each other reference with a kind of above-described partition method of Kafka.
Fig. 2 is a kind of partition system structural schematic diagram of Kafka provided by the embodiment of the present application, which can be with
Include:
Module 100 is obtained, for when detecting subregion variation, obtaining the practical total number of partitions in partition queue, current
The corresponding practical number of partitions of consumer's container of the consumer's container number and each survival of survival；
First determining module 200, it is each for being determined according to consumer's container number of the total number of partitions of the reality and survival
The corresponding average deserved number of partitions of consumer's container；
Second determining module 300, for determining each institute according to the practical number of partitions and the average deserved number of partitions
State the corresponding subregion difference of consumer's container；
Division module 400, for removing a subregion from the maximum first consumer container of the subregion difference, distribution is extremely
Second consumer container of the subregion difference less than 0, and update the first consumer container and second consumer appearance
The corresponding subregion difference of device；
Loop module 500 enters the division module for repeating, until the corresponding reality of any two consumer container
When the absolute value of the difference of the number of partitions is less than or equal to 1, terminate subregion.
Based on the above embodiment, as preferred embodiment, the partition system can also include:
The useless subregion is put into point by recycling module for recycling the dead corresponding useless subregion of consumer's container
Area's queue to be allocated.
Based on the above embodiment, as preferred embodiment, the partition system can also include:
Offline processing module, for stopping as consumer's container allocation subregion in down status or from being in down
Consumer's container of linear state removes subregion.
Based on the above embodiment, as preferred embodiment, the division module 400 may include:
Subregion counting unit is counted for removing a subregion from the maximum first consumer container of the subregion difference
Device, the corresponding first subregion difference of the first consumer container subtract one, and it is poor to the subregion that the subregion counter is distributed
It is worth second consumer's container less than 0, the corresponding second subregion difference of the second consumer container adds one；
Then correspondingly, the loop module 500 may include:
Cycling element enters the subregion counting unit for repeating, until the corresponding reality of any two consumer container
When the absolute value of the difference of the border number of partitions is less than or equal to 1, according to the corresponding subregion counter number of each consumer's container into
Row subregion.
Based on the above embodiment, as preferred embodiment, the partition system can also include:
Module is locked, for saving the corresponding mirror image of all subregions in the partition queue to duplication partition queue,
When being assigned subregion, the subregion being assigned is locked in the duplication partition queue；Wherein, in the duplication subregion
The subregion being locked in queue is forbidden being reallocated in the partition queue.
Present invention also provides a kind of computer readable storage mediums, have computer program thereon, the computer program
The step of being performed the partition method that a kind of Kafka provided by above-described embodiment may be implemented.The storage medium can wrap
It includes: USB flash disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access
Memory, RAM), the various media that can store program code such as magnetic or disk.
Present invention also provides a kind of partitioned servers, may include memory and processor, have in the memory
Computer program when the processor calls the computer program in the memory, may be implemented above-described embodiment and be provided
A kind of Kafka partition method the step of.Certain partitioned server can also include various network interfaces, the groups such as power supply
Part.
Each embodiment is described in a progressive manner in specification, the highlights of each of the examples are with other realities
The difference of example is applied, the same or similar parts in each embodiment may refer to each other.For embodiment provide system and
Speech, since it is corresponding with the method that embodiment provides, so being described relatively simple, related place is referring to method part illustration
?.
Specific examples are used herein to illustrate the principle and implementation manner of the present application, and above embodiments are said
It is bright to be merely used to help understand the present processes and its core concept.It should be pointed out that for the ordinary skill of the art
For personnel, under the premise of not departing from the application principle, can also to the application, some improvement and modification can also be carried out, these improvement
It is also fallen into the protection scope of the claim of this application with modification.
It should also be noted that, in the present specification, relational terms such as first and second and the like be used merely to by
One entity or operation are distinguished with another entity or operation, without necessarily requiring or implying these entities or operation
Between there are any actual relationship or orders.Moreover, the terms "include", "comprise" or its any other variant meaning
Covering non-exclusive inclusion, so that the process, method, article or equipment for including a series of elements not only includes that
A little elements, but also including other elements that are not explicitly listed, or further include for this process, method, article or
The intrinsic element of equipment.In the absence of more restrictions, the element limited by sentence "including a ...", is not arranged
Except there is also other identical elements in the process, method, article or apparatus that includes the element.
Claims (10)
1. a kind of partition method of Kafka characterized by comprising
S1: when detecting subregion variation, the practical always number of partitions in acquisition partition queue, the consumer's container number currently survived
And the corresponding practical number of partitions of consumer's container of each survival；
S2: it determines that each consumer's container is corresponding according to consumer's container number of the total number of partitions of the reality and survival and averagely answers
Scoring area number；
S3: the corresponding subregion of each consumer's container is determined according to the practical number of partitions and the average deserved number of partitions
Difference；
S4: a subregion is removed from the maximum first consumer container of the subregion difference, distribution is to the subregion difference less than 0
Second consumer's container, and update the first consumer container and the corresponding subregion difference of the second consumer container；
S5: repeating S4, until the absolute value of the difference of the corresponding practical number of partitions of any two consumer container is less than or equal to 1
When, terminate subregion.
2. partition method according to claim 1, which is characterized in that when consumer's container death, further includes:
The dead corresponding useless subregion of consumer's container is recycled, the useless subregion is put into subregion queue to be allocated.
3. partition method according to claim 1, which is characterized in that when consumer's container is in down status,
Further include:
Stop holding for consumer's container allocation subregion in down status or from the consumer in down status
Device removes subregion.
4. partition method according to claim 2 or 3, which is characterized in that S4 includes:
S401: a subregion counter, first consumer are removed from the maximum first consumer container of the subregion difference
The corresponding first subregion difference of container subtracts one, distributes the subregion counter to the second consumption to the subregion difference less than 0
Person's container, the corresponding second subregion difference of the second consumer container add one；
Then correspondingly, S5 includes:
S501: repeating S401, until the absolute value of the difference of the corresponding practical number of partitions of any two consumer container is less than
When equal to 1, subregion is carried out according to the corresponding subregion counter number of each consumer's container.
5. partition method according to claim 4, which is characterized in that further include:
By the corresponding mirror image of all subregions in the partition queue save to duplication partition queue, when being assigned subregion,
The subregion being assigned is locked in the duplication partition queue；Wherein, point being locked in the duplication partition queue
Forbid being reallocated in the partition queue in area.
6. a kind of partition system of Kafka characterized by comprising
Module is obtained, for when detecting subregion variation, obtaining the practical total number of partitions in partition queue, disappearing of currently surviving
The corresponding practical number of partitions of consumer's container of the person's of expense container number and each survival；
First determining module, for determining that each consumer holds according to consumer's container number of the total number of partitions of the reality and survival
The corresponding average deserved number of partitions of device；
Second determining module, for determining each consumer according to the practical number of partitions and the average deserved number of partitions
The corresponding subregion difference of container；
Division module is distributed for removing a subregion from the maximum first consumer container of the subregion difference to described point
Second consumer container of area's difference less than 0, and update the first consumer container and the second consumer container correspondence
Subregion difference；
Loop module, for judging whether that the absolute value of the difference of the corresponding practical number of partitions of any two consumer's container is less than
In 1；If it is not, the division module is returned to, if so, terminating subregion.
7. partition system according to claim 6, which is characterized in that the division module includes:
Subregion counting unit, for removing a subregion counter, institute from the maximum first consumer container of the subregion difference
It states the corresponding first subregion difference of first consumer's container and subtracts one, the subregion counter is distributed to the subregion difference and is less than
0 second consumer's container, the corresponding second subregion difference of the second consumer container add one；
Then correspondingly, the loop module includes:
Cycling element, for judging whether that the absolute value of the difference of the corresponding practical number of partitions of any two consumer's container is less than
In 1；If it is not, returning to the subregion counting unit；If so, according to the subregion counter subregion and terminating.
8. partition system according to claim 7, which is characterized in that further include:
Module is locked, for saving the corresponding mirror image of all subregions in the partition queue to duplication partition queue, whenever
When subregion is assigned, the subregion being assigned is locked in the duplication partition queue；Wherein, in the duplication partition queue
In the subregion that is locked forbid being reallocated in the partition queue.
9. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program quilt
The step of processor realizes partition method as described in any one in claim 1-5 when executing.
10. a kind of partitioned server, which is characterized in that including memory and processor, have computer journey in the memory
Sequence, the processor realize subregion as described in any one in claim 1-5 when calling the computer program in the memory
The step of method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811512234.9A CN109656684A (en) | 2018-12-11 | 2018-12-11 | A kind of partition method of Kafka, partition system and relevant apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811512234.9A CN109656684A (en) | 2018-12-11 | 2018-12-11 | A kind of partition method of Kafka, partition system and relevant apparatus |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109656684A true CN109656684A (en) | 2019-04-19 |
Family
ID=66112960
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811512234.9A Pending CN109656684A (en) | 2018-12-11 | 2018-12-11 | A kind of partition method of Kafka, partition system and relevant apparatus |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109656684A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110401721A (en) * | 2019-08-06 | 2019-11-01 | 北京达佳互联信息技术有限公司 | The method, apparatus and system of content data distribution |
| CN110990436A (en) * | 2019-12-04 | 2020-04-10 | 中国农业银行股份有限公司 | Consumption information stream processing method and system for Apache Kafka |
| CN111142880A (en) * | 2019-12-25 | 2020-05-12 | 北京奇艺世纪科技有限公司 | Partition allocation method and device in message system and electronic equipment |
| CN112765121A (en) * | 2021-01-08 | 2021-05-07 | 北京虹信万达科技有限公司 | Administration and application system based on big data service |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102968503A (en) * | 2012-12-10 | 2013-03-13 | 曙光信息产业(北京)有限公司 | Data processing method for database system, and database system |
| CN105045877A (en) * | 2015-07-20 | 2015-11-11 | 深圳市深信服电子科技有限公司 | Database data fragmentation storage method and apparatus and data query method and apparatus |
| CN106897116A (en) * | 2017-02-27 | 2017-06-27 | 郑州云海信息技术有限公司 | A kind of virtual machine migration method and device |
-
2018
- 2018-12-11 CN CN201811512234.9A patent/CN109656684A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102968503A (en) * | 2012-12-10 | 2013-03-13 | 曙光信息产业(北京)有限公司 | Data processing method for database system, and database system |
| CN105045877A (en) * | 2015-07-20 | 2015-11-11 | 深圳市深信服电子科技有限公司 | Database data fragmentation storage method and apparatus and data query method and apparatus |
| CN106897116A (en) * | 2017-02-27 | 2017-06-27 | 郑州云海信息技术有限公司 | A kind of virtual machine migration method and device |
Non-Patent Citations (1)
| Title |
|---|
| 王郑合等: "一种优化的Kafka消费者/客户端负载均衡算法", 《计算机应用研究》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110401721A (en) * | 2019-08-06 | 2019-11-01 | 北京达佳互联信息技术有限公司 | The method, apparatus and system of content data distribution |
| CN110401721B (en) * | 2019-08-06 | 2022-07-08 | 北京达佳互联信息技术有限公司 | Method, device and system for distributing content data |
| CN110990436A (en) * | 2019-12-04 | 2020-04-10 | 中国农业银行股份有限公司 | Consumption information stream processing method and system for Apache Kafka |
| CN110990436B (en) * | 2019-12-04 | 2024-02-23 | 中国农业银行股份有限公司 | Consumption information stream processing method and system for Apache Kafka |
| CN111142880A (en) * | 2019-12-25 | 2020-05-12 | 北京奇艺世纪科技有限公司 | Partition allocation method and device in message system and electronic equipment |
| CN111142880B (en) * | 2019-12-25 | 2024-03-12 | 北京奇艺世纪科技有限公司 | Partition allocation method and device in message system and electronic equipment |
| CN112765121A (en) * | 2021-01-08 | 2021-05-07 | 北京虹信万达科技有限公司 | Administration and application system based on big data service |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109656684A (en) | A kind of partition method of Kafka, partition system and relevant apparatus | |
| EP2372551B1 (en) | Real time backup storage node assignment | |
| CN106095589B (en) | A kind of method, apparatus and system for distributing subregion | |
| CN110413412B (en) | GPU (graphics processing Unit) cluster resource allocation method and device | |
| CN106598735B (en) | A kind of distributed computing method, main controlled node and computing system | |
| CN110515704A (en) | Resource regulating method and device based on Kubernetes system | |
| CN104407926B (en) | A kind of dispatching method of cloud computing resources | |
| CN107241281A (en) | A kind of data processing method and its device | |
| CN107818012B (en) | Data processing method and device and electronic equipment | |
| CN108804383B (en) | Support point parallel enumeration method and device based on measurement space | |
| CN111124643A (en) | Task deletion scheduling method, system and related device in distributed storage | |
| US20240061712A1 (en) | Method, apparatus, and system for creating training task on ai training platform, and medium | |
| CN109800067A (en) | Database connection optimization method, device and relevant device based on cloud monitoring | |
| CN112162852A (en) | Multi-architecture CPU node management method, device and related components | |
| CN108694083B (en) | Data processing method and device for server | |
| CN109800261A (en) | Dynamic control method, device and the relevant device of double data library connection pool | |
| CN115658311A (en) | Resource scheduling method, device, equipment and medium | |
| CN107015876A (en) | A kind of service request processing method and device | |
| CN110442454A (en) | A kind of resource regulating method, device and computer equipment | |
| CN108595259A (en) | A kind of internal memory pool managing method based on global administration | |
| CN109788013B (en) | Method, device and equipment for distributing operation resources in distributed system | |
| CN111290858A (en) | Input/output resource management method, device, computer equipment and storage medium | |
| CN108616583A (en) | A kind of memory allocation method based on computer cloud | |
| CN115599300A (en) | Task allocation method, device, equipment and medium | |
| CN109040491A (en) | On-hook behavior processing method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190419 |
|
| RJ01 | Rejection of invention patent application after publication |