CN107451147A - A kind of method and apparatus of kafka clusters switching at runtime - Google Patents
A kind of method and apparatus of kafka clusters switching at runtime Download PDFInfo
- Publication number
- CN107451147A CN107451147A CN201610375007.0A CN201610375007A CN107451147A CN 107451147 A CN107451147 A CN 107451147A CN 201610375007 A CN201610375007 A CN 201610375007A CN 107451147 A CN107451147 A CN 107451147A
- Authority
- CN
- China
- Prior art keywords
- cluster
- configuration information
- client
- information
- cluster configuration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
Abstract
The present invention provides a kind of method and apparatus of kafka clusters switching at runtime, can monitor the operation conditions of cluster in real time, and dynamic changes the configuration information of cluster to realize the switching at runtime of cluster.The method of the kafka cluster switching at runtime of the present invention includes：Authentication parameter is sent to configuration service center to obtain the cluster configuration information of respective cluster by client；Receive the cluster configuration information that the configuration service center returns and the previous cluster configuration information with being preserved in the client compares；If the cluster configuration information received is different from the previously cluster configuration information, with the cluster configuration information renewal received the previously cluster configuration information and corresponding cluster is accessed according to the cluster configuration information after renewal.
Description
Technical field
The present invention relates to computer and computer software technical field, a kind of particularly kafka clusters switching at runtime
Method and apparatus.
Background technology
In distributed post subscribes to message system kafka, when the switching of kafka clusters, adjustment different pieces of information subregion are read
When resource occupation, often through modification kafka (distributed post subscription message system) client-side program configuration parameter come
Realize.
Under normal circumstances, the kafka image parameters in cluster can be set by changing client-side program, as site is believed
Cease, the module location information topic and password token of partition information incidence relation, efficiency of transmission, cluster configuration, backup number
Replications, fragment segment sizes etc. are measured, and after the completion of parameter modification, client-side program compiling packing is laid equal stress on
Task is opened to realize the functions such as kafka clusters switching.
The difference realized according to kafka client functionalitys, kafka clients can be divided into production client and customer
End.
If certain kafka client is production client, is i.e. producer, the task of data generation is performed, then is being carried out
Operating procedure when kafka clusters switch is as follows：
1st, determine to need to carry out the cluster configuration information after the producer of cluster switching task and switching (such as to wrap
Include：Distributed application program coordination service device zookeeper root zkRoot, the partition information brokers on server,
Module location information topic of partition information etc.)；
2nd, the program of corresponding production client is changed by program compilation tool, finds corresponding kafka (distributed posts
Subscribe to message system) cluster configuration parameter, recompilates packing after modification；
3rd, the program after packing is uploaded onto the server and updates original program；
4th, game server, stop and restart producer task；
5th, whether test cluster switching task succeeds, and passes through kafka (distributed post subscription message system) administration interface
Flow amount data are monitored, if the group of topic corresponding to production task has created, and starts to write data, illustrate that cluster is cut
Change work(into.
If certain kafka client is customer end, i.e. consumer, the task of data consumption is performed, then is being carried out
Operating procedure when kafka clusters switch is as follows：
1st, determine to need to carry out the consumer of cluster switching task and amended cluster configuration information (main bag
Include：Distributed application program coordination service device zookeeper root zkRoot, the partition information brokers on server,
Module location information topic of partition information etc.)；
2nd, the program at kafka (distributed post subscription message system) customer end is opened by program compilation tool,
Cluster configuration parameter corresponding to finding, and the cluster configuration parameter after renewal is revised as, recompilate packing；
3rd, the program after compiling is uploaded onto the server, and updates original program；
4th, by kafka (distributed post subscription message system) cluster monitoring interface, monitor corresponding to the consumption task
Data, when backlog is 0, illustrate that the data on former cluster are consumed completely with the presence or absence of overstocking；
5th, after the data on former cluster are consumed completely, ssh (order that server redirects in linux) instrument is passed through
Game server, original consumption program is cut off, changes and the corresponding parameter information of task is consumed in program, then restart this
Consumer (consumption) task；
6th, by kafka (distributed post subscription message system) cluster monitoring interface, test and confirm that task is from new
Cluster consumption data, if topic corresponding to consumption task (the division module positional information on coordination service device) group is
Create, and start consumption data, illustrate that cluster switches successfully.
However, when carrying out the switching of kafka clusters using existing technical scheme, with the following drawback that：
Had to when the 1st, switching cluster first by the program stopped of client, then change the module of site information, partition information
Positional information topic and password token incidence relation, efficiency of transmission, cluster configuration, backup quantity replications, piece
The data such as section segment sizes, compiling packing and then restarting program are preserved, the real-time continuous switching of cluster can not be realized；
2nd, the occupation condition of current producer tasks and consumer tasks can not be checked, can not reasonable arrangement
Existing resource is distributed, the availability of resource is not high.
The content of the invention
In view of this, the present invention provides a kind of method and apparatus of kafka clusters switching at runtime, can monitor cluster in real time
Operation conditions, and dynamic changes the configuration information of cluster to realize the switching at runtime of cluster.
To achieve the above object, according to an aspect of the invention, there is provided a kind of side of kafka clusters switching at runtime
Method.
A kind of method of kafka clusters switching at runtime, including：Client by authentication parameter be sent to configuration service center with
Obtain the cluster configuration information of respective cluster；Receive the cluster configuration information that the configuration service center returns and with the client
The previous cluster configuration information preserved on end compares；If the cluster configuration information received is believed with the previously cluster configuration
Breath is different, then updates the previously cluster configuration information with the cluster configuration information received and believed according to the cluster configuration after renewal
Breath accesses corresponding cluster.
Alternatively, the client includes production client and customer end, and the production client creation data is simultaneously
The data are write into the cluster, the customer end reads data from the cluster and consumes the data.
Alternatively, the client timing sends the heartbeat message of the client to the configuration service center, described
Heartbeat message includes：Rate information, data processing type and number of data, the data processing type include production and consumption.
Alternatively, the configuration service center receives the authentication parameter of client transmission using connecting interface and authenticated
Operation, and the cluster configuration information is returned after authentication passes through.
Alternatively, in addition to：For customer end, when switching to a new cluster from previous cluster, if described new
Cluster is the backup cluster of the previously cluster, then the site information of the configuration service central record current consumption and is saved in
In database, then the cluster configuration information of the previously cluster is updated to the cluster configuration information of the new cluster；Otherwise,
The configuration service center checks the message backlog of consumption task, and when message backlog is 0, the previously cluster
Cluster configuration information is updated to the cluster configuration information of the new cluster.
Alternatively, the cluster configuration information includes：Should in cluster message addresses, clustered partition mark and zookeeper
The address of cluster.
According to another aspect of the present invention, there is provided a kind of device of kafka clusters switching at runtime.
A kind of device of kafka clusters switching at runtime, including：Data obtaining module, authentication parameter is sent out for client
Configuration service center is given to obtain the cluster configuration information of respective cluster；Information contrast module, for receiving the configuration clothes
The cluster configuration information and the previous cluster configuration information with being preserved in the client that business center returns compare；Information updating
Module, if different from the previously cluster configuration information for the cluster configuration information received, with the collection flock mating received
Put previous cluster configuration information described in information updating and corresponding cluster is accessed according to the cluster configuration information after renewal.
Alternatively, the client includes production client and customer end, and the production client creation data is simultaneously
The data are write into the cluster, the customer end reads data from the cluster and consumes the data.
Alternatively, the client timing sends the heartbeat message of the client to the configuration service center, described
Heartbeat message includes：Rate information, data processing type and number of data, the data processing type include production and consumption.
Alternatively, the configuration service center receives the authentication parameter of client transmission using connecting interface and authenticated
Operation, and the cluster configuration information is returned after authentication passes through.
Alternatively, in addition to：For customer end, when switching to a new cluster from previous cluster, if described new
Cluster is the backup cluster of the previously cluster, then the site information of the configuration service central record current consumption and is saved in
In database, then the cluster configuration information of the previously cluster is updated to the cluster configuration information of the new cluster；Otherwise,
The configuration service center checks the message backlog of consumption task, and when message backlog is 0, the previously cluster
Cluster configuration information is updated to the cluster configuration information of the new cluster.
Alternatively, the cluster configuration information includes：Should in cluster message addresses, clustered partition mark and zookeeper
The address of cluster.
According to another aspect of the invention, there is provided a kind of device of kafka clusters switching at runtime.
A kind of device of kafka clusters switching at runtime, including：Memory and processor, wherein, the memory storage refers to
Order；Instruct and be used for described in the computing device：Authentication parameter is sent to configuration service center to obtain respective episode by client
The cluster configuration information of group；Receive cluster configuration information that the configuration service center returns and with being preserved in the client
Previous cluster configuration information compares；If the cluster configuration information received is different from the previously cluster configuration information,
With the cluster configuration information renewal of the reception previously cluster configuration information and phase is accessed according to the cluster configuration information after renewal
The cluster answered.
Technique according to the invention scheme, taken by the configuration that the configuration information of kafka clusters is encapsulated in cluster outer layer
Business center is simultaneously shown in the form of client, can monitor the occupation condition and operation information of cluster in real time, and can basis
The concrete condition for the new cluster to be switched carrys out switching at runtime cluster, so as to the dynamic in the case of the message backlog of monitoring cluster
Switch cluster, realize and more efficiently use cluster；In addition, the configuration information of kafka clusters is encapsulated in outside cluster by the present invention
The configuration service center of layer, it can be achieved to pass through configuration service center dynamic configuration collection swarm parameter, when carrying out cluster switching, without
Suspend reset routine, read configuration parameter in real time and be applied in the production and consumption task of message, cluster easy switching
Quickly.
Brief description of the drawings
Accompanying drawing is used to more fully understand the present invention, does not form inappropriate limitation of the present invention.Wherein：
Fig. 1 is the key step schematic diagram of the method for kafka cluster switching at runtime according to embodiments of the present invention；
Fig. 2 is the Organization Chart of kafka cluster switching at runtime systems according to embodiments of the present invention；
Fig. 3 is the main modular schematic diagram of the device of kafka clusters switching at runtime according to an embodiment of the invention；
Fig. 4 is the main modular schematic diagram of the device of kafka clusters switching at runtime according to another embodiment of the present invention.
Embodiment
The one exemplary embodiment of the present invention is explained below in conjunction with accompanying drawing, including the various of the embodiment of the present invention
Details should think them only exemplary to help understanding.Therefore, those of ordinary skill in the art should recognize
Arrive, various changes and modifications can be made to the embodiments described herein, without departing from scope and spirit of the present invention.Together
Sample, for clarity and conciseness, the description to known function and structure is eliminated in following description.
The present invention proposes following technical scheme：By corresponding to the production and consumption task of message in kafka systems
The configuration information of cluster is encapsulated in outside kafka clusters, forms configuration service center, and the exhibition in the form of single client
It is existing, to monitor the operation information of cluster.The interface that production client and customer end are provided by configuration service center enters
The lookup of row proof of identity and cluster configuration information., can real-time update when client finds that cluster configuration information changes
Cluster configuration information in client, and it is switched to production or consumption that new cluster carries out message.
Fig. 1 is the key step schematic diagram of the method for kafka cluster switching at runtime according to embodiments of the present invention.Such as Fig. 1
Shown, the method for kafka cluster switching at runtime of the invention mainly includes steps S11 to step S13.
Step S11：Authentication parameter is sent to configuration service center to obtain the cluster configuration of respective cluster letter by client
Breath；
Step S12：Receive cluster configuration information that the configuration service center returns and with being preserved in the client
Previous cluster configuration information compares；
Step S13：If the cluster configuration information received is different from the previously cluster configuration information, with what is received
The cluster configuration information renewal previously cluster configuration information simultaneously accesses corresponding cluster according to the cluster configuration information after renewal.
Wherein, the client includes production client and customer end, and the production client creation data simultaneously will
The data write the cluster, and the customer end reads data from the cluster and consumes the data.
Technique according to the invention scheme, the client timing send the client to the configuration service center
Heartbeat message, the heartbeat message include：Rate information, data processing type and number of data, the data processing type bag
Include production and consumption.
In the present invention, the configuration service center receives the authentication parameter of client transmission using connecting interface and reflected
Power operation, and the cluster configuration information is returned after authentication passes through.
Also, for customer end, when switching to a new cluster from previous cluster, if the new cluster is described
The backup cluster of previous cluster, then the site information of the configuration service central record current consumption and it is saved in database,
Then the cluster configuration information of the previously cluster is updated to the cluster configuration information of the new cluster；Otherwise, the configuration
Service centre checks the message backlog of consumption task, and when message backlog is 0, the cluster configuration of the previously cluster
Information updating is the cluster configuration information of the new cluster.
In addition, heretofore described cluster configuration information includes：Cluster message addresses, clustered partition mark and
The address of the cluster in zookeeper.
Fig. 2 is the Organization Chart of kafka cluster switching at runtime systems according to embodiments of the present invention.It is as shown in Fig. 2 of the invention
Outside kafka clusters, configuration service center is encapsulated based on kafka, and the configuration service center provides user for client
Authentication and the interface for obtaining cluster configuration information.Client can be verified by authentication parameter username and password, and
After verification passes through, kafka cluster configuration informations are read in real time.
In the embodiment of the present invention, kafka clients are packaged, by kafka partitioned parameters " the module position of partition information
Confidence ceases topic " and " zk information (IP of each server in cluster in distributed application program coordination service device zookeeper
Location information) " etc. be packaged as appId (user name) and token (password).In addition, (the distribution of zk information can be used in authentication parameter
In formula application program coordination service device zookeeper in cluster each server IP address information), the partition information on server
Brokers, the module location information topic of partition information, consumption group name etc., but parameter can be readily accessible by the user, and security is not high.
Therefore subscription authentication is carried out using the appId (user name) and token (password) that are obtained after encapsulation in the present invention.Wherein, user
Name is generally system domain name；Password is to be generated by mark of system domain name and topic data etc. according to certain rule, with dividing
The module location information topic of area's information is information-related.Specifically for example：If client is production client, password is by being
Three system domain name, cluster information mark, authentication type fields carry out coding post package and obtained；If client is customer end,
Then password is to carry out coding post package by four system domain name, cluster information mark, authentication type, consumption group fields to obtain.Its
In, authentication type generally refers to production and consumption.
In addition, the mode of timed task can also be set in the embodiment of the present invention in client, pass through configuration service center
The interface of offer, regularly to configuration service center send client heartbeat message (such as rate information, data processing type and
Number of data etc., the data processing type include production and consumption), and obtain appId (user name) and token (password)
Corresponding newest cluster configuration information.Wherein, " timing " can flexibly be set according to the situation that cluster is run, such as collected
It is arranged to 10 minutes etc. in the case that group is stable.
Client will be believed after newest cluster configuration information is got with the previous cluster configuration preserved in the client
Breath is compared, and as found to change, then updates the previous cluster configuration information with the cluster configuration information obtained and adds automatically
Carry configuration information so that configuration take-effective, then, corresponding cluster is accessed according to the cluster configuration information after renewal.
The implementation process of the present invention will be introduced in terms of production client and customer end two below.
For producing client, before cluster switching, the module location information topic of the partition information of new cluster need to be believed
Breath establishes incidence relation with existing token；Then, the address information renewal of new kafka clusters is taken to packaged configuration
In the allocation list at business center.Herein, the address information of new kafka clusters refers to the module location information topic of partition information
With the partition information brokers on server.When production client proceeds by data write-in, cluster configuration will be read in real time
Information, it is compared after reading cluster configuration information with the previous cluster configuration information that active client uses.If read
The cluster configuration information arrived is different from previous cluster configuration information, and production client (such as client1 in figure) can be moved in real time
The cluster configuration information that state loading parsing is read, and parameter assignment is carried out, the configuration information of new cluster is applied in program,
So as to realize the cluster configuration information after dynamic load switching.
, equally need to be by the module location information of the partition information of new cluster before cluster switching for customer end
Topic information establishes incidence relation with existing token.In addition, it is also necessary to according to the new cluster to be switched and previous cluster
Which kind of mode is relation determine this to switch cluster.
If new cluster is the backup cluster of previous cluster, the site information of configuration service central record current consumption and guarantor
It is stored in database, then the cluster configuration information of previous cluster is updated to the cluster configuration information of new cluster.Sometimes in number
During according to storage, data can be backed up, such as：Have two kafka clusters A and B, B be A backup cluster, data in B with
Data in A are identical.Assuming that such as the customer end clienta in Fig. 2, the data in positive consumption cluster A are, it is necessary to switch at present
To consume cluster B data.So, it is necessary to which clienta recorded into database in the current site of cluster A consumption datas first
In, then clienta is switched to the data for consuming cluster B.When carrying out cluster switching, need first by the Distributed Application of new cluster
The IP address information of each server, the module location information topic kimonos of partition information in Program Coordination server zookeeper
The configuration information updates such as the partition information brokers being engaged on device pass through in configuration service to configuration service center, then client
The interface that the heart provides obtains newest cluster configuration information and loaded automatically to cause configuration take-effective.Start to disappear after cluster switching
Before taking data, the site of data in clienta consumption clusters A is first read from database, finds corresponding data in cluster B afterwards
Corresponding site, the finally consumption data since the B site.Do not consider that message to be consumed overstocks in this way, may be implemented in
In the case of, switch cluster at any time, and ensure data without loss.
If new cluster is not the backup cluster of previous cluster, that is, the data in new cluster and the data in previous cluster are not
Same or incomplete same, then configuration service center need to check the message backlog of consumption task, and when message backlog is 0,
The cluster configuration information of previous cluster is updated to the cluster configuration information of new cluster.Detailed process is：Configuration service center leads to
The method in the site of overwriting current consumption data monitors the backlog of consumption task in real time, will be new when message backlog is 0
The distributed application program coordination service device zookeeper of cluster IP address information, the module location information of partition information
The configuration information updates such as the partition information brokers on topic and server to configuration service center, then client by with
The interface for putting service centre's offer obtains newest cluster configuration information and loaded automatically to cause configuration take-effective.It is switched to new collection
After group carries out information consumption, client consumes the cluster after switching from performing check since the first data in cluster, with
Ensure data without loss.In this way, in the case of may be implemented in message without overstocking, switch cluster at any time and ensure data without losing
Lose.
Fig. 3 is the main modular schematic diagram of the device of kafka clusters switching at runtime according to an embodiment of the invention.Such as
Shown in Fig. 3, the device 30 of kafka cluster switching at runtime of the invention mainly includes data obtaining module 31, information contrast module
32 and information updating module 33.
Authentication parameter is sent to configuration service center to obtain respective cluster by data obtaining module 31 for client
Cluster configuration information；Information contrast module 32 be used to receiving the cluster configuration information that the configuration service center returns and with it is described
The previous cluster configuration information preserved in client compares；If information updating module 33 is used for the cluster configuration letter received
Breath is different from the previously cluster configuration information, then with the cluster configuration information renewal received the previously cluster configuration information simultaneously
Corresponding cluster is accessed according to the cluster configuration information after renewal.
In the present invention, the client includes production client and customer end, the production client creation data
And the data are write into the cluster, the customer end reads data from the cluster and consumes the data.
Also, the client timing sends the heartbeat message of the client, the heart to the configuration service center
Hop-information includes：Rate information, data processing type and number of data, the data processing type include production and consumption.
Technical scheme according to embodiments of the present invention, the configuration service center receive client using connecting interface and sent
Authentication parameter and carry out authentication operations, and the cluster configuration information is returned to after authentication passes through.
Also, for customer end, when switching to a new cluster from previous cluster, if the new cluster is described
The backup cluster of previous cluster, then the site information of the configuration service central record current consumption and it is saved in database,
Then the cluster configuration information of the previously cluster is updated to the cluster configuration information of the new cluster；Otherwise, the configuration
Service centre checks the message backlog of consumption task, and when message backlog is 0, the cluster configuration of the previously cluster
Information updating is the cluster configuration information of the new cluster.
Wherein, the cluster configuration information includes：The collection in cluster message addresses, clustered partition mark and zookeeper
The address of group.
Fig. 4 is the main modular schematic diagram of the device of kafka clusters switching at runtime according to another embodiment of the present invention.Such as
Shown in Fig. 4, the device 40 of kafka cluster switching at runtime of the invention mainly includes memory 41 and processor 42.
Wherein, the store instruction of memory 41；Processor 42 performs described instruct and is used for：Authentication parameter is sent to by client
Configuration service center is to obtain the cluster configuration information of respective cluster；Receive the cluster configuration letter that the configuration service center returns
Cease and the previous cluster configuration information with being preserved in the client compares；If the cluster configuration information received with it is described
Previous cluster configuration information is different, then with the cluster configuration information renewal received the previously cluster configuration information and according to renewal
Cluster configuration information afterwards accesses corresponding cluster.
Technical scheme according to embodiments of the present invention, by the way that the configuration information of kafka clusters is encapsulated in into cluster outer layer
Configuration service center is simultaneously shown in the form of client, can monitor the occupation condition and operation information of cluster in real time, and
Can be according to the concrete condition for the new cluster to be switched come switching at runtime cluster, so that the situation of the message backlog in monitoring cluster
Lower switching at runtime cluster, realize and more efficiently use cluster；In addition, the configuration information of kafka clusters is encapsulated in by the present invention
It the configuration service center of cluster outer layer, can be achieved to pass through configuration service center dynamic configuration collection swarm parameter, carrying out cluster switching
When, without suspending reset routine, configuration parameter is read in real time and is applied in the production and consumption task of message, cluster is cut
Change fast and easy.
Above-mentioned embodiment, does not form limiting the scope of the invention.Those skilled in the art should be bright
It is white, depending on design requirement and other factors, various modifications, combination, sub-portfolio and replacement can occur.It is any
Modifications, equivalent substitutions and improvements made within the spirit and principles in the present invention etc., should be included in the scope of the present invention
Within.
Claims (13)
- A kind of 1. method of kafka clusters switching at runtime, it is characterised in that including：Authentication parameter is sent to configuration service center to obtain the cluster configuration information of respective cluster by client；Receive the cluster configuration information that the configuration service center returns and the previous cluster configuration with being preserved in the client Information compares；If the cluster configuration information received is different from the previously cluster configuration information, with the cluster configuration information received Update the previously cluster configuration information and corresponding cluster is accessed according to the cluster configuration information after renewal.
- 2. according to the method for claim 1, it is characterised in that the client includes production client and customer The data are simultaneously write the cluster by end, the production client creation data, and the customer end is read from the cluster Access evidence simultaneously consumes the data.
- 3. according to the method for claim 1, it is characterised in that the client timing is sent to the configuration service center The heartbeat message of the client, the heartbeat message include：Rate information, data processing type and number of data, the number Include production and consumption according to processing type.
- 4. according to the method for claim 1, it is characterised in that the configuration service center receives client using connecting interface Hold the authentication parameter sent and carry out authentication operations, and the cluster configuration information is returned after authentication passes through.
- 5. according to the method for claim 2, it is characterised in that also include：For customer end, cut when from previous cluster When shifting to a new cluster, if the new cluster is the backup cluster of the previously cluster, the configuration service central record The site information of current consumption is simultaneously saved in database, then the cluster configuration information of the previously cluster is updated to described The cluster configuration information of new cluster；Otherwise, the configuration service center checks the message backlog of consumption task, and when message backlog is 0, described in The cluster configuration information of previous cluster is updated to the cluster configuration information of the new cluster.
- 6. according to the method for claim 1, it is characterised in that the cluster configuration information includes：Cluster message addresses, collection Group partition identification and zookeeper in the cluster address.
- A kind of 7. device of kafka clusters switching at runtime, it is characterised in that including：Data obtaining module, authentication parameter is sent to configuration service center to obtain the collection flock mating of respective cluster for client Confidence ceases；Information contrast module, for receiving the cluster configuration information of the configuration service center return and with being protected in the client The previous cluster configuration information deposited compares；Information updating module, if different from the previously cluster configuration information for the cluster configuration information received, use The cluster configuration information renewal previously cluster configuration information of reception simultaneously accesses corresponding according to the cluster configuration information after renewal Cluster.
- 8. device according to claim 7, it is characterised in that the client includes production client and customer The data are simultaneously write the cluster by end, the production client creation data, and the customer end is read from the cluster Access evidence simultaneously consumes the data.
- 9. device according to claim 7, it is characterised in that the client timing is sent to the configuration service center The heartbeat message of the client, the heartbeat message include：Rate information, data processing type and number of data, the number Include production and consumption according to processing type.
- 10. device according to claim 7, it is characterised in that the configuration service center receives visitor using connecting interface The authentication parameter of family end transmission simultaneously carries out authentication operations, and the cluster configuration information is returned after authentication passes through.
- 11. device according to claim 8, it is characterised in that also include：For customer end, when from previous cluster When switching to a new cluster,If the new cluster is the backup cluster of the previously cluster, the site of the configuration service central record current consumption Information is simultaneously saved in database, then the cluster configuration information of the previously cluster is updated to the collection flock mating of the new cluster Confidence ceases；Otherwise, the configuration service center checks the message backlog of consumption task, and when message backlog is 0, described in The cluster configuration information of previous cluster is updated to the cluster configuration information of the new cluster.
- 12. device according to claim 7, it is characterised in that the cluster configuration information includes：Cluster message addresses, Clustered partition identifies the address with the cluster in zookeeper.
- A kind of 13. device of kafka clusters switching at runtime, it is characterised in that including：Memory and processor, wherein,The memory store instruction；Instruct and be used for described in the computing device：Authentication parameter is sent to configuration service center to obtain the cluster configuration information of respective cluster by client；Receive the cluster configuration information that the configuration service center returns and the previous cluster configuration with being preserved in the client Information compares；If the cluster configuration information received is different from the previously cluster configuration information, with the cluster configuration information received Update the previously cluster configuration information and corresponding cluster is accessed according to the cluster configuration information after renewal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610375007.0A CN107451147B (en) | 2016-05-31 | 2016-05-31 | Method and device for dynamically switching kafka clusters |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610375007.0A CN107451147B (en) | 2016-05-31 | 2016-05-31 | Method and device for dynamically switching kafka clusters |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107451147A true CN107451147A (en) | 2017-12-08 |
| CN107451147B CN107451147B (en) | 2020-07-31 |
Family
ID=60485833
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610375007.0A Active CN107451147B (en) | 2016-05-31 | 2016-05-31 | Method and device for dynamically switching kafka clusters |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107451147B (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108170527A (en) * | 2017-12-15 | 2018-06-15 | 北京奇艺世纪科技有限公司 | A kind of strange land distributed message consuming method and device mostly living |
| CN108322358A (en) * | 2017-12-15 | 2018-07-24 | 北京奇艺世纪科技有限公司 | Strange land distributed message transmission, processing, consuming method and device mostly living |
| CN108667930A (en) * | 2018-05-09 | 2018-10-16 | 聚龙股份有限公司 | A kind of configuration of dynamic data source and automation switching method |
| CN108900572A (en) * | 2018-05-31 | 2018-11-27 | 康键信息技术（深圳）有限公司 | Primary database domain name method for handover control, computer equipment and storage medium |
| CN109861878A (en) * | 2019-01-17 | 2019-06-07 | 平安科技（深圳）有限公司 | The monitoring method and relevant device of the topic data of kafka cluster |
| CN110515748A (en) * | 2019-08-28 | 2019-11-29 | 腾讯科技（深圳）有限公司 | A kind of method and relevant apparatus of Message Processing |
| CN110716813A (en) * | 2019-09-17 | 2020-01-21 | 贝壳技术有限公司 | Data stream processing method and device, readable storage medium and processor |
| CN111163172A (en) * | 2019-12-31 | 2020-05-15 | 北京奇艺世纪科技有限公司 | Message processing system, method, electronic device and storage medium |
| CN111726388A (en) * | 2019-03-22 | 2020-09-29 | 苏宁易购集团股份有限公司 | Cross-cluster high-availability implementation method, device, system and equipment |
| CN112671590A (en) * | 2020-12-31 | 2021-04-16 | 北京奇艺世纪科技有限公司 | Data transmission method and device, electronic equipment and computer storage medium |
| CN112948346A (en) * | 2019-11-26 | 2021-06-11 | 马上消费金融股份有限公司 | Database switching method, device, equipment and computer readable storage medium |
| CN113110926A (en) * | 2021-04-19 | 2021-07-13 | 上海华兴数字科技有限公司 | Dynamic adjustment method, device, medium and electronic equipment for data consumption thread |
| CN114650283A (en) * | 2020-12-21 | 2022-06-21 | 京东科技控股股份有限公司 | Method and device for dynamically switching cloud storage service, electronic equipment and storage medium |
| CN115396339A (en) * | 2022-08-24 | 2022-11-25 | 银清科技有限公司 | Abnormal message processing method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102255752A (en) * | 2011-06-30 | 2011-11-23 | 北京新媒传信科技有限公司 | Configuration management system and method of server cluster |
| US20130054521A1 (en) * | 2011-08-29 | 2013-02-28 | Huawei Technologies Co., Ltd. | Method and device for automactic migration of system configuration item |
| CN103516538A (en) * | 2012-06-28 | 2014-01-15 | 中兴通讯股份有限公司 | Method and device for uniform configuration of carrier-class clustered applications |
| CN104092718A (en) * | 2013-12-12 | 2014-10-08 | 腾讯数码（天津）有限公司 | Distributed system and configuration information updating method in distributed system |
| CN105279603A (en) * | 2015-09-11 | 2016-01-27 | 福建师范大学 | Dynamically configured big data analysis system and method |
| CN105610987A (en) * | 2016-03-18 | 2016-05-25 | 车智互联（北京）科技有限公司 | Method, application and system for managing server cluster |
-
2016
- 2016-05-31 CN CN201610375007.0A patent/CN107451147B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102255752A (en) * | 2011-06-30 | 2011-11-23 | 北京新媒传信科技有限公司 | Configuration management system and method of server cluster |
| US20130054521A1 (en) * | 2011-08-29 | 2013-02-28 | Huawei Technologies Co., Ltd. | Method and device for automactic migration of system configuration item |
| CN103516538A (en) * | 2012-06-28 | 2014-01-15 | 中兴通讯股份有限公司 | Method and device for uniform configuration of carrier-class clustered applications |
| CN104092718A (en) * | 2013-12-12 | 2014-10-08 | 腾讯数码（天津）有限公司 | Distributed system and configuration information updating method in distributed system |
| CN105279603A (en) * | 2015-09-11 | 2016-01-27 | 福建师范大学 | Dynamically configured big data analysis system and method |
| CN105610987A (en) * | 2016-03-18 | 2016-05-25 | 车智互联（北京）科技有限公司 | Method, application and system for managing server cluster |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108170527B (en) * | 2017-12-15 | 2021-06-22 | 北京奇艺世纪科技有限公司 | Remote multi-activity distributed message consumption method and device |
| CN108322358A (en) * | 2017-12-15 | 2018-07-24 | 北京奇艺世纪科技有限公司 | Strange land distributed message transmission, processing, consuming method and device mostly living |
| CN108170527A (en) * | 2017-12-15 | 2018-06-15 | 北京奇艺世纪科技有限公司 | A kind of strange land distributed message consuming method and device mostly living |
| CN108322358B (en) * | 2017-12-15 | 2020-09-01 | 北京奇艺世纪科技有限公司 | Method and device for sending, processing and consuming multi-live distributed messages in different places |
| CN108667930A (en) * | 2018-05-09 | 2018-10-16 | 聚龙股份有限公司 | A kind of configuration of dynamic data source and automation switching method |
| CN108667930B (en) * | 2018-05-09 | 2020-10-30 | 聚龙股份有限公司 | Dynamic data source configuration and automatic switching method |
| CN108900572A (en) * | 2018-05-31 | 2018-11-27 | 康键信息技术（深圳）有限公司 | Primary database domain name method for handover control, computer equipment and storage medium |
| CN108900572B (en) * | 2018-05-31 | 2022-08-09 | 康键信息技术（深圳）有限公司 | Main database domain name switching control method, computer equipment and storage medium |
| CN109861878A (en) * | 2019-01-17 | 2019-06-07 | 平安科技（深圳）有限公司 | The monitoring method and relevant device of the topic data of kafka cluster |
| CN109861878B (en) * | 2019-01-17 | 2022-03-25 | 平安科技（深圳）有限公司 | Method for monitoring topic data of kafka cluster and related equipment |
| CN111726388A (en) * | 2019-03-22 | 2020-09-29 | 苏宁易购集团股份有限公司 | Cross-cluster high-availability implementation method, device, system and equipment |
| CN110515748A (en) * | 2019-08-28 | 2019-11-29 | 腾讯科技（深圳）有限公司 | A kind of method and relevant apparatus of Message Processing |
| CN110515748B (en) * | 2019-08-28 | 2022-02-01 | 腾讯科技（深圳）有限公司 | Message processing method and related device |
| CN110716813A (en) * | 2019-09-17 | 2020-01-21 | 贝壳技术有限公司 | Data stream processing method and device, readable storage medium and processor |
| CN112948346A (en) * | 2019-11-26 | 2021-06-11 | 马上消费金融股份有限公司 | Database switching method, device, equipment and computer readable storage medium |
| CN112948346B (en) * | 2019-11-26 | 2023-05-02 | 马上消费金融股份有限公司 | Database switching method, device, equipment and computer readable storage medium |
| CN111163172A (en) * | 2019-12-31 | 2020-05-15 | 北京奇艺世纪科技有限公司 | Message processing system, method, electronic device and storage medium |
| CN114650283A (en) * | 2020-12-21 | 2022-06-21 | 京东科技控股股份有限公司 | Method and device for dynamically switching cloud storage service, electronic equipment and storage medium |
| CN112671590A (en) * | 2020-12-31 | 2021-04-16 | 北京奇艺世纪科技有限公司 | Data transmission method and device, electronic equipment and computer storage medium |
| CN112671590B (en) * | 2020-12-31 | 2023-01-20 | 北京奇艺世纪科技有限公司 | Data transmission method and device, electronic equipment and computer storage medium |
| CN113110926A (en) * | 2021-04-19 | 2021-07-13 | 上海华兴数字科技有限公司 | Dynamic adjustment method, device, medium and electronic equipment for data consumption thread |
| CN115396339A (en) * | 2022-08-24 | 2022-11-25 | 银清科技有限公司 | Abnormal message processing method and device |
| CN115396339B (en) * | 2022-08-24 | 2024-06-25 | 银清科技有限公司 | Abnormal message processing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107451147B (en) | 2020-07-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107451147A (en) | A kind of method and apparatus of kafka clusters switching at runtime | |
| CN109729143B (en) | Deploying a network-based cloud platform on a terminal device | |
| CN111858054B (en) | Resource scheduling system and method based on edge computing in heterogeneous environment | |
| WO2019210580A1 (en) | Access request processing method, apparatus, computer device, and storage medium | |
| CN105162879B (en) | Realize the method, apparatus and system of multimachine room data consistency | |
| CN110209484B (en) | Credible cloud task scheduling system and method based on multiple clouds and security mapping | |
| CN110611707B (en) | Task scheduling method and device | |
| CN112416581B (en) | Distributed calling system for timed tasks | |
| CN107040578A (en) | Method of data synchronization, device and system | |
| CN109408286A (en) | Data processing method, device, system, computer readable storage medium | |
| CN113204353B (en) | Big data platform assembly deployment method and device | |
| CN112737800A (en) | Service node fault positioning method, call chain generation method and server | |
| CN111858007A (en) | Task scheduling method and device based on message middleware | |
| CN108073423A (en) | A kind of accelerator loading method, system and accelerator loading device | |
| CN113220431A (en) | Cross-cloud distributed data task scheduling method, device and storage medium | |
| CN107203429A (en) | A kind of method and device that distributed task scheduling is loaded based on distributed lock | |
| CN112698952A (en) | Unified management method and device for computing resources, computer equipment and storage medium | |
| CN112328448A (en) | Zookeeper-based monitoring method, monitoring device, equipment and storage medium | |
| CN115757611A (en) | Big data cluster switching method and device, electronic equipment and storage medium | |
| CN117608825A (en) | Resource management method based on multi-cloud management platform and related equipment | |
| CN108132807B (en) | Parameter loading method and device | |
| CN115129423A (en) | Resource management method, device, equipment and storage medium | |
| CN109286617B (en) | Data processing method and related equipment | |
| CN110569124A (en) | Task allocation method and device | |
| CN112866314A (en) | Method for switching slave nodes in distributed master-slave system, master node device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |