CN107451147B - Method and device for dynamically switching kafka clusters - Google Patents
Method and device for dynamically switching kafka clusters Download PDFInfo
- Publication number
- CN107451147B CN107451147B CN201610375007.0A CN201610375007A CN107451147B CN 107451147 B CN107451147 B CN 107451147B CN 201610375007 A CN201610375007 A CN 201610375007A CN 107451147 B CN107451147 B CN 107451147B
- Authority
- CN
- China
- Prior art keywords
- cluster
- configuration information
- information
- previous
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention provides a method and a device for dynamically switching kafka clusters, which can monitor the running condition of the clusters in real time and dynamically modify the configuration information of the clusters to realize the dynamic switching of the clusters. The method for dynamically switching the kafka cluster comprises the following steps: the client sends the authentication parameters to a configuration service center to acquire cluster configuration information of a corresponding cluster; receiving cluster configuration information returned by the configuration service center and comparing the cluster configuration information with previous cluster configuration information stored on the client; and if the received cluster configuration information is different from the previous cluster configuration information, updating the previous cluster configuration information by using the received cluster configuration information and accessing the corresponding cluster according to the updated cluster configuration information.
Description
Technical Field
The invention relates to the technical field of computers and computer software, in particular to a method and a device for dynamically switching kafka clusters.
Background
In the distributed publish-subscribe messaging system kafka, resource occupation when kafka clusters switch and adjust different data partition reading is often realized by modifying configuration parameters of kafka (distributed publish-subscribe messaging system) client programs.
In general, parameters of the kafka object in the cluster, such as association relation between location information, module location information topic of partition information, and a password token, transmission efficiency, cluster configuration, backup quantity replicons, segment size, and the like, can be set by modifying the client program, and after the parameters are modified, the client program is compiled, packaged, and restarted to implement functions such as kafka cluster switching.
The kafka client can be divided into a production client and a consumption client according to different functions of the kafka client.
If a certain kafka client is a production client, namely, a producer, and performs a task of data generation, the operation steps when performing kafka cluster switching are as follows:
1. determining the task of the producer needing cluster switching and the cluster configuration information after switching (for example, including a root directory zkRoot of a distributed application program coordinating server zookeeper, partition information brokers on the server, module position information topic of the partition information, and the like);
2. modifying a program corresponding to a production client through a program compiling tool, finding out corresponding kafka (distributed publishing and subscribing message system) cluster configuration parameters, and recompiling and packaging after modification;
3. uploading the packaged program to a server and updating the original program;
4. logging in a server, and stopping and restarting a producer task;
5. testing whether the cluster switching task is successful, monitoring flow data through a kafka (distributed publish-subscribe message system) management interface, and if a topic group corresponding to the production task is established and data writing is started, indicating that the cluster switching is successful.
If a certain kafka client is a consuming client, i.e. a consumer, executing a task of data consumption, the operation steps when performing kafka cluster switching are as follows:
1. determining tasks of a provider needing cluster switching and modified cluster configuration information (mainly comprising a root directory zkRoot of a distributed application program coordinating server zookeeper, partition information brokers on the server, module position information topic of the partition information and the like);
2. opening a program of a kafka (distributed publish-subscribe message system) consumption client through a program compiling tool, finding a corresponding cluster configuration parameter, modifying the cluster configuration parameter into an updated cluster configuration parameter, and recompiling and packaging;
3. uploading the compiled program to a server, and updating the original program;
4. monitoring whether backlog exists in data corresponding to the consumption task through a kafka (distributed publish-subscribe message system) cluster monitoring interface, and when the backlog amount is 0, indicating that the data on the original cluster is completely consumed;
5. after the data on the original cluster is completely consumed, logging in a server through an ssh (command of server jump in linux) tool, stopping the original consumption program, modifying parameter information corresponding to a consumption task in the program, and then starting the consumer (consumption) task;
6. through a kafka (distributed publish-subscribe message system) cluster monitoring interface, testing and confirming that the task consumes data from a new cluster, and if a group of topic (partition module position information on a coordination server) corresponding to the consumption task is created and starts to consume the data, indicating that cluster switching is successful.
However, when the kafka cluster switching is performed by using the existing technical scheme, the following disadvantages are caused:
1. when the cluster is switched, the program of the client must be stopped, then the incidence relation, the transmission efficiency, the cluster configuration, the backup quantity replenishments, the segment size and other data of the position information, the module position information topic of the partition information and the password token are modified, the program is saved, compiled and packaged and then restarted, and the real-time continuous switching of the cluster cannot be realized;
2. the resource occupation conditions of the current producer task and the consumer task cannot be checked, the existing resources cannot be reasonably arranged and distributed, and the availability of the resources is not high.
Disclosure of Invention
In view of this, the present invention provides a method and an apparatus for dynamically switching a kafka cluster, which can monitor an operation status of the cluster in real time and dynamically modify configuration information of the cluster to implement dynamic switching of the cluster.
To achieve the above object, according to one aspect of the present invention, a method for kafka cluster dynamic switching is provided.
A method for dynamically switching kafka clusters comprises the following steps: the client sends the authentication parameters to a configuration service center to acquire cluster configuration information of a corresponding cluster; receiving cluster configuration information returned by the configuration service center and comparing the cluster configuration information with previous cluster configuration information stored on the client; and if the received cluster configuration information is different from the previous cluster configuration information, updating the previous cluster configuration information by using the received cluster configuration information and accessing the corresponding cluster according to the updated cluster configuration information.
Optionally, the clients include a production client and a consumption client, the production client produces data and writes the data into the cluster, and the consumption client reads data from the cluster and consumes the data.
Optionally, the client sends heartbeat information of the client to the configuration service center at regular time, where the heartbeat information includes: rate information, data processing type, including production and consumption, and number of data pieces.
Optionally, the configuration service center receives an authentication parameter sent by the client by using the connection interface, performs an authentication operation, and returns the cluster configuration information after the authentication is passed.
Optionally, the method further comprises: for a consumption client, when a previous cluster is switched to a new cluster, if the new cluster is a backup cluster of the previous cluster, the configuration service center records the current consumption site information and stores the current consumption site information in a database, and then updates the cluster configuration information of the previous cluster into the cluster configuration information of the new cluster; otherwise, the configuration service center checks the message backlog amount of the consumption task, and updates the cluster configuration information of the previous cluster to the cluster configuration information of the new cluster when the message backlog amount is 0.
Optionally, the cluster configuration information includes: the cluster message address, the cluster partition identification, and the address of the cluster in zookeeper.
According to another aspect of the present invention, an apparatus for dynamically switching kafka clusters is provided.
An apparatus for kafka cluster dynamic switching, comprising: the information acquisition module is used for the client side to send the authentication parameters to the configuration service center so as to acquire cluster configuration information of a corresponding cluster; the information comparison module is used for receiving the cluster configuration information returned by the configuration service center and comparing the cluster configuration information with the previous cluster configuration information stored on the client; and the information updating module is used for updating the previous cluster configuration information by using the received cluster configuration information and accessing the corresponding cluster according to the updated cluster configuration information if the received cluster configuration information is different from the previous cluster configuration information.
Optionally, the clients include a production client and a consumption client, the production client produces data and writes the data into the cluster, and the consumption client reads data from the cluster and consumes the data.
Optionally, the client sends heartbeat information of the client to the configuration service center at regular time, where the heartbeat information includes: rate information, data processing type, including production and consumption, and number of data pieces.
Optionally, the configuration service center receives an authentication parameter sent by the client by using the connection interface, performs an authentication operation, and returns the cluster configuration information after the authentication is passed.
Optionally, the method further comprises: for a consumption client, when a previous cluster is switched to a new cluster, if the new cluster is a backup cluster of the previous cluster, the configuration service center records the current consumption site information and stores the current consumption site information in a database, and then updates the cluster configuration information of the previous cluster into the cluster configuration information of the new cluster; otherwise, the configuration service center checks the message backlog amount of the consumption task, and updates the cluster configuration information of the previous cluster to the cluster configuration information of the new cluster when the message backlog amount is 0.
Optionally, the cluster configuration information includes: the cluster message address, the cluster partition identification, and the address of the cluster in zookeeper.
According to another aspect of the present invention, an apparatus for dynamically switching kafka clusters is provided.
An apparatus for kafka cluster dynamic switching, comprising: a memory and a processor, wherein the memory stores instructions; the processor executing the instructions to: the client sends the authentication parameters to a configuration service center to acquire cluster configuration information of a corresponding cluster; receiving cluster configuration information returned by the configuration service center and comparing the cluster configuration information with previous cluster configuration information stored on the client; and if the received cluster configuration information is different from the previous cluster configuration information, updating the previous cluster configuration information by using the received cluster configuration information and accessing the corresponding cluster according to the updated cluster configuration information.
According to yet another aspect of the invention, an electronic device for dynamic switching of kafka clusters is provided.
An electronic device for kafka cluster dynamic switching, comprising: one or more processors; and the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors realize the method for dynamically switching the kafka cluster, provided by the invention.
According to yet another aspect of the present invention, a computer-readable medium is provided.
A computer readable medium, on which a computer program is stored, which when executed by a processor implements the method for kafka cluster dynamic switching provided by the present invention.
According to the technical scheme of the invention, the configuration information of the kafka cluster is packaged in the configuration service center on the outer layer of the cluster and is displayed in the form of the client, so that the resource occupation condition and the operation information of the cluster can be monitored in real time, and the cluster can be dynamically switched according to the specific condition of a new cluster to be switched, so that the cluster is dynamically switched under the condition of monitoring the message backlog of the cluster, and the cluster is more efficiently used; in addition, the configuration information of the kafka cluster is encapsulated in the configuration service center on the outer layer of the cluster, cluster parameters can be dynamically configured through the configuration service center, a restarting program does not need to be suspended when cluster switching is carried out, the configuration parameters are read in real time and are applied to production and consumption tasks of messages, and the cluster switching is convenient and rapid.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
fig. 1 is a schematic diagram of main steps of a method for dynamically switching kafka clusters according to an embodiment of the present invention;
FIG. 2 is an architecture diagram of a kafka cluster dynamic switching system according to an embodiment of the present invention;
fig. 3 is a schematic diagram of main modules of an apparatus for dynamically switching kafka clusters according to an embodiment of the present invention;
fig. 4 is a schematic diagram of main modules of an apparatus for dynamically switching kafka clusters according to another embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
The invention provides the following technical scheme: and encapsulating the configuration information of the cluster corresponding to the production and consumption tasks of the messages in the kafka system outside the kafka cluster to form a configuration service center, and displaying the configuration service center in the form of a single client so as to monitor the operation information of the cluster. And the production client and the consumption client perform identity verification and search of cluster configuration information through an interface provided by the configuration service center. When the client finds that the cluster configuration information changes, the cluster configuration information in the client can be updated in real time, and the client is switched to a new cluster to produce or consume the message.
Fig. 1 is a schematic diagram of main steps of a method for dynamically switching kafka clusters according to an embodiment of the present invention. As shown in fig. 1, the method for dynamically switching kafka clusters of the present invention mainly includes the following steps S11 to S13.
Step S11: the client sends the authentication parameters to a configuration service center to acquire cluster configuration information of a corresponding cluster;
step S12: receiving cluster configuration information returned by the configuration service center and comparing the cluster configuration information with previous cluster configuration information stored on the client;
step S13: and if the received cluster configuration information is different from the previous cluster configuration information, updating the previous cluster configuration information by using the received cluster configuration information and accessing the corresponding cluster according to the updated cluster configuration information.
The client comprises a production client and a consumption client, the production client produces data and writes the data into the cluster, and the consumption client reads the data from the cluster and consumes the data.
According to the technical scheme of the invention, the client regularly sends the heartbeat information of the client to the configuration service center, wherein the heartbeat information comprises: rate information, data processing type, including production and consumption, and number of data pieces.
In the invention, the configuration service center receives the authentication parameters sent by the client by using the connection interface, performs authentication operation and returns the cluster configuration information after the authentication is passed.
For the consumption client, when a previous cluster is switched to a new cluster, if the new cluster is a backup cluster of the previous cluster, the configuration service center records the current consumption site information and stores the current consumption site information into a database, and then updates the cluster configuration information of the previous cluster into the cluster configuration information of the new cluster; otherwise, the configuration service center checks the message backlog amount of the consumption task, and updates the cluster configuration information of the previous cluster to the cluster configuration information of the new cluster when the message backlog amount is 0.
In addition, the cluster configuration information in the present invention includes: the cluster message address, the cluster partition identification, and the address of the cluster in zookeeper.
Fig. 2 is an architecture diagram of a kafka cluster dynamic switching system according to an embodiment of the present invention. As shown in fig. 2, the present invention encapsulates a configuration service center outside the kafka cluster based on kafka, and the configuration service center provides a user authentication and an interface for obtaining cluster configuration information for the client. The client can verify the authentication parameters by the user name and the password, and read the kafka cluster configuration information in real time after the verification is passed.
In the embodiment of the invention, a kafka client is packaged, and kafka partition parameters, namely ' module position information topic of partition information ' and ' zk information ' (IP address information of each server in a distributed application program coordinating server zookeeper ' and the like, are packaged into appId (user name) and token (password). In addition, zk information (IP address information of each server in a cluster in a distributed application program coordination server zookeeper), partition information brokers on the server, module position information topic of the partition information, a consumption group name and the like can be used as the authentication parameters, but the parameters can be acquired by a user and the security is not high. Therefore, the invention adopts the packaged appId (user name) and token (password) to carry out user authentication. Wherein, the user name is a system domain name generally; the password is generated by a system domain name, a topic data identifier and the like according to a certain rule and is related to module position information topic information of the partition information. Specific examples thereof include: if the client is a production client, the password is obtained by encoding and then packaging three fields of a system domain name, a cluster information identifier and an authentication type; if the client is a consumption client, the password is obtained by encoding and packaging four fields of a system domain name, a cluster information identifier, an authentication type and a consumption group. Wherein the authentication type mainly refers to production and consumption.
In addition, in the embodiment of the present invention, a timed task mode may be set at the client, and heartbeat information (for example, rate information, data processing type, number of pieces of data, and the like, where the data processing type includes production and consumption) of the client is sent to the configuration service center at a fixed time through an interface provided by the configuration service center, and the latest cluster configuration information corresponding to the appId (user name) and token (password) is obtained. The "timing" can be flexibly set according to the cluster operation condition, for example, set to 10 minutes in the case of stable cluster operation.
After the client acquires the latest cluster configuration information, the latest cluster configuration information is compared with the previous cluster configuration information stored on the client, if a change is found, the previous cluster configuration information is updated by the acquired cluster configuration information and the configuration information is automatically loaded, so that the configuration is effective, and then the corresponding cluster is accessed according to the updated cluster configuration information.
The implementation of the present invention will be described below in terms of both production and consumer clients.
For a production client, before cluster switching, establishing an association relationship between module position information topic information of partition information of a new cluster and an existing token; then, the address information of the new kafka cluster is updated to the configuration table of the packaged configuration service center. Here, the address information of the new kafka cluster refers to the module location information topic of the partition information and the partition information brokers on the server. When the production client starts to write data, the cluster configuration information is read in real time and is compared with the previous cluster configuration information used by the current client after being read. If the read cluster configuration information is different from the previous cluster configuration information, the production client (e.g., client1 in the figure) dynamically loads, analyzes and reads the cluster configuration information in real time, and performs parameter assignment to apply the configuration information of the new cluster to the program, thereby dynamically loading the switched cluster configuration information.
For the consuming client, before cluster switching, an association relationship needs to be established between the module position information topic of the partition information of the new cluster and the existing token. In addition, it is also necessary to determine which way to select to switch the cluster according to the relationship between the new cluster to be switched and the previous cluster.
If the new cluster is a backup cluster of the previous cluster, the configuration service center records the current consumption site information and stores the current consumption site information into the database, and then updates the cluster configuration information of the previous cluster into the cluster configuration information of the new cluster. Sometimes, when data is stored, data is backed up, for example: there are two kafka clusters a and B, B being the backup cluster for a, the data in B being the same as the data in a. Suppose that a consuming client clienta, as in fig. 2, is currently consuming data in cluster a, and needs to switch to consuming data in cluster B. Then, it is necessary to first record the clienta in the database at the current location where cluster a consumes the data, and then switch the clienta to consume the data of cluster B. When switching clusters, configuration information such as IP address information of each server, module position information topic of partition information, partition information brokers on the server, and the like in a distributed application program coordination server zookeeper of a new cluster needs to be updated to a configuration service center, and then a client acquires latest cluster configuration information through an interface provided by the configuration service center and automatically loads the latest cluster configuration information so that configuration becomes effective. Reading the data site of the client consumption cluster A from the database before starting to consume the data after the cluster is switched, then finding the corresponding data site in the cluster B, and finally starting to consume the data from the corresponding data site of the cluster B. Therefore, the cluster can be switched at any time under the condition of not considering the backlog of the messages to be consumed, and the data is ensured not to be lost.
If the new cluster is not a backup cluster of the previous cluster, that is, the data in the new cluster is different from or not identical to the data in the previous cluster, the configuration service center needs to check the message backlog amount of the consuming task, and update the cluster configuration information of the previous cluster to the cluster configuration information of the new cluster when the message backlog amount is 0. The specific process is as follows: the method comprises the steps that a configuration service center monitors the backlog of a consumption task in real time by a method of recording the current consumption data, when the message backlog is 0, configuration information such as IP address information of a zookeeper, module position information topic of partition information and partition information brokers on a server of a distributed application program coordination server of a new cluster is updated to the configuration service center, and then a client side obtains the latest cluster configuration information through an interface provided by the configuration service center and automatically loads the latest cluster configuration information to enable configuration to take effect. After the cluster is switched to a new cluster for message consumption, the client side automatically checks the switched cluster and starts to consume from the first data in the cluster so as to ensure that the data is not lost. Therefore, the cluster can be switched at any time under the condition that the message is not backlogged, and the data is ensured not to be lost.
Fig. 3 is a schematic diagram of main modules of an apparatus for dynamically switching kafka clusters according to an embodiment of the present invention. As shown in fig. 3, the apparatus 30 for dynamically switching kafka clusters mainly includes an information obtaining module 31, an information comparing module 32 and an information updating module 33.
The information obtaining module 31 is configured to send the authentication parameter to the configuration service center by the client to obtain cluster configuration information of a corresponding cluster; the information comparison module 32 is configured to receive the cluster configuration information returned by the configuration service center and compare the cluster configuration information with the previous cluster configuration information stored on the client; the information updating module 33 is configured to update the previous cluster configuration information with the received cluster configuration information and access the corresponding cluster according to the updated cluster configuration information if the received cluster configuration information is different from the previous cluster configuration information.
In the invention, the client comprises a production client and a consumption client, the production client produces data and writes the data into the cluster, and the consumption client reads the data from the cluster and consumes the data.
And the client regularly sends heartbeat information of the client to the configuration service center, wherein the heartbeat information comprises: rate information, data processing type, including production and consumption, and number of data pieces.
According to the technical scheme of the embodiment of the invention, the configuration service center receives the authentication parameters sent by the client by using the connection interface, performs authentication operation and returns the cluster configuration information after the authentication is passed.
For the consumption client, when a previous cluster is switched to a new cluster, if the new cluster is a backup cluster of the previous cluster, the configuration service center records the current consumption site information and stores the current consumption site information into a database, and then updates the cluster configuration information of the previous cluster into the cluster configuration information of the new cluster; otherwise, the configuration service center checks the message backlog amount of the consumption task, and updates the cluster configuration information of the previous cluster to the cluster configuration information of the new cluster when the message backlog amount is 0.
Wherein the cluster configuration information includes: the cluster message address, the cluster partition identification, and the address of the cluster in zookeeper.
Fig. 4 is a schematic diagram of main modules of an apparatus for dynamically switching kafka clusters according to another embodiment of the present invention. As shown in fig. 4, the apparatus 40 for dynamically switching kafka clusters of the present invention mainly includes a memory 41 and a processor 42.
Wherein the memory 41 stores instructions; processor 42 executes the instructions to: the client sends the authentication parameters to a configuration service center to acquire cluster configuration information of a corresponding cluster; receiving cluster configuration information returned by the configuration service center and comparing the cluster configuration information with previous cluster configuration information stored on the client; and if the received cluster configuration information is different from the previous cluster configuration information, updating the previous cluster configuration information by using the received cluster configuration information and accessing the corresponding cluster according to the updated cluster configuration information.
According to the technical scheme of the embodiment of the invention, the configuration information of the kafka cluster is packaged in the configuration service center on the outer layer of the cluster and is displayed in the form of the client, so that the resource occupation condition and the operation information of the cluster can be monitored in real time, and the cluster can be dynamically switched according to the specific condition of a new cluster to be switched, so that the cluster is dynamically switched under the condition of monitoring the message backlog of the cluster, and the cluster is more efficiently used; in addition, the configuration information of the kafka cluster is encapsulated in the configuration service center on the outer layer of the cluster, cluster parameters can be dynamically configured through the configuration service center, a restarting program does not need to be suspended when cluster switching is carried out, the configuration parameters are read in real time and are applied to production and consumption tasks of messages, and the cluster switching is convenient and rapid.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (15)
1. A method for dynamically switching kafka clusters is characterized by comprising the following steps:
the client sends the authentication parameters to a configuration service center to acquire cluster configuration information of a corresponding cluster;
receiving cluster configuration information returned by the configuration service center and comparing the cluster configuration information with previous cluster configuration information stored on the client;
and if the received cluster configuration information is different from the previous cluster configuration information, updating the previous cluster configuration information by using the received cluster configuration information and accessing the corresponding cluster according to the updated cluster configuration information according to the relationship between the cluster corresponding to the received cluster configuration information and the previous cluster and the resource occupation condition and the operation information of the previous cluster recorded by the configuration service center.
2. The method of claim 1, wherein the clients comprise production clients that produce data and write the data to the cluster and consumption clients that read data from the cluster and consume the data.
3. The method according to claim 1, wherein the client periodically sends heartbeat information of the client to the configuration service center, and the heartbeat information includes: rate information, data processing type, including production and consumption, and number of data pieces.
4. The method of claim 1, wherein the configuration service center receives an authentication parameter sent by a client through a connection interface, performs an authentication operation, and returns the cluster configuration information after the authentication is passed.
5. The method of claim 2, further comprising: for a consumption client, when a previous cluster is switched to a new cluster, if the new cluster is a backup cluster of the previous cluster, the configuration service center records the current consumption site information and stores the current consumption site information in a database, and then updates the cluster configuration information of the previous cluster into the cluster configuration information of the new cluster;
otherwise, the configuration service center checks the message backlog amount of the consumption task, and updates the cluster configuration information of the previous cluster to the cluster configuration information of the new cluster when the message backlog amount is 0.
6. The method of claim 1, wherein the cluster configuration information comprises: the cluster message address, the cluster partition identification, and the address of the cluster in zookeeper.
7. An apparatus for dynamically switching kafka clusters, comprising:
the information acquisition module is used for the client side to send the authentication parameters to the configuration service center so as to acquire cluster configuration information of a corresponding cluster;
the information comparison module is used for receiving the cluster configuration information returned by the configuration service center and comparing the cluster configuration information with the previous cluster configuration information stored on the client;
and the information updating module is used for updating the previous cluster configuration information by using the received cluster configuration information and accessing the corresponding cluster according to the updated cluster configuration information according to the relationship between the cluster corresponding to the received cluster configuration information and the previous cluster and the resource occupation condition and the operation information of the previous cluster recorded by the configuration service center if the received cluster configuration information is different from the previous cluster configuration information.
8. The apparatus of claim 7, wherein the clients comprise production clients that produce data and write the data to the cluster and consumption clients that read data from the cluster and consume the data.
9. The apparatus according to claim 7, wherein the client periodically sends heartbeat information of the client to the configuration service center, and the heartbeat information includes: rate information, data processing type, including production and consumption, and number of data pieces.
10. The apparatus of claim 7, wherein the configuration service center receives an authentication parameter sent by a client via a connection interface, performs an authentication operation, and returns the cluster configuration information after the authentication is passed.
11. The apparatus of claim 8, further comprising: for a consuming client, when switching from a previous cluster to a new cluster,
if the new cluster is a backup cluster of the previous cluster, the configuration service center records the current consumption site information and stores the current consumption site information in a database, and then updates the cluster configuration information of the previous cluster into the cluster configuration information of the new cluster;
otherwise, the configuration service center checks the message backlog amount of the consumption task, and updates the cluster configuration information of the previous cluster to the cluster configuration information of the new cluster when the message backlog amount is 0.
12. The apparatus of claim 7, wherein the cluster configuration information comprises: the cluster message address, the cluster partition identification, and the address of the cluster in zookeeper.
13. An apparatus for dynamically switching kafka clusters, comprising: a memory and a processor, wherein,
the memory stores instructions;
the processor executing the instructions to:
the client sends the authentication parameters to a configuration service center to acquire cluster configuration information of a corresponding cluster;
receiving cluster configuration information returned by the configuration service center and comparing the cluster configuration information with previous cluster configuration information stored on the client;
and if the received cluster configuration information is different from the previous cluster configuration information, updating the previous cluster configuration information by using the received cluster configuration information and accessing the corresponding cluster according to the updated cluster configuration information according to the relationship between the cluster corresponding to the received cluster configuration information and the previous cluster and the resource occupation condition and the operation information of the previous cluster recorded by the configuration service center.
14. An electronic device for dynamically switching kafka clusters, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.
15. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610375007.0A CN107451147B (en) | 2016-05-31 | 2016-05-31 | Method and device for dynamically switching kafka clusters |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610375007.0A CN107451147B (en) | 2016-05-31 | 2016-05-31 | Method and device for dynamically switching kafka clusters |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107451147A CN107451147A (en) | 2017-12-08 |
| CN107451147B true CN107451147B (en) | 2020-07-31 |
Family
ID=60485833
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610375007.0A Active CN107451147B (en) | 2016-05-31 | 2016-05-31 | Method and device for dynamically switching kafka clusters |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107451147B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108322358B (en) * | 2017-12-15 | 2020-09-01 | 北京奇艺世纪科技有限公司 | Method and device for sending, processing and consuming multi-live distributed messages in different places |
| CN108170527B (en) * | 2017-12-15 | 2021-06-22 | 北京奇艺世纪科技有限公司 | Remote multi-activity distributed message consumption method and device |
| CN108667930B (en) * | 2018-05-09 | 2020-10-30 | 聚龙股份有限公司 | Dynamic data source configuration and automatic switching method |
| CN108900572B (en) * | 2018-05-31 | 2022-08-09 | 康键信息技术（深圳）有限公司 | Main database domain name switching control method, computer equipment and storage medium |
| CN109861878B (en) * | 2019-01-17 | 2022-03-25 | 平安科技（深圳）有限公司 | Method for monitoring topic data of kafka cluster and related equipment |
| CN111726388A (en) * | 2019-03-22 | 2020-09-29 | 苏宁易购集团股份有限公司 | Cross-cluster high-availability implementation method, device, system and equipment |
| CN110515748B (en) * | 2019-08-28 | 2022-02-01 | 腾讯科技（深圳）有限公司 | Message processing method and related device |
| CN110716813A (en) * | 2019-09-17 | 2020-01-21 | 贝壳技术有限公司 | Data stream processing method and device, readable storage medium and processor |
| CN112948346B (en) * | 2019-11-26 | 2023-05-02 | 马上消费金融股份有限公司 | Database switching method, device, equipment and computer readable storage medium |
| CN111163172B (en) * | 2019-12-31 | 2022-04-22 | 北京奇艺世纪科技有限公司 | Message processing system, method, electronic device and storage medium |
| CN114650283A (en) * | 2020-12-21 | 2022-06-21 | 京东科技控股股份有限公司 | Method and device for dynamically switching cloud storage service, electronic equipment and storage medium |
| CN112671590B (en) * | 2020-12-31 | 2023-01-20 | 北京奇艺世纪科技有限公司 | Data transmission method and device, electronic equipment and computer storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102255752A (en) * | 2011-06-30 | 2011-11-23 | 北京新媒传信科技有限公司 | Configuration management system and method of server cluster |
| CN103516538A (en) * | 2012-06-28 | 2014-01-15 | 中兴通讯股份有限公司 | Method and device for uniform configuration of carrier-class clustered applications |
| CN104092718A (en) * | 2013-12-12 | 2014-10-08 | 腾讯数码（天津）有限公司 | Distributed system and configuration information updating method in distributed system |
| CN105610987A (en) * | 2016-03-18 | 2016-05-25 | 车智互联（北京）科技有限公司 | Method, application and system for managing server cluster |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2675127B1 (en) * | 2011-08-29 | 2015-07-15 | Huawei Technologies Co., Ltd. | Method and device for automatically migrating system configuration item |
| CN105279603B (en) * | 2015-09-11 | 2020-02-07 | 福建师范大学 | Dynamically configurable big data analysis system and method |
-
2016
- 2016-05-31 CN CN201610375007.0A patent/CN107451147B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102255752A (en) * | 2011-06-30 | 2011-11-23 | 北京新媒传信科技有限公司 | Configuration management system and method of server cluster |
| CN103516538A (en) * | 2012-06-28 | 2014-01-15 | 中兴通讯股份有限公司 | Method and device for uniform configuration of carrier-class clustered applications |
| CN104092718A (en) * | 2013-12-12 | 2014-10-08 | 腾讯数码（天津）有限公司 | Distributed system and configuration information updating method in distributed system |
| CN105610987A (en) * | 2016-03-18 | 2016-05-25 | 车智互联（北京）科技有限公司 | Method, application and system for managing server cluster |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107451147A (en) | 2017-12-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107451147B (en) | Method and device for dynamically switching kafka clusters | |
| CN107729139B (en) | Method and device for concurrently acquiring resources | |
| US9535754B1 (en) | Dynamic provisioning of computing resources | |
| CN111124277B (en) | Deep learning data set caching method, system, terminal and storage medium | |
| US9396039B1 (en) | Scalable load testing using a queue | |
| CN107491371B (en) | Deployment monitoring method and device | |
| US10063644B1 (en) | Managing operation of instances | |
| US20160182320A1 (en) | Techniques to generate a graph model for cloud infrastructure elements | |
| CN110611707B (en) | Task scheduling method and device | |
| CN104166589A (en) | Heartbeat package processing method and device | |
| CN113204353B (en) | Big data platform assembly deployment method and device | |
| US20220374282A1 (en) | Container runtime optimization | |
| CN109257396B (en) | Distributed lock scheduling method and device | |
| CN115525406A (en) | Distributed asynchronous task scheduling system | |
| US10185613B2 (en) | Error determination from logs | |
| CN109684051B (en) | Method and system for asynchronously submitting hybrid big data task | |
| CN111399999A (en) | Computer resource processing method and device, readable storage medium and computer equipment | |
| US10110502B1 (en) | Autonomous host deployment in managed deployment systems | |
| US9244736B2 (en) | Thinning operating systems | |
| CN110297706A (en) | A kind of inserting method based on Eureka-Server project | |
| CN115617459A (en) | Method, device and equipment for resource scheduling | |
| CN115080309A (en) | Data backup system, method, storage medium, and electronic device | |
| CN114327835A (en) | Distributed task scheduling method and device, processor and electronic equipment | |
| US20170316035A1 (en) | Rule-governed entitlement data structure change notifications | |
| CN113407629A (en) | Data synchronization method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |