CN116346578A - Kafka double-cluster real-time flow automatic switching method and device and electronic equipment - Google Patents
Kafka double-cluster real-time flow automatic switching method and device and electronic equipment Download PDFInfo
- Publication number
- CN116346578A CN116346578A CN202310323573.7A CN202310323573A CN116346578A CN 116346578 A CN116346578 A CN 116346578A CN 202310323573 A CN202310323573 A CN 202310323573A CN 116346578 A CN116346578 A CN 116346578A
- Authority
- CN
- China
- Prior art keywords
- cluster
- address
- data
- release
- traffic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000004519 manufacturing process Methods 0.000 claims abstract description 25
- 238000003860 storage Methods 0.000 claims abstract description 15
- 230000004044 response Effects 0.000 claims abstract description 12
- 238000004590 computer program Methods 0.000 claims description 12
- 238000004891 communication Methods 0.000 claims description 10
- 238000009825 accumulation Methods 0.000 claims description 4
- 238000009826 distribution Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 230000002159 abnormal effect Effects 0.000 description 4
- 230000007613 environmental effect Effects 0.000 description 4
- 238000005192 partition Methods 0.000 description 4
- 230000009977 dual effect Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0663—Performing the actions predefined by failover planning, e.g. switching to standby network elements
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The disclosure relates to the technical field of computers, and in particular relates to an automatic switching method and device of Kafka dual-cluster real-time traffic, electronic equipment and a storage medium. The specific implementation scheme is as follows: the method comprises the steps that a production client creates a first data release instance and a second data release instance and is respectively connected with a main cluster and a standby cluster; the production client calls an interface of the data bus cluster management and control platform to acquire a target release address corresponding to each group of traffic of the first data release example release; judging whether the target release address corresponding to each group of flow is consistent with the address of the main cluster; responding to the inconformity of all the target release addresses and the addresses of the main clusters and the correspondence of the target release addresses and the addresses of the standby clusters, and switching to a second data release instance to release each group of flow to the standby clusters; in response to at least one of the target published addresses being consistent with the address of the primary cluster, no handoff is performed. Therefore, the fast and automatic switching of the Kafka double-cluster real-time flow is realized, and the problem of long switching time consumption is solved.
Description
Technical Field
The disclosure relates to the technical field of computers, and in particular relates to an automatic switching method and device of Kafka dual-cluster real-time traffic, electronic equipment and a storage medium.
Background
Kafka is an event stream platform that combines three key functions, publish and subscribe to event streams, store event streams that are persistent and reliable, process event streams at the time of event occurrence or retrospectively. These functions allow the user to implement end-to-end event stream use cases. These functions are provided in a distributed, highly scalable, highly resilient, fault tolerant and secure manner. Kafka may be deployed on physical machines, virtual machines, and containers.
The Kafka can only guarantee the stability inside a single cluster at present, and once all physical machines of the whole Kafka cluster crash at the same time, for example, the whole machine room is powered down, the Kafka cluster cannot serve outside. When the Kafka cluster crashes or wants to maintain the cluster, if all the docking systems are to switch to the standby cluster, the switching participated by the upstream and downstream systems together takes a lot of time.
Disclosure of Invention
The disclosure provides a Kafka double-cluster real-time flow automatic switching method and device, electronic equipment and a storage medium.
According to a first aspect of the present disclosure, there is provided an automatic switching method of Kafka dual-cluster real-time traffic, including:
the method comprises the steps that a production client creates a first data release instance and a second data release instance and is respectively connected with a main cluster and a standby cluster;
the production client calls an interface of a data bus cluster management and control platform to acquire a target release address corresponding to each group of traffic released by the first data release example;
judging whether the target release address corresponding to each group of flow is consistent with the address of the main cluster;
responding to the fact that all the target release addresses are inconsistent with the addresses of the main clusters and consistent with the addresses of the standby clusters, switching to the second data release instance to release all the groups of traffic to the standby clusters;
and responding to the fact that at least one target release address is consistent with the address of the main cluster, not executing switching, and continuing to release the groups of traffic to the main cluster through the first data release instance. .
According to a second aspect of the present disclosure, there is provided an automatic switching method of Kafka dual cluster real-time traffic, including:
the consumption client creates a data subscription instance and is connected with the main cluster;
the consumption client calls an interface of a data bus cluster management and control platform to acquire a target subscription address corresponding to each group of traffic subscribed by the data subscription instance;
judging whether the target subscription address corresponding to each group of the traffic is consistent with the address of the main cluster;
responding to the fact that all the target subscription addresses are inconsistent with the addresses of the main clusters and consistent with the addresses of the standby clusters, switching the consumption client to be connected with the standby clusters, and subscribing the groups of traffic from the standby clusters;
and responding to the fact that at least one target subscription address is consistent with the address of the main cluster, not executing switching, and continuing to subscribe the groups of traffic to the main cluster through the data subscription instance.
According to a third aspect of the present disclosure, there is provided an automatic switching device for Kafka dual cluster real-time traffic, including:
the first creating module is configured to create a first data release instance and a second data release instance and is respectively connected with the main cluster and the standby cluster;
the first acquisition module is configured to call an interface of the data bus cluster management and control platform to acquire a target release address corresponding to each group of traffic released by the first data release example;
the first judging module is configured to judge whether the target release address corresponding to each group of traffic is consistent with the address of the main cluster;
the first execution module is configured to respond to the fact that all the target release addresses are inconsistent with the addresses of the main clusters and consistent with the addresses of the standby clusters, and switch to the second data release instance to release the groups of traffic to the standby clusters;
and the first execution module responds to the fact that at least one target release address is consistent with the address of the main cluster, does not execute switching, and continues to release the flows of each group to the main cluster through the first data release example.
According to a fourth aspect of the present disclosure, there is provided an automatic switching device for Kafka dual cluster real-time traffic, including:
the second creation module is configured to create a data subscription instance and is connected with the main cluster;
the second acquisition module is configured to call an interface of the data bus cluster management and control platform, and acquire a target subscription address corresponding to each group of traffic subscribed by the data subscription instance;
the second judging module is configured to judge whether the target subscription address corresponding to each group of traffic is consistent with the address of the main cluster;
the second execution module is configured to respond to the fact that all the target subscription addresses are inconsistent with the addresses of the main clusters and consistent with the addresses of the standby clusters, the consumption client switches to be connected with the standby clusters, and subscribes to the groups of traffic from the standby clusters;
and the second execution module does not execute switching and continues to subscribe the groups of traffic to the main cluster through the data subscription instance in response to the fact that at least one target subscription address is consistent with the address of the main cluster.
According to a fifth aspect of the present disclosure, there is provided an electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of the above claims.
According to a sixth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of the above-mentioned technical solutions.
According to a seventh aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements a method according to any of the above-mentioned technical solutions.
The invention provides a Kafka double-cluster real-time flow automatic switching method, a Kafka double-cluster real-time flow automatic switching device, electronic equipment and a Kafka double-cluster real-time flow automatic switching storage medium.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.
Drawings
The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:
FIG. 1 is a schematic diagram of steps of a method for automatic handoff of a production client in an embodiment of the present disclosure;
FIG. 2 is a schematic diagram of a production client auto-switching method in an embodiment of the present disclosure;
FIG. 3 is a schematic diagram of steps of a method for automatic switching of a consuming client in an embodiment of the disclosure;
FIG. 4 is a schematic diagram of a method for automatic switching of a consuming client in an embodiment of the present disclosure;
FIG. 5 is a schematic diagram of the overall principle of the Kafka dual-cluster real-time traffic automatic switching method in an embodiment of the present disclosure;
FIG. 6 is a functional block diagram of a production client automatic switching device in an embodiment of the present disclosure;
fig. 7 is a functional block diagram of a consumer client automatic switching device in an embodiment of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
The present disclosure provides an automatic switching method for Kafka dual-cluster real-time traffic, as shown in fig. 1, including:
in step S101, the production client creates a first data publishing instance and a second data publishing instance, and connects the primary cluster and the standby cluster, respectively.
Step S102, a production client calls an interface of a data bus cluster management and control platform to acquire a target release address corresponding to each group of traffic released by a first data release example.
Step S103, judging whether the target release address corresponding to each group of flow is consistent with the address of the main cluster.
Step S104, responding to all the target release addresses and the addresses of the main clusters, and consistent with the addresses of the standby clusters, switching to the second data release instance to release each group of traffic to the standby clusters.
Step S105, in response to the at least one target publishing address being consistent with the address of the primary cluster, not executing the switching, and continuing to publish each group of traffic to the primary cluster through the first data publishing instance.
As shown in fig. 2, when the production client 201 uses the encapsulated SDK (Software Development Kit, software development tool) to create a Producer object, an interface of the cluster management and control platform 202 is called to obtain addresses of the main cluster 203 and the standby cluster 204, and then two objects, namely, a data publishing instance 1 and a data publishing instance 2, are created according to the returned main and standby cluster addresses, so as to respectively connect the main cluster 203 and the standby cluster 204. The production client 201 obtains a target release address corresponding to each group of Topic level traffic currently released by the production client 201, namely a connected cluster address (a main cluster or a standby cluster) by calling an interface of the primary cluster management platform 202. When data is transmitted each time, firstly comparing the cluster address of a Topic flow cache for transmitting the data with the addresses of the main and standby clusters, and if the target release address of the Topic flow is the same as the main cluster address, using the object (the first data release instance 205) of the main cluster 203 to transmit the data; if the target release address of the Topic traffic is the same as that of the backup cluster 204, the object (the second data release instance 206) of the backup cluster 204 is used to send data, and if the target release address is not the same as that of both the primary and backup clusters, the two Producer objects are re-created and the primary and backup cluster addresses are re-acquired. Through the technical scheme, when the main cluster is abnormal, the flow issued by the production client can be timely switched to the standby cluster, so that the switching time is reduced, and the stability of the cluster is ensured.
As an alternative embodiment, the data bus cluster management platform is configured to implement flow control on the Topic level, and may control whether each Topic of each cluster is connected to a primary cluster or a backup cluster. The user can enter the cluster switching management page through logging in the display module of the data bus cluster management platform. Environmental information can be newly added to the page, the environmental information is modified, a flow control rule is created, and the publishing and subscribing flow of Topic in the environment is controlled. At the same time, the list of topics can be clicked and refreshed, and all topics of the main cluster in the environment can be obtained by one key. The list of cluster groups may be clicked to refresh and all groups of the master cluster in the environment may be retrieved by one-touch.
As an optional implementation manner, the method for the production client to call the interface of the data bus cluster management and control platform, and the method for obtaining the target release address corresponding to each group of traffic released by the first data release example includes: as shown in fig. 2, a first timing task thread 207 is created, and the interface of the data bus cluster management platform 202 is called every other preset time based on the first timing task thread 207, so as to obtain the target release addresses corresponding to each group of traffic. For example, the preset time can be set to 10s, and the production client calls the interface of the data bus cluster management and control platform every 10s to acquire the cluster address connected with the current issued Topic level traffic, so that the real-time monitoring and switching of the cluster address connected with the traffic are realized.
As an alternative embodiment, in response to at least one of the target published addresses not being consistent with the address of the primary cluster and not being consistent with the address of the backup cluster, the first data published instance and the second data published instance are recreated and the address of the primary cluster and the address of the backup cluster are reacquired. When the cluster addresses of all the Topic traffic are inconsistent with the main cluster and the standby cluster, the target release address of the Topic is possibly modified and cannot be connected to the main cluster and the standby cluster, at this time, the object is required to be re-created and the addresses of the main cluster and the standby cluster are required to be re-acquired, so that the production client and the cluster reestablish a data transmission channel.
The present disclosure provides an automatic switching method for Kafka dual-cluster real-time traffic, as shown in fig. 3, including:
in step S301, a consuming client creates a data subscription instance and connects to a primary cluster.
Step S302, a consumption client calls an interface of a data bus cluster management and control platform to acquire a target subscription address corresponding to each group of traffic of a data subscription instance subscription.
Step S303, judging whether the target subscription address corresponding to each group of flow is consistent with the address of the main cluster.
Step S304, in response to the fact that all the target subscription addresses are inconsistent with the addresses of the main clusters and consistent with the addresses of the standby clusters, the consumption client switches to be connected with the standby clusters, and subscribes each group of traffic from the standby clusters.
In step S305, in response to the at least one target subscription address being consistent with the address of the primary cluster, no handover is performed, and each group of traffic is continuously subscribed to the primary cluster through the data subscription instance.
As shown in fig. 4, the Consumer client 208 uses the pre-packaged SDK, when creating a Consumer, invokes the interface of the data bus cluster management platform 202 to obtain the addresses of the primary cluster 203 and the backup cluster 204, and then creates a Consumer object (i.e. the data subscription instance 209) according to the returned primary and backup cluster addresses, so as to connect the primary cluster. The consuming client 208 obtains the cluster address of the currently published Topic traffic connection by invoking an interface of the data bus cluster management platform 202. Whenever the poll method is executed (the poll method returns a group of messages of the subscribed subject or partition, if no message exists in the subscribed subject or partition, an empty message set is returned), a check is made on the local cached Topic connection address and the current cluster address, and if multiple topics are consumed simultaneously, one or more of the target subscription addresses of the topics are inconsistent with the current running cluster address, but not all of the topics are inconsistent with the current running cluster address, the switch is temporarily not made. And if all the target subscription addresses of the Topic are inconsistent with the current running cluster address, cluster switching is performed.
As an optional implementation manner, the consuming client calls an interface of the data bus cluster management and control platform, and the obtaining the target subscription address corresponding to each group of traffic subscribed by the data subscription instance includes: as shown in fig. 4, a second timed task thread 210 is created, and based on the second timed task thread 210 calling the interface of the data bus cluster management platform at intervals of a preset time, the data subscription instance 209 subscribes to the target subscription addresses corresponding to each group of traffic. When the consumer client 208 creates the consumer object, a timed task thread, namely a second timed task thread 210, is started at the same time, and the interface of the data bus cluster management platform 202 is called once every a preset time (for example, 10 s), so as to obtain the cluster address of the current issued Topic traffic connection, thereby realizing real-time monitoring and switching of the cluster address of the traffic connection.
As an alternative embodiment, before the consumer client 208 switches to connect to the backup cluster 204 and subscribes to each group of traffic from the backup cluster 204, the method further includes: and verifying the consumption accumulation amount of each group of traffic, and switching to be connected with the standby cluster 204 by the consumption client when the acquired traffic of each group reaches the preset times and the consumption accumulation amount is zero. After determining that the cluster needs to be switched, verifying the stacking amount consumed by the Topic traffic, and if the continuous poll acquires data 10 times and the stacking amount is 0, starting to switch the cluster.
As an alternative embodiment, switching the consuming client to connect to the standby cluster comprises: the consumption client closes the current data subscription instance; re-acquiring the address of the main cluster and the address of the standby cluster; creating a new data subscription instance; the backup cluster is connected based on the new data subscription instance and groups of traffic are subscribed from the backup cluster. In the present embodiment, in the process of switching clusters, the original consumer object (data subscription instance 209) is closed, the cluster address is obtained again, and a new consumer object is created.
Through the above technical solution, as shown in fig. 5, both the production client 201 and the consumption client 208 can determine whether the currently running main cluster 203 is abnormal by acquiring the cluster address connected with the Topic traffic in real time, and quickly and automatically switch to the standby cluster 204 when the main cluster is abnormal, so that the switching time can be reduced, and the stability of the clusters can be ensured.
The present disclosure provides an automatic switching device 600 for Kafka dual-cluster real-time traffic, as shown in fig. 6, including:
the first creating module 601 is configured to create a first data distribution instance and a second data distribution instance, and connect the primary cluster and the backup cluster, respectively.
The first obtaining module 602 is configured to call an interface of the data bus cluster management and control platform, and obtain a target release address corresponding to each group of traffic released by the first data release example.
The first determining module 603 is configured to determine whether the target issue address corresponding to each group of traffic is consistent with the address of the primary cluster.
The first execution module 604 is configured to switch to the second data publishing instance to publish each set of traffic to the standby cluster in response to all the target publishing addresses not being consistent with the addresses of the primary cluster and with the addresses of the standby cluster.
The first execution module 604, in response to at least one target publishing address being consistent with the address of the primary cluster, does not perform a switch, i.e., does not perform a switch when all target publishing addresses are consistent with the address of the primary cluster or a portion of the target publishing addresses are consistent with the address of the primary cluster, and continues to publish each set of traffic to the primary cluster through the first data publishing instance.
As shown in fig. 2, the first creation module 601, the first acquisition module 602, the first judgment module 603, and the first execution module 604 may be provided in the production client. The first creating module 601 uses the encapsulated SDK, when creating the Producer object, the first obtaining module 602 obtains the addresses of the main cluster 203 and the standby cluster 204 by calling the interface of the cluster management and control platform 202, and then the first creating module 601 creates two objects, namely, the data publishing instance 1 and the data publishing instance 2, according to the returned main and standby cluster addresses, and connects the main cluster 203 and the standby cluster 204 respectively. The production client 201 obtains a target release address corresponding to each group of Topic level traffic currently released by the production client 201, namely a connected cluster address (a main cluster or a standby cluster) by calling an interface of the primary cluster management platform 202. When data is sent, the first judging module 603 compares the cluster address of the Topic flow cache of the sent data with the address of the main and standby clusters, if the target release address of the Topic flow is the same as the main cluster address, the first executing module 604 sends the data by using the object (data release example 1) of the main cluster 203; if the target release address of the Topic traffic is the same as that of the backup cluster 204, the first execution module 604 sends data using the object of the backup cluster 204 (data release instance 2), and if the target release address is not the same as that of both the primary and backup clusters, the two Producer objects are recreated and the primary and backup cluster addresses are reacquired. Through the technical scheme, when the main cluster is abnormal, the flow issued by the production client can be timely switched to the standby cluster, so that the switching time is reduced, and the stability of the cluster is ensured.
As an alternative embodiment, the data bus cluster management platform is configured to implement flow control on the Topic level, and may control whether each Topic of each cluster is connected to a primary cluster or a backup cluster. The user can enter the cluster switching management page through logging in the display module of the data bus cluster management platform. Environmental information can be newly added to the page, the environmental information is modified, a flow control rule is created, and the publishing and subscribing flow of Topic in the environment is controlled. At the same time, the list of topics can be clicked and refreshed, and all topics of the main cluster in the environment can be obtained by one key. The list of cluster groups may be clicked to refresh and all groups of the master cluster in the environment may be retrieved by one-touch.
The present disclosure also provides an automatic switching device 700 for Kafka dual-cluster real-time traffic, as shown in fig. 7, including:
a second creation module 701 configured to create a data subscription instance and connect the primary cluster.
The second obtaining module 702 is configured to call an interface of the data bus cluster management platform, and obtain a target subscription address corresponding to each group of traffic subscribed by the data subscription instance.
A second determining module 703 is configured to determine whether the target subscription address corresponding to each group of traffic is consistent with the address of the primary cluster.
A second execution module 704 is configured to subscribe to groups of traffic from the backup cluster in response to all target subscription addresses not coinciding with the addresses of the primary cluster and coinciding with the addresses of the backup cluster, the consuming client switching to connect to the backup cluster.
The second execution module 704 continues to subscribe groups of traffic to the primary cluster through the data subscription instance without performing a handoff in response to the at least one target subscription address coinciding with the address of the primary cluster.
As shown in fig. 4, the second creation module 701, the second acquisition module 702, the second judgment module 703, and the second execution module 704 may be provided at the consumption client 208. The Consumer client 208 uses the pre-packaged SDK, and when creating a Consumer, it invokes the interface of the data bus cluster management platform 202 to obtain the addresses of the main cluster 203 and the standby cluster 204, and then creates a Consumer object (i.e. a data subscription instance 209) according to the returned main standby cluster address, so as to connect the main cluster. The consuming client 208 obtains the cluster address of the currently published Topic traffic connection by invoking an interface of the data bus cluster management platform 202. Whenever the poll method is executed (the poll method returns a group of messages of the subscribed subject or partition, if no message exists in the subscribed subject or partition, an empty message set is returned), a check is made on the local cached Topic connection address and the current cluster address, and if multiple topics are consumed simultaneously, one or more of the target subscription addresses of the topics are inconsistent with the current running cluster address, but not all of the topics are inconsistent with the current running cluster address, the switch is temporarily not made. And if all the target subscription addresses of the Topic are inconsistent with the current running cluster address, cluster switching is performed.
According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
In particular, electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
The apparatus includes a computing unit that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) or a computer program loaded from a storage unit into a Random Access Memory (RAM). In the RAM, various programs and data required for the operation of the device may also be stored. The computing unit, ROM and RAM are connected to each other by a bus. An input/output (I/O) interface is also connected to the bus.
A plurality of components in a device are connected to an I/O interface, comprising: an input unit such as a keyboard, a mouse, etc.; an output unit such as various types of displays, speakers, and the like; a storage unit such as a magnetic disk, an optical disk, or the like; and communication units such as network cards, modems, wireless communication transceivers, and the like. The communication unit allows the device to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The computing unit may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing units include, but are not limited to, central Processing Units (CPUs), graphics Processing Units (GPUs), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processors, controllers, microcontrollers, and the like. The computing unit performs the respective methods and processes described above, such as the automatic switching method in the above-described embodiments. For example, in some embodiments, the automatic switching method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device via the ROM and/or the communication unit. One or more steps of the automatic switching method described above may be performed when the computer program is loaded into RAM and executed by a computing unit. Alternatively, in other embodiments, the computing unit may be configured to perform the automatic switching method by any other suitable means (e.g. by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out the automatic switching methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially, or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.
The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.
Claims (13)
1. The automatic switching method of the Kafka double-cluster real-time traffic is characterized by comprising the following steps of:
the method comprises the steps that a production client creates a first data release instance and a second data release instance and is respectively connected with a main cluster and a standby cluster;
the production client calls an interface of a data bus cluster management and control platform to acquire a target release address corresponding to each group of traffic released by the first data release example;
judging whether the target release address corresponding to each group of flow is consistent with the address of the main cluster;
responding to the fact that all the target release addresses are inconsistent with the addresses of the main clusters and consistent with the addresses of the standby clusters, switching to the second data release instance to release all the groups of traffic to the standby clusters;
and responding to the fact that at least one target release address is consistent with the address of the main cluster, not executing switching, and continuing to release the groups of traffic to the main cluster through the first data release instance.
2. The method of claim 1, wherein the production client invoking an interface of a data bus cluster management platform, the obtaining a target publication address corresponding to each set of traffic for the first data publication example comprises:
and creating a first timing task thread, calling an interface of the data bus cluster management and control platform at intervals of a preset time based on the first timing task thread, and acquiring the target release addresses corresponding to the groups of traffic.
3. The method of claim 1, further comprising: and in response to at least one of the target published addresses not being consistent with the address of the main cluster and not being consistent with the address of the standby cluster, recreating the first data published instance and the second data published instance, and reacquiring the address of the main cluster and the address of the standby cluster.
4. The method of claim 1, wherein the production client creating a first data distribution instance and a second data distribution instance and connecting a primary cluster and a backup cluster, respectively, comprises:
calling an interface of the data bus cluster management and control platform to acquire the address of the main cluster and the address of the standby cluster;
the first data publishing instance establishes a communication connection with the primary cluster based on the address of the primary cluster, and the second data publishing instance establishes a communication connection with the backup cluster based on the address of the backup cluster.
5. The automatic switching method of the Kafka double-cluster real-time traffic is characterized by comprising the following steps of:
the consumption client creates a data subscription instance and is connected with the main cluster;
the consumption client calls an interface of a data bus cluster management and control platform to acquire a target subscription address corresponding to each group of traffic subscribed by the data subscription instance;
judging whether the target subscription address corresponding to each group of the traffic is consistent with the address of the main cluster;
responding to the fact that all the target subscription addresses are inconsistent with the addresses of the main clusters and consistent with the addresses of the standby clusters, switching the consumption client to be connected with the standby clusters, and subscribing the groups of traffic from the standby clusters;
and responding to the fact that at least one target subscription address is consistent with the address of the main cluster, not executing switching, and continuing to subscribe the groups of traffic to the main cluster through the data subscription instance.
6. The method of claim 5, wherein the consuming client invoking an interface of a data bus cluster management platform, the obtaining a target subscription address corresponding to each group of traffic subscribed by the data subscription instance comprises:
creating a second timed task thread, calling an interface of the data bus cluster management and control platform at intervals of a preset time based on the second timed task thread, and acquiring the target subscription addresses corresponding to the data subscription examples subscribing the groups of traffic.
7. The method of claim 5, wherein the consumer client switches to connect to a backup cluster, and further comprising, prior to subscribing to the groups of traffic from the backup cluster:
and verifying consumption accumulation amounts of the flows of each group, and switching to be connected with the standby cluster by the consumption client when the flows of each group are obtained to reach the preset times and the consumption accumulation amounts are zero.
8. The method of any of claims 5-7, wherein the consumer client switching to connect to the standby cluster comprises:
the consumption client closes the current data subscription instance;
re-acquiring the address of the main cluster and the address of the standby cluster;
creating a new data subscription instance;
and connecting the standby cluster based on the new data subscription instance, and subscribing the groups of traffic from the standby cluster.
9. An automatic switching device of Kafka double-cluster real-time flow, which is characterized by comprising:
the first creating module is configured to create a first data release instance and a second data release instance and is respectively connected with the main cluster and the standby cluster;
the first acquisition module is configured to call an interface of the data bus cluster management and control platform to acquire a target release address corresponding to each group of traffic released by the first data release example;
the first judging module is configured to judge whether the target release address corresponding to each group of traffic is consistent with the address of the main cluster;
the first execution module is configured to respond to the fact that all the target release addresses are inconsistent with the addresses of the main clusters and consistent with the addresses of the standby clusters, and switch to the second data release instance to release the groups of traffic to the standby clusters;
and the first execution module responds to the fact that at least one target release address is consistent with the address of the main cluster, does not execute switching, and continues to release the flows of each group to the main cluster through the first data release example.
10. An automatic switching device of Kafka double-cluster real-time flow, which is characterized by comprising:
the second creation module is configured to create a data subscription instance and is connected with the main cluster;
the second acquisition module is configured to call an interface of the data bus cluster management and control platform, and acquire a target subscription address corresponding to each group of traffic subscribed by the data subscription instance;
the second judging module is configured to judge whether the target subscription address corresponding to each group of traffic is consistent with the address of the main cluster;
the second execution module is configured to respond to the fact that all the target subscription addresses are inconsistent with the addresses of the main clusters and consistent with the addresses of the standby clusters, the consumption client switches to be connected with the standby clusters, and subscribes to the groups of traffic from the standby clusters;
and the second execution module does not execute switching and continues to subscribe the groups of traffic to the main cluster through the data subscription instance in response to the fact that at least one target subscription address is consistent with the address of the main cluster.
11. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.
12. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-8.
13. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310323573.7A CN116346578A (en) | 2023-03-29 | 2023-03-29 | Kafka double-cluster real-time flow automatic switching method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310323573.7A CN116346578A (en) | 2023-03-29 | 2023-03-29 | Kafka double-cluster real-time flow automatic switching method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116346578A true CN116346578A (en) | 2023-06-27 |
Family
ID=86892713
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310323573.7A Pending CN116346578A (en) | 2023-03-29 | 2023-03-29 | Kafka double-cluster real-time flow automatic switching method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116346578A (en) |
-
2023
- 2023-03-29 CN CN202310323573.7A patent/CN116346578A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104408071A (en) | Distributive database high-availability method and system based on cluster manager | |
| CN109245908B (en) | Method and device for switching master cluster and slave cluster | |
| CN107666415B (en) | Optimization method and device of FC-AE-1553 protocol bridge | |
| CN112181683A (en) | Concurrent consumption method and device for message middleware | |
| CN112527567A (en) | System disaster tolerance method, device, equipment and storage medium | |
| CN111865632A (en) | Switching method of distributed data storage cluster and switching instruction sending method and device | |
| CN111418187A (en) | Scalable statistics and analysis mechanism in cloud networks | |
| CN114448989B (en) | Method, device, electronic equipment, storage medium and product for adjusting message distribution | |
| CN116016276A (en) | Method and device for detecting health state of server | |
| CN112965847B (en) | Fault processing method, device, equipment and storage medium of micro-service architecture | |
| CN112817992B (en) | Method, apparatus, electronic device and readable storage medium for executing change task | |
| US9703646B2 (en) | Centralized database system | |
| CN116346578A (en) | Kafka double-cluster real-time flow automatic switching method and device and electronic equipment | |
| JP7307766B2 (en) | Traffic adjustment method, apparatus, electronic equipment, computer readable recording medium and computer program | |
| CN115934304A (en) | Data processing method and device, computer equipment and readable storage medium | |
| CN113747423A (en) | Cloud mobile phone state synchronization method, device, equipment, storage medium and program product | |
| CN111858260A (en) | Information display method, device, equipment and medium | |
| CN107644035B (en) | Database system and deployment method thereof | |
| CN113297002B (en) | Database working mode switching method and device | |
| CN116938826B (en) | Network speed limiting method, device and system and electronic equipment | |
| CN115629918B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN114567540B (en) | Master-slave node switching method, device, equipment, medium and program product | |
| CN112596922B (en) | Communication management method, device, equipment and medium | |
| CN110519393B (en) | Self-service equipment supervision method, device, equipment, server and medium | |
| CN116800592A (en) | Multi-purpose multi-standby control method, device, equipment and medium based on Ethernet interface |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |