CN117834421A - Cluster configuration management method and system based on RocketMQ - Google Patents
Cluster configuration management method and system based on RocketMQ Download PDFInfo
- Publication number
- CN117834421A CN117834421A CN202311691061.2A CN202311691061A CN117834421A CN 117834421 A CN117834421 A CN 117834421A CN 202311691061 A CN202311691061 A CN 202311691061A CN 117834421 A CN117834421 A CN 117834421A
- Authority
- CN
- China
- Prior art keywords
- node
- namesrv
- configuration
- cluster
- rocketmq
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title abstract description 49
- 238000000034 method Methods 0.000 claims abstract description 19
- 238000012795 verification Methods 0.000 claims description 4
- 230000000977 initiatory effect Effects 0.000 claims 1
- 230000002159 abnormal effect Effects 0.000 abstract description 7
- 238000012423 maintenance Methods 0.000 abstract description 7
- 230000000694 effects Effects 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 238000012986 modification Methods 0.000 description 5
- 230000001360 synchronised effect Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001902 propagating effect Effects 0.000 description 2
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000012508 change request Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 229910052802 copper Inorganic materials 0.000 description 1
- 239000010949 copper Substances 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 229910052751 metal Inorganic materials 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0806—Configuration setting for initial configuration or provisioning, e.g. plug-and-play
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
- G06F9/526—Mutual exclusion algorithms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
- H04L41/0266—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using meta-data, objects or commands for formatting management information, e.g. using eXtensible markup language [XML]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/04—Network management architectures or arrangements
- H04L41/042—Network management architectures or arrangements comprising distributed management centres cooperatively managing the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0813—Configuration setting characterised by the conditions triggering a change of settings
- H04L41/082—Configuration setting characterised by the conditions triggering a change of settings the condition being updates or upgrades of network functionality
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0823—Configuration setting characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability
- H04L41/0836—Configuration setting characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability to enhance reliability, e.g. reduce downtime
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/30—Profiles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/562—Brokering proxy services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention belongs to the technical field of middleware, and particularly relates to a cluster configuration management method and system based on RocketMQ, wherein a management side is connected with NameSrv nodes through interfaces, and configuration information is stored in the NameSrv nodes; sending a PUT KV request to a NameSrv node through a management side; starting a brooker node according to the PUT KV request to acquire configuration information from NameSrv nodes, and managing the configuration of all the brooker nodes of the cluster; the method provides a unified view angle for cluster configuration management, reduces the difficulty and cost of operation and maintenance, improves the stability of the RocketMQ cluster, and avoids abnormal problems caused by inconsistent metadata. The user does not perceive the effect of realizing the whole cluster configuration change; reliability and flexibility, multiple copies of configuration information are stored, and multiple storage systems are supported; automatic maintenance of a large number of node configurations is supported, manual participation is not needed, operation and maintenance management difficulty is reduced, and automatic synchronization of configuration for dynamically increasing the number of nodes is supported.
Description
Technical Field
The invention belongs to the technical field of middleware, and particularly relates to a cluster configuration management method and system based on RocketMQ.
Background
Apache RocketMQ is a low latency, high concurrency, high availability, high reliability distributed message. The method is widely adopted by a plurality of enterprise developers and cloud manufacturers due to the characteristics of simple architecture, rich service functions, extremely strong expandability and the like
The deployment architecture of the Apache RocketMQ is simple, and each process is independently deployed and independent. The RocketMQ Broker is responsible for the storage and computation of messages. RocketMQ NameSrv is a simple Topic routing registry supporting dynamic registration and discovery of Topic and Broker. NameSrv typically has multiple instances deployed, with no information communication between each instance.
The configuration of each node of the RocketMQ is independent of a local configuration file, and an independent configuration modification interface is arranged, so that the consistency of metadata of each node of the cluster cannot be ensured, more abnormal scenes can be caused, and the normal operation and use of the cluster are affected. Each node configuration in the cluster is independently stored and updated, and partial node update failures exist, so that cluster metadata are inconsistent, and unpredictable cluster abnormality occurs.
Disclosure of Invention
The invention aims to provide a cluster configuration management method and system based on a RocketMQ, which can improve the stability of the RocketMQ cluster, avoid abnormal problems caused by inconsistent metadata and solve the problems in the background technology.
In order to achieve the above purpose, the invention adopts the following technical scheme: a cluster configuration management method based on RocketMQ comprises the following steps: establishing connection between a management side and a NameSrv node through an interface, and storing configuration information on the NameSrv node; sending a PUT KV request to the NameSrv node through the management side; and starting a brooker node according to the PUT KV request to acquire the configuration information from the NameSrv node, and managing the configuration of all the brooker nodes of the cluster.
Preferably, the management side uses the existing interface of the NameSrv to Store the configuration information in the NameSrv attribute, and the NameSrv node synchronizes to the entire NameSrv cluster through the distributed KV Store module.
Preferably, the interface of the KV Store module comprises KEYS, DELETE and LOCK; the KEYS and DELETE only clean data when the managed clusters are cleaned; the LOCK includes GET and PUT.
Preferably, the method includes actively querying version information on the brooker node when the starting brooker node acquires the configuration information from the NameSrv node.
Preferably, the querying version information on the brooker node includes:
if version information is not found, locking the current node, synchronizing the full configuration information to the node through the existing interface of the open source, and updating the version information of the node after the operation is finished;
if version information is found and is lower than the latest version information, replaying the incremental operation steps to the target node once for the theme and the subscription group, updating the full quantity once for other configurations, and updating the version information of the node after the operation is finished;
if the node version information is found to be greater than or equal to the latest version information, ending the operation.
Preferably, the externally exposed interface of the KV Store module receives a PUT request and a GET request.
Preferably, the KV Store module is connected to an ETCD type KV system or an object storage system.
Preferably, the KV value of the KV Store module is set according to the configuration information of the brooker node.
Preferably, the header of the PUT request is provided with a security verification field.
On the other hand, the invention provides a cluster configuration management method based on RocketMQ, which comprises the following steps:
the storage module is used for establishing connection between the management side and the NameSrv node through an interface and storing configuration information on the NameSrv node;
the request sending module is used for sending a PUT KV request to the NameSrv node through the management side;
and the configuration management module is used for starting the brooker node to acquire the configuration information from the NameSrv node according to the PUT KV request and managing the configuration of all the brooker nodes of the cluster.
The invention has the technical effects and advantages that: compared with the prior art, the cluster configuration management method and system based on the RocketMQ have the following advantages:
the invention establishes connection between the management side and NameSrv node through the interface, and stores configuration information on the NameSrv node; sending a PUT KV request to a NameSrv node through a management side; starting a brooker node according to the PUT KV request to acquire configuration information from NameSrv nodes, and managing the configuration of all the brooker nodes of the cluster; the method provides a unified view angle for cluster configuration management, reduces the difficulty and cost of operation and maintenance, improves the stability of the RocketMQ cluster, and avoids abnormal problems caused by inconsistent metadata.
Drawings
FIG. 1 is a block diagram of a method for managing configuration of a RocketMQ-based cluster in accordance with the present invention;
FIG. 2 is a schematic diagram of an externally exposed interface structure of the KV Store module of the present invention;
FIG. 3 is a schematic diagram of a KV Store module of the present invention with various implementations based on a scene;
FIG. 4 is a schematic diagram of a RAFT-based KV Store implementation of the present invention;
FIG. 5 is a graph showing the KV value setting according to the present invention;
FIG. 6 is a flow chart of a management configuration of the present invention;
FIG. 7 is a flow chart of a method for managing cluster configuration based on RocketMQ in accordance with the present invention;
fig. 8 is a schematic structural diagram of a cluster configuration management system based on a dockmq of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. The specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention provides a cluster configuration management method based on RocketMQ as shown in FIG. 7, which comprises the following steps:
step one: establishing connection between a management side and a NameSrv node through an interface, and storing configuration information on the NameSrv node;
specifically, the management side uses the existing interface of NameSrv to Store the configuration information in the Namespace attribute, and the NameSrv node synchronizes to the whole NameSrv cluster through the distributed KV Store module.
Further, the interface of the KV Store module comprises KEYS, DELETE and LOCK; the KEYS and DELETE only clean data when the managed clusters are cleaned; the LOCK includes GET and PUT.
Step two: sending a PUT KV request to the NameSrv node through the management side;
step three: and starting a brooker node according to the PUT KV request to acquire the configuration information from the NameSrv node, and managing the configuration of all the brooker nodes of the cluster.
Specifically, when the starting browser node acquires the configuration information from the NameSrv node, the version information on the browser node is actively inquired.
The querying version information on the brooker node comprises the following steps:
if version information is not found, locking the current node, synchronizing the full configuration information to the node through the existing interface of the open source, and updating the version information of the node after the operation is finished;
if version information is found and is lower than the latest version information, replaying the incremental operation steps to the target node once for the theme and the subscription group, updating the full quantity once for other configurations, and updating the version information of the node after the operation is finished;
if the node version information is found to be greater than or equal to the latest version information, ending the operation.
And the externally exposed interface of the KV Store module receives the PUT request and the GET request, and a message header of the PUT request is provided with a security verification field. The KV Store module is connected into an ETCD type KV system or an object storage system. The KV value of the KV Store module is set according to the configuration information of the brooker node.
Establishing connection between a management side and a NameSrv node through an interface, and storing configuration information on the NameSrv node; sending a PUT KV request to a NameSrv node through a management side; starting a brooker node according to the PUT KV request to acquire configuration information from NameSrv nodes, and managing the configuration of all the brooker nodes of the cluster; the method provides a unified view angle for cluster configuration management, reduces the difficulty and cost of operation and maintenance, improves the stability of the RocketMQ cluster, and avoids abnormal problems caused by inconsistent metadata.
On the other hand, the present invention proposes a method for cluster configuration management based on a RocketMQ, as shown in FIG. 8, including: the device comprises a storage module, a request sending module and a configuration management module.
Specifically, the storage module is used for establishing connection between the management side and the NameSrv node through an interface, and storing configuration information on the NameSrv node;
specifically, the request sending module is used for sending a PUT KV request to the NameSrv node through the management side;
specifically, the configuration management module is configured to start the brooker node according to the PUT KV request to obtain the configuration information from the NameSrv node, and manage the configuration of all the brooker nodes of the cluster.
In addition, when the storage module, the request sending module and the configuration management module are executed, other functions of the cluster configuration management method based on the RocketMQ can be realized. Specifically, the following is described.
The functions of a cluster configuration center are realized by utilizing distributed KV storage at a NameSrv node layer without changing a cluster deployment architecture and completely compatible with open source ecology conditions, and the capabilities of automatic synchronous configuration information, dynamic node addition and the like are supported.
The management side uses the existing interface of NameSrv to store the configuration information under a specific Namespace attribute. The NameSrv nodes are synchronized to the whole NameSrv cluster through distributed KV storage, so that consistency and high availability of configuration information are ensured. And maintaining configuration information of all the brooker nodes of the cluster by using a registration mechanism of the open source brooker, so as to ensure the consistency of the information.
By uniformly placing configuration information on the NameSrv node. And then, managing all the brooker node configurations of the cluster through a mechanism that the brooker node can register on the NameSrv node when being started, and ensuring that the metadata information of the cluster is kept consistent.
The cluster configuration management provides a unified view angle, and reduces the difficulty and cost of operation and maintenance. The stability of the RocketMQ cluster is improved, and abnormal problems caused by inconsistent metadata are avoided. Cloud protogenesis, supporting automatic telescoping and expansion. The cost is low, the simplified storage format is adopted, and the requirement on the storage space is very low.
As shown in fig. 3, the distributed KV Store module is configured to Store configuration information of the cluster. The KV Store module itself needs to guarantee high availability and consistency of distributed clusters.
Only PUT and GET requests are exposed to the outside to access the KV Store module. Because the KV Store module has clear functions and boundaries, different implementations can be adopted based on different scenes.
One or more KV Store implementations may be configured. When multiple implementations are performed, all the implementations are written and read according to the configuration sequence. The multi-implementation scenario can be used to enhance data security as well as seamless migration from environment to environment.
As shown in FIG. 4, based on a conventional bare metal deployment, the machine resources are substantially unchanged. The synchronization of KV values and the election function can be realized based on RAFT. The data is compressed and stored on a local store. Based on the cloud primary or container scene, KV storage can be accessed to a KV system such as ETCD or an object storage system. Therefore, the NameSrv node does not store any state and can be expanded at will.
As shown in fig. 5, the KV value design of the KV Store is mainly determined based on the configuration information required by the Broker.
The capability of managing a plurality of RocketMQ clusters simultaneously is supported, and configuration at the cluster level and configuration information management at the node level are supported. Priority level: node > cluster. Having version information can prevent the problem of ABA. For part of the configuration information only the latest value needs to be read. This part only needs to maintain a full set of the latest values and current versions.
The following examples:
fileReservedTime＝168
DeleteWhen＝04
for partial configuration information, there is an ABA problem. Such as a topic or subscription group configuration. Then a version-based delta content needs to be maintained as follows:
[
{"topicName":"test","state":"create","version":"1"},
{"topicName":"test","state":"delete","version":"2"},
{"topicName":"test","state":"create","version":"3"}
]。
configuration management of the entire RocektMQ cluster provides a unified management portal without fear of abnormal problems due to configuration inconsistencies.
The management side directly interfaces with NameSrv without each node accessing once to obtain complete cluster information. The management side sends a PUT KV request to NameSrv, and only needs to write the PUT KV request into the distributed KV Store completely.
The Broker node will register information on the NameSrv just started and periodically. At this time, the NameSrv may be triggered to perform metadata version checking on the node, and ensure that the elements of the node remain consistent with the version of the cluster. The update is based on version, so the entire update operation is idempotent. The metadata of the clusters may also be automatically synchronized for newly added/abnormally restored nodes.
As shown in fig. 6, a configuration management module is extended over Apache RocektMQ.
The definition of interfaces for KV Store is increased. Only GET, PUT, KEYS, DELETE and LOCK. The KEYS and DELETE interfaces are used for cleaning data, and are only used when the managed clusters are cleaned, so that the core functions are not affected. The LOCK interface is formed by combining GET and PUT interfaces, and needs to realize the combined operation of GET and PUT of a set of atoms, and can also be realized by utilizing the characteristic of external storage.
And adding KV Store. And the realization of external systems such as the docking ETCD and the object storage is realized by only calling a corresponding interface. For the implementation of local storage, a KV cached data format is implemented. The KEY values are all cached in the memory, and the data type of TreeMap is adopted. The VALUE assumes a latency loaded memory object. Specific values are placed on local storage at ordinary times and need to be temporarily loaded into memory. And if the memory is not used for a certain time, releasing the memory. The KEY/VALUE storage adopts a compressed JSON file mode. Not all content is read for loading into memory during use. And splitting the KV content into a plurality of files for storage according to the KEY value. Based on the local KV storage, a RAFT synchronization protocol is added, so that the operation on the local KV storage is synchronized to all nodes. And realizing RAFT-based distributed KV Store realization.
And expanding the existing GET/PUT KV interface of the open source NAmeserver, and transferring to the corresponding KV STORE interface when the special meta-manager is detected correspondingly. This is an interface for externally exposed operation and maintenance management. For the PUT interface, for security reasons, an additional security verification field needs to be added to the header. The VALUE of the PUT KV interface is not the configuration VALUE itself, but is actually an operational data structure as follows: { "action": "create_tpoic", "params": { "topicName": "test" }.
Preventing the problem of concurrent execution requires locking the target KEY value. This required KV STORE storage provides this capability. A common implementation is a mechanism implementation using TTL or temporary nodes.
When configuration is modified, LOCK interface locking needs to be called first. Followed by a logical modification of the target KEY VALUE. For the configuration information of the brooker configuration file and the role control, only the latest full-quantity value needs to be maintained. For information such as topics and subscription groups, incremental content information needs to be saved. After the modification, version information of the cluster also needs to be updated.
The Nameserver can actively inquire version information on the browser node when receiving a configuration change request or a registration request of the browser node.
If no version information is found, a lock is required for this node. The full amount of configuration information is synchronized to this node through the existing interface of the open source. After the operation is finished, the version information of the node is updated.
If version information is found, and is lower than the latest version information: the incremental steps of operation are replayed once to the target node for the topic and subscription group. The full update is made once for other configurations. After the operation is finished, the version information of the node is updated.
If the node version information is found to be greater than or equal to the latest version information, the operation is ended.
The problem that a large number of Rocker MQ nodes are inconvenient to manage and configure in actual operation is solved. And simultaneously, the capability of synchronizing configuration information under the scene of dynamically expanding the nodes is also provided. Providing basic capabilities for other extended functions: metadata management of the cloud primary scene; realize distributed lock of cluster level, and provide distributed transaction message across nodes, etc.
In addition, the embodiment of the invention also provides a terminal device, and the cluster configuration management method based on the RocketMQ is mainly applied to the terminal device, wherein the terminal device can be a PC, a portable computer, a mobile terminal and other devices with display and processing functions.
In particular, the terminal device may include a processor (e.g., CPU), a communication bus, a user interface, a network interface, and a memory. Wherein the communication bus is used for realizing connection communication among the components; the user interface may include a Display screen (Display), an input unit such as a Keyboard (Keyboard); the network interface may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface); the memory may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory, or alternatively may be a storage device independent of the aforementioned processor.
The storage is used for storing a readable storage medium, the readable storage medium is used for storing a cluster configuration management program, and the processor can call the cluster configuration management program stored in the storage and execute the cluster configuration management method based on the RocketMQ.
It will be appreciated that the readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.
The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.
Computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, c++ or the like and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present disclosure are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information of computer readable program instructions, which can execute the computer readable program instructions.
Finally, it should be noted that: the foregoing description is only illustrative of the preferred embodiments of the present invention, and although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that modifications may be made to the embodiments described, or equivalents may be substituted for elements thereof, and any modifications, equivalents, improvements or changes may be made without departing from the spirit and principles of the present invention.
Claims (10)
1. A method for managing cluster configuration based on a dockmq, comprising:
establishing connection between a management side and a NameSrv node through an interface, and storing configuration information on the NameSrv node;
sending a PUT KV request to the NameSrv node through the management side;
and starting a brooker node according to the PUT KV request to acquire the configuration information from the NameSrv node, and managing the configuration of all the brooker nodes of the cluster.
2. The method for managing cluster configuration based on RocketMQ according to claim 1, wherein the management side uses an existing interface of NameSrv to Store configuration information in a Namespace attribute, and the NameSrv node synchronizes to the whole NameSrv cluster through a distributed KV Store module.
3. The method for managing cluster configuration based on RocketMQ of claim 2, wherein the interface of the KV Store module comprises KEYS, DELETE and LOCK;
the KEYS and DELETE only clean data when the managed clusters are cleaned;
the LOCK includes GET and PUT.
4. The method for managing cluster configuration based on RocketMQ according to claim 1, wherein the initiating browser node actively queries version information on the browser node when obtaining the configuration information from a NameSrv node.
5. The method for managing cluster configuration based on a RocketMQ according to claim 4, wherein the querying version information on the broker node comprises:
if version information is not found, locking the current node, synchronizing the full configuration information to the node through the existing interface of the open source, and updating the version information of the node after the operation is finished;
if version information is found and is lower than the latest version information, replaying the incremental operation steps to the target node once for the theme and the subscription group, updating the full quantity once for other configurations, and updating the version information of the node after the operation is finished;
if the node version information is found to be greater than or equal to the latest version information, ending the operation.
6. The method for managing cluster configuration based on the RocketMQ according to claim 2, wherein an externally exposed interface of the KV Store module receives a PUT request and a GET request.
7. The method for managing cluster configuration based on RocketMQ according to claim 6, wherein the KV Store module is connected to an ETCD type KV system or an object storage system.
8. The method for managing cluster configuration based on RocketMQ according to claim 7, wherein the KV value of the KV Store module is set according to configuration information of a brooker node.
9. The method for managing cluster configuration based on RocketMQ as recited in claim 6, wherein the header of the PUT request is provided with a security verification field.
10. A method for managing cluster configuration based on a dockmq, comprising:
the storage module is used for establishing connection between the management side and the NameSrv node through an interface and storing configuration information on the NameSrv node;
the request sending module is used for sending a PUT KV request to the NameSrv node through the management side;
and the configuration management module is used for starting the brooker node to acquire the configuration information from the NameSrv node according to the PUT KV request and managing the configuration of all the brooker nodes of the cluster.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311691061.2A CN117834421A (en) | 2023-12-11 | 2023-12-11 | Cluster configuration management method and system based on RocketMQ |
| PCT/CN2023/143253 WO2024187918A1 (en) | 2023-12-11 | 2023-12-29 | Cluster configuration management method and system based on rocketmq |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311691061.2A CN117834421A (en) | 2023-12-11 | 2023-12-11 | Cluster configuration management method and system based on RocketMQ |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117834421A true CN117834421A (en) | 2024-04-05 |
Family
ID=90518079
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311691061.2A Pending CN117834421A (en) | 2023-12-11 | 2023-12-11 | Cluster configuration management method and system based on RocketMQ |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN117834421A (en) |
| WO (1) | WO2024187918A1 (en) |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110795503A (en) * | 2019-10-18 | 2020-02-14 | 北京达佳互联信息技术有限公司 | Multi-cluster data synchronization method and related device of distributed storage system |
| US11861405B2 (en) * | 2020-04-29 | 2024-01-02 | Kyndryl, Inc. | Multi-cluster container orchestration |
| CN112612545A (en) * | 2020-12-23 | 2021-04-06 | 北京浪潮数据技术有限公司 | Configuration hot loading system, method, equipment and medium of server cluster |
| CN115509673A (en) * | 2021-06-07 | 2022-12-23 | 中移动信息技术有限公司 | IPv4/IPv6 dual-stack support method and device for multi-version container cluster |
-
2023
- 2023-12-11 CN CN202311691061.2A patent/CN117834421A/en active Pending
- 2023-12-29 WO PCT/CN2023/143253 patent/WO2024187918A1/en unknown
Also Published As
| Publication number | Publication date |
|---|---|
| WO2024187918A1 (en) | 2024-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110597910A (en) | Remote data synchronization method, device and system | |
| US9262324B2 (en) | Efficient distributed cache consistency | |
| CN110149382A (en) | Data synchronization method, system, main server, synchronization client and medium | |
| CN104156361B (en) | A kind of method and system for realizing data syn-chronization | |
| US10915555B2 (en) | Systems and methods for adaptive data replication | |
| CN115517009B (en) | Cluster management method, cluster management device, storage medium and electronic equipment | |
| JP2010518490A (en) | Synchronization framework for irregularly connected applications | |
| CN111708738B (en) | Method and system for realizing interaction of hadoop file system hdfs and object storage s3 data | |
| CN113452617B (en) | Dynamic gateway route management method, device and storage medium | |
| CN113656195A (en) | Service message channel management method and device and electronic equipment | |
| CN109347936B (en) | Redis proxy client implementation method, system, storage medium and electronic device | |
| CN111581239A (en) | Cache refreshing method and electronic equipment | |
| CN113079098B (en) | Method, device, equipment and computer readable medium for updating route | |
| CN113050890B (en) | Data migration method and device | |
| US9871863B2 (en) | Managing network attached storage | |
| CN113742376A (en) | Data synchronization method, first server and data synchronization system | |
| CN112000850A (en) | Method, device, system and equipment for data processing | |
| CN115189931A (en) | Distributed key management method, device, equipment and storage medium | |
| CN117834421A (en) | Cluster configuration management method and system based on RocketMQ | |
| US11314599B2 (en) | Method, apparatus and computer program product for data replication | |
| CN110851192B (en) | Method and device for responding to degraded switch configuration | |
| CN113761075A (en) | Method, device, equipment and computer readable medium for switching databases | |
| CN114072768A (en) | Controller for bridging database architectures | |
| US11693578B2 (en) | Method and system for handoff with portable storage devices | |
| CN114531322B (en) | Method, system, device and medium for synchronizing multi-gateway data of Internet of things |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |