CN114363350B - Service management system and method - Google Patents
Service management system and method Download PDFInfo
- Publication number
- CN114363350B CN114363350B CN202111524029.6A CN202111524029A CN114363350B CN 114363350 B CN114363350 B CN 114363350B CN 202111524029 A CN202111524029 A CN 202111524029A CN 114363350 B CN114363350 B CN 114363350B
- Authority
- CN
- China
- Prior art keywords
- node
- master
- service
- master node
- slave
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims description 18
- 238000007726 management method Methods 0.000 claims abstract description 37
- 238000012544 monitoring process Methods 0.000 claims abstract description 19
- 238000012790 confirmation Methods 0.000 claims abstract description 9
- 230000001360 synchronised effect Effects 0.000 claims abstract description 7
- 239000012634 fragment Substances 0.000 claims description 42
- 238000004364 calculation method Methods 0.000 claims description 11
- 230000001419 dependent effect Effects 0.000 claims description 10
- 238000013500 data storage Methods 0.000 claims description 4
- 230000001121 heart beat frequency Effects 0.000 claims description 4
- 238000005067 remediation Methods 0.000 claims description 4
- 230000007958 sleep Effects 0.000 claims description 4
- 238000011084 recovery Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 description 9
- 238000013461 design Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 8
- 239000000178 monomer Substances 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000007251 Prelog reaction Methods 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a service management system and a service management method, wherein the system comprises a server module, a client module and a monitoring alarm module, the server module comprises a plurality of machine groups consisting of a master node and a plurality of slave nodes managed by the master node, a control node is determined in the master node through an election module, and the control node preferentially receives a service request of the client; before the master node in the election module does not receive the voting results from other master nodes, the master node throws the votes to the master node, otherwise, the master node throws the same votes according to the received voting results; the data registry is stored on different master nodes through data slicing, the master nodes send data slicing information to slave nodes through a synchronous module, and when more than half of slave nodes receive the data slicing information and return confirmation characters, the data information can be stored locally; the invention solves the problem that the data table stored by a single node is too huge, has strong read-write performance, improves the stability of the system and ensures the high availability of the system.
Description
Technical Field
The invention relates to a service management system and a method.
Background
In the era of internet, as the business system becomes more and more complex, the monomer system becomes more and more massive, and related codes are continuously piled on the monomer architecture, so that the monomer services need to be split, new problems are caused after the splitting, and the services can be centrally managed, and the services can be perceived as existence of other services, so that a service management system is needed to be integrated to centrally manage the services.
At present, service management systems mainly have poor timeliness in service discovery, and generally can discover a service within seconds or minutes, or sense the offline of a service; if the service discovery time is shortened through configuration, a great network bottleneck overhead is caused, and each server needs to bear a great amount of requests due to the peer-to-peer architecture adopted by the current service management system; meanwhile, the registry is required to be copied between the nodes, the consistency of the data is poor, the final consistency can be achieved, and generally, tens of seconds or even one minute are required to complete the data synchronization.
In addition, at present, a service management system adopts a reverse push mode to realize concrete implementation, each service instance monitors a specific service directory, once service is online or offline, the change of the service registry is pushed to hundreds or thousands of service instances reversely, and if a machine is frequently online or offline, the service management system can cause higher load on a network card when broadcasting information, so that the system stability is deteriorated.
Disclosure of Invention
The invention aims to: the invention aims to provide a high-timeliness and high-availability service management system and method, which can perform centralized discovery and management on services, reduce resource consumption, improve system robustness and eliminate single-point faults.
The technical scheme is as follows: the service management system of the invention comprises:
the server side module comprises a plurality of machine groups consisting of a master node and a plurality of slave nodes managed by the master node, wherein a control node is determined in the master node through the election module, and the control node preferentially receives the service request of the client side; each master node in the election module throws the vote to itself before not receiving the voting results from other master nodes, otherwise, the same vote is thrown according to the received voting results; the master node with the ticket number exceeding half of the total number of the master nodes is the control node;
the client module is used for registering and discovering the service to the server module;
and the monitoring module is used for monitoring the heartbeat request and the running state of the machine, and sending out alarm notification when the state value exceeds the threshold range.
In the election module, each master node throws the vote to the master node in the first round of voting, then each master node can randomly sleep for a period of time, the master node which is preferentially awakened throws the vote to the master node, the voting result is notified to other master nodes, and the other master nodes throw the same vote according to the received voting result after awakening.
Further, the data registry is stored on different master nodes through data slicing, and the master nodes send data slicing information to slave nodes through a synchronization module for data storage; in the synchronization module, a master node sends data fragment information and a submitting request to slave nodes, and the master node marks the request as submitted and sends the submitted request to the slave nodes as long as receiving more than half of the confirmation characters returned by the slave nodes within a certain time, and the slave nodes store the data fragment information to the local; otherwise, the master node resends the data fragment information and submits the request.
Further, when the system is started, determining the node as a master node or a slave node through the command line parameters or the configuration file; when the system operates, the master node sends a heartbeat request to the slave node every second, and if the slave node does not receive the heartbeat request within a certain time, the slave node acts as the master node; if there are only 1 slave node, the slave node acts as a master node; if a plurality of slave nodes exist, the slave node with the latest synchronized data slicing information time stamp is selected to serve as the master node, and if the time stamps are consistent, the slave node with the largest number is selected to serve as the master node.
When the client module registers the service to the server module, a main node is randomly connected to inquire the access address of the control node, then the control node is connected, the control node performs hash calculation according to the service name, and the service is fragmented into the appointed machine group.
The service is fragmented to a main node in a designated machine group, the main node firstly performs pre-writing journaling on service operation, and after a certain time, memory data in the main node are written into a data snapshot and stored into a disk; when restarting recovery, the registry information is recovered using the data snapshot and the pre-written log.
The monitoring module monitors the heartbeat request of the system, and if the service end module does not receive the heartbeat request sent by the service where the client module is located within a certain time, the service end module removes the service from the registry; if a certain number of slave nodes cannot receive the heartbeat request in the slave nodes managed by the master node, the monitoring module calculates the heartbeat frequency to judge whether the heartbeat frequency is a master node fault, and reselects the master node if the master node fault is the master node fault; and if the heartbeat between the main node and the control node is lost, triggering the election module to redetermine the control node.
The master node establishes connection with a master node with a smaller number than the master node, and then connects with a slave node associated with the master node. And after all the master-slave nodes are started and established, selecting the control node from the master nodes.
The service treatment method of the invention comprises the following steps:
(1) The client is connected with a control node of the server, performs hash calculation according to the service name and distributes the hash calculation to a designated master node, and is connected with the master node and sends service information to the master node; the control node is obtained by selecting from the master nodes, one master node or the control node is connected with a plurality of slave nodes to form a machine group, and the server side comprises a plurality of machine groups;
(2) The master node writes the service information into a log file, and constructs a memory registry to disable the local read-write cache;
(3) The slave node regularly pulls the registration information table to the master node; when more than half of the slave nodes receive the registry information and return confirmation characters to the master node, the master node sends a confirmation submitting request to the slave nodes;
(4) The background thread regularly detects whether the read-write buffer memory is invalid, if so, the registry is pulled from the memory to store the read-write buffer memory and the read-write buffer memory;
(5) The client service pulls the dependent service address from the master node for local access.
The beneficial effects are that: compared with the prior art, the invention has the advantages that: (1) The node load is small, the registration information table is subjected to data slicing and stored in different master nodes, and only a small amount of data slicing is needed to be synchronized to the slave nodes, so that the problem that the data table stored in a single node is too huge is solved; (2) The read-write performance is strong, and the data synchronization is performed by adopting two-stage submission and half confirmation modes, so that the consistency of the data is ensured, the write-in performance is also ensured, and the write-in efficiency is greatly improved; (3) The network load is low, a concurrent and safe hash table, a read-only hash Map and a multi-level cache mechanism of the read-write hash Map are arranged in the registry, the client only subscribes to the service concerned by the client, and only needs to push a small amount of configuration information when registering and connecting the service to the online, so that the information of the full registry is not required to be pulled, the timeliness is high, and the network load is greatly reduced; (4) The system has strong stability, the control nodes are determined in a voting mode, the randomness of the election is guaranteed, each main node is possibly used as the control node, and the stability of the system is integrally improved; (5) The system is high in availability, and when the node is down, other nodes are guaranteed to take over the role of the down node rapidly by the other nodes, so that the stability of the whole service management system is integrally guaranteed; when service registration and service discovery are carried out, the information of the upper line, the lower line and the heartbeat of each node is formed into a pre-written log, files are prevented from being tampered in a redundancy check sum mode, a registry is exported to be an image file at regular time for storage, and the recovery efficiency of a memory registry is greatly improved;
Drawings
FIG. 1 is a schematic diagram of a service management system according to the present invention;
FIG. 2 is a flow chart of server-side setup and startup according to the present invention;
FIG. 3 is a client side online flow chart of the present invention;
FIG. 4 is a flow chart of a main node downtime process according to the present invention
Fig. 5 is a flow chart of downtime processing of a control node of the present invention.
Detailed Description
The technical scheme of the invention is further described below with reference to the accompanying drawings.
As shown in fig. 1, the service management system of the present invention includes a server module, a client module and a monitoring module; the server side module is used for enabling the service to be managed to register, carrying out centralized management on the online and offline of the service, and enabling other services to sense the online and offline behaviors of the other services; the client module integrates the service to be managed, is convenient to register with the service end, and can sense the change of other corresponding services, such as online or offline; and the monitoring module is used for regularly collecting information such as CPU, disk, memory and the like of the service management system machine. When a machine where the service management system is located has a relatively large load or is abnormally down, an alarm can be given in a targeted manner.
And (one) service end module service design:
since the entire service remediation system is of a distributed architecture, the distributed remediation system is formed from a plurality of machine nodes together.
(a) Role and group partitioning
The server side module of the system comprises three roles of a master node, a slave node and a control node.
The master node is mainly responsible for processing read-write requests of client service, the slave node is mainly redundant backup of the master node, and when the master node is down, the slave node can quickly serve as the master node. The control node is selected from a plurality of main nodes, wherein the control node is selected from the plurality of main nodes through a distributed consistency protocol, the node receives service registration requests of other main nodes, and when a client needs to perform service registration and service discovery, the control node is accessed preferentially. The control node will tell the client the location of the data fragment of the service it needs to serve, in which particular node; the control node will preferentially initialize 10240 data slices and then perform an average allocation according to the number of master nodes in the node. Similar to node 1 being responsible for the number 1-2048 slices, node 2 being responsible for the number 2049-4096 slices, node 3 being responsible for the number 4097-6144 slices, node 4 being responsible for the number 6145-8192 slices, and node 5 being responsible for the number 8193-10240 slices.
(b) Election module
Roles between nodes can be translated and the control node determined by the election module. When the master node is started, one control node is selected from a plurality of master nodes, voting is initiated in the election process, the votes are cast to the master node in a first round, the master node hopes to become the control node, then the votes are delivered to other nodes, and three machines A, B and C are assumed, so that after the first round of voting is finished, the votes received by each node are A (1), B (1) and C (1). Since the number of votes obtained by the machine is the same, a second round of voting needs to be initiated, each node sleeps randomly for a period of time before the second round of voting is performed, the node which wakes up preferentially still delivers the votes to the machine, and supposing that the machine B wakes up preferentially, after the voting information is delivered to the machine A and the machine C, after the machine A wakes up, the votes are not delivered to the machine A, but the votes are directly delivered to the machine B, and the votes are also delivered to other nodes, and similarly, when the machine C wakes up, the node which delivers the votes to the machine B is also found, then the machine C does not cast the votes to the machine A, but directly casts the votes to the machine B. After the second round of delivery is completed, the people can perform the respective unified calculation of the votes, wherein the votes 2 received by the node A and the node C are cast to the node B, and the votes 2 received by the node A and the node B are cast to the node B. In this case, most nodes (most nodes are defined as 2 nodes if 3 nodes, and most nodes are 3 nodes if 5 nodes) cast votes to the node B, and the node B acts as a control node.
The determination of identity between the master node and the slave node is accomplished in the following manner. First, when the master node and the slave node are started, the role of the current node is specified preferentially through command line parameters or configuration files. During operation, the master node sends heartbeats to the slave node every second, and if the slave node does not receive the heartbeat request of the master node in more than 5 seconds, the slave node considers that the master node may be down. The slave nodes are needed to take over the master node to become a new master node, and if only 1 slave node exists, the slave nodes can serve as the master node directly without reelecting. If there are multiple slave nodes, it is determined whether the time stamp of the data of the registry to which the current slave node has synchronized is up-to-date, and if so, the current slave node may act as a master node. If all the slave nodes receive consistent data time stamps from the master node, the node with the largest number is selected as the master node. Once a new master node is elected, it will also register with the control node that has been elected before. During the operation of the control node, the control node is down due to memory overflow or machine reasons, and then the control node can trigger re-election. Since the control node itself is also the master node. And sending heartbeat requests every second between the master nodes, and triggering the control nodes to reselect after the heartbeat requests cannot be monitored between the control nodes and other master nodes after the heartbeat requests exceed 5 seconds. The whole process of election is completed based on the remaining master nodes. Before the control node is elected, the client cannot sense the position information of other services, so that the whole service management system can not provide the service for a short time in the re-election process. The whole election process is similar to the prior starting, and the mode of voting and ballot returning is adopted for carrying out the formula election. Once a control node has elected, other master nodes register with the control node, and the restored master node registers with the new control node.
(c) Data slicing and machine set
A master node and a plurality of slave nodes together form a machine group. When the external service is provided, one service can be selected from the external service, the master node provides the read-write service, and the slave node only provides the read service. If there is a corresponding write request to the sending slave node, the slave node will forward the request to the master node for processing. In order to make the service information related to the registry be dispersed on different machines, a data slicing design is adopted, the general transverse expansion of the registry generally does not exceed 100, 10240 slices are designed for storage, in this embodiment, the service management system has 15 machines, every three machines form a master and two slaves, and the total number of the machines is 5, so that each machine group is responsible for about 200 slices, the data volume of the whole data slices is invariable, and once the setting is invariable. Each machine that performs service registration performs hash computation according to the service name (ServiceB) and routes the service to a particular data fragment to which a particular machine of the service administration system is bound, and all registration information for the service instance is registered. The data fragments are firstly stored on the master node, and the master node synchronizes the data fragment information to the slave node for data storage. If a plurality of services ServiceC are deployed and a plurality of client services depend on the ServiceC, a large number of connection requests may be connected to a specific machine group in service management, and then problems such as hot keys are easy to occur. At this time, the slave nodes can be added to disperse connection requests, so that related problems such as hot key and data tilting can be solved.
After the master node deposits the data fragments of the service registry, the master node needs to synchronize the data in the data fragments to the slave nodes. The bottom layer of the whole synchronous process adopts a mode of adding half of acknowledgements to two-stage submission. Firstly, a master node receives a service registration request sent by a client service, saves service information to a local area, and then sends the request to be submitted to each slave node as a first stage; after each slave node receives the request to be submitted, ack (Acknowledge character ) is returned to the master node, if the master node receives more than half of the acks of the slave nodes, the master node marks the request as submitted, sends the submitted request to each slave node, and makes the submitted request to submit the message, and in the process, the client service can perceive the message of the online specific service as long as the master node submits the message once. After the slave node also submits the message, the slave node can acquire the latest registry data slicing information.
(d) Weight balance design
The first aspect is mainly that when the service management system needs to perform capacity expansion or capacity reduction, the original service management center cannot bear registration and discovery of mass services, capacity expansion is needed at this time, corresponding to expanding machines, new machines are added to the whole service management center cluster, if the whole machine resources are surplus, reduction is needed at this time, and machines designated by the lower line are required. Once a particular machine is brought on-line or off-line, the data fragments need to be migrated. The service name is internally re-hashed, and the data fragments to be migrated are migrated to the designated machine. After the rebalancing is completed, the overall storage is subjected to load balancing again. After the data rebalancing is completed, there is a part that the requested rebalancing and the rebalancing of long connections are needed. Every time a new service starts to register, a designated master node is found to establish a long connection, and when a new node is added, the long connection also needs to be switched again because the data fragments may not be stored in the node, wherein the slave node bears a part of responsibilities of service discovery, but after the data fragments migrate, the corresponding long connection also needs to be disconnected to establish a connection with a machine in the new machine group. After the data fragment migration is completed, the machine for managing the service is reversely pushed to be disconnected actively so as to connect with the designated new machine.
(e) Registry design
Firstly, the registry information is evenly dispersed in the data fragments by the aid of the data fragments and stored in the registry, a master node and a slave node in each machine group are in charge of a part of the data fragments, and concurrent secure hash maps adopted by the registry in each machine are realized. And the multi-level cache is mainly adopted for processing, wherein the multi-level cache is mainly divided into a most original concurrent secure hash table (memory registration information table), and only a read cache hash Map and a read-write cache hash Map are provided. The whole processing flow is that firstly, data is read from the read-only buffer Map, if the data is not read, the data is read from the read-write Map, and if the data is not read, the data (memory registration information table) is read from the hash table of the core. If new service registration is finished, the hash Map (memory registration information table) of the core is changed, at this time, the read-write buffer Map is expired and emptied, a thread is arranged at the background to compare whether the information in the information tables in the read-only buffer and the read-write buffer are consistent or not, if not, the read-only buffer is emptied, and then reloading is performed from the core registration table next time. By the design of the mode, lock competition is greatly reduced, concurrent access capacity is improved, and access efficiency is greatly improved.
The whole registry is needed to be provided for the client to use, the client needs to know which service he depends on through the registry, and specific ip addresses and ports in the services are what, so that the client service can only initiate requests to machines where the services are located, and related services are requested. The traditional way to obtain the registry is to pull the registry at intervals of 30 seconds or one minute. The general pull strategy is that a full pull is used for the first time, and an incremental pull is used later. However, this has a problem that service C only care about service a, but it pulls down all other service changes (service E, service F goes offline), which inevitably increases the network load. At present, the method is that the client service is used for subscribing only the service concerned by the client, and when the service of the integrated client is started, the client needs to inform the service which the current service depends on, and directly sends a subscription request aiming at the called service which the client needs. For example, after the service C starts, it needs to know the relevant machine and address information of the service a, then it will find a machine at random, ask which control node is, then request the control node, obtain the data fragment of the service a on which machine through the control node, at this time, it is equivalent to the service C directly subscribing to the specific service a, the service C can set up a long connection with a machine in a certain machine group (including the node of the data fragment of the service a) of the service administration system, at this time, if the service is online and offline, only need to push a small amount of configuration information, and need not pull the whole amount of registry information, and the timeliness will be very high. The network load is greatly reduced.
(f) High availability design
The whole service management system adopts a distributed architecture, which is provided with a plurality of machine groups, wherein partial data fragments are stored in each machine group, and each machine group is organized together in a master-multiple-slave mode. The interior of each machine group itself ensures high availability. The inside of which can realize corresponding redundancy backup through the redundancy of the slave nodes. When the master node is down, selecting the node which receives the latest data from the slave nodes as the master node, and if the received data fragments are the same, and no difference exists between the data fragments, selecting the machine with the largest number as the master node. The pieces of information of the registration tables in the master node and the slave node are not stored in the memory only, and if the pieces of information are stored in the memory only, if the machine is in the master node or the slave node is down, the data can be lost. Therefore, when a new machine initiates registration, it is preferable to first write a log in advance, and record the operations of registration, heartbeat, offline, etc. of each service in the file. After a period of time, the memory data in the master node is written into a data snapshot, and all the memory data are written into the disk file. When the later restart is restored, the data snapshot is preferentially used for restoration, then the latest registry information is restored by combining the pre-written log and adopting a forward rolling strategy according to the time stamp, and any snapshot file written into the disk or the pre-logged file can generate a checksum through the crc32 to prevent the data file from being forcibly tampered.
When the control node in the whole service management system is in downtime, a new node is selected from the rest of the main nodes through the distributed consistency protocol to serve as the control node. The steps of all data slices in the entire cluster are managed. The client knows by the control node which node the fragment to which it belongs is at, so that the message can be forwarded to the designated node by the route of the request.
(II) client service design:
when any service needs to provide service to the outside, the service management client needs to be integrated, and the functions of service registration, service discovery and the like can be realized by using the client. When the service where the client is located is started, the client is used for registering with the service management system. The client can randomly find a host machine from the cluster to inquire whether the current node is a control node or not, the current node informs the client because the address of the current control node is known in the main node, the access address of the control node is sent to the control node by the client, after the client is connected with the control node, the control node performs hash calculation according to the service name, and relevant information of the service is fragmented into a designated machine group. The data shards are responsible for by the designated set of machines. After service registration is completed, this corresponds to registration over the entire service administration system. If other services need to be called remotely, the calling path of the service management system can be obtained from the specific data fragments through the control node of the service management system. When the current client starts, the client also depends on some other services, and at this time, after the service where the client is located starts successfully, the corresponding service discovery operation is performed. The control node is used for finding out the data fragments of the services which the service depends on in which machine groups, and at the moment, the current service can initiate long connection to the machine groups of the fragments by using the client side to acquire the specific addresses of the dependent services. When these services are needed during execution of the business execution flow, a request can be initiated to a specified service using a specific access policy. The access mode can adopt a polling access mode or a random access mode, and the access is realized by a hash access mode. The requests are largely evenly distributed to the individual service nodes.
And (III) monitoring module design:
the whole monitoring module can monitor the running condition of the whole service management system cluster, and the monitoring of the heartbeat comprises three aspects:
in the first aspect, the service where the client is located sends a heartbeat request to the server at regular time through the client, the server also knows whether the current service survives according to the heartbeat, if the client of a certain service exceeds 5 seconds, the heartbeat is still not sent to the server, and the current service may be down. At this point, the server removes it from the registry. But if at some time the master node finds that 90 percent of the machines it manages are not able to receive heartbeats, it is likely that problems will occur with itself. The monitoring module can intelligently calculate the frequency number of the heartbeat and comprehensively feed back according to the calculation result, if the problems possibly occur in the main node, the reelection of the main node is immediately triggered, and the current main node pauses to provide services. After waiting to elect a new node, the service is provided to the outside.
In a second aspect, the master node and the slave node are also kept connected through heartbeat, if the master node does not receive the heartbeat of the slave node within 5 seconds, the slave node is removed, and if the slave node does not receive the heartbeat of the master node, the master node is triggered to reselect.
In a third aspect, the following monitoring module triggers a reselection of the control node if a heartbeat between the master node and the control node is lost.
The monitoring module can collect the whole state of the whole service management system cluster in real time, and according to the machine load state, the current utilization rate of the cpu, the disk read-write speed, the network flow speed and the gc frequency in the whole service management system, and the time spent by full gc each time. When a certain machine exceeds the alarm threshold value set before, an alarm notification is sent out, and related operation and maintenance personnel can optimize system parameters or check the problems of hardware according to alarm content.
The service management method of the invention comprises the following contents:
and carrying out configuration operation on the files of the server according to the requirements according to the field environment. After the configuration is completed, the server is started, and the whole service management system consists of a plurality of machines which are internally provided with a plurality of machine groups. Each machine group is started in turn, and after the starting is completed. The master node can be actively disconnected with machines with smaller numbers than the master node, a round of voting is initiated among the master nodes after all the machines of the master node are connected, the first round of voting is given to the master node by default, and the calculation of the votes taken by the master node is carried out after each round of voting is finished. If most nodes in the machine are delivered to a certain node, the control node is the node, and if the nodes in the middle of the main node do not acquire the votes of the most nodes in the current round, the nodes randomly sleep for a period of time at the moment, and then immediately perform the votes of the next round. After multiple rounds of voting, there will be one node that gets votes for most of the nodes, which are elected as control nodes. After the control node is selected, all the main nodes register with the control node, the control node segments 10240 data fragments according to the number of the current nodes and distributes the data fragments to the main nodes evenly, and the main nodes are responsible for some of 10240 data fragments. The master node and the control node have heartbeats all the time, and the connection between the master node and the control node is maintained. During election of a control node, the slave node will initiate a connection to the configured master node by default. The slave node timing and master node synchronize the data slicing and heartbeat information.
After integrating the client with an application service, when the application service is started, the client service initiates a service registration request to the service administration server. The client will randomly pick a server, query its control node, and the master node will forward the request for client service to the control node. After receiving the request, the control node performs hash calculation according to the service name of the client, finds the main node designated to be responsible for the stored data fragments, tells the client the address of the main node, and after knowing the address of the main node, the client initiates connection to the machine group of the main node again, and the main node in the machine group writes the relevant basic information of the service of the client into the data fragments responsible for management. When the data fragments are written, the pre-written log is written first, and then the pre-written log is written on the disk. After 5 seconds, the slave nodes in the machine group pull information such as the pre-written log and the like, and synchronize the data slicing information.
When the client service accesses the control node after the client service completes service registration, the client service can acquire the corresponding master node information of the service on which the client service depends from the control node. The client initiates a request to a machine group of the relied service, and one machine processing request is randomly distributed in the machine group, which may be a master node or a slave node. The client service establishes a long connection with the service. The client service is equivalent to the service which only subscribes to be called, and when the service is online and offline, nodes in the dependent service can timely inform the client service. The internal delay is very low and the timeliness is very high. The client service synchronizes its registry information for dependent services at regular time. When access is required, access can be performed according to different access policies.
Specifically, the server side establishment and start-up flow, the client side online flow and the exception handling service flow are as follows.
(1) As shown in FIG. 2, the server side establishment and start-up procedure includes the following steps
(1.1) loading a configuration file and analyzing each parameter in the configuration file.
And (1.2) starting a monitoring thread, and monitoring a master node with a larger number than the master node to establish connection with the master node, wherein the master node can actively connect with a master node with a smaller number than the master node. Waiting for all master nodes to establish a connection.
(1.3) after all the master nodes complete the connection establishment, the listening thread is started to wait for the slave nodes associated with the master node, and the number of the slave nodes may be one or more. Waiting for the slave node and the master node to complete the connection establishment.
(1.4) when all the master-slave nodes are started and the establishment is completed, the control node is selected from the master nodes.
(1.5) the control node elects by voting, which may require multiple rounds in the middle. And if the votes of other nodes are received, the votes are defaulted to the next round of votes which are directly cast to the machine which receives the first vote at rest. After one round of voting is finished, each node can check the votes received by the nodes and perform voting classification aggregation, if a certain machine is selected by most nodes, the machine is the control node, and if one round of voting is not selected, the control node can be selected from the main nodes after a plurality of rounds of voting.
(1.6) once the control node elects, all master nodes need to actively establish connections with the control node.
(1.7) if the start is the first time, the control node will make a data slicing policy, and if there are 5 master nodes, 10240 slices are allocated to the 5 machines. Each machine is responsible for 2048 slices. If the starting is not the first starting, the master node can load the snapshot file of the data fragment of the local disk preferentially and restore the complete data fragment in the memory table according to the content in the pre-log. Because the master node already knows the location of the control node, a data request needs to be sent to the control node at this time, and the distribution of the fragmented data is sent to the control node. After receiving the request, the control node reconstructs the step and mapping relation between the data fragments and the main node in the own memory.
(1.8) after the above procedure is completed, a heartbeat request is sent between the master node and the control node every 1 second. The master node and the slave node in the same machine group also send heartbeat requests every second.
(2) As shown in FIG. 3, the client-side online process comprises the following steps of
The integral client service online is mainly divided into two parts, namely service registration and service discovery.
(2.1) services that need to be registered with the service administration system are first integrated with the incoming client. The service governance client will start with the service.
(2.2) the client reads the addresses of all the master nodes from the configuration file in the starting process, and then randomly selects one node from the master nodes for connection.
(2.3) after the connection is successful, checking whether the node is a control node, and if not, acquiring the related information of the control node from the master node. And then initiate a connection to the control node.
(2.4) after the client and the control node establish a connection, the client may attempt to obtain relevant data information for the data shards from the control node. The control node performs hash operation according to the service name currently required to be registered. And allocating the designated master node for processing.
(2.5) after the client service obtains the feedback of the information about the control node, it will initiate a connection request to the master node. After the connection with the master node is established, the client transmits its own related service information to the specific master node.
And (2.6) after receiving the related information, the master node preferentially writes the service information into the file of the pre-written log, constructs a new memory registry, and causes the local read-write cache to be invalid so as to ensure the consistency of the data.
(2.7) the slave node pulls the latest registry information from the master node every 5 seconds, synchronizes the pre-written logs, and keeps the registry information of the slave node consistent with the master node through the pre-written logs.
And (2.8) checking whether the read-write buffer memory is invalid by the background thread every 30 seconds, if so, directly loading a piece of latest registry information from the memory to enter the read-write buffer memory, and covering the data in the original read buffer memory by using the data in the read-write buffer memory.
(2.9) after the service registration is completed, the client service may obtain the master node address of the data fragment where the service it needs to invoke is located from the control node. The client initiates connection to the service node and establishes a long connection operation in such a way that connection can be established with the master node or with the slave node.
(2.10) once the connection is established, the client service can pull the dependent service address from the node, and can perform polling or hash access in combination with the local access policy, if the dependent service is online or offline, the master node or the slave node connected with the dependent service can directly push the dependent service back to inform the dependent service. Only certain services are pushed and the load of the whole cluster will be low.
(3) The processing service flow of the main node exception and the control node exception comprises the following steps of
(3.1) As shown in FIG. 4, the service flow of the exception handling in which the master node (non-control node) is down is as follows
(3.1.1) there is a heartbeat interaction between the slave node and the master node in the machine group every second. If both parties exceed 5 seconds, no heartbeat is received by both parties. At this time, the master node will be actively disconnected. And elects one of the remaining slave nodes as the master node.
(3.1.2) comparing the latest time stamp in the registry pulled in the slave nodes, if the time stamp of the current node is latest in all the slave nodes, the current slave node will be selected as the master node, if the time stamp in some of the slave nodes is the same. Then the slave node with the largest number is selected as the master node.
(3.1.3) there is also a case where a switch of the master node and the slave node is triggered, and the slave node can receive the heartbeat of the master node, but the master node cannot receive the heartbeat information of the slave node. Possibly when a unidirectional network fails. The master node then actively refuses service and attempts to let the slave node initiate a master-slave switching action.
(3.1.4) after the handover is completed, the original master node is added to the machine group with the identity of the slave node.
(3.2) As shown in FIG. 5, the exception handling service flow of the master node (control node) downtime is as follows
(3.2.1) the control node and other related master nodes perform heartbeat communication regularly, and if the master node cannot sense the heartbeat of the control node within 5 seconds, the control node may be down.
(3.2.2) at this point the master node initiates the operation of selecting a new control node in conjunction with the other master nodes. And carrying out multi-round voting again, and selecting the node with the most votes as the control node in the voting stage.
(3.2.3) after the control node elects, all other master nodes register with the control node.
Claims (6)
1. A service remediation system, comprising:
the server side module comprises a plurality of machine groups consisting of a master node and a plurality of slave nodes managed by the master node, wherein a control node is determined in the master node through the election module, and the control node preferentially receives the service request of the client side; each master node in the election module throws the vote to itself before not receiving the voting results from other master nodes, otherwise, the same vote is thrown according to the received voting results; the master node with the ticket number exceeding half of the total number of the master nodes is the control node;
The client module is used for registering and discovering the service to the server module;
the monitoring module is used for monitoring the heartbeat request and the running state of the machine, and sending out alarm notification when the state value exceeds the threshold range;
the data registry is stored on different master nodes through data slicing, and the master nodes send data slicing information to slave nodes through a synchronization module for data storage; in the synchronization module, a master node sends data fragment information and a submitting request to slave nodes, and the master node marks the request as submitted and sends the submitted request to the slave nodes as long as receiving more than half of the confirmation characters returned by the slave nodes within a certain time, and the slave nodes store the data fragment information to the local; otherwise, the master node resends the data fragment information and submits the request;
when the system is started, determining the node as a master node or a slave node through command line parameters or configuration files; when the system operates, the master node sends a heartbeat request to the slave node every second, and if the slave node does not receive the heartbeat request within a certain time, the slave node acts as the master node; if there are only 1 slave node, the slave node acts as a master node; if a plurality of slave nodes exist, the slave node with the latest synchronized data fragment information time stamp is selected to serve as a master node, and if the time stamps are consistent, the slave node with the largest number is selected to serve as the master node;
When the client module registers the service to the server module, a main node is randomly connected to inquire the access address of the control node, then the control node is connected, the control node performs hash calculation according to the service name, and the service is fragmented into the appointed machine group.
2. The service management system according to claim 1, wherein in the election module, each master node throws a vote to itself in the first round of voting, then each master node sleeps randomly for a period of time, the master node that is preferentially awakened throws the vote to itself, and notifies other master nodes of the voting result, and the other master nodes throw the same vote according to the received voting result after awakening.
3. The service management system according to claim 1, wherein the service is fragmented to a master node in a designated machine group, the master node first pre-writes a log of service operations, and after a certain time, writes memory data in the master node as a data snapshot and stores the data snapshot into a disk; when restarting recovery, the registry information is recovered using the data snapshot and the pre-written log.
4. The system of claim 1, wherein the monitoring module monitors the heartbeat request of the system, and if the service module does not receive the heartbeat request sent by the service where the client module is located within a certain period of time, the service module removes the service from the registry; if a certain number of slave nodes cannot receive the heartbeat request in the slave nodes managed by the master node, the monitoring module calculates the heartbeat frequency to judge whether the heartbeat frequency is a master node fault, and reselects the master node if the master node fault is the master node fault; and if the heartbeat between the main node and the control node is lost, triggering the election module to redetermine the control node.
5. The service management system of claim 1, wherein the master node establishes a connection with a master node that is less than its own number and then connects with its own associated slave node.
6. A method of service remediation comprising the steps of:
(1) The client side firstly inquires the access address of the control node by randomly connecting a main node of the server side, then connects the control node of the server side, carries out hash calculation according to the service name and distributes the hash calculation to the appointed main node, and the client side is connected with the main node and sends service information to the main node; the control node is obtained by selecting from the master nodes, one master node or the control node is connected with a plurality of slave nodes to form a machine group, and the server side comprises a plurality of machine groups;
determining the node as a master node or a slave node through command line parameters or configuration files; the master node sends a heartbeat request to the slave node every second, and if the slave node does not receive the heartbeat request within a certain time, the slave node acts as the master node; if there are only 1 slave node, the slave node acts as a master node; if a plurality of slave nodes exist, the slave node with the latest synchronized data fragment information time stamp is selected to serve as a master node, and if the time stamps are consistent, the slave node with the largest number is selected to serve as the master node;
(2) The master node writes the service information into a log file, and constructs a memory registry to disable the local read-write cache;
(3) The slave node regularly pulls the registration information table to the master node; when more than half of the slave nodes receive the registry information and return confirmation characters to the master node, the master node sends a confirmation submitting request to the slave nodes;
the register table is stored on different master nodes through data slicing, the master nodes send data slicing information to the slave nodes for data storage, the master nodes send the data slicing information and submit requests to the slave nodes, and the master nodes mark the requests as submitted and send the submitted requests to the slave nodes as long as the master nodes receive more than half of confirmation characters returned by the slave nodes within a certain time, and the slave nodes store the data slicing information to the local; otherwise, the master node resends the data fragment information and submits the request;
(4) The background thread regularly detects whether the read-write buffer memory is invalid, if so, the registry is pulled from the memory to store the read-write buffer memory and the read-write buffer memory;
(5) The client service pulls the dependent service address from the master node for local access.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111524029.6A CN114363350B (en) | 2021-12-14 | 2021-12-14 | Service management system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111524029.6A CN114363350B (en) | 2021-12-14 | 2021-12-14 | Service management system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114363350A CN114363350A (en) | 2022-04-15 |
| CN114363350B true CN114363350B (en) | 2024-04-16 |
Family
ID=81099663
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111524029.6A Active CN114363350B (en) | 2021-12-14 | 2021-12-14 | Service management system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114363350B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114760659B (en) * | 2022-04-18 | 2024-08-27 | 杭州魔迅科技有限公司 | Parallel issuing method and device for block data, computer equipment and storage medium |
| CN116400853B (en) * | 2023-02-21 | 2023-11-07 | 北京志凌海纳科技有限公司 | Distributed block storage system and manufacturing-oriented fault recovery time shortening method |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105873164A (en) * | 2016-06-21 | 2016-08-17 | 广州大学 | Improved GAF (geographical adaptive fidelity) topology design method for wireless sensor network |
| CN106331098A (en) * | 2016-08-23 | 2017-01-11 | 东方网力科技股份有限公司 | Server cluster system |
| CN107832138A (en) * | 2017-09-21 | 2018-03-23 | 南京邮电大学 | A kind of implementation method of the High Availabitity namenode models of flattening |
| CN107846318A (en) * | 2017-11-15 | 2018-03-27 | 郑州云海信息技术有限公司 | A kind of distributed type assemblies and distributed type assemblies management method |
| CN110837408A (en) * | 2019-09-16 | 2020-02-25 | 中国科学院软件研究所 | High-performance server-free computing method and system based on resource cache |
| CN111124301A (en) * | 2019-12-18 | 2020-05-08 | 深圳供电局有限公司 | Data consistency storage method and system of object storage device |
| CN112054926A (en) * | 2020-08-31 | 2020-12-08 | 深圳前海微众银行股份有限公司 | Cluster management method and device, electronic equipment and storage medium |
| CN112416889A (en) * | 2020-10-27 | 2021-02-26 | 中科曙光南京研究院有限公司 | Distributed storage system |
| CN112860386A (en) * | 2019-11-27 | 2021-05-28 | 上海哔哩哔哩科技有限公司 | Method for switching nodes in distributed master-slave system |
| CN112865992A (en) * | 2019-11-27 | 2021-05-28 | 上海哔哩哔哩科技有限公司 | Method and device for switching master nodes in distributed master-slave system and computer equipment |
| CN112865995A (en) * | 2019-11-27 | 2021-05-28 | 上海哔哩哔哩科技有限公司 | Distributed master-slave system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10523498B2 (en) * | 2016-12-23 | 2019-12-31 | Sierra Nevada Corporation | Multi-broker messaging and telemedicine database replication |
| CN111274317A (en) * | 2020-01-07 | 2020-06-12 | 书生星际(北京)科技有限公司 | Method and device for synchronizing multi-node data and computer equipment |
-
2021
- 2021-12-14 CN CN202111524029.6A patent/CN114363350B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105873164A (en) * | 2016-06-21 | 2016-08-17 | 广州大学 | Improved GAF (geographical adaptive fidelity) topology design method for wireless sensor network |
| CN106331098A (en) * | 2016-08-23 | 2017-01-11 | 东方网力科技股份有限公司 | Server cluster system |
| CN107832138A (en) * | 2017-09-21 | 2018-03-23 | 南京邮电大学 | A kind of implementation method of the High Availabitity namenode models of flattening |
| CN107846318A (en) * | 2017-11-15 | 2018-03-27 | 郑州云海信息技术有限公司 | A kind of distributed type assemblies and distributed type assemblies management method |
| CN110837408A (en) * | 2019-09-16 | 2020-02-25 | 中国科学院软件研究所 | High-performance server-free computing method and system based on resource cache |
| CN112860386A (en) * | 2019-11-27 | 2021-05-28 | 上海哔哩哔哩科技有限公司 | Method for switching nodes in distributed master-slave system |
| CN112865992A (en) * | 2019-11-27 | 2021-05-28 | 上海哔哩哔哩科技有限公司 | Method and device for switching master nodes in distributed master-slave system and computer equipment |
| CN112865995A (en) * | 2019-11-27 | 2021-05-28 | 上海哔哩哔哩科技有限公司 | Distributed master-slave system |
| CN111124301A (en) * | 2019-12-18 | 2020-05-08 | 深圳供电局有限公司 | Data consistency storage method and system of object storage device |
| CN112054926A (en) * | 2020-08-31 | 2020-12-08 | 深圳前海微众银行股份有限公司 | Cluster management method and device, electronic equipment and storage medium |
| CN112416889A (en) * | 2020-10-27 | 2021-02-26 | 中科曙光南京研究院有限公司 | Distributed storage system |
Non-Patent Citations (3)
| Title |
|---|
| Block Proposer Election Method Based on Verifiable Random Function in Consensus Mechanism;H. Wang and W. Tan;《2020 IEEE International Conference on Progress in Informatics and Computing (PIC), Shanghai, China》;20210216;304-308 * |
| 一种改进的主从节点选举算法用于实现集群负载均衡;任乐乐;何灵敏;;中国计量学院学报;20150915(第03期);全文 * |
| 无人机编队组网关键技术研究;张彤;《中国博士学位论文全文数据库(电子期刊) 信息科技辑》;20210115;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114363350A (en) | 2022-04-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11853263B2 (en) | Geographically-distributed file system using coordinated namespace replication over a wide area network | |
| AU2019236685B2 (en) | Distributed file system using consensus nodes | |
| US11120044B2 (en) | System and method for maintaining a master replica for reads and writes in a data store | |
| CN113010496B (en) | Data migration method, device, equipment and storage medium | |
| US9846704B2 (en) | Distributed file system using consensus nodes | |
| US9495381B2 (en) | Geographically-distributed file system using coordinated namespace replication over a wide area network | |
| CN107832138B (en) | Method for realizing flattened high-availability namenode model | |
| CN114363350B (en) | Service management system and method | |
| US20080077635A1 (en) | Highly Available Clustered Storage Network | |
| US20070061379A1 (en) | Method and apparatus for sequencing transactions globally in a distributed database cluster | |
| CN110365750A (en) | Service registration system and method | |
| GB2484086A (en) | Reliability and performance modes in a distributed storage system | |
| CN107623703B (en) | Synchronization method, device and system for Global Transaction Identifier (GTID) | |
| CN103207867A (en) | Method for processing data blocks, method for initiating recovery operation and nodes | |
| CN105493474A (en) | System and method for supporting partition level journaling for synchronizing data in a distributed data grid | |
| CN114124650A (en) | Master-slave deployment method of SPTN (shortest Path bridging) network controller | |
| CN105323271B (en) | Cloud computing system and processing method and device thereof | |
| CN109726211A (en) | A kind of distribution time series database | |
| CN117909136A (en) | Distributed log storage backup method, device, equipment and storage medium | |
| CN115562849A (en) | Cache data method and system based on high availability |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |