TWI740885B - Service node switching method and device of distributed storage system - Google Patents

Abstract

The embodiment of the present invention provides a service node switching method of a distributed storage system. The service node includes a current service node and a standby service node. The method includes: monitoring the response status of the service node to a service request; If the response state of the node is abnormal, the communication between the current service node and the standby service node is stopped, and the switching process of the current service node is triggered. Through the service node inspection logic, logical judgment and data statistics are made for various factors that affect the response status of the service node. In the event of service timeout, service unavailability, service abnormality, etc. caused by hardware failure or software defect, Realize autonomous switching and recovery of service nodes, enhancing service availability.

Description

Service node switching method and device of distributed storage system

The present invention relates to the field of Internet technology, in particular to a service node switching method of a distributed storage system and a service node switching device of a distributed storage system.

In the current large-scale distributed storage system, in order to achieve centralized authority authentication and quota control, a centralized metadata management method is mainly adopted, that is, the metadata of all data in the entire system is concentrated in several metadata service nodes for storage.

The availability of metadata service nodes in such an architecture is directly related to the availability of the entire system. Therefore, in a distributed storage system, the availability of metadata service nodes is usually improved through redundancy. At present, there are two main methods to improve the availability of metadata service nodes. For example, the metadata service (Name Node) uses HA (High Avai1ablity, high availability), and the standby service node (Slave node) is used to switch the current service in abnormal state. Node (Primary node); or, for example, the Alibaba Cloud Feitian distributed system and the Pangu file storage system use the Paxos protocol to switch the service node.

In the above two service node switching methods, the service will only be triggered when the current service node cannot normally send a heartbeat confirmation to the standby service node due to server downtime, service process restart, network disconnection, etc. Node switching processing, in other abnormal situations such as duplex network single-way disconnection, some network protocol abnormalities, slow disk response, etc., due to the lock maintenance mechanism and heartbeat mechanism, the standby service node will still consider the current service node It is in a normal working state without triggering the switching process of the service node.

However, in fact, the current service node in an abnormal state will cause the response to the user’s service request to time out, fail to provide complete metadata, and fail to store the log on the shared storage device. This has actually affected the current service node’s The quality of service provided by users, but the current service node switching method cannot restore normal and stable metadata services accordingly. Therefore, the current service node switching method has a problem that the metadata service recovery efficiency is low, which affects the user experience.

In view of the above problems, embodiments of the present invention are proposed to provide a method for switching service nodes of a distributed storage system and a corresponding service node switching device of a distributed storage system that overcome the above problems or at least partially solve the above problems.

In order to solve the above problems, the present invention discloses a service node switching method of a distributed storage system. The service node includes a current service node and a standby service node. The method includes: monitoring the response status of the service node to a service request; If the response status of the current service node is abnormal, the communication between the current service node and the standby service node is stopped, and the switching process of the current service node is triggered.

Optionally, the method further includes: if the response status of a certain service node is abnormal, adding an abnormal node identifier for marking that it does not participate in the handover processing of the current service node to the service node.

Optionally, the step of triggering the handover process of the current service node includes: triggering to select at least one service node that does not carry the abnormal node identifier as the new current service node, and replace the current service node that responds to an abnormal state.

Optionally, the step of monitoring the response status of the service node to the service request includes: monitoring the response status of the service node to the service request through multiple threads.

Optionally, the step of monitoring the response status of the service node to the service request through multiple threads includes: obtaining, through a first check thread, the time point T1 at which the service node recently took out the service request from the service request queue, and The time interval D1 from the current time point N1 is taken as the response time; it is determined whether the response time is greater than the first preset response time threshold, and if so, it is determined that the response state of the service node is abnormal.

Optionally, the step of monitoring the response status of the service node to the service request through multiple threads includes: judging through a second check thread whether the storage unit of the service node carries a storage response timeout identifier; The time interval D2 between the addition time point T2 and the current time point N2 of the identifier corresponding to the storage response timeout identifier is used as the response time, and if the response time is greater than the second preset response time threshold, the response of the service node is determined The status is abnormal.

Optionally, before the step of monitoring the response status of the service node to the service request through multithreading, the method further includes: obtaining the start and end time of the log writing of the storage unit of the service node through a log recording thread, and Use the time interval between the start and end times as the response time of the storage unit; determine whether the response time of the storage unit is greater than a third preset response time threshold, and if so, add the storage response timeout flag for the storage unit, and respond accordingly Record the time point T2 when the logo is added.

Optionally, the method further includes: if the storage unit response time is less than the third preset response time threshold, and the storage unit already carries the storage response timeout flag, deleting the storage response Timeout indicator.

Optionally, before the step of monitoring the response status of the service node to the service request, the method further includes: registering a monitoring result of at least one of the service nodes in a monitoring result registration list; The step of the service node's response status to the service request is: polling the registered monitoring result at the start position of the monitoring result registration list.

Optionally, the method further includes: dividing the number of the multi-threads by the preset response time threshold as the frequency of monitoring the response state of the service node.

In order to solve the above problems, the present invention also discloses a service node switching device of a distributed storage system. The service node includes a current service node and a standby service node. The device includes: a service node response status monitoring module for monitoring The response status of the service node to the service request; the current service node switching trigger module is used to stop the communication between the current service node and the standby service node if the response status of the current service node is abnormal, and trigger Handover processing of the current service node.

Optionally, the device further includes: an abnormal node identification adding module, which is used to add an abnormal node for marking the service node not to participate in the handover process of the current service node if the response status of a certain service node is abnormal Logo.

Optionally, the current service node switching trigger module includes: a trigger selection sub-module for triggering the selection of at least one service node that does not carry the abnormal node identifier as the new current service node, replacing the current service node with an abnormal response state Service node.

Optionally, the service node response status monitoring module includes: a multi-thread monitoring sub-module configured to monitor the response status of the service node to the service request through multiple threads.

Optionally, the multi-thread monitoring submodule includes: a first check thread subunit, configured to obtain, through the first check thread, the time point T1 at which the service node recently took out the service request from the service request queue, and compare it with The time interval D1 of the current time point N1 is used as the response time; the first preset response time threshold judging subunit is used to judge whether the response time is greater than the first preset response time threshold, and if so, determine the response of the service node The status is abnormal.

Optionally, the multi-thread monitoring sub-module includes: a second check thread subunit, which is used to determine whether the storage unit of the service node carries a storage response timeout flag through the second check thread; if so, call the second check thread A preset response time threshold judging subunit; a second preset response time threshold judging subunit for adding the identifier corresponding to the stored response timeout identifier to the time interval D2 between the time point T2 and the current time point N2 as the response Time, if the response time is greater than the second preset response time threshold, it is determined that the response state of the service node is abnormal.

Optionally, the device further includes: a storage unit response time determination module, configured to obtain the start and end time of the log writing of the storage unit of the service node through the log recording thread, and use the time interval of the start and end time as the storage unit Response time; storage response timeout flag adding module, used to determine whether the storage unit response time is greater than the third preset response time threshold, if so, add the storage response timeout flag for the storage unit, and corresponding Record the time point T2 when the logo is added.

Optionally, the device further includes: a storage response timeout flag deletion module, configured to: if the storage unit response time is less than the third preset response time threshold, and the storage unit already carries the storage In response to the timeout flag, the storage response timeout flag is deleted.

Optionally, the device further includes: a monitoring result registration module, configured to register a monitoring result of at least one of the service nodes in a monitoring result registration list; the service node response status monitoring module includes: a monitoring result wheel The inquiry sub-module is used for polling the registered monitoring result at the start position of the monitoring result registration list.

Optionally, the device further includes: a monitoring frequency determination module, configured to divide the number of the multithreads by the preset response time threshold as the frequency of monitoring the response state of the service node.

The embodiment of the present invention includes the following advantages: The embodiment of the present invention monitors the response status of the service node to the service request, and stops the communication between the current service node and the standby service node for the current service node whose response status is abnormal, thereby triggering the current service node Handover processing. Through the service node inspection logic, logical judgment and data statistics are made for various factors that affect the response status of the service node. In the event of service timeout, service unavailability, service abnormality, etc. caused by hardware failure or software defect, The autonomous switching and recovery of service nodes are realized, service availability is enhanced, service recovery efficiency is improved, and user experience is improved.

Secondly, the embodiment of the present invention can monitor one or more factors that affect the response status of the service node, and the multi-dimensional monitoring method improves the comprehensiveness and scalability of service recovery.

Further, the embodiment of the present invention does not directly initialize the abnormal current service node, but uses a relatively conservative manner of stopping communication with the standby service node to trigger the service node switch. When a misdiagnosis accident occurs, it is misdiagnosed as an abnormal response state. The current service node also has the opportunity to re-function as the new current service node and continue to provide services, avoiding the negative impact of misdiagnosis on the entire system.

Further, the embodiment of the present invention adds an abnormal node identifier to the service node with an abnormal response state, which avoids the problem that the service node with an abnormal response state is selected as the current service node, thereby failing to achieve the purpose of service node switching. Moreover, the abnormal service node is excluded, and the switching processing of the service node can ensure the stability of the new current service node, avoid system fluctuations caused by multiple service node switching, and improve the stability of service recovery.

501‧‧‧Service node response status monitoring module

502‧‧‧Current service node switching trigger module

601‧‧‧Monitoring result registration module

602‧‧‧Service node response status monitoring module

603‧‧‧Current service node switching trigger module

604‧‧‧Abnormal node identification add module

605‧‧‧Monitoring frequency determination module

Fig. 1 is a step flow chart of Embodiment 1 of a service node switching method of a distributed storage system of the present invention; Fig. 2 is a step flow chart of Embodiment 2 of a service node switching method of a distributed storage system of the present invention; Fig. 3 Is a step flow chart of the third embodiment of the service node switching method of a distributed storage system of the present invention; FIG. 4 is a step flow chart of the fourth embodiment of the service node switching method of a distributed storage system of the present invention; FIG. 5 is the present invention A structural block diagram of the first embodiment of a service node switching device of a distributed storage system of the invention; FIG. 6 is a structural block diagram of a second embodiment of the service node switching device of a distributed storage system of the present invention.

In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Among the more commonly used methods to improve the availability of metadata service nodes, for example, in the Hadoop distributed file system, the metadata service utilizes a backup service node to switch the current service node in an abnormal state. Specifically: through the distributed lock service, the service node that has obtained the distributed lock is used as the current service node, and the current service node provides it to the outside world, and stores the generated log on the shared storage device, and other standby service nodes do not provide it to the outside world The metadata service only reads logs from the shared storage device and applies them to the random access memory, keeping the random access memory in sync with the current service node. The standby service node checks the lock status from time to time. When the lock is released, it indicates that the current service node is in abnormal conditions such as server downtime, service process restart, network disconnection, etc. The standby service node acquires the distributed lock and upgrades it to a new one. The current service node and provide metadata services to the outside world.

In another commonly used way to improve availability, multiple backup service nodes use Paxos protocol to elect the current service node, from which a current service node that provides metadata services to the outside world is generated, and the user requests metadata services from the current service node. The current service node The log generated after the response is stored locally and sent to all standby service nodes. After receiving the log, the standby service node stores it locally and applies it to the random access memory, keeping it in sync with the current service node. At the same time, when the current service node is working normally, it can send heartbeat confirmation information to the backup service node, and the backup service node confirms the survival of the current service node through the heartbeat mechanism. If the current service node encounters abnormal conditions such as server downtime, service process restart, network disconnection, etc., the current service node cannot send heartbeat confirmation information to the standby service node, and the standby service node initiates the current service node switching process. A new current service node is elected from the standby service nodes to provide metadata services to the outside world.

It can be seen from the above that in the current methods of improving the availability of metadata service nodes, triggering the switching of service nodes depends on serious failures that cause the current service node to completely fail to work. Other failures that cause abnormal conditions such as slow response of the current service node will not Trigger the switching of the service node, but the current service node in an abnormal state has affected the quality of the service provided to the user.

Therefore, the current service node switching method has a problem that the metadata service recovery efficiency is low, which affects the user experience. Moreover, according to the current service node switching method, even if the service node switching process is triggered, it may be switched to a service node that is already in an abnormal state again. The purpose of service node switching cannot be achieved, which affects the efficiency of metadata service recovery. In order to solve the above-mentioned problems, several embodiments of service node switching methods are proposed below.

1, there is shown a step flow chart of Embodiment 1 of a service node switching method for a distributed storage system of the present invention. The service node includes a current service node and a standby service node. The method may specifically include the following steps:

Step 101: Monitor the response status of the service node to the service request.

It should be noted that the service node may be a service node that provides metadata services. Metadata, also known as intermediary data, metadata, is data about data, mainly information describing properties of data, and is used to support information such as indicating storage location, historical data, resource search, File recording and other functions.

In a specific implementation, for different check points, a number of check threads can be set up in the system to monitor whether the response status of the service node is abnormal. For example, setting a first inspection thread focusing on abnormal response time of the service request queue and/or setting a second inspection thread focusing on abnormal response time of the storage unit.

It should be noted that whether the response time is abnormal can be determined by comparing with the preset response time threshold. For example, the response time threshold can be preset to 10 seconds. If the service node responds to a certain service request in the service request queue If the time exceeds 10 seconds, or the response time of the storage unit of the service node for log reading and writing exceeds 10 seconds, it can be understood that the response status of the service node is abnormal, and its response to the service request requires the user to wait for a long time, or even fails to serve normally. Has affected the user experience.

As a preferred example of the embodiment of the present invention, the response status of the service node to the service request can be monitored through multithreading. Because in actual applications, the response status of the service node may be affected by many factors, those skilled in the art can combine to set up multiple inspection threads according to the actual situation to monitor the response status of the service node to the service request. For example, Set up a check thread that focuses on whether the storage unit is close to full.

Preferably, the check thread used to monitor the service node may not have the ability to execute logic, but is only used for logical judgment and data statistics for checking the response state. Keep the check thread as lightweight as possible to avoid processing tasks that require a large amount of calculation and time-consuming. For example, it is possible not to perform RPC (Remote Procedure Call Protocol, remote procedure call protocol) operations or wait for a long time for lock operations. As a result, the subsequent service node switching processing is severely delayed or even cannot be executed effectively, which may eventually cause the entire inspection mechanism to lose its due effect.

Step 102: If the response state of the current service node is abnormal, stop the communication between the current service node and the backup service node, and trigger the switching process of the current service node.

In a specific implementation, by monitoring the response status of the service node, it can be determined whether the response status of the current service node and the standby service node is abnormal. For different monitoring results of different service nodes, corresponding operations can be performed. For the current service node that responds to an abnormal state, the communication between it and multiple standby service nodes can be stopped.

There can be many ways to stop communicating with each other. For example, by stopping the heartbeat confirmation between the current service node and the standby service node, when the standby service node has not received the heartbeat confirmation information sent by the current service node for a certain period of time, it can be considered as the current The service node is in an abnormal state and is invalid, and the handover process of the current service node needs to be initiated.

The standby service node cannot communicate with the current service node normally. It can be considered that the current service node is in an abnormal state, which triggers a new current service node election operation, and replaces the newly elected current service node with the abnormal current service node to complete the current service node. Handover processing of the service node. The election operation of the current service node can be implemented through the Paxos protocol.

In practical applications, other methods can also be used to trigger the switching process of the current service node, such as using distributed lock service. If the standby service node thinks that the current service node is in an abnormal state and is invalid, it triggers the current service node to release the lock. The standby service node performs a lock grab operation, and the abnormal current service node is replaced by the standby service node that has acquired the lock to complete the switching process of the current service node.

For the current service node and standby service node that respond to abnormal status, an abnormal node identifier can also be added, and the service node carrying the identifier will not be switched to the new current service node during the switching process of the service node, so as not to Taking the abnormal service node as the new current service node cannot achieve the effect of service restoration.

As a preferred example of the embodiment of the present invention, the monitoring result of at least one of the service nodes may be registered to the monitoring result registration list; the registered monitoring result can be polled at the beginning of the monitoring result registration list.

In practical applications, checkpoint execution modules can be set to perform corresponding operations for different monitoring results. Each inspection thread can generate monitoring results during system startup, and register the monitoring results in the monitoring result registration list of the checkpoint execution module. The checkpoint execution module can be a system background thread, which can start to poll the monitoring results one by one at the beginning of the registration list, and perform corresponding processing according to the monitoring results. For example, in a certain monitoring result, the first check thread judges that the response status of the current service node is abnormal according to the service request processing response time, and the checkpoint execution module can stop it from sending heartbeat confirmation information to the standby service node and add An abnormal node identification; for another example, in another monitoring result, the second check thread judges that the response status of the standby service node is abnormal according to the timeout of the storage unit's read and write log, and the checkpoint execution module is added for the standby service node The identifier of the abnormal node.

It should be noted that the checkpoint execution module does not need to pay attention to the logic of each check thread to determine how to implement it, that is, it does not need to pay attention to how the check thread specifically monitors whether the service node is abnormal, and only pays attention to the service node response reflected by the monitoring result Whether the status is abnormal. Specifically, whether the response status of the service node is abnormal can be represented by True and False, and the check thread may only register the value of True or False as the monitoring result in the registration list of the checkpoint execution module.

In addition, the result of dividing the number of the multi-threads by the preset response time threshold may be used as the frequency of monitoring the response state of the service node.

Because in actual applications, if the monitoring frequency is too low, the monitoring results reflecting abnormal service nodes may be omitted, and it is impossible to guarantee timely switching when the current service node is abnormal. Therefore, in order to improve the inspection accuracy, the monitoring execution interval cannot be greater than the preset response time threshold set by any inspection thread to determine whether the response status is abnormal. For example, if the preset response time threshold is 10 seconds, the monitoring execution interval can be set to 1 second. In order to facilitate the determination of the monitoring frequency, the number of inspection threads to be monitored can be divided by the preset response time threshold, and the result can be used as the monitoring frequency. For example, there are 10 inspection threads, and the preset response time threshold is 10 seconds, the monitoring frequency is 1 per second, that is, the checkpoint execution module can retrieve a monitoring result from the monitoring result registration list every 1 second to Perform the corresponding processing.

Those skilled in the art can apply the methods provided in the embodiments of the present invention to various distributed file systems and computing and storage platforms, such as HDFS (Hadoop Distributed File System) and ODPS computing platforms (Open Data Processing Service, Open data processing service), OSS storage platform (Object Storage Service, open object storage service), OTS storage platform (Open Table Service, open table service structured data service), ECS computing platform (Elastic Compute Service, elastic computing service), etc. Wait.

Compared with the current service node switching method, the embodiment of the present invention monitors the response status of the service node to the service request, and stops the communication between the current service node and the standby service node for the current service node whose response status is abnormal, thereby triggering the current Handover processing of the service node. Through the service node inspection logic, logical judgment and data statistics are made for various factors that affect the response status of the service node. In the event of service timeout, service unavailability, service abnormality, etc. caused by hardware failure or software defect, The autonomous switching and recovery of service nodes are realized, service availability is enhanced, service recovery efficiency is improved, and user experience is improved.

Secondly, the embodiment of the present invention can monitor one or more factors that affect the response status of the service node, and the multi-dimensional monitoring method improves the comprehensiveness and scalability of service recovery.

Further, the embodiment of the present invention does not directly initialize the abnormal current service node, but uses a relatively conservative manner of stopping communication with the standby service node to trigger the service node switch. When a misdiagnosis accident occurs, it is misdiagnosed as an abnormal response state. The current service node also has the opportunity to re-function as the new current service node and continue to provide services, avoiding the negative impact of misdiagnosis on the entire system.

2, there is shown a flow chart of the second embodiment of a service node switching method for a distributed storage system of the present invention. The service node includes a current service node and a standby service node. The method may specifically include the following steps:

Step 201: Obtain the time point T1 at which the service node recently took out the service request from the service request queue through the first check thread, and use the time interval D1 from the current time point N1 as the response time.

It should be noted that the above-mentioned first check thread may be a check thread that focuses on whether the response time of the service request queue is abnormal. When a user submits a service request to the current service node, the service request will be queued in the service request queue first, waiting to be processed by the current service node one by one. When the current service node takes out the service request from the queue, it can record the time point T1 at this time.

The first check thread can periodically check the service request queue. When there are service requests waiting to be processed in the service request queue, the first check thread obtains the previously recorded time point T1 at which the service request was recently taken out, and compares the current time point N1 with T1. The time interval D1 is used as the response time of the current service node.

Step 202: Determine whether the response time is greater than a first preset response time threshold, and if so, determine that the response state of the service node is abnormal.

The response time can be compared with the first preset response time threshold. If the response time is greater than the first preset response time threshold, it indicates that the current service node is blocked while processing the user's service request, causing the user to wait for a long time for service. Condition. Therefore, regardless of whether other aspects of the current service node are normal, it can also be considered that the response state of the current service node is abnormal.

In practical applications, the first preset response time threshold can be set to 10 seconds, that is, if the service request submitted by the user is not successfully responded within 10 seconds, it can be understood that the response status of the current service node is abnormal. Of course, Those skilled in the art can set the first preset response time threshold according to actual conditions, which is not limited in the embodiment of the present invention.

The first check thread may register the abnormal response state or the normal monitoring result in the monitoring result registration list of the checkpoint execution module, and the checkpoint execution module will take corresponding processing according to the monitoring result.

Step 203: If the response state of the current service node is abnormal, stop the communication between the current service node and the backup service node, and trigger the switching process of the current service node.

Step 204: If the response status of a certain service node is abnormal, add an abnormal node identifier for marking that it does not participate in the handover process of the current service node to the service node.

The checkpoint execution module can stop the communication with multiple standby service nodes for the current service node that responds to the abnormal state. For the current service node and the standby service node that respond to the abnormal state, an abnormal node identifier can also be added, and the service node carrying the identifier will not be switched to the new current service node during the switching process of the service node.

As a preferred example of the embodiment of the present invention, the step of triggering the handover processing of the current service node may include: triggering the selection of at least one service node that does not carry the abnormal node identifier as the new current service node, and replacing the one that responds to the abnormal state. The current service node.

In practical applications, if the current service node is elected through the Paxos protocol, the service node carrying the identifier of the abnormal node will not participate in the election. When the standby service node triggers the switching process of the current service node, the service node carrying the abnormal node identifier does not participate in the election, and will not be selected as the new current service node. If the service node is switched through the distributed lock service, the service node carrying the identifier of the abnormal node will not perform the lock grab operation, and only the normal service node participates in the lock grab.

The embodiment of the present invention adds an abnormal node identifier to the service node with an abnormal response state, which avoids the problem that the service node with an abnormal response state is selected as the current service node, thereby failing to achieve the purpose of switching the service node. Moreover, the abnormal service node is excluded, and the switching processing of the service node can ensure the stability of the new current service node, avoid system fluctuations caused by multiple service node switching, and improve the stability of service recovery.

3, there is shown a step flow chart of Embodiment 3 of a service node switching method for a distributed storage system of the present invention. The service node includes a current service node and a standby service node. The method may specifically include the following steps:

Step 301: Obtain the start and end time of log writing by the storage unit of the service node through the log recording thread, and use the time interval of the start and end time as the storage unit response time.

It should be noted that when a user submits a service request, a log will be generated. Both the current service node and the standby service node need to record the log in the storage unit through the logging thread, and then return a notification that the user's service request has been successfully processed. Response time directly affects the response time to service requests.

In a specific implementation, the log recording thread records the start and end time points when the service node starts to write the log and the end time point after the log writing is completed, and the time interval of the start time point is used as the storage unit response time.

Step 302: Determine whether the response time of the storage unit is greater than a third preset response time threshold, and if so, add the storage response timeout flag for the storage unit, and record the time point T2 when the flag is added accordingly.

It is determined whether the storage unit response time is greater than the third preset response time threshold. If it is, it indicates that the storage unit of the service node is abnormal, a storage response timeout flag can be added to the storage unit, and the time point T2 when the flag is added is recorded. If the storage unit of the service node already carries the storage response timeout identifier, there is no need to add the identifier.

Step 303: If the storage unit response time is less than the third preset response time threshold and the storage unit already carries the storage response timeout flag, delete the storage response timeout flag.

In practical applications, the inspection thread needs to pay attention to the abnormal situation of continuous slow response of the storage unit. The slow response of a single storage unit may be caused by accidental factors and can be temporarily ignored to avoid misdetection. Therefore, if the response time of the storage unit is less than the third preset response time threshold, and the storage response timeout flag is already carried, the flag can be deleted.

Step 304: Determine whether the storage unit of the service node carries a storage response timeout flag through a second check thread.

Step 305: If yes, use the time interval D2 between the time point T2 and the current time point N2 corresponding to the storage response timeout identifier as the response time, and if the response time is greater than the second preset response time threshold, It is determined that the response status of the service node is abnormal.

The second checking thread can judge whether the storage unit carries a storage response timeout flag, and perform corresponding processing according to the judgment result.

If the storage response timeout flag is carried, the time point T2 of adding the flag is obtained from the log recording thread, and the current time point N2 is subtracted to obtain the time interval D2 between the two as the response time of the service node. If the response time is greater than the second preset response time threshold, it indicates that it takes too long for the service node to record the log to the storage unit, which affects the response time to the service request. Therefore, regardless of whether other aspects of the current service node are normal, it can also be considered that the response state of the current service node is abnormal.

The second check thread registers the abnormal response state or normal monitoring results in the monitoring result registration list of the checkpoint execution module, and the checkpoint execution module takes corresponding processing according to the monitoring results. For example, if the second checking thread detects that the response time of the random access memory storage unit within 30 seconds has not decreased to less than 30 milliseconds, it can be determined that the storage unit of the service node is abnormal, resulting in an abnormal response state of the service node.

Step 306: If the response state of the current service node is abnormal, stop the communication between the current service node and the backup service node, and trigger the switching process of the current service node.

Step 307: If the response status of a certain service node is abnormal, add an abnormal node identifier for marking that it does not participate in the handover process of the current service node to the service node.

The checkpoint execution module can stop the communication with multiple standby service nodes for the current service node that responds to the abnormal state. For the current service node and the standby service node that respond to the abnormal state, an abnormal node identifier can also be added, and the service node carrying the identifier will not be switched to the new current service node during the switching process of the service node.

It should be noted that steps 301 to 303 can be performed in a loop, repeating statistics and comparisons of the response time of the storage unit to write logs, and specifically determine whether there is an abnormal situation in which the storage unit continuously responds slowly, and update the storage of the storage unit accordingly. Responding to the timeout identifier, so that the second check thread performs corresponding processing according to the identifier.

It should be noted that for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the described sequence of actions, because According to the embodiments of the present invention, certain steps may be performed in other order or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

4, there is shown a step flow chart of Embodiment 4 of a service node switching method for a distributed storage system of the present invention. The service node includes a current service node and a standby service node. The method may specifically include the following steps:

Step 401: Monitor the response status of the service node to the service request through multithreading.

It is possible to monitor whether the response status of the service node to the service request is abnormal through multiple inspection threads focusing on different aspects of the service node. Because in practical applications, the response status of the service node may be affected by many factors, whether it is a unilateral factor or the combined effect of multiple factors, when it affects the response status of the service node, it can be monitored in a targeted manner. Therefore, in order to monitor the service node more comprehensively and flexibly, the service node can be monitored through a combination of multiple threads. Of course, the number of threads and the specific combination of threads can be determined by those skilled in the art according to actual conditions.

As a preferred example of the embodiment of the present invention, the step 401 may specifically include the following sub-steps:

In sub-step S11, the time point T1 at which the service node recently took out the service request from the service request queue is obtained through the first check thread, and the time interval D1 from the current time point N1 is used as the response time.

In sub-step S12, it is determined whether the response time is greater than a first preset response time threshold, and if so, it is determined that the response state of the service node is abnormal.

In sub-step S13, a second check thread is used to determine whether the storage unit of the service node carries a storage response timeout flag.

Sub-step S14, if yes, add the time interval D2 between the time point T2 and the current time point N2 of the identifier corresponding to the stored response timeout identifier as the response time, if the response time is greater than the second preset response time threshold , It is determined that the response status of the service node is abnormal.

The aforementioned first check thread may be a thread that focuses on whether the response time of the service request queue is abnormal. By monitoring the service node by the first check thread, the abnormality of the service node caused by the too slow response time of the processing service request queue can be monitored. The above-mentioned second check thread may be a thread that focuses on whether the response time of the storage unit is abnormal. By monitoring the service node by the second check thread, the abnormal situation of the service node caused by the slow log writing of the storage unit can be monitored. It should be noted that the above sub-steps are not prioritized, that is, the first inspection thread and the second inspection thread can be used for monitoring at the same time.

Step 402: If the response status of the current service node is abnormal, stop the communication between the current service node and the standby service node, and trigger the switching process of the current service node.

Step 403: If the response status of a certain service node is abnormal, add an abnormal node identifier for marking that it does not participate in the handover process of the current service node to the service node.

Through the above-mentioned first inspection thread and second inspection thread at the same time to monitor the response status of the service node to the service request, the response time of the processing service request queue and the response time of the storage unit writing log can be monitored at the same time. It can trigger the handover processing of the service node, and add the abnormal node identifier in a targeted manner. Thus, the comprehensiveness and scalability of service recovery are improved through multi-dimensional monitoring methods.

5, there is shown a structural block diagram of Embodiment 1 of a service node switching device for a distributed storage system of the present invention. The service node includes a current service node and a backup service node. The device may specifically include the following modules :

The service node response status monitoring module 501 is used to monitor the response status of the service node to the service request.

The current service node switching trigger module 502 is configured to stop the communication between the current service node and the standby service node if the response state of the current service node is abnormal, and trigger the switching process of the current service node.

The embodiment of the present invention monitors the response status of the service node to the service request, and stops the communication between the current service node and the standby service node for the current service node whose response status is abnormal, thereby triggering the switching process of the current service node. Through the service node inspection logic, logical judgment and data statistics are made for various factors that affect the response status of the service node. In the event of service timeout, service unavailability, service abnormality, etc. caused by hardware failure or software defect, The autonomous switching and recovery of service nodes are realized, service availability is enhanced, service recovery efficiency is improved, and user experience is improved.

Secondly, the embodiment of the present invention can monitor one or more factors that affect the response status of the service node, and the multi-dimensional monitoring method improves the comprehensiveness and scalability of service recovery.

Further, the embodiment of the present invention does not directly initialize the abnormal current service node, but uses a relatively conservative manner of stopping communication with the backup service node to trigger the service node switching. When a misdiagnosis accident occurs, it is misdiagnosed as an abnormal response state. The current service node also has the opportunity to re-function as the new current service node and continue to provide services, avoiding the negative impact of misdiagnosis on the entire system.

6, there is shown a structural block diagram of the second embodiment of a service node switching device for a distributed storage system of the present invention. The service node includes the current service node and the standby service node. The device may specifically include the following modules :

The monitoring result registration module 601 is configured to register the monitoring result of at least one of the service nodes to the monitoring result registration list.

The service node response status monitoring module 602 is used to monitor the response status of the service node to the service request.

The current service node switching trigger module 603 is configured to stop the communication between the current service node and the standby service node if the response state of the current service node is abnormal, and trigger the switching process of the current service node.

The abnormal node identifier adding module 604 is configured to add an abnormal node identifier for marking the service node not to participate in the switching process of the current service node if the response state of a certain service node is abnormal.

The monitoring frequency determination module 605 is configured to divide the number of the multi-threads by the preset response time threshold as the frequency of monitoring the response state of the service node.

As a preferred example of the embodiment of the present invention, the device may further include: a storage unit response time determination module, which is used to obtain the start and end time of the storage unit writing log of the service node through a log recording thread, and to set the start and end The time interval is used as the response time of the storage unit.

The storage response timeout flag adding module is used to determine whether the storage unit response time is greater than the third preset response time threshold; if so, the storage response timeout flag is added to the storage unit, and the corresponding record adds the flag The time point T2.

The storage response timeout flag deletion module is configured to delete the storage if the storage unit response time is less than the third preset response time threshold and the storage unit already carries the storage response timeout flag Response timeout indicator.

As a preferred example of the embodiment of the present invention, the current service node switching trigger module 603 may include the following sub-modules: a trigger selection sub-module for triggering the selection of at least one service node that does not carry the abnormal node identifier as The new current service node replaces the current service node that responds to an abnormal state.

As a preferred example of the embodiment of the present invention, the service node response status monitoring module 602 may include the following sub-modules: a multi-thread monitoring sub-module for monitoring the response status of the service node to service requests through multiple threads .

As a preferred example one of the embodiment of the present invention, the multi-threaded monitoring sub-module may include the following sub-units: The first check thread sub-unit is used to obtain the service node from the service request queue by the first check thread Take out the time point T1 of the service request, and use the time interval D1 from the current time point N1 as the response time.

The first preset response time threshold judging subunit is configured to judge whether the response time is greater than the first preset response time threshold, and if so, determine that the response state of the service node is abnormal.

As a second preferred example of the embodiment of the present invention, the multi-threaded monitoring sub-module may include the following sub-units: a second check thread sub-unit for determining whether the storage unit of the service node carries storage through the second check thread Response timeout flag; if yes, call the second preset response time threshold judging subunit.

The second preset response time threshold judgment subunit is configured to add the time interval D2 between the time point T2 and the current time point N2 of the identifier corresponding to the storage response timeout identifier as the response time, if the response time is greater than the first 2. When the response time threshold is preset, it is determined that the response state of the service node is abnormal.

As a preferred example of the embodiment of the present invention, the service node response status monitoring module 602 may include the following sub-modules: a monitoring result polling sub-module for polling registration at the beginning of the monitoring result registration list Monitoring results.

The embodiment of the present invention adds an abnormal node identifier to the service node with an abnormal response state, which avoids the problem that the service node with an abnormal response state is selected as the current service node, thereby failing to achieve the purpose of switching the service node. Moreover, the abnormal service node is excluded, and the switching processing of the service node can ensure the stability of the new current service node, avoid system fluctuations caused by multiple service node switching, and improve the stability of service recovery.

As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

Those skilled in the art should understand that the embodiments of the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may adopt computer program products implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program codes. form.

In a typical configuration, the computer device includes one or more processors (CPUs), input/output interfaces, network interfaces, and random access memory. Random access memory may include non-permanent memory in computer-readable media, random access memory (RAM) and/or non-volatile random access memory, such as read-only memory (ROM) Or flash memory (flash RAM). Random access memory is an example of computer-readable media. Computer-readable media includes permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), and other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other random access memory technology, CD-ROM ), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic tape storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by computing devices. According to the definition in this article, computer-readable media does not include non-persistent computer-readable media (transitory media), such as modulated data signals and carrier waves.

The embodiments of the present invention are described with reference to the flowcharts and/or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing terminal equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal equipment A device for realizing the functions specified in a flow or multiple flows in the flowchart and/or a block or multiple blocks in the block diagram is generated.

These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing terminal equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including the instruction device , The instruction device realizes the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be loaded on a computer or other programmable data processing terminal equipment, so that a series of operation steps are executed on the computer or other programmable terminal equipment to generate computer-implemented processing, so that the computer or other programmable terminal equipment The instructions executed above provide steps for implementing functions specified in a flow or multiple flows in the flowchart and/or a block or multiple blocks in the block diagram.

Although the preferred embodiments of the embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concept. Therefore, the scope of the attached patent application is intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

Finally, it should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities. Or there is any such actual relationship or sequence between operations. Moreover, the terms "including", "including" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements not only includes those elements, but also includes those elements that are not explicitly listed. Other elements listed, or also include elements inherent to this process, method, article, or terminal device. Without more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, article, or terminal device that includes the element.

The foregoing provides a detailed introduction to the service node switching method of the distributed storage system and the service node switching device of the distributed storage system provided by the present invention. Specific examples are used in this article to illustrate the principle and implementation of the present invention. The description of the above embodiments is only used to help understand the method and core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation and the scope of application In summary, the content of this specification should not be construed as a limitation of the present invention.

Claims (16)

A method for switching a service node of a distributed storage system. The service node includes a current service node and a standby service node. The method includes: monitoring the response status of the service node to a service request, including: monitoring the service through multiple threads The response status of the node to the service request; and if the response status of the current service node is abnormal, stop the communication between the current service node and the standby service node, and trigger the switching process of the current service node; wherein, the method It also includes the frequency of monitoring the response state of the service node as a result of dividing the number of the multi-threads by the preset response time threshold. The method according to item 1 of the scope of patent application, further comprising: if the response status of a certain service node is abnormal, adding an abnormal node identifier for marking that it does not participate in the handover processing of the current service node to the service node. The method according to item 2 of the scope of patent application, wherein the step of triggering the handover processing of the current service node includes: triggering the selection of at least one service node that does not carry the abnormal node identifier as the new current service node, and replacing the response The current service node with abnormal status. The method according to item 1 of the scope of patent application, wherein the step of monitoring the response status of the service node to the service request through multi-threading The steps include: obtaining the time point T1 at which the service node recently took out the service request from the service request queue through the first checking thread, and taking the time interval D1 from the current time point N1 as the response time; judging whether the response time is greater than the first A preset response time threshold, if yes, it is determined that the response state of the service node is abnormal. The method according to item 1 of the scope of patent application, wherein the step of monitoring the response status of the service node to the service request through multiple threads includes: judging whether the storage unit of the service node carries storage through a second check thread Response timeout flag; if yes, add the time interval D2 between the time point T2 and the current time point N2 to the storage response timeout flag corresponding to the time interval D2 as the response time, if the response time is greater than the second preset response time Threshold, it is determined that the response status of the service node is abnormal. The method according to item 5 of the scope of patent application, wherein, before the step of monitoring the response status of the service node to the service request through multiple threads, the method further includes: obtaining the service node through a logging thread The storage unit writes the start and end time of the log, and uses the time interval between the start and end times as the response time of the storage unit; determine whether the response time of the storage unit is greater than the third preset response time threshold, and if so, add it for the storage unit The storage response timeout flag is stored, and the time point T2 when the flag is added is recorded accordingly. The method according to item 6 of the scope of patent application, further comprising: if the storage unit response time is less than the third preset response time threshold, and the storage unit already carries the storage response timeout flag , Delete the storage response timeout flag. The method according to item 1 of the scope of patent application, wherein, before the step of monitoring the response status of the service node to the service request, the method further includes: registering the monitoring result of at least one of the service nodes To the monitoring result registration list; the step of monitoring the response status of the service node to the service request is: polling the registered monitoring result at the start position of the monitoring result registration list. A service node switching device of a distributed storage system. The service node includes a current service node and a standby service node. The device includes: a service node response status monitoring module for monitoring the response status of the service node to a service request , Which includes: a multi-threaded monitoring sub-module for monitoring the response status of the service node to a service request through multi-threading; and a current service node switching trigger module for if the response status of the current service node is abnormal, Then stop the communication between the current service node and the standby service node, and trigger the switching process of the current service node; Wherein, the device further includes: a monitoring frequency determination module, configured to divide the number of the multi-threads by the preset response time threshold as the frequency of monitoring the response state of the service node. The device according to item 9 of the scope of patent application, wherein the device further includes: an abnormal node identification adding module, which is used to add an abnormal node identifier to the service node if the response status of a certain service node is abnormal. The identifier of the abnormal node participating in the handover process of the current service node. The device according to item 10 of the scope of patent application, wherein the current service node switching trigger module includes: a trigger selection sub-module for triggering the selection of at least one service node that does not carry the abnormal node identifier as a new The current service node replaces the current service node with an abnormal response state. The device according to item 9 of the scope of patent application, wherein the multi-thread monitoring sub-module includes: a first check thread subunit for obtaining the service node from the service request queue by the first check thread The time point T1 of the service request, and the time interval D1 from the current time point N1 as the response time; the first preset response time threshold judging subunit is used to judge whether the response time is greater than the first preset response time threshold, If yes, it is determined that the response status of the service node is abnormal. The device according to item 9 of the scope of patent application, wherein the multi-thread monitoring sub-module includes: The second check thread subunit is used for judging whether the storage unit of the service node carries the storage response timeout flag through the second check thread; if so, call the second preset response time threshold judging subunit; the second preset response The time threshold judging subunit is configured to add the time interval D2 between the time point T2 and the current time point N2 to the identifier corresponding to the storage response timeout identifier as the response time, if the response time is greater than the second preset response time Threshold, it is determined that the response status of the service node is abnormal. The device according to item 13 of the scope of patent application, wherein the device further includes: a storage unit response time determining module, configured to obtain the start and end time of the storage unit of the service node to write logs through a log recording thread, and The time interval between the start and end times is used as the storage unit response time; the storage response timeout flag adding module is used to determine whether the storage unit response time is greater than the third preset response time threshold, and if so, add it for the storage unit The storage response timeout flag is stored, and the time point T2 when the flag is added is recorded accordingly. The device according to item 14 of the scope of patent application, wherein the device further includes: a storage response timeout flag deletion module, configured to: if the storage unit response time is less than the third preset response time threshold, and If the storage unit already carries the storage response timeout flag, the storage response timeout flag is deleted. According to the device described in item 9 of the scope of patent application, the The device further includes: a monitoring result registration module for registering the monitoring result of at least one of the service nodes to a monitoring result registration list; the service node response status monitoring module includes: a monitoring result polling sub-module, Polling the registered monitoring result at the start position of the monitoring result registration list.

Images