CN110535692A - Fault handling method, device, computer equipment, storage medium and storage system - Google Patents

Fault handling method, device, computer equipment, storage medium and storage system Download PDF

Info

Publication number
CN110535692A
CN110535692A CN201910741190.5A CN201910741190A CN110535692A CN 110535692 A CN110535692 A CN 110535692A CN 201910741190 A CN201910741190 A CN 201910741190A CN 110535692 A CN110535692 A CN 110535692A
Authority
CN
China
Prior art keywords
malfunction
memory node
node
target
virtual machine
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910741190.5A
Other languages
Chinese (zh)
Other versions
CN110535692B (en
Inventor
陶毅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to CN201910741190.5A priority Critical patent/CN110535692B/en
Publication of CN110535692A publication Critical patent/CN110535692A/en
Priority to PCT/CN2020/102302 priority patent/WO2021027481A1/en
Application granted granted Critical
Publication of CN110535692B publication Critical patent/CN110535692B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06—Management of faults, events, alarms or notifications
    • H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

The invention discloses a kind of distributed memory system fault handling method, device, computer equipment, storage medium and storage systems, belong to troubleshooting technical field.The memory node that this method breaks down according at least one of the multiple memory node, determine the malfunction of the distributed memory system, without when the equal failure of all memory nodes, just determine the malfunction of distributed memory system, after having determined malfunction, malfunction can be sent by each memory node into distributed memory system immediately, so that each memory node carries out troubleshooting according to determining malfunction, restore the normal time used so as to reduce distributed memory system.

Description

Fault handling method, device, computer equipment, storage medium and storage system
Technical field
The present invention relates to troubleshooting technical field, in particular to a kind of distributed memory system fault handling method, dress It sets, computer equipment, storage medium and distributed memory system.
Background technique
With the development of big data technology, in order to store more data and prevent loss of data, distributed storage system System increasingly by the favor of enterprise, the memory node in distributed memory system with the increase of use time, unavoidably Will appear failure, when the memory node in distributed memory system breaks down, in order to guarantee the memory node of failure Normal business is not influenced, and the calculate node for providing business for user can carry out event to the failure in distributed memory system Barrier processing.
Wherein, troubleshooting can be following procedure: in distributed memory system, client is sent out to multiple memory nodes Small computer system interface (small computer system interface, SCSI) is sent to request；When distributed storage system When all memory nodes in system break down, if memory node occurs that recoverable failure in the short time, store Node will not respond SCSI request, and when the response of any memory node has not been obtained in client, then client deposits the distribution The malfunction of storage system is determined as abnormal (all path down, the APD) state of All Paths, and APD state is that VMWare is virtual A kind of malfunction for memory node that machine defines is asked for indicating that all paths of rear end memory node cannot respond to host It asks, client hangs untreated SCSI request, and technical staff is waited to repair the failure in distributed memory system；When storage saves When the failure of unrepairable in the short time occurs in point, memory node returns to the abnormal message of storage to client, then client terminates After the message for receiving storage exception, the malfunction of the distributed memory system is determined as permanent device and is lost (permanent device lost, PDL) state, wherein PDL state is a kind of memory node that VMWare virtual machine defines Malfunction can be led for indicating that rear end memory node is long-term or permanent fault since memory node breaks down for a long time The file system corruption in distributed memory system is caused, when the malfunction that client determines is PDL state, which will It is electric under file system, and technical staff is waited to repair failure.
In above-mentioned fault treating procedure, only when memory nodes all in distributed memory system break down, Client can just execute the process of above-mentioned troubleshooting, still, when event occurs in the part memory node in distributed memory system When barrier, the process of above-mentioned troubleshooting is not carried out in client.However, part memory node occurs in distributed memory system Failure is a kind of relatively common phenomenon, once part memory node breaks down, if not to depositing in distributed memory system It stores up node and carries out fault diagnosis, then client and can not determine whether the memory node in distributed memory system is faulty, from And technical staff cannot know the message of memory node failure from client in time, it also cannot be immediately to failure Memory node repaired, to extend the time that distributed memory system restores normal used.
Summary of the invention
The embodiment of the invention provides a kind of distributed memory system fault handling method, device, computer equipments, storage Medium and storage system can reduce distributed memory system and restore the normal time used.The technical solution is as follows:
In a first aspect, providing a kind of distributed memory system fault handling method, the distributed memory system includes Multiple memory nodes；This method comprises:
According to the memory node that at least one of the multiple memory node breaks down, the distributed storage is determined The malfunction of system；Can the malfunction is used to indicate at least one described memory node to break down pre- first If being all repaired in duration；
Into the multiple memory node, each memory node sends the malfunction.
Based on above-mentioned implementation, according to the memory node that at least one of the multiple memory node breaks down, The malfunction for determining the distributed memory system, without just determining distributed when the equal failure of all memory nodes The malfunction of storage system, after having determined malfunction, each storage section that can immediately into distributed memory system Point sends malfunction, so that each memory node carries out troubleshooting according to determining malfunction, so as to reduce point Cloth storage system restores the normal time used.
In a kind of possible implementation, the method also includes:
According to the malfunction of the distributed memory system, troubleshooting is carried out.
It is described to be deposited according to what at least one of the multiple memory node broke down in a kind of possible implementation Node is stored up, determines that the malfunction of the distributed memory system includes:
Determine at least one memory node to break down in the distributed memory system and it is described at least one deposit The target data that can not be accessed in storage node；
According at least one described memory node and the target data, the malfunction is determined.
In a kind of possible implementation, described at least one memory node according to and the target data, really Determining the malfunction includes:
When the number of at least one memory node is greater than the redundancy of the distributed memory system, and the target When the data volume of data meets preset condition, the malfunction is determined as Fisrt fault state, the Fisrt fault state Being used to indicate at least one described memory node can all be repaired in first preset duration.
Based on above-mentioned possible implementation, under Fisrt fault state, target device will not descend electric file system, once At least one memory node can be repaired in the first preset time, then also can to avoid lower electric file system, from And the time for repairing file system can be reduced, allow distributed memory system to restore business as early as possible, to guarantee Service Quality Amount.
In a kind of possible implementation, described at least one memory node according to and the target data, really Determining the malfunction includes:
When the number of at least one memory node is greater than the redundancy of the distributed memory system, and the target When the data volume of data meets preset condition, according to the fault scenes of the distributed memory system, the malfunction is determined, The fault scenes be used to indicate at least one described memory node whether and meanwhile break down.
In a kind of possible implementation, the preset condition includes any one of following:
Ratio between the data volume of the target data and the first preset data amount is greater than default ratio, and described first is pre- If data volume is the total amount of data of all data of distributed memory system storage；
The data volume of the target data is greater than the second preset data amount.
In a kind of possible implementation, described at least one memory node according to and the target data, really After the fixed malfunction, the method also includes:
When the malfunction is the Fisrt fault state, if in first preset duration it is described at least one Memory node is not repaired all, then the malfunction is updated to the second malfunction by Fisrt fault state, and described Two malfunctions are used to indicate at least one described memory node and cannot all be repaired in first preset duration.
In a kind of possible implementation, the fault scenes according to the distributed memory system determine described point Before the malfunction of cloth storage system, the method also includes:
According to the time that at least one described memory node breaks down, the fault scenes are determined.
In a kind of possible implementation, the time that described at least one memory node according to breaks down is determined The fault scenes include:
When at least one described memory node breaks down in target duration, the fault scenes are determined as Otherwise the fault scenes are determined as the second fault scenes, the Fisrt fault scene is used to indicate described by one fault scenes At least one memory node breaks down simultaneously, and second fault scenes, which are used to indicate at least one described memory node, to be occurred The time of failure is different.
In a kind of possible implementation, the fault scenes according to the distributed memory system determine described point The malfunction of cloth storage system includes:
If the fault scenes are Fisrt fault scene, according to each memory node at least one described memory node Fault type, determine the malfunction, the Fisrt fault scene is used to indicate at least one described memory node simultaneously It breaks down, can the failure that the fault type is used to indicate a memory node be repaired in the second preset duration；
If the fault scenes are the second fault scenes, according to what is broken down the latest at least one described memory node The fault type of first memory node, determines the malfunction, second fault scenes be used to indicate it is described at least one The time that memory node breaks down is different.
In a kind of possible implementation, the event of each memory node in described at least one memory node according to Hinder type, determines that the malfunction includes:
It, will when the fault type of each memory node at least one described memory node is Fisrt fault type The malfunction is determined as Fisrt fault state, and the failure that the Fisrt fault type is used to indicate a memory node can be It is repaired in second preset duration, the Fisrt fault state is used to indicate at least one described memory node can be described It is all repaired in first preset duration；
When the fault type of the memory node of target number at least one described memory node is the second fault type, If the target number is less than or equal to the redundancy of the distributed memory system, the malfunction is determined as described the Otherwise the malfunction is determined as the second malfunction, second fault type is used to indicate one by one malfunction The failure of memory node cannot be repaired in second preset duration, second malfunction be used to indicate it is described at least One memory node cannot be all repaired in first preset duration.
In a kind of possible implementation, break down the latest in described at least one memory node according to first The fault type of memory node determines that the malfunction includes:
When the fault type of first memory node is Fisrt fault type, the malfunction is determined as first Malfunction, the failure that the Fisrt fault type is used to indicate a memory node can be repaired in second preset duration Multiple, the Fisrt fault state is used to indicate at least one described memory node and can all be repaired in first preset duration It is multiple；
When the fault type of first memory node is the second fault type, then the malfunction is determined as the Two malfunctions, the failure that second fault type is used to indicate a memory node cannot be in second preset duration It is repaired, second malfunction is used to indicate at least one described memory node cannot be complete in first preset duration Portion is repaired.
In a kind of possible implementation, the fault scenes according to the distributed memory system determine described point Before the malfunction of cloth storage system, the method also includes:
For any memory node at least one described memory node, when any memory node appearance is preset Network failure, preset powered-off fault failure, preset maloperation failure, preset hardware fault or preset software fault When, the fault type of any memory node is determined as Fisrt fault type, otherwise, by the event of any memory node Barrier type is determined as the second fault type, and the failure that the Fisrt fault type is used to indicate a memory node can be described the It is repaired in two preset durations, the failure that second fault type is used to indicate a memory node cannot be pre- described second If being repaired in duration.
In a kind of possible implementation, the malfunction according to the distributed memory system is carried out at failure After reason, the method also includes:
When at least one described memory node reparation is completed, each equipment in Xiang Suoshu distributed memory system is sent It repairs and completes response, the reparation is completed response and is used to indicate in the distributed memory system without faulty equipment.
Second aspect, provides a kind of distributed memory system fault handling method, and the distributed memory system includes Multiple memory nodes；This method comprises:
Target storage node into the distributed memory system sends access request；
Receive the response that the target storage node returns；It include the failure of the distributed memory system in the response State；The malfunction be used to indicate at least one failure memory node can in the first preset duration whole quilts It repairs.
In a kind of possible implementation, after the response for receiving the target storage node return, the method Further include:
Based on the malfunction for including in the response, troubleshooting is carried out.
In a kind of possible implementation, the failure identification of the malfunction includes Fisrt fault mark or the second failure Any of mark, wherein the Fisrt fault mark is used to indicate Fisrt fault state, and second failure identification is used for Indicate the second malfunction, the Fisrt fault state is used to indicate at least one memory node can be complete in the first preset duration Portion is repaired, and second malfunction is used to indicate at least one described memory node cannot be in first preset duration It is all repaired, the memory node is the memory node to break down in the distributed memory system.
It is described based on the malfunction for including in the response in a kind of possible implementation, carry out troubleshooting packet It includes:
When the access request by the destination client in the distributed memory system based on target virtual machine send, and When the target virtual machine is VMWare virtual machine, if the malfunction is the Fisrt fault state, not to the target Virtual machine responds the access request, and the Fisrt fault state is used to indicate at least one memory node can be when first is default It is all repaired in long；
When the access request is sent by the destination client based on target virtual machine, and the target virtual machine is When VMWare virtual machine, if the malfunction is second malfunction, it is different that Xiang Suoshu target virtual machine returns to storage Normal message, second malfunction is used to indicate at least one described memory node cannot be in first preset duration All it is repaired.
It is described based on the malfunction for including in the response in a kind of possible implementation, carry out troubleshooting packet It includes:
When the access request by the destination client in the distributed memory system based on target virtual machine send, and When the target virtual machine is not VMWare virtual machine, if the malfunction is the Fisrt fault state, Xiang Suoshu target Virtual machine transmission retries request, it is described retry request and be used to indicate re-issue the access request, the Fisrt fault state Being used to indicate at least one memory node can all be repaired in the first preset duration；
When the access request is sent by the destination client based on target virtual machine, and the target virtual machine is not When VMWare virtual machine, if the malfunction is second malfunction, Xiang Suoshu target virtual machine returns to the target The identifiable target error of virtual machine, the target error are used to indicate storage media failure, and second malfunction is used for Indicate that at least one described memory node cannot be all repaired in first preset duration.
Based on above-mentioned possible implementation, different troubleshooting modes is provided, so that the present invention is implemented Troubleshooting mode provided by example has more universality.
In a kind of possible implementation, any memory node into distributed memory system sends access request Before, the method also includes:
Receive the target access request that the destination client in the distributed memory system is sent；
The target storage node into distributed memory system sends access request
It is requested based on the target access, the target storage node into distributed memory system sends the access and asks It asks.
In a kind of possible implementation, after the response for receiving the target storage node return, the method Further include:
It receives the reparation that target storage node returns and completes response, the reparation completes response and is used to indicate the distribution Without faulty equipment in storage system.
The third aspect, provides a kind of distributed memory system, and the distributed memory system includes supervisory node and more A memory node；
The supervisory node is used for:
The distributed storage is determined according to the memory node that at least one of the multiple memory node breaks down The malfunction of system；Can the malfunction is used to indicate at least one described memory node to break down pre- first If being all repaired in duration；
Into the multiple memory node, each memory node sends the malfunction；
Each of the multiple memory node memory node, for receiving the malfunction.
In one possible implementation, the failure identification of the malfunction includes Fisrt fault mark and the second event Any of barrier mark, the Fisrt fault mark are used to indicate the Fisrt fault state, and second failure identification is used In indicating the second malfunction, the Fisrt fault state is used to indicate at least one described memory node can be pre- described first If being all repaired in duration, second malfunction is used to indicate at least one described memory node cannot be described first It is all repaired in preset duration.
In one possible implementation, each of the multiple memory node memory node is also used to connect After receiving the malfunction, if receiving the access request again, the access request is hung, is based on received failure shape State carries out troubleshooting.
Fourth aspect provides a kind of fault treating apparatus, for executing above-mentioned distributed memory system troubleshooting side Method.Specifically, which includes any optional side for executing above-mentioned first aspect or above-mentioned first aspect The functional module for the fault handling method that formula provides.
5th aspect, provides a kind of fault treating apparatus, for executing above-mentioned distributed memory system troubleshooting side Method.Specifically, which includes any optional side for executing above-mentioned second aspect or above-mentioned second aspect The functional module for the fault handling method that formula provides.
6th aspect, provides a kind of computer equipment, which includes processor and memory, in the memory It is stored at least one instruction, which is loaded by the processor and executed to realize such as above-mentioned distributed memory system failure Operation performed by reason method.
7th aspect, provides a kind of storage medium, at least one instruction is stored in the storage medium, the instruction is by handling Device is loaded and is executed to realize the operation as performed by above-mentioned distributed memory system fault handling method.
Detailed description of the invention
To describe the technical solutions in the embodiments of the present invention more clearly, make required in being described below to embodiment Attached drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the invention, for For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings other Attached drawing.
Fig. 1 is a kind of schematic diagram of distributed memory system provided in an embodiment of the present invention；
Fig. 2 is a kind of schematic diagram of the network environment of distributed memory system provided in an embodiment of the present invention；
Fig. 3 is the interaction schematic diagram in a kind of distributed memory system provided in an embodiment of the present invention between each equipment
Fig. 4 is a kind of structural schematic diagram of computer equipment provided in an embodiment of the present invention；
Fig. 5 is a kind of flow chart of distributed memory system fault handling method provided in an embodiment of the present invention；
Fig. 6 is a kind of flow chart of distributed memory system fault handling method provided in an embodiment of the present invention；
Fig. 7 is a kind of structural schematic diagram of fault treating apparatus provided in an embodiment of the present invention；
Fig. 8 is a kind of structural schematic diagram of fault treating apparatus provided in an embodiment of the present invention；
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, below in conjunction with attached drawing to embodiment party of the present invention Formula is described in further detail.
Fig. 1 is a kind of schematic diagram of distributed memory system provided in an embodiment of the present invention, and referring to Fig. 1, which is deposited Storage system includes at least one client 101, multiple memory nodes 102 and supervisory node 103.Wherein, client 101 is used for The business of data storage and reading data is provided for user, that is to say, the data that client 101 can upload user store In memory node 102, data can also be read from memory node 102.
Memory node 102 is also used to return to 101 returned data of client for storing the data of the write-in of client 101 The data returned can be client 101 and request the data read, can also return to what supervisory node 103 issued to client 101 The malfunction of distributed memory system, so that client 101 is according to the malfunction of the distributed memory system, processing occurs Failure.
Whether supervisory node 103 breaks down for memory node 102 each in monitoring distributed storage system, when point It, may shadow when the number of the memory node to break down in cloth storage system is higher than the redundancy of the distributed memory system The normal operation for the business of arriving is rung, therefore, which can be according to the storage section to break down in distributed memory system Point determines the malfunction of the distributed memory system, and determining malfunction is issued to all memory nodes 102, Memory node 102 it is possible to notify that 101 distributed memory system of client malfunction so that client 101 Or memory node 102 can carry out troubleshooting according to the malfunction of the distributed memory system.
The baseboard management controller (baseboard management controller, BMC) of each memory node 102 It can monitor whether respective memory node breaks down in real time, any storage section at least one memory node 102 When the BMC of point monitors any memory node failure, which can store the original that any memory node breaks down Because and break down time, can also by any memory node break down the reason of and failure time send out Supervisory node 103 is given, so that supervisory node 103 can know whether any memory node breaks down.
When the reason of the BMC of memory node does not break down memory node and the time of failure is sent to supervision When node 103, supervisory node 103 can be by accessing each memory node 102, to know whether each memory node 102 is deposited In failure, failure, then can obtain the time of the reason of breaking down and failure from the memory node of failure if it exists.
The reason of failure that the memory node no matter supervisory node 103 is reception failure is sent and failure Time, still initiatively from the memory node of failure obtain send failure the reason of and failure time, should The time of the reason of supervisory node 103 can break down to the memory node of failure and failure stores, with After an action of the bowels continue can according to the memory node of failure send failure the reason of and failure time, determine failure classes Type and fault scenes, wherein to the description of fault type see below in step 603, to the descriptions of fault scenes referring to Following steps 602.
The time of the reason of supervisory node 103 can break down to the memory node of failure and failure carries out The time of the reason of storage may include: the failure by the memory node transmission of failure and failure is stored in failure In table, which can store number, the mark of memory node, fault time and failure cause, wherein number is for referring to Show which memory node to break down, the mark of a memory node is for uniquely indicating a memory node, the mark The Internet protocol address (internet protocol address, IP) that can be memory node, can be with memory node Media access control address (media access control address, MAC), can also be the memory node in the distribution Number in formula storage system, the embodiment of the present invention are not specifically limited the mark of the memory node.In addition, fault time is The time that memory node breaks down, failure cause are the reason of memory node breaks down.
Such as bug list shown in table 1, as known from Table 1, the memory node to break down in current distributed memory system There are 2, be respectively identified as the memory node of X and be identified as the memory node of Y, wherein the memory node for being identified as X occurs The time of failure is A, is D the reason of failure；The time for being identified as the memory node failure of Y is B, failure Reason is C.
Table 1
Number The mark of memory node Fault time Failure cause
01 X A D
02 Y C E
It should be noted that the supervisory node can be in the event after memory node of the record in bug list is repaired Hinder in table, the relevant information for being repaired the memory node of completion is deleted, so that supervisory node can be according to last in bug list The number of a memory node determines the number of the memory node to break down in distributed memory system.
When the memory node in distributed memory system breaks down, the file in distributed memory system may cause The damage of system, when file system corruption, the metadata in file system is likely to occur mistake, due to metadata refer to for The distribution of data block in the system data of the feature of one file, such as access authority, file owner and file is described Information etc., when metadata when the error occurs, data block in file indicated by metadata possibly can not be accessed.
In some implementations, when client accesses the failure of the data block in any memory node, which can be incited somebody to action The Data Identification of the data volume and unique identification of the data block data block is sent to supervisory node, which receives After the data volume and Data Identification, the data volume and Data Identification that receive are stored, specifically can store In tables of data, which can be used for storing data total amount, Data Identification and data volume corresponding with each Data Identification, Wherein, total amount of data is the data volume for all data that can not be currently accessed in distributed memory system.
Such as tables of data shown in table 2, as known from Table 2, the data that cannot be currently accessed in distributed memory system are Data in data block indicated by data and Data Identification N in data block indicated by Data Identification M, wherein current The total amount of data for the data that cannot be accessed is 30 kilobytes (kilobyte, KB), and the data that cannot be currently accessed include number The data of 20KB in data block indicated by data and Data Identification N according to 10KB in data block indicated by mark M.
Table 2
It should be noted that when client accesses the data block that cannot be accessed again, if the client can visit Ask success, then the client sends the access data success response for carrying the Data Identification of the data block, the visit to supervisory node Ask that the data that data success response is used to indicate in the data block can be accessed, when receiving the access data success response Afterwards, in the tables of data, which can delete the corresponding data volume of the Data Identification, and the data in more new data table Total amount.For example, the access data success response carry Data Identification M, then the supervisory node delete tables of data in Data Identification M Relevant information, and total amount of data is updated to 20KB.It should be noted that can also be stored in tables of data and Data Identification The mark of corresponding memory node cannot be accessed with that data block indicated in which memory node.
In some embodiments, since the responsible portfolio of distributed memory system is bigger, client 101 and storage section The number of point 102 may be relatively more, in order to facilitate the data transmission between client 101 and memory node 102, client 101 At least one service switch has can be set in the application layer at place, with the interaction between client 101 and memory node 102. For the ease of the data transmission between memory node 102, at least one can be set in the accumulation layer where memory node 102 and deposit Interchanger is stored up, to realize the interaction between each memory node 102.For the ease of between supervisory node 103 and memory node 102 Data transmission, supervision switch can be set, to realize the interaction between supervisory node 103 and memory node 102.
From above description it is found that in the distributed memory system, other than needing to provide business service, it is also necessary to Monitoring service is provided, different services can be realized by different networks.In order to realize client, memory node with And the network connection between supervisory node, at least one can be installed in the client, memory node and supervisory node Network interface, at least one network interface can be used for connecting different networks, and different networks can transmit the data of different services, should At least one network interface can be the service network port for connecting business network, the supervision network interface for connecting supervisory network, and connection respectively The BMC network interface of BMC network.
In order to illustrate the network environment in distributed memory system, shown in Figure 2 provided in an embodiment of the present invention one The schematic diagram of the network environment of kind of distributed memory system, in the distributed memory system network may include business network, Supervisory network and BMC network.
Wherein, business network is used network, example when being used for heartbeat, data synchronization and mirror image between memory node Such as, when the data block stored in memory node 11 is synchronized to memory node 2, memory node 1 can pass through service network port, In In the business network, data block 1 is sent to memory node 2, then memory node 2 can receive by the service network port of oneself It is stored in memory node 2 to data block 1, and by data block 1.
Supervisory network is whether monitoring memory node breaks down and carry out used network when information inquiry, is supervising In managed network, the malfunction for the distributed memory system that supervisory node issues can be transmitted, failure can also be inquired Memory node.In some possible embodiments, supervisory node can by supervise supervision network interface of the network interface to memory node, In supervisory network, the malfunction of distributed memory system is sent, memory node can be by the supervision network interface of oneself from prison The malfunction that supervisory node is received in managed network, when the service request that memory node receives client again (that is to say hereafter Middle SCSI request) after, received service request can not be handled, returns to the malfunction issued, to client directly so as to visitor Family end can carry out corresponding troubleshooting according to malfunction.
BMC network is the network for managing BMC, and supervisory node can monitor BMC by the BMC network interface of the access BMC network State can determine whether memory node faulty according to the state of the BMC of monitoring.It should be noted that BMC network is Optional network can not monitor the whether faulty of memory node in some embodiments by BMC, but can be with By other means, whether faulty to monitor memory node, therefore, the BMC can also be not provided in distributed memory system Network, directly by supervisory network, to realize monitoring.
When the memory node in distributed memory system is equal and business network, supervisory network and BMC are connected to the network, prison Pipe node then can from three net in real-time reception memory node status information, for example whether the information of failure.
In order to further illustrate interactive process between client, memory node and supervisory node, sheet shown in Figure 3 Interaction schematic diagram in a kind of distributed memory system that inventive embodiments provide between each equipment, in Fig. 3, a storage At least one object storage (object storage device, OSD) process, SCSI treatment progress can be installed in node And monitoring nodes service (node monitor service, NMS) agent process.
Wherein, an OSD process can correspond to one or more storage mediums in memory node for storing data, should Storage medium can be hard disk, and OSD process is used to manage the access request for one or more storage mediums, and access request is used Data to be processed are handled in instruction, wherein carrying out processing to data to be processed may include that reading is described at least The data block stored in one or more storage mediums, data block to be processed includes data to be processed, to number to be processed It can further include that data to be processed are written at least one or more described storage medium according to processing is carried out, when access request can When using SCSI to send, which can be considered as SCSI request.
The SCSI request that SCSI treatment progress is sent for obtaining client from business network, and convert and decompose SCSI Request obtains multiple SCSI requests, and the request of multiple SCSI is issued to corresponding OSD process.For example, SCSI request is taken Logical block addresses (logical block address, LBA) with data to be read are 100-200, due to LBA 100- In the storage medium 1 of memory node 1, storage location indicated by LBA 151-200 is storing storage location indicated by 150 In the storage medium 2 of node 1, then SCSI request can be converted and be decomposed into 2 SCSI requests by SCSI treatment progress, In, SCSI request 1 is used to indicate request and reads the data stored at LBA 100-150 in storage medium 1, SCSI request 2 It is used to indicate request and reads the data stored at LBA 151-200 in storage medium 2, so that SCSI treatment progress can be by SCSI Son request 1 is sent as the corresponding OSD process of storage medium 1, and SCSI request 2 is sent as the corresponding OSD of storage medium 2 Process.
One NMS agent process is used to receive the malfunction for the distributed memory system that supervisory node issues, and to one All OSD processes of a memory node issue received malfunction.For example, client is sent out by supervisory network to memory node The malfunction of distributed memory system is sent, the NMS agent process in memory node obtains supervisory node hair from supervisory network The malfunction sent, and the malfunction that will acquire is issued in each OSD process in the memory node, when any OSD into After journey receives malfunction, if receive again SCSI treatment progress transmission SCSI request or SCSI request, directly to The malfunction that SCSI treatment progress transmits and receives, to install the equipment of SCSI treatment progress malfunction based on the received, Carry out troubleshooting.
It should be noted that in some embodiments, SCSI treatment progress is not installed in memory node, but pacify In client, the embodiment of the present invention is not specifically limited the equipment for installing the SCSI treatment progress.For example, client The LBA that SCSI request in SCSI treatment progress carries data to be read is 0-100, the storage as indicated by LBA 0-50 Position is in memory node 1, and storage location indicated by LBA 51-100 is in memory node 2, then SCSI treatment progress can be with 2 SCSI requests are converted and be decomposed into SCSI request, wherein SCSI request 1 is used to indicate request and reads memory node 1 The data stored at middle LBA 0-50, SCSI request 2 are used to indicate request and read storage at LBA 51-100 in memory node 2 Data, so that SCSI treatment progress can request SCSI 1 to send as the OSD process in memory node 1, by SCSI Request 2 is sent as the OSD process in memory node 2.
Client, memory node and supervisory node may each be computer equipment, in order to further illustrate computer is set Standby hardware configuration, a kind of structural schematic diagram of computer equipment provided in an embodiment of the present invention shown in Figure 4, computer Equipment 400 includes that can generate bigger difference because configuration or performance are different, may include one or more processors (central processing units, CPU) 401 and one or more memory 402, wherein the memory 402 In be stored at least one instruction, this at least one instruction by the processor 401 load and execute with realize under troubleshooting The method that embodiment of the method provides.Certainly, the computer equipment 400 can also have wired or wireless network interface, keyboard with And the components such as input/output interface, to carry out input and output, which can also set including other The component of standby function, this will not be repeated here.
In the exemplary embodiment, a kind of computer readable storage medium is additionally provided, the memory for example including instruction, Above-metioned instruction can be executed by the processor in terminal to complete the fault handling method in following embodiments.For example, the computer Readable storage medium storing program for executing can be read-only memory (read-only memory, ROM), random access memory (random Access memory, RAM), CD-ROM (compact disc read-only memory, CD-ROM), tape, floppy disk and Optical data storage node etc..
In embodiments of the present invention, supervisory node can be according to the memory node of failure, to determine distributed storage Determining malfunction is simultaneously issued to storage section all in distributed memory system by the malfunction of system, supervisory node Point；Client after the read request or write request for receiving user, can into distributed memory system memory node hair SCSI request is sent, for completing the read request or write request of user；It is any after any memory node receives SCSI request Memory node wouldn't handle received SCSI request, and return to received malfunction to client, so that client can be with base In the malfunction that any memory node returns, troubleshooting is carried out.And in some embodiments, any memory node can also be with Troubleshooting is carried out based on malfunction, in one possible implementation, when any memory node receives supervisory node When the malfunction of transmission, if receiving the SCSI request of client transmission again, any memory node can be according to failure shape State carries out troubleshooting.
In order to further illustrate the above process, deposited referring to a kind of distribution provided in an embodiment of the present invention as shown in Figure 5 The flow chart of storage system fault handling method, this method specifically include:
501, supervisory node determine at least one memory node to break down in distributed memory system and this at least The target data that one memory node can not be accessed.
The target data can also be data that are storing in the distributed memory system and cannot being accessed, the number of targets According to can also be only data that can not be accessed that at least one memory node is stored.The embodiment of the present invention is with the number of targets According to being that the data instance that cannot be accessed at least one memory node is illustrated.
The supervisory node can determine at least one memory node and the target data, In by way of inquiry In a kind of possible implementation, which can pass through the 8th preset duration with every, inquire bug list and tables of data, from The memory node to break down, the data that determination can not be accessed from tables of data are determined in the bug list.8th it is default when Length can be 10 minutes or 1 hour, and the embodiment of the present invention is not specifically limited the 8th preset duration.
It should be noted that above to from the bug list determine break down memory node mode and from The mode for the data that determination can not be accessed in tables of data is described, and this will not be repeated here.
It, can also be according at least one memory node and the target after supervisory node has executed this step 501 Data determine malfunction, then, determine at least one memory node to break down in distributed memory system and this extremely The target data that a few memory node can not be accessed；According at least one memory node and the target data, determine The process of malfunction, that is to say, according to the memory node that at least one of the multiple memory node breaks down, determine The process of the malfunction of the distributed memory system.Wherein, occurred according at least one of the multiple memory node The memory node of failure determines that the process of the malfunction of the distributed memory system can be by shown in following step 502 Process realize.
502, when the number of at least one memory node is greater than the redundancy of the distributed memory system, and the number of targets According to data volume meet preset condition when, the supervisory node malfunction of the distributed memory system is then determined as first therefore Barrier state.
The redundancy is the redundancy of the data stored in distributed memory system, be that is to say in the distributed memory system The copy number of the data of storage, can the malfunction of distributed memory system is used to indicate at least one memory node It is all repaired in first preset duration, malfunction may include any in Fisrt fault state and the second malfunction It is a, wherein the second malfunction is used to indicate at least one memory node and cannot all be repaired in first preset duration It is multiple.It that is to say, when at least one memory node can be repaired in the first preset duration, it is believed that distributed storage Failure in system can repair in a short time, and distributed memory system is in Fisrt fault state, the Fisrt fault state It that is to say node short trouble (transient node down, TND) state.When at least one memory node can be When being repaired in one preset duration, then it is assumed that the failure in distributed memory system cannot repair in a short time, distributed Storage system is in the second malfunction, the second malfunction that is to say node chromic trouble (permanent node down, PND) state.First preset duration can be 20 minutes or 2 two hours, the embodiment of the present invention to first preset duration not It is specifically limited.
When the data stored in distributed memory system are more and the number of memory node is also more, if distributed storage The number of the memory node to break down in system is less than the redundancy of the distributed memory system, in the distributed memory system The memory node of non-failure can reconstruct the number that cannot be accessed in the memory node of failure according to the data that can be accessed According to therefore, when the number of the memory node to break down in distributed memory system is less than the redundancy of the distributed memory system When spending, the regular traffic of the distributed memory system is not interfered with, then, it is also just not necessarily to repair the memory node of failure.But It is, if the number of the memory node to break down is greater than the redundancy of the distributed memory system, in the distributed memory system Without failure memory node can not be according to the data that can be accessed, reconstructing in the memory node of failure cannot be accessed Data, so that normal business in distributed memory system may be influenced whether.And consider distributed memory system It is also traffic affecting key factor that can the data of interior storage, which be accessed, when the mesh that can not be accessed in distributed memory system When marking overabundance of data, the influence to business provided by distributed memory system is also just bigger, may influence whether business It operates normally, therefore, which can first determine the malfunction of distributed memory system, so as to quickly according to event Barrier state carries out troubleshooting, to reduce the influence degree to business to greatest extent.And when the target data it is less when, It is also relatively small to service impact, the normal operation of business may not be influenced, in order to be that user persistently provides industry Business, the supervisory node can wouldn't carry out troubleshooting.
Supervisory node by preset condition, can determine whether the data volume of target data can influence the normal of business Operation, the preset condition may include any one of following: the ratio between the data volume of target data and the first preset data amount Greater than default ratio, which is the total amount of data of all data of distributed memory system storage；The mesh The data volume for marking data is greater than the second preset data amount.Ratio between the data volume and the first preset data amount of target data When greater than default ratio, or when the data volume of the target data is greater than the second preset data amount, illustrate the number of target data It is bigger according to measuring, the normal operation of business may be also influenced whether, so that the data volume when target data meets preset condition When, which first can set Fisrt fault state for the malfunction of distributed memory system, so as to quick According to malfunction, troubleshooting is carried out, to reduce the influence degree to business.If this at least one more than the first preset time A memory node is not repaired all and is completed, then the malfunction can be updated to the second malfunction again.It should be noted that The default ratio can be 0.4,0.5 or 0.6, the embodiment of the present invention to this preset ratio, the first preset data amount and Second preset data amount is not specifically limited.
It, should in order to which the number of at least one memory node compares with the redundancy of the distributed memory system Supervisory node can inquire the bug list that the supervisory node is stored, and the number of at least one memory node is determined from bug list Mesh.In order to determine whether the data volume of target data meets preset condition, which can inquire stored number According to table, from the data volume for determining the target data that can not be accessed at present in the distributed memory system in tables of data.For example, supervision Node can determine that target data includes the data and number of 10KB in data block indicated by Data Identification M by inquiry table 2 According to the data of 20KB in data block indicated by mark N.
It should be noted that being determined to from bug list in the distributed memory system when introducing bug list above The process of the number of the memory node of interior failure is described, here, the embodiment of the present invention is determined to from bug list The process of the number of the memory node to break down in the distributed memory system does not repeat them here.
It should be noted that process shown in this step 502 that is to say according at least one memory node and described Target data determines the process of the malfunction of the distributed memory system.
503, all memory nodes transmission of the supervisory node into the distributed memory system is used to indicate Fisrt fault The Fisrt fault of state identifies.
The failure identification of malfunction includes any of Fisrt fault mark and second failure identification, wherein first Failure identification is used to indicate Fisrt fault state, and the second failure identification is used to indicate the second malfunction, Fisrt fault mark Can be different with the second failure identification, for example, Fisrt fault mark can be s, the second failure identification can be t, and the present invention is real Example is applied to be not specifically limited the representation of Fisrt fault mark and the second failure identification.
The supervisory node can send Fisrt fault mark to the NMS agent process of each memory node, to realize Fisrt fault mark is sent to all memory nodes, to inform the current failure shape of all memory node distributed memory systems State is Fisrt fault state.
It should be noted that process shown in this step 503 that is to say that supervisory node includes to distributed memory system Each memory node sends the process of the malfunction in multiple memory nodes.
It should be noted that in some embodiments, supervisory node, can also be according to after having determined malfunction The malfunction of storage system, carries out troubleshooting, and the process that the embodiment of the present invention carries out troubleshooting to supervisory node is not done It is specific to limit.
504, the target storage node in the distributed memory system receives Fisrt fault mark.
The target storage node is any memory node in the distributed memory system, every in the target storage node A OSD process can obtain Fisrt fault mark from the NMS agent process of the target storage node, thus target storage section The available Fisrt fault mark of each OSD process of point.It should be noted that each in the distributed memory system Memory node can carry out this step 504, can receive Fisrt fault mark for the memory node of failure, Fisrt fault mark may not received.
505, target device sends access request to the target storage node,
The access request, which is used to indicate, reads the data that are stored of the target storage node or to the target storage node Data are written.The target device is the equipment for being equipped with SCSI treatment progress, can be the destination client, can also be target Memory node, wherein destination client is any client in the distributed memory system.This step 506 can be by target SCSI treatment progress in equipment is realized.
Before 505 before this step, the destination client in distributed memory system can send mesh to target device Access request is marked, target access request is handled first object data for executing, and wherein first object data include Data indicated by the access request, target access request can carry target storage address, which can be with It is the storage address of first object data.The target virtual machine that can be installed by destination client send target access request Lai. Specifically, the target virtual machine can the SCSI treatment progress into the target device send the target access request.Target is empty Quasi- machine sends target access request to SCSI treatment progress and can be triggered by the movement of user, for example, when user is in client Interface in input data to be read storage address, and click when reading button, trigger the target virtual machine in client Target access request is sent to SCSI treatment progress, to request to read the data of the storage address storage of user's input.
Then, which receives the target access request that the destination client in distributed memory system is sent, this Step 506 can lead to following manner to realize: target device is requested based on the target access, the mesh into distributed memory system It marks memory node and sends access request.Specifically, after SCSI treatment progress receives target access request, according to target Location is converted and is decomposed to target access request, obtains multiple access requests, each access request can carry the target Partial address in address, this partial address can be the storage that any OSD process in the target storage node is managed and be situated between Offset address in matter, so that OSD process of the SCSI treatment progress into target storage node sends corresponding access request, it will The process of access request conversion access request that is to say the process of conversion SCSI request above-mentioned.
506, after the target storage node in the distributed memory system receives Fisrt fault mark, if the target Memory node receives access request again, which hangs the access request, sends first event to target device Barrier mark.
This step 506 can be realized by receiving the OSD process of the access request in the target storage node.When the target After memory node receives Fisrt fault mark, illustrate that the target storage node has known in the distributed memory system have The memory node of failure, and current malfunction is Fisrt fault state, due to the storage in current distributed memory system Node breaks down, then the target storage node can hang the access request, wouldn't handle the access request, waits failure Memory node be automatically repaired or repaired manually.
In order to enable target device can also know the malfunction of distributed memory system, then the target storage node is connecing When receiving the access request of target device transmission, which can identify output for the Fisrt fault and set to target It is standby.Specifically, the access that any OSD process in the target storage node receives the transmission of target device SCSI treatment progress is asked After asking, any OSD process sends Fisrt fault mark to the SCSI treatment progress.Certainly, in some embodiments, supervision section After point has determined the malfunction of the distributed memory system, malfunction can also be transmitted directly to target device, so as to The malfunction of the available distributed memory system of target device, also just without sending event to target device by memory node Barrier state.
It should be noted that process shown in this step 506 that is to say when the target storage in the distributed memory system After node receives the failure identification, if the target storage node receives access request again, the process of failure identification is exported.
507, target device receives the Fisrt fault mark that the target storage node is returned based on the access request.
This step 507 to be realized by the SCSI treatment progress in the target device, process shown in this step 507 namely It is the process for receiving the failure identification that the target storage node is returned based on the access request.When any OSD process is to the SCSI Treatment progress sends the Fisrt fault state, which can receive Fisrt fault mark.It should be noted that Process shown in this step 507, that is to say, receive the response that the target storage node returns；Comprising described in the response The malfunction of distributed memory system；Can the memory node that the malfunction is used to indicate at least one failure The process being all repaired in first preset duration, wherein include the failure shape of the distributed memory system in the response State that is to say failure identification.
508, target device is identified based on received Fisrt fault, carries out troubleshooting.
This step 508 can be executed by the SCSI treatment progress installed in target device, when the SCSI treatment progress connects After receiving Fisrt fault mark, which can will be identified based on Fisrt fault, troubleshooting be carried out, so as to mesh It is able to carry out in mark client and correspondingly handles.
In some embodiments, docking target virtual machine with the SCSI treatment progress in destination client may be VMWare Virtual machine, it is also possible to it is not VMWare virtual machine, since the mode that different virtual machine target devices carries out troubleshooting is different, Therefore, this step 508 can be realized by formula either in following manner 1-2.
Mode 1, when the access request by the destination client in the distributed memory system based on target virtual machine send, And the target virtual machine is when being VMWare virtual machine, if the malfunction is the Fisrt fault state, SCSI treatment progress not to The target virtual machine responds the access request.
When SCSI treatment progress receives Fisrt fault mark, illustrate that distributed memory system is in Fisrt fault shape State, for VMWare virtual machine, Fisrt fault state that is to say ADP state.And due to the received visit of SCSI treatment progress Ask that request is what the target virtual machine was sent, then, in order to enable VMWare virtual machine can perceive distributed memory system APD state in oneself definition, SCSI treatment progress is not responding to target virtual machine, and because owns in distributed memory system As long as SCSI treatment progress issue access request to OSD process, can receive Fisrt fault mark, and handle access request Each SCSI treatment progress will not respond target virtual machine, so as to all links in simulation distribution formula storage system without Response (DOWN) is not received by the response of SCSI treatment progress for target virtual machine, and target virtual machine will continue Access request is sent, thus even if when memory node in distributed memory system is without whole failures, it can also basis The malfunction that VMWare virtual machine defines carries out troubleshooting.
Mode 2, when the access request is sent by the destination client based on target virtual machine, and the target virtual machine is not When VMWare virtual machine, if the malfunction is the Fisrt fault state, SCSI treatment progress is sent to the target virtual machine Request is retried, this, which retries request and is used to indicate, re-issues the access request.
Retrying the detection keyword carried in request can be Unit Attention (0x6) error code, Unit Attention (0x6) error code can indicate that storage medium or link state in memory node are changed, and that is to say out Failure is showed, which receives after this retries request, and target virtual machine re-issues one to the SCSI treatment progress A access request, for realizing the processing mode in aforesaid way 1.
For different virtual machines, the embodiment of the invention provides different troubleshooting modes, so that this hair Troubleshooting mode provided by bright embodiment has more universality.
It should be noted that process shown in this step 508 that is to say based on the malfunction for including in response, event is carried out Hinder the process of processing.
If 509, at least one memory node is all repaired in first preset duration, supervisory node is to the distribution Each equipment in storage system, which sends to repair, completes response, which completes response and be used to indicate in the distributed memory system There is no faulty equipment.
Each equipment includes memory node and client.When at least one storage section in first preset duration Point all repair when, illustrate in distributed memory system at this time without faulty equipment, be used for if the supervisory node stores this The Fisrt fault mark of Fisrt fault state is identified, which can delete Fisrt fault mark, and to the distribution Each equipment in storage system, which sends to repair, completes response, has informed in each equipment distributed memory system without failure Equipment can work normally, then after each equipment receives reparation completion response, received failure mark before deletion Know, and can start to work normally.It should be noted that process shown in step 509 that is to say at least one storage section when this When point repairs completion, each equipment into the distributed memory system sends the process repaired and complete response.
In the prior art, when the response of any memory node has not been obtained in client, directly think distributed storage system The malfunction of system is APD state, just will be considered that distributed storage system when memory node clearly returns to the abnormal message of storage The malfunction of system is determined as PDL state, if chromic trouble occurs in memory node, it is different may to return to storage without normal direction client Therefore normal message, determines distributed storage so that malfunction may be determined as APD state by client in the prior art The malfunction of system is simultaneously inaccurate, and if PDL state is mistakenly considered APD state, will lead to business side repair can not open Exhibition finally extends fault restoration duration instead.And in embodiment provided by the invention, it is deposited each of in distributed memory system Storage node knows that the malfunction of distributed memory system can be based on for without failure memory node SCSI request returns to malfunction to target device, so that target device can explicitly know the failure of distributed memory system State, and then the precision that target device determines malfunction can be improved.
510, when the malfunction is Fisrt fault state, if at least one storage section in first preset duration Point is not repaired all, then the malfunction is updated to the second malfunction by Fisrt fault state by supervisory node.
Since the first preset duration is only a preset duration, in first preset duration, at least one storage Node possibly can not be repaired all, then, when at least one memory node may be repaired all, it is possible to need The longer time repairs the memory node that do not repair also.It is uncertain as repairing duration used in the memory node that do not repair also, May relatively long, therefore, which directly can be updated to the second failure by Fisrt fault state for the malfunction State.
511, when the malfunction is updated to second malfunction by Fisrt fault state, supervisory node is to this point All memory nodes in cloth storage system send the second failure identification for being used to indicate second failure.
Supervisory node sends all memory nodes in the mode and step 503 of the second failure identification to all memory nodes and sends out Send Fisrt fault identify mode similarly, here, the embodiment of the present invention does not repeat them here this step 511.Shown in this step 511 Process that is to say that into multiple memory node each memory node sends the process of the malfunction.
512, target storage node receives second failure identification.
Target storage node receives the mode of reception Fisrt fault mark in second failure identification and step 504 similarly, Here, the embodiment of the present invention does not repeat them here this step 512.
513, target device sends access request to the target storage node.
Target device has associated description to the mode that the target storage node sends access request in step 505, here, The embodiment of the present invention does not repeat them here this step 513.
514, when the target storage node in the distributed memory system receives second failure identification, if the target Memory node receives access request again, which hangs the access request, exports second failure identification.
The target in the second failure identification mode and step 506 is requested and exported to any memory node hanging use The mode of Fisrt fault mark is requested and exported to memory node hanging use similarly, here, the embodiment of the present invention is to this Step 514 does not repeat them here.
515, target device receives the second failure identification that the target storage node is returned based on the access request.
The mode that target device receives the mode of second failure identification and reception Fisrt fault identifies in step 507 is same Reason, here, the embodiment of the present invention does not repeat them here this.
516, target device is based on received second failure identification, carries out troubleshooting.
This step 516 can be executed by the SCSI treatment progress of target device, in some embodiments, in client with It may be VMWare virtual machine that the SCSI treatment progress, which docks target virtual machine, it is also possible to not be VMWare virtual machine, due to not The mode that same virtual machine target device carries out troubleshooting is different, and therefore, this step 516 can be by following manner 3-4 Either formula realize.
Mode 3, when the access request is sent by the destination client based on target virtual machine, and the target virtual machine is When VMWare virtual machine, if the malfunction is second malfunction, SCSI treatment progress is deposited to VMWare virtual machine return Store up abnormal message.
When second failure identification of failure identification, illustrate that at least one memory node cannot be complete in the first preset time Portion is repaired, and needs the longer time, and target virtual machine can carry out the troubleshooting under PDL state, for target virtual machine PDL state can be perceived, VMWare virtual machine customized SK 0x0, ASC&ASCQ can be carried by storing abnormal message SCSI mistake, the SCSI mistakes such as 0x0200 or SK 0x5, ASC&ASCQ 0x2500 can indicate the shape in distributed memory system State is PDL state, so that the message that target virtual machine receives storage exception can perceive PDL state, then the mesh Marking virtual machine electricity, waiting technical staff can will repair distributed memory system under the file system in the distributed memory system Interior failure, or select more excellent troubleshooting mode to carry out the storage of handling failure according to the customized troubleshooting mode of user Node.For example, by electric under the memory node of failure.
It should be noted that when there is the failure that cannot be repaired in short-term due to the memory node in distributed memory system, It may cause file system exception, after guaranteeing that the failure of the memory node in distributed memory system is repaired, file system System can be by normal use, then needing first will be electric under file system.When repairing completion, powered on by file system, and Repair file system.
Mode 4, when the access request is sent by the destination client based on target virtual machine, and the target virtual machine is not When VMWare virtual machine, if the malfunction is second malfunction, which returns to the mesh to the target virtual machine The identifiable target error of virtual machine is marked, which is used to indicate storage media failure.
The target error can be Sense key 0x3 mistake, that is to say storage medium mistake (Medium Error), one As virtual machine can identify, when target virtual machine receives the target error, illustrate the shape of distributed memory system at this time State is the second malfunction, then the target device electricity, waiting technical staff can will repair distribution under distributed file system Failure in formula storage system, or select more excellent troubleshooting mode to handle according to the customized troubleshooting mode of user The memory node of failure.
It should be noted that process shown in this step 516 that is to say that, based on received failure identification, target device carries out The process of troubleshooting that is to say based on the malfunction for including in the response, carry out the process of troubleshooting.
517, when at least one memory node reparation is completed, each equipment into the distributed memory system is sent It repairs and completes response.
This step 517 and step 509 similarly, do not repeat them here this step 517 in this embodiment of the present invention.It needs to illustrate It is that the failure of each memory node can be to be repaired by itself, can also be repaired by technical staff, the embodiment of the present invention pair The repair mode of memory node is not specifically limited.
It should be noted that when client receives reparation completion response, if file system has descended electricity, the visitor Is repaired to this document system electrification by this document system for family end.Due to being stored with a large amount of metadata in file system, when to this It when file system is repaired, needs to scan all metadata in this document system, and modifies the wrong metadata scanned, The process of the general reparation file system needs to consume part-time, and under Fisrt fault state, client will not descend text part System also can be to avoid lower text once at least one memory node can be repaired in the first preset time Part system allows distributed memory system to restore business as early as possible so as to reduce the time for repairing file system, to protect Demonstrate,prove service quality.
Method shown in the embodiment of the present invention, the storage broken down according at least one of the multiple memory node Node determines the malfunction of the distributed memory system, without just determining and dividing when the equal failure of all memory nodes The malfunction of cloth storage system can deposit each of into distributed memory system immediately after having determined malfunction It stores up node and sends malfunction, so that each memory node carries out troubleshooting according to determining malfunction, so as to drop Low distributed memory system restores the normal time used.Also, for different virtual machines, the embodiment of the invention provides not Same troubleshooting mode, so that troubleshooting mode provided by the embodiment of the present invention has more universality.Also, Each memory node in distributed memory system knows the malfunction of distributed memory system, for without failure For memory node, malfunction can be returned to target device based on access request, so that target device can explicitly be known The malfunction of road distributed memory system, and then the precision that target device determines malfunction can be improved.Also, first Under malfunction, target device will not descend electric file system, once at least one memory node is equal in the first preset time Can be repaired, then also can to avoid lower electric file system, and after at least one memory node recovery, file system with And business can recover immediately, so as to reduce repair file system time, allow distributed memory system as early as possible Recovery business, to guarantee service quality.
Since the failure that memory node occurs can may repair in a short time, it is also possible to need to repair for a long time, and every The repair time of a memory node can influence the repair time of entire distributed memory system, therefore, in some implementations It, can also be according to the fault type of each memory node, to determine the malfunction of distributed memory system in example.In order into one Step illustrates this process, a kind of distributed memory system fault handling method provided by the embodiment of the present invention shown in Figure 6 Flow chart, the process of this method may comprise steps of.
601, supervisory node determine at least one memory node to break down in distributed memory system and this at least The target data that one memory node can not be accessed.
This step 601 and step 501 similarly, do not repeat them here step 601 in this embodiment of the present invention.
602, the time that supervisory node breaks down according at least one memory node, the distributed memory system is determined Fault scenes, the fault scenes be used to indicate at least one memory node whether and meanwhile break down.
The fault scenes may include any of Fisrt fault scene and the second fault scenes, wherein first event Barrier scene is used to indicate at least one memory node while breaking down, which is used to indicate this at least one The time that memory node breaks down is different.
The supervisory node can determine the time that each memory node breaks down from the bug list of storage, thus should Whether the time that supervisory node can break down according at least one memory node is identical, determines fault scenes.
In one possible implementation, when at least one memory node breaks down in target duration, The fault scenes are determined as Fisrt fault scene by the supervisory node, otherwise, which are determined as the second fault scenes. It should be noted that process shown in this step 602 that is to say the time broken down according at least one described memory node, Determine the process of the fault scenes.
603, for any memory node at least one memory node, when any memory node appearance is preset Network failure, preset powered-off fault failure, preset maloperation failure, preset hardware fault or preset software fault When, which is determined as Fisrt fault type for the fault type of any memory node, and otherwise, which is saved The fault type of point is determined as the second fault type.
The fault type of memory node is for indicating that can the failure of a memory node be repaired in the second preset duration Multiple, fault type may include any of Fisrt fault type and the second fault type, and the second fault type is used to indicate The failure of one memory node cannot be repaired in second preset duration, which can be less than or equal to the One preset duration, the embodiment of the present invention are not specifically limited second preset duration.
Wherein, which may include any one of following 1.1-1.7:
1.1st, the service network port of any memory node can not access, the supervision network interface of any memory node can Access, the service network port are the nets of used business network when being used for heartbeat, data synchronization and mirror image between memory node Mouthful, which is whether monitoring memory node breaks down and carry out the net of used supervisory network when information inquiry Mouthful.
The supervisory node can be sent to the service network port of any memory node because of spy by the service network port of supervisory node Net packet survey meter (ping) request, ping request establishes connection for requesting, if can connect success, then it is assumed that this is any to deposit Store up node service network can mouth access, it is on the contrary, then it is assumed that the service network port of any memory node can not access.Similarly, should Supervisory node can send ping request to the supervision network interface of any memory node by the supervision network interface of supervisory node, if can be with Successful connection, then it is assumed that the supervision network interface of any memory node is accessible, otherwise it is assumed that the supervision of any memory node Network interface can not access.
If supervisory node can not access any memory node by service network port, illustrate that net occurs in any memory node Network failure, but any memory node is able to access that by supervising network interface, the failure for illustrating that any memory node occurs can To repair in a short time, then there is preset network failure in any memory node.
1.2nd, when the business network and the supervisory network are the same target network, any memory node is at this There is the packet loss of the first preset number or the Abnormal Packet of the second preset number in the data packet transmitted in target network, and this any is deposited The service network port, supervision network interface and baseboard management controller BMC network interface for storing up node are inaccessible, and BMC network interface is management BMC BMC network network interface.
The supervisory node can send ping to the BMC network interface of any memory node by the BMC network interface of supervisory node and ask It asks, if can connect success, then it is assumed that the BMC network interface of any memory node is accessible, otherwise it is assumed that any storage section The BMC network interface of point cannot access.
In the delivery stage, technical staff can configure the same network for business network and supervisory network, that is to say mesh Network is marked, when the business network is the same target network with the supervisory network, if any memory node is in the target There is the packet loss of the first preset number or the Abnormal Packet of the second preset number in the data packet transmitted in network, illustrates any storage There is network failure in node, if the supervisory node can not access any memory node, illustrates that any memory node occurs Failure can repair in a short time, then there is preset network failure in any memory node.
1.3rd, when the business network and the supervisory network are the same target network, any memory node is at this The data packet transmitted in target network occurs being greater than the packet loss of the first preset number or the Abnormal Packet greater than the second preset number, and The time delay that any memory node transmits data in target network is greater than third preset duration.
It should be noted that when supervisory node is when sending ping request to any memory node, if connection is unsuccessful, Any memory node can send connection failure response to the supervisory node, and connection failure response is used to indicate connection failure, And Delay can be carried in connection failure response, which is used to indicate any memory node in target network The time delay of interior transmission data, thus when the supervisory node may determine that whether time delay indicated by Delay is preset greater than third It is long.
1.4th, when the business network and the supervisory network are the same target network, be somebody's turn to do in the target network The flow control PFC message of the priority of any memory node is greater than third preset number, and any memory node can not visit It asks.
Supervisory node can detecte flow control (the priority-based flow of each priority in target network Control, PFC) message, it is pre- to may thereby determine that whether the number for the PFC message that any memory node is sent is greater than third If number, which is assured that whether any memory node meets preset condition.The embodiment of the present invention to this Three preset numbers are not specifically limited.
1.5th, when the business network and the supervisory network are the same target network, be somebody's turn to do in the target network Any memory node sends the flow control PFC message of the priority of third preset number, and any memory node is in the mesh The time delay for marking transmission data in network is greater than the 4th preset duration.
It should be noted that the embodiment of the present invention is not specifically limited four preset duration.
1.6th, when the business network and the supervisory network are the same target network, it is incumbent out in the target network Broadcast storm caused by one memory node, and the service network port of any memory node, supervision network interface and BMC network interface can not Access.
It should be noted that send a large amount of broadcast packet in the target network when any memory node, then the target network Broadcast storm is likely to occur in network.
1.7th, when the business network and the supervisory network are the same target network, be somebody's turn to do in the target network Broadcast storm caused by any memory node, and the time delay of any memory node in the target network be greater than the 5th it is default when It is long.
It should be noted that the embodiment of the present invention is not specifically limited five preset duration.
The preset powered-off fault failure may include any one of following 2.1-2.2:
2.1st, the service network port of all memory nodes in machine frame, supervision network interface and BMC network interface are inaccessible, The machine frame includes any memory node.
It may include at least one memory node in one machine frame, when all memory node service network ports, supervision network interface And BMC network interface it is inaccessible when, it is believed that in machine frame all memory nodes by lower electricity, then, if this any is deposited Node is stored up in the machine frame, illustrates any memory node also by lower electricity, as long as powering on to machine frame, so that it may repair this and any deposit Store up the failure of node, then it is assumed that preset powered-off fault failure occurs in any memory node.
2.2nd, in the 7th preset duration, the service network port of the memory node of first object number, supervision network interface with And BMC network interface is inaccessible, the memory node of the first object number includes any memory node.
In the 7th preset duration, service network port, supervision network interface and the BMC net of the memory node of first object number When mouth is inaccessible, the memory node of first object number, which can consider, has there is preset powered-off fault failure.It needs Illustrate, the embodiment of the present invention is not specifically limited the 7th preset duration.
The preset maloperation failure may include: any memory node by actively lower electricity.For example, appointing when the user clicks When the power-off button or reboot button of one memory node, memory node is thought by the lower electricity of active, and by the information of the lower electricity of active It is sent to supervisory node, so that supervisory node determines that preset maloperation failure occurs in any memory node.
The preset hardware fault includes: that any memory node exits extremely, and the BMC network interface of any memory node can Access, and there is the component loosened in any memory node.
When any memory node exits extremely, the information exited extremely can be sent to supervisory node, to indicate oneself Exit extremely.Since abnormal exit may be that internal superstructure loosening causes.Either component loosening can be memory bar And strip etc., the component of loosening can be recovered immediately by way of plug, that is, occur that short trouble.It needs It is noted that illustrating that any memory node has pine when any memory node detects either component bad connection Dynamic component, then any memory node can send the information of loose part to supervisory node, so that supervisory node can root According to the information of loose part, determine that preset maloperation failure occurs in any memory node.
The preset software fault may include any one of following 3.1-3.3:
3.1st, the operating system of any memory node lead to any memory node exceptional reset extremely.
When the low memory of any memory node, cause the operating system of any memory node that can not continue to run, Need to be reset or any memory node exceptional reset etc. of house dog triggering, when any memory node occur it is abnormal When reset, which can send the message of exceptional reset to supervisory node, so that the supervisory node can be known Any memory node exceptional reset then illustrates that preset software fault occurs in any memory node.
3.2nd, the software anomaly of any memory node cause the target process of any memory node to exit.
The target process can be OSD process, when there is exceptional reset in any memory node, any memory node The message that target process exits can be sent to supervisory node, so that the supervisory node can know the mesh of any memory node Mark process exits, then illustrates that preset software fault occurs in any memory node.
3.3rd, the software anomaly of any memory node cause the operating system of any memory node to reset.
Due to software anomaly, when which resets, which can be to prison Pipe node sends the message that operating system resets, so that the supervisory node can know that the operating system of any memory node is multiple Position, then illustrate that preset software fault occurs in any memory node.
It should be noted that supervisory node is just when each memory node at least one memory node breaks down The fault type that each memory node can be judged by this step 603 is Fisrt fault type or the second fault type, and The fault type of each memory node is stored in bug list, to need supervisory node to need the failure of any memory node When type, can directly it be obtained from bug list.
It should be noted that various faults type identification method embodied in this step 603, can be accurately determined every The fault type of a memory node, so it is true according to the fault type of each memory node, and can be more accurate determine is distributed The malfunction of formula storage system.
604, when the number of at least one memory node is greater than the redundancy of the distributed memory system, and the number of targets According to data volume meet preset condition when, if the fault scenes be the Fisrt fault scene, the supervisory node according to this at least The fault type of each memory node, determines the malfunction in one memory node.
Since Fisrt fault scene is for indicating at least one memory node while occurring, then the supervisory node can root According to each memory node, to determine the malfunction of the distributed memory system.
In one possible implementation, supervisory node is then saved according to each storage at least one memory node The fault type of point determines that the malfunction may include: the event when each memory node at least one memory node When barrier type is Fisrt fault type, which is determined as the Fisrt fault state for the malfunction, first event Barrier state is used to indicate at least one memory node and can all be repaired in first preset duration；When this at least one When the fault type of the memory node of target number is the second fault type in memory node, if the target number is less than or waits When the redundancy of the distributed memory system, which is determined as the Fisrt fault state for the malfunction, otherwise, The malfunction is determined as second malfunction, which is used to indicate at least one described memory node not It can be all repaired in first preset duration.
It, can be with when the fault type of each memory node at least one memory node is Fisrt fault type Think that the failure in distributed memory system can be repaired in a short time, then the malfunction can be determined as to the first event Barrier state.Although the fault type of the memory node of target number is the second fault type, when target number is less than or waits In the redundancy of the distributed memory system, illustrate that influence of the memory node of target number to distributed memory system is little, The malfunction can be then determined as to Fisrt fault state.Once target number is greater than the redundancy of the distributed memory system Degree, illustrates that the memory node of target number is affected to distributed memory system, then can be determined as the malfunction Second malfunction, to repair failure rapidly.
It should be noted that whether the number at least one memory node is greater than the superfluous of the distributed memory system Remaining, and whether the data volume of the target data meets the description of preset condition, there is embodiment above, implements to this present invention Example does not repeat them here.
It should be noted that supervisory node can be by presetting in step 603 when fault scenes are Fisrt fault scene Network failure, any preset failure in preset powered-off fault failure or preset maloperation failure judges each storage The fault type of node.
605, when the number of at least one memory node is greater than the redundancy of the distributed memory system, and the number of targets According to data volume meet preset condition when, if the fault scenes be the second fault scenes, supervisory node according to this at least one The fault type for the first memory node that the last one in memory node breaks down, determines the malfunction.
Since the time that the second fault scenes are used to indicate that at least one memory node to break down is different, then the supervision Node can be according to the fault type of the memory node at least to break down with the last one in a memory node, to determine The malfunction of the distributed memory system, wherein the last one memory node to break down that is to say the first memory node.
In one possible implementation, when the fault type of first memory node is Fisrt fault type, prison The malfunction is then determined as Fisrt fault state by pipe node, which is used to indicate at least one storage section Point can be all repaired in first preset duration；When the fault type of first memory node is second fault type When, which is then determined as the second malfunction by supervisory node, which is used to indicate this at least one Memory node cannot be all repaired in first preset duration.
It should be noted that process shown in this step 604 and 605 that is to say the event according to the distributed memory system Hinder scene, determines the process of the malfunction of the distributed memory system.
It should be noted that supervisory node is only it needs to be determined that the first storage section when fault scenes are the second fault scenes The fault type of point, does not need to determine the fault type so memory node, can pass through network event preset in step 603 Hinder, any in preset powered-off fault failure, preset maloperation failure, preset hardware fault or preset software fault Preset failure, to judge the fault type of the first memory node.
606, all memory nodes transmission of the supervisory node into the distributed memory system is used to indicate malfunction Failure identification.
When the malfunction is Fisrt fault state, all storages of the supervisory node into the distributed memory system Node sends Fisrt fault mark, when the malfunction is the second malfunction, to the supervisory node to the distributed storage All memory nodes in system send the second failure identification, and similarly, this will not be repeated here for specific implementation procedure and step 503. It should be noted that process shown in this step 606 that is to say that each memory node is sent into the multiple memory node The process of the malfunction.
607, the target storage node in the distributed memory system receives failure identification.
When the failure identification is that Fisrt fault identifies, target storage node receives Fisrt fault mark, when the failure When being identified as the second failure identification, target storage node receives the second failure identification, and specific implementation procedure and step 504 are same Reason, this will not be repeated here.
608, target device sends access request to the target storage node.
Similarly, the embodiment of the present invention does not repeat them here this step 608 to process shown in this step 608 and step 505.
609, after the target storage node in the distributed memory system receives the failure identification, if the target stores Node receives access request again, which hangs the access request, exports the failure identification.
When the failure identification is that Fisrt fault identifies, target storage node is identified to target device output Fisrt fault, When the failure identification is the second failure identification, target storage node exports the second failure identification to target device, specific to execute Similarly, this will not be repeated here for process and step 506.
610, target device receives the failure identification that the target storage node is returned based on the access request.
Similarly, the embodiment of the present invention does not repeat them here this step 610 to process shown in this step 610 and step 505.It needs Illustrate, process shown in this step 610, which that is to say, receives the response that the target storage node returns；It is wrapped in the response Malfunction containing the distributed memory system；The malfunction is used to indicate the memory node of at least one failure The process that can be all repaired in the first preset duration.Wherein, the malfunction for including in the response that is to say failure Mark.
611, target device is based on received failure identification, carries out troubleshooting.
When the failure identification is that Fisrt fault identifies, target device is identified based on received Fisrt fault, carries out failure Process shown in processing, specific implementation procedure and step 508 is similarly.When the failure identification is the second failure identification, target is set It is standby to be based on received second failure identification, carry out troubleshooting, process shown in specific implementation procedure and step 516 similarly, In This, the embodiment of the present invention does not repeat them here this step 611.
It should be noted that each memory node in distributed memory system knows the failure of distributed memory system State can return to malfunction to target device based on access request for without failure memory node, thus Target device can explicitly know the malfunction of distributed memory system, and then target device can be improved and determine failure shape The precision of state.
It should be noted that for different virtual machines, the embodiment of the invention provides different troubleshooting modes, from And make troubleshooting mode provided by the embodiment of the present invention with more universality.
It should be noted that process shown in this step 611 that is to say based on the malfunction for including in the response, into The process of row troubleshooting.
612, when at least one memory node reparation is completed, supervisory node is each into the distributed memory system Equipment, which sends to repair, completes response.
This step 612 and step 509 similarly, do not repeat them here this step 612 in this embodiment of the present invention.It needs to illustrate It is, when the malfunction is Fisrt fault state, if at least one memory node whole quilt in first preset duration Repairing intact then can directly execute this step 612, if at least one memory node is not whole in first preset duration It repairs, then the malfunction is updated to the second malfunction by Fisrt fault state by supervisory node, and jumps execution step 606.It should be noted that client will not descend electric file system under Fisrt fault state, once in the first preset time At least one memory node can be repaired, then also text can be repaired to avoid lower electric file system so as to reduce The time of part system, distributed memory system is allowed to restore business as early as possible, to guarantee service quality.
Method shown in the embodiment of the present invention, the storage broken down according at least one of the multiple memory node Node determines the malfunction of the distributed memory system, without just determining and dividing when the equal failure of all memory nodes The malfunction of cloth storage system can deposit each of into distributed memory system immediately after having determined malfunction It stores up node and sends malfunction, so that each memory node carries out troubleshooting according to determining malfunction, so as to drop Low distributed memory system restores the normal time used.Also, for different virtual machines, the embodiment of the invention provides not Same troubleshooting mode, so that troubleshooting mode provided by the embodiment of the present invention has more universality.Also, Each memory node in distributed memory system knows the malfunction of distributed memory system, for without failure For memory node, malfunction can be returned to target device based on access request, so that target device can explicitly be known The malfunction of road distributed memory system, and then the precision that target device determines malfunction can be improved.Also, first Under malfunction, target device will not descend electric file system, once at least one memory node is equal in the first preset time It can be repaired, then also can be to avoid lower electric file system, so as to reduce the time for repairing file system, so that distribution Formula storage system can restore business as early as possible, to guarantee service quality.Also, various faults class embodied in this step 603 Type method of discrimination can be accurately determined the fault type of each memory node, and then according to the failure classes of each memory node Type is true, the malfunction of determination distributed memory system that can be more accurate.
Fig. 7 is a kind of structural schematic diagram of fault treating apparatus provided in an embodiment of the present invention, is applied to distributed storage System, the distributed memory system include multiple memory nodes, which includes:
Determining module 701, the memory node for being broken down according at least one of the multiple memory node, really The malfunction of the fixed distributed memory system；The malfunction is used to indicate the storage of at least one failure Can node all be repaired in the first preset duration；
Sending module 702, for each memory node transmission malfunction into the multiple memory node.
Optionally, described device further include:
Processing module, for each memory node transmission malfunction into the multiple memory node.
Optionally, the determining module 701, the determining module include:
First determination unit, for executing above-mentioned steps 501；
Second determination unit, for determining the event according at least one described memory node and the target data Barrier state.
Optionally, second determination unit is used for:
When the number of at least one memory node is greater than the redundancy of the distributed memory system, and the target When the data volume of data meets preset condition, the malfunction is determined as Fisrt fault state, the Fisrt fault state Being used to indicate at least one described memory node can all be repaired in first preset duration.
Optionally, second determination unit is used for:
When the number of at least one memory node is greater than the redundancy of the distributed memory system, and the target When the data volume of data meets preset condition, according to the fault scenes of the distributed memory system, the malfunction is determined, The fault scenes be used to indicate at least one described memory node whether and meanwhile break down.
Optionally, the preset condition includes any one of following:
Ratio between the data volume of the target data and the first preset data amount is greater than default ratio, and described first is pre- If data volume is the total amount of data of all data of distributed memory system storage；
The data volume of the target data is greater than the second preset data amount.
Optionally, described device further include:
Update module is used for when the malfunction is the Fisrt fault state, if in first preset duration At least one interior described memory node is not repaired all, then the malfunction is updated to the second event by Fisrt fault state Barrier state, second malfunction is used to indicate at least one described memory node cannot be complete in first preset duration Portion is repaired.
Optionally, second determination unit, for executing above-mentioned steps 602.
Optionally, second determination unit, for occurring in target duration when at least one described memory node When failure, the fault scenes are determined as Fisrt fault scene, otherwise, the fault scenes are determined as the second failure field Scape, the Fisrt fault scene are used to indicate at least one described memory node while breaking down, second fault scenes It is different to be used to indicate the time that at least one described memory node breaks down.
Optionally, second determination unit includes:
First determines subelement, for executing above-mentioned steps 604；
Second determines subelement, for executing above-mentioned steps 605.
Optionally, described first determine that subelement is used for:
It, will when the fault type of each memory node at least one described memory node is Fisrt fault type The malfunction is determined as Fisrt fault state, and the failure that the Fisrt fault type is used to indicate a memory node can be It is repaired in second preset duration, the Fisrt fault state is used to indicate at least one described memory node can be described It is all repaired in first preset duration；
When the fault type of the memory node of target number at least one described memory node is the second fault type, If the target number is less than or equal to the redundancy of the distributed memory system, the malfunction is determined as described the Otherwise the malfunction is determined as the second malfunction, second fault type is used to indicate one by one malfunction The failure of memory node cannot be repaired in second preset duration, second malfunction be used to indicate it is described at least One memory node cannot be all repaired in first preset duration.
Optionally, described second determine that subelement is used for:
When the fault type of first memory node is Fisrt fault type, the malfunction is determined as first Malfunction, the failure that the Fisrt fault type is used to indicate a memory node can be repaired in second preset duration Multiple, the Fisrt fault state is used to indicate at least one described memory node and can all be repaired in first preset duration It is multiple；
When the fault type of first memory node is the second fault type, then the malfunction is determined as the Two malfunctions, the failure that second fault type is used to indicate a memory node cannot be in second preset duration It is repaired, second malfunction is used to indicate at least one described memory node cannot be complete in first preset duration Portion is repaired.
Optionally, the determining module 701, is also used to execute step 603.
Optionally, the sending module 702, is also used to execute above-mentioned steps 509.
Fig. 8 is a kind of structural schematic diagram of fault treating apparatus provided in an embodiment of the present invention, the distributed storage system System includes multiple memory nodes；The device includes:
Sending module 801, for executing above-mentioned steps 608；
Receiving module 802, the response returned for receiving the target storage node；It include the distribution in the response The malfunction of formula storage system；Can the memory node that the malfunction is used to indicate at least one failure first It is all repaired in preset duration.
Optionally, the device further include:
Processing module, for carrying out troubleshooting based on the malfunction for including in the response.
Optionally, the failure identification of the malfunction includes any in Fisrt fault mark or the second failure identification It is a, wherein the Fisrt fault mark is used to indicate Fisrt fault state, and second failure identification is used to indicate the second failure State, the Fisrt fault state are used to indicate at least one memory node and can all be repaired in the first preset duration, institute Stating the second malfunction and being used to indicate at least one described memory node cannot all be repaired in first preset duration, The memory node is the memory node to break down in the distributed memory system.
Optionally, the processing module is used for:
When the access request by the destination client in the distributed memory system based on target virtual machine send, and When the target virtual machine is VMWare virtual machine, if the malfunction is the Fisrt fault state, not to the target Virtual machine responds the access request, and the Fisrt fault state is used to indicate at least one memory node can be when first is default It is all repaired in long；
When the access request is sent by the destination client based on target virtual machine, and the target virtual machine is When VMWare virtual machine, if the malfunction is second malfunction, it is different that Xiang Suoshu target virtual machine returns to storage Normal message, second malfunction is used to indicate at least one described memory node cannot be in first preset duration All it is repaired.
Optionally, the processing module is used for:
When target access request is sent out by the destination client in the distributed memory system based on target virtual machine It send, and when the target virtual machine is not VMWare virtual machine, if the malfunction is the Fisrt fault state, Xiang Suoshu Target virtual machine transmission retries request, it is described retry request and be used to indicate re-issue the access request, the Fisrt fault State is used to indicate at least one memory node and can all be repaired in the first preset duration；
When target access request is based on target virtual machine transmission, and the target virtual machine by the destination client When not being VMWare virtual machine, if the malfunction be second malfunction, Xiang Suoshu target virtual machine return described in The identifiable target error of target virtual machine, the target error are used to indicate storage media failure, second malfunction Being used to indicate at least one described memory node cannot all be repaired in first preset duration.
Optionally, the receiving module 802 is also used to receive the destination client in the distributed memory system and sends Target access request, target access request, which is used to indicate, handles first object data, the first object number According to including the target data；
The sending module 801, the target storage for being requested based on the target access, into distributed memory system Node sends the access request.
Optionally, receiving module 802, response is completed in the reparation for receiving target storage node return, described to have repaired It is used to indicate in the distributed memory system at response without faulty equipment.
The embodiment of the present invention also provides a kind of distributed memory system, the distributed memory system include supervisory node and Multiple memory nodes；
The supervisory node is used for:
The distributed storage is determined according to the memory node that at least one of the multiple memory node breaks down The malfunction of system；Can the malfunction is used to indicate at least one described memory node to break down pre- first If being all repaired in duration；
Into the multiple memory node, each memory node sends the malfunction；
Each of the multiple memory node memory node, for receiving the malfunction.
Optionally, the failure identification of the malfunction includes any in Fisrt fault mark and the second failure identification A, the Fisrt fault mark is used to indicate the Fisrt fault state, and second failure identification is used to indicate the second failure State, the Fisrt fault state be used to indicate at least one described memory node can in first preset duration whole quilts It repairs, second malfunction is used to indicate at least one described memory node cannot be in first preset duration all It is repaired.
Optionally, each of the multiple memory node memory node is also used to that the failure identification ought be received Afterwards, if receiving the access request again, the access request is hung, is based on received malfunction, carries out troubleshooting.
Each equipment in the distributed memory system for the above-mentioned offer for needing to illustrate may each be in embodiment 5 and 6 Equipment.
All the above alternatives can form the alternative embodiment of the disclosure, herein no longer using any combination It repeats one by one.
It should be understood that fault treating apparatus provided by the above embodiment is in handling failure, only with above-mentioned each function The division progress of module can according to need and for example, in practical application by above-mentioned function distribution by different function moulds Block is completed, i.e., the internal structure of device is divided into different functional modules, to complete all or part of function described above Energy.In addition, distributed memory system fault handling method embodiment provided by the above embodiment belongs to same design, it is specific real Existing process is detailed in embodiment of the method, and which is not described herein again.
Those of ordinary skill in the art will appreciate that realizing that all or part of the steps of above-described embodiment can pass through hardware It completes, relevant hardware can also be instructed to complete by program, the program can store in a kind of computer-readable In storage medium, storage medium mentioned above can be read-only memory, disk or CD etc..
The foregoing is merely presently preferred embodiments of the present invention, is not intended to limit the invention, it is all in spirit of the invention and Within principle, any modification, equivalent replacement, improvement and so on be should all be included in the protection scope of the present invention.

Claims (22)

1. a kind of distributed memory system fault handling method, which is characterized in that the distributed memory system includes multiple deposits Store up node；The described method includes:
According to the memory node that at least one of the multiple memory node breaks down, the distributed memory system is determined Malfunction；Can the malfunction is used to indicate at least one described memory node to break down when first is default It is all repaired in long；
Into the multiple memory node, each memory node sends the malfunction.
2. the method according to claim 1, wherein the method also includes:
According to the malfunction of the storage system, troubleshooting is carried out.
3. the method according to claim 1, wherein described according at least one of the multiple memory node The memory node of failure determines that the malfunction of the distributed memory system includes:
Determine at least one memory node to break down in the distributed memory system and at least one described storage section The target data that can not be accessed in point；
According at least one described memory node and the target data, the malfunction is determined.
4. according to the method described in claim 3, it is characterized in that, described at least one memory node according to and described Target data determines that the malfunction includes:
When the number of at least one memory node is greater than the redundancy of the distributed memory system, and the target data Data volume when meeting preset condition, the malfunction is determined as Fisrt fault state, the Fisrt fault state is used for Indicate that at least one described memory node can be all repaired in first preset duration.
5. according to the method described in claim 3, it is characterized in that, described at least one memory node according to and described Target data determines that the malfunction includes:
When the number of at least one memory node is greater than the redundancy of the distributed memory system, and the target data Data volume when meeting preset condition, according to the fault scenes of the distributed memory system, determine the malfunction, it is described Fault scenes be used to indicate at least one described memory node whether and meanwhile break down.
6. a kind of distributed memory system fault handling method, which is characterized in that the distributed memory system includes multiple deposits Store up node；The described method includes:
Target storage node into the distributed memory system sends access request；
Receive the response that the target storage node returns；It include the failure shape of the distributed memory system in the response State；Can the memory node that the malfunction is used to indicate at least one failure all be repaired in the first preset duration It is multiple.
7. according to the method described in claim 6, it is characterized in that, it is described receive response that the target storage node returns it Afterwards, the method also includes:
Based on the malfunction for including in the response, troubleshooting is carried out.
8. the method according to the description of claim 7 is characterized in that described based on the malfunction for including in the response, into Row troubleshooting includes:
It is and described when the access request is sent by the destination client in the distributed memory system based on target virtual machine When target virtual machine is VMWare virtual machine, if the malfunction is the Fisrt fault state, not to the destination virtual Machine responds the access request, and the Fisrt fault state is used to indicate at least one memory node can be in the first preset duration All it is repaired；
When the access request is sent by the destination client based on target virtual machine, and the target virtual machine is VMWare When virtual machine, if the malfunction is second malfunction, Xiang Suoshu target virtual machine returns to disappearing for storage exception Breath, second malfunction be used to indicate at least one described memory node cannot in first preset duration whole quilts It repairs.
9. the method according to the description of claim 7 is characterized in that described based on the malfunction for including in the response, into Row troubleshooting includes:
It is and described when the access request is sent by the destination client in the distributed memory system based on target virtual machine When target virtual machine is not VMWare virtual machine, if the malfunction is the Fisrt fault state, Xiang Suoshu destination virtual Machine transmission retries request, it is described retry request and be used to indicate re-issue the access request, the Fisrt fault state is used for Indicate that at least one memory node can be all repaired in the first preset duration；
When the access request is sent by the destination client based on target virtual machine, and the target virtual machine is not When VMWare virtual machine, if the malfunction is second malfunction, Xiang Suoshu target virtual machine returns to the target The identifiable target error of virtual machine, the target error are used to indicate storage media failure, and second malfunction is used for Indicate that at least one described memory node cannot be all repaired in first preset duration.
10. a kind of distributed memory system, which is characterized in that the distributed memory system includes supervisory node and multiple storages Node；
The supervisory node is used for:
The distributed memory system is determined according to the memory node that at least one of the multiple memory node breaks down Malfunction；Can the malfunction is used to indicate at least one described memory node to break down when first is default It is all repaired in long；
Into the multiple memory node, each memory node sends the malfunction；
Each of the multiple memory node memory node, for receiving the malfunction.
11. system according to claim 10, which is characterized in that each of the multiple memory node storage section Point, is also used to after receiving the failure identification, if receiving the access request again, hangs the access request, is based on Received malfunction carries out troubleshooting.
12. a kind of fault treating apparatus, which is characterized in that be applied to distributed memory system, the distributed memory system packet Containing multiple memory nodes, described device includes:
Determining module, the memory node for being broken down according at least one of the multiple memory node, determine described in The malfunction of distributed memory system；The malfunction is used to indicate the memory node energy of at least one failure It is no to be all repaired in the first preset duration；
Sending module, for each memory node transmission malfunction into the multiple memory node.
13. device according to claim 12, which is characterized in that described device further include:
Processing module, for each memory node transmission malfunction into the multiple memory node.
14. device according to claim 12, which is characterized in that the determining module includes:
First determination unit, for determining at least one memory node to break down in the distributed memory system and institute State the target data that can not be accessed at least one memory node；
Second determination unit, for determining the failure shape according at least one described memory node and the target data State.
15. device according to claim 14, which is characterized in that second determination unit is used for:
When the number of at least one memory node is greater than the redundancy of the distributed memory system, and the target data Data volume when meeting preset condition, the malfunction is determined as Fisrt fault state, the Fisrt fault state is used for Indicate that at least one described memory node can be all repaired in first preset duration.
16. device according to claim 14, which is characterized in that second determination unit is used for:
When the number of at least one memory node is greater than the redundancy of the distributed memory system, and the target data Data volume when meeting preset condition, according to the fault scenes of the distributed memory system, determine the malfunction, it is described Fault scenes be used to indicate at least one described memory node whether and meanwhile break down.
17. a kind of fault treating apparatus, which is characterized in that be applied to distributed memory system, the distributed memory system packet Containing multiple memory nodes, described device includes:
Sending module sends access request, the distribution for the target storage node into the distributed memory system Storage system includes multiple memory nodes；
Receiving module, the response returned for receiving the target storage node；It include the distributed storage in the response The malfunction of system；Can the memory node that the malfunction is used to indicate at least one failure when first is default It is all repaired in long.
18. device according to claim 17, which is characterized in that described device further include:
Processing module, for carrying out troubleshooting based on the malfunction for including in the response.
19. device according to claim 18, which is characterized in that the processing module is used for:
It is and described when the access request is sent by the destination client in the distributed memory system based on target virtual machine When target virtual machine is VMWare virtual machine, if the malfunction is the Fisrt fault state, not to the destination virtual Machine responds the access request, and the Fisrt fault state is used to indicate at least one memory node can be in the first preset duration All it is repaired；
When the access request is sent by the destination client based on target virtual machine, and the target virtual machine is VMWare When virtual machine, if the malfunction is second malfunction, Xiang Suoshu target virtual machine returns to disappearing for storage exception Breath, second malfunction be used to indicate at least one described memory node cannot in first preset duration whole quilts It repairs.
20. device according to claim 18, which is characterized in that the processing module is used for:
When the target access request by the destination client in the distributed memory system be based on target virtual machine transmission, and When the target virtual machine is not VMWare virtual machine, if the malfunction is the Fisrt fault state, Xiang Suoshu target Virtual machine transmission retries request, it is described retry request and be used to indicate re-issue the access request, the Fisrt fault state Being used to indicate at least one memory node can all be repaired in the first preset duration；
When target access request is based on target virtual machine transmission by the destination client, and the target virtual machine is not When VMWare virtual machine, if the malfunction is second malfunction, Xiang Suoshu target virtual machine returns to the target The identifiable target error of virtual machine, the target error are used to indicate storage media failure, and second malfunction is used for Indicate that at least one described memory node cannot be all repaired in first preset duration.
21. a kind of computer equipment, which is characterized in that the computer equipment includes processor and memory, the memory In be stored at least one instruction, described instruction is loaded by the processor and is executed to realize as claim 1 to right is wanted Ask operation performed by 9 described in any item distributed memory system fault handling methods.
22. a kind of storage medium, which is characterized in that be stored at least one instruction in the storage medium, described instruction by Reason device is loaded and is executed to realize as claim 1 to right wants 9 described in any item distributed memory system troubleshooting sides Operation performed by method.
CN201910741190.5A 2019-08-12 2019-08-12 Fault processing method and device, computer equipment, storage medium and storage system Active CN110535692B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201910741190.5A CN110535692B (en) 2019-08-12 2019-08-12 Fault processing method and device, computer equipment, storage medium and storage system
PCT/CN2020/102302 WO2021027481A1 (en) 2019-08-12 2020-07-16 Fault processing method, apparatus, computer device, storage medium and storage system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910741190.5A CN110535692B (en) 2019-08-12 2019-08-12 Fault processing method and device, computer equipment, storage medium and storage system

Publications (2)

Publication Number Publication Date
CN110535692A true CN110535692A (en) 2019-12-03
CN110535692B CN110535692B (en) 2020-12-18

Family

ID=68662506

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910741190.5A Active CN110535692B (en) 2019-08-12 2019-08-12 Fault processing method and device, computer equipment, storage medium and storage system

Country Status (2)

Country Link
CN (1) CN110535692B (en)
WO (1) WO2021027481A1 (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112187919A (en) * 2020-09-28 2021-01-05 腾讯科技（深圳）有限公司 Storage node management method and related device
WO2021027481A1 (en) * 2019-08-12 2021-02-18 华为技术有限公司 Fault processing method, apparatus, computer device, storage medium and storage system
WO2021164163A1 (en) * 2020-02-21 2021-08-26 苏州浪潮智能科技有限公司 Request processing method and apparatus, device and storage medium
CN113326251A (en) * 2021-06-25 2021-08-31 深信服科技股份有限公司 Data management method, system, device and storage medium
WO2021249173A1 (en) * 2020-06-12 2021-12-16 华为技术有限公司 Distributed storage system, abnormality processing method therefor, and related device
CN114584454A (en) * 2022-02-21 2022-06-03 苏州浪潮智能科技有限公司 Server information processing method and device, electronic equipment and storage medium
CN116382850A (en) * 2023-04-10 2023-07-04 北京志凌海纳科技有限公司 Virtual machine high availability management device and system using multi-storage heartbeat detection
WO2023208228A1 (en) * 2022-04-28 2023-11-02 华为技术有限公司 Storage device and data processing method

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11544139B1 (en) * 2021-11-30 2023-01-03 Vast Data Ltd. Resolving erred 10 flows

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104935481A (en) * 2015-06-24 2015-09-23 华中科技大学 Data recovery method based on redundancy mechanism in distributed storage
CN108984107A (en) * 2017-06-02 2018-12-11 伊姆西Ip控股有限责任公司 Improve the availability of storage system
US20190004845A1 (en) * 2017-06-28 2019-01-03 Vmware, Inc. Virtual machine placement based on device profiles

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103092712B (en) * 2011-11-04 2016-03-30 阿里巴巴集团控股有限公司 A kind of tasks interrupt restoration methods and equipment
CN109831342A (en) * 2019-03-19 2019-05-31 江苏汇智达信息科技有限公司 A kind of fault recovery method based on distributed system
CN110535692B (en) * 2019-08-12 2020-12-18 华为技术有限公司 Fault processing method and device, computer equipment, storage medium and storage system

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104935481A (en) * 2015-06-24 2015-09-23 华中科技大学 Data recovery method based on redundancy mechanism in distributed storage
CN108984107A (en) * 2017-06-02 2018-12-11 伊姆西Ip控股有限责任公司 Improve the availability of storage system
US20190004845A1 (en) * 2017-06-28 2019-01-03 Vmware, Inc. Virtual machine placement based on device profiles

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021027481A1 (en) * 2019-08-12 2021-02-18 华为技术有限公司 Fault processing method, apparatus, computer device, storage medium and storage system
WO2021164163A1 (en) * 2020-02-21 2021-08-26 苏州浪潮智能科技有限公司 Request processing method and apparatus, device and storage medium
WO2021249173A1 (en) * 2020-06-12 2021-12-16 华为技术有限公司 Distributed storage system, abnormality processing method therefor, and related device
CN112187919A (en) * 2020-09-28 2021-01-05 腾讯科技（深圳）有限公司 Storage node management method and related device
CN112187919B (en) * 2020-09-28 2024-01-23 腾讯科技（深圳）有限公司 Storage node management method and related device
CN113326251A (en) * 2021-06-25 2021-08-31 深信服科技股份有限公司 Data management method, system, device and storage medium
CN113326251B (en) * 2021-06-25 2024-02-23 深信服科技股份有限公司 Data management method, system, device and storage medium
CN114584454A (en) * 2022-02-21 2022-06-03 苏州浪潮智能科技有限公司 Server information processing method and device, electronic equipment and storage medium
CN114584454B (en) * 2022-02-21 2023-08-11 苏州浪潮智能科技有限公司 Processing method and device of server information, electronic equipment and storage medium
WO2023208228A1 (en) * 2022-04-28 2023-11-02 华为技术有限公司 Storage device and data processing method
CN116382850A (en) * 2023-04-10 2023-07-04 北京志凌海纳科技有限公司 Virtual machine high availability management device and system using multi-storage heartbeat detection
CN116382850B (en) * 2023-04-10 2023-11-07 北京志凌海纳科技有限公司 Virtual machine high availability management device and system using multi-storage heartbeat detection

Also Published As

Publication number Publication date
WO2021027481A1 (en) 2021-02-18
CN110535692B (en) 2020-12-18

Similar Documents

Publication Publication Date Title
CN110535692A (en) Fault handling method, device, computer equipment, storage medium and storage system
CN100417081C (en) Method, system for checking and repairing a network configuration
JP6476348B2 (en) Implementing automatic switchover
US6742059B1 (en) Primary and secondary management commands for a peripheral connected to multiple agents
US7961594B2 (en) Methods and systems for history analysis for access paths in networks
CN101390336B (en) Disaster recovery architecture
US9137148B2 (en) Information processing system and information processing apparatus
US20070050417A1 (en) Storage management method and a storage system
AU7868994A (en) Determination of network topology
CN104503965A (en) High-elasticity high availability and load balancing realization method of PostgreSQL (Structured Query Language)
US8189458B2 (en) Monitoring system, monitoring device, monitored device, and monitoring method
US20140201356A1 (en) Monitoring system of managing cloud-based hosts and monitoring method using for the same
US20020124079A1 (en) System for inference of presence of network infrastructure devices
JP2005327271A (en) Reconfiguration of logical setting in storage system
US7499987B2 (en) Deterministically electing an active node
CN102752146B (en) Cluster topological graph generation method and server
JP6033189B2 (en) COMMUNICATION DEVICE, COMMUNICATION METHOD, AND COMMUNICATION PROGRAM
CN105573872B (en) The HD management method and apparatus of data-storage system
CN109150602A (en) A kind of method, apparatus, computer storage medium and the terminal of adaptation storage equipment
JP4761978B2 (en) Hard disk redundancy management device, method, program, and monitoring control system
CN115632706B (en) FC link management method, device, equipment and readable storage medium
JPH0728667A (en) Fault-tolerant computer system
JP2003032257A (en) Method of specifying installed spot of lan component unit and retrieval device
JP2000090028A (en) Network managing device and computer readble recording medium which records program
US6990609B2 (en) System and method for isolating faults in a network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant