CN108958973A - Distributed file system correcting and eleting codes data reconstruction memory node selection method and device - Google Patents
Distributed file system correcting and eleting codes data reconstruction memory node selection method and device Download PDFInfo
- Publication number
- CN108958973A CN108958973A CN201810677871.5A CN201810677871A CN108958973A CN 108958973 A CN108958973 A CN 108958973A CN 201810677871 A CN201810677871 A CN 201810677871A CN 108958973 A CN108958973 A CN 108958973A
- Authority
- CN
- China
- Prior art keywords
- data
- memory
- node
- file system
- memory node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1479—Generic software techniques for error detection or fault masking
- G06F11/1489—Generic software techniques for error detection or fault masking through recovery blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
- G06F11/1451—Management of the data involved in backup or backup restore by selection of backup contents
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the present invention discloses a kind of distributed file system correcting and eleting codes data reconstruction memory node selection method and device, wherein, method includes: for any memory node in distributed file system, when the memory node has corrupted data, according to the reading record sheet of memory nodes all in distributed file system, all data needed for obtaining trade-off decision algorithm；According to all data needed for trade-off decision algorithm, the priority weighting of each candidate storage node is obtained, each candidate storage node is other each memory nodes in distributed file system other than this memory node；The priority weighting of each candidate storage node is ranked up, the maximum preset quantity candidate storage node of priority weighting is chosen from each candidate storage node, and reads data from the preset quantity candidate storage node of selection and carries out data reconstruction.The embodiment of the present invention is randomly selected compared to existing, can be improved the speed of reading data in data reconstruction.
Description
Technical field
The present embodiments relate to distributed memory system technical fields, and in particular to a kind of distributed file system, which is entangled, deletes
Code data reconstruction memory node selection method and device.
Background technique
Since monokaryon computing capability is limited, more and more applications start to realize by the way of distributed computing, such as
Solve complicated mathematical problem, password cracking, gene coding and protein structure research etc..In addition Internet company rises abruptly
It rises, the prevalence of e-commerce is but also more and more data are stored on large-scale distributed cluster.Distributed computing
Task is often run on the basis of distributed file system.Distributed file system can establish system based on more independent machines
One storage system provides unified storage service for program.The HDFS of the GFS of Google, Hadoop project and Microsoft
Windows AzureStorage belongs to distributed file system.
In large-scale distributed file system, delay machine, mistake and exception etc. failure are unavoidable.It is desirable to
It can keep high reliability and high availability in use day by day.Therefore it needs to set up fault tolerant mechanism, so that because
The data lost for mechanical disorder can be resumed.General way first is that more parts of data copy, there are different ground
Side, such as this scheme that version before Hadoop 3.0 all uses.Triple duplications are common way, many data centers
All use this way.The harm of this way is to occupy more additional memory space, leads to the expense mistake of system resource
Greatly.Nowadays each data center needs the data volume stored to explode, and the expense of book copying is just more difficult to bear.Correcting and eleting codes
(erasure coding) can obtain stronger appearance under lesser extra storage expense compared to the conventional method of book copying
Wrong ability.Nowadays correcting and eleting codes are more and more applied in distributed document, to reinforce the reliability of cluster.
When occurring mechanical disorder in distributed file system, the influence of the speed of data reconstruction to the overall performance of cluster
It is huge.Correcting and eleting codes needs when carrying out data reconstruction read data from multiple nodes and carry out the data that repair data is damaged node.
And hard disk I/O (input and output) pressure and network transmission pressure of each memory node in current distributed file system cluster
Power is different, therefore selects suitable memory node, is particularly important with alap delay to read data.Existing rank
The research that section reads memory node to selection in data reconstruction is also very weak, most of system realize such as ceph be with
Several nodes are selected in the slave alternate node of machine, cannot further optimize the speed of data reconstruction.
In consideration of it, how memory node is selected in distributed file system correcting and eleting codes data reconstruction, to improve data weight
The speed of reading data becomes the current technical issues that need to address in structure.
Summary of the invention
Since existing method is there are the above problem, the embodiment of the present invention proposes a kind of distributed file system correcting and eleting codes data
Reconstruct memory node selection method and device.
In a first aspect, the embodiment of the present invention proposes a kind of distributed file system correcting and eleting codes data reconstruction memory node selection
Method, comprising:
For any memory node in distributed file system, when the memory node has corrupted data, according to distribution
The reading record sheet of all memory nodes in formula file system, all data needed for obtaining trade-off decision algorithm；
According to all data needed for the trade-off decision algorithm, the priority weighting of each candidate storage node, institute are obtained
Stating each candidate storage node is other each memory nodes in distributed file system other than this memory node；
The priority weighting of each candidate storage node is ranked up, chooses priority from each candidate storage node
The maximum preset quantity candidate storage node of weight, and from the preset quantity candidate storage node of selection read data into
Row data reconstruction.
Optionally, all data needed for the trade-off decision algorithm, comprising: the total data reconstruction of this memory node is read
Number, this memory node read the number of data and the performance parameter of other each memory nodes from other each memory nodes, described
Other each memory nodes are other each memory nodes in distributed file system other than this memory node.
Optionally, for any memory node in distributed file system, when the memory node has corrupted data,
According to the reading record sheet of memory nodes all in distributed file system, all data needed for obtaining trade-off decision algorithm it
Before, the method also includes:
Each memory node in distributed file system safeguards a reading record sheet, record the memory node from other
Each memory node reads the average retardation of data, the memory node reads the number of data from other each memory nodes and is somebody's turn to do
The total data reconstruction reading times of memory node.
Optionally, the trade-off decision algorithm, comprising: game theory Slot Machine Bandit UCB algorithm.
Optionally, the method also includes:
When reading data progress data reconstruction from the preset quantity candidate storage node of selection, record is read each time
The delay time of data, the memory node for updating each memory node corresponding record in reading record sheet are saved from other each storages
The average retardation that point reads data reads the number of data with the memory node from other each memory nodes.
Second aspect, the embodiment of the present invention also propose a kind of distributed file system correcting and eleting codes data reconstruction memory node choosing
Select device, comprising:
First obtains module, for having number in the memory node for any memory node in distributed file system
When according to damage, according to the reading record sheet of memory nodes all in distributed file system, obtain needed for trade-off decision algorithm
All data；
Second acquisition module obtains each candidate storage section for all data according to needed for the trade-off decision algorithm
The priority weighting of point, each candidate storage node be in distributed file system other than this memory node other are each
Memory node；
Module is chosen, for the priority weighting of each candidate storage node to be ranked up, from each candidate storage section
The maximum preset quantity candidate storage node of priority weighting is chosen in point, and from the preset quantity candidate storage section of selection
Data are read in point carries out data reconstruction.
Optionally, described device further include:
Maintenance module safeguards that a reading record sheet, record should for each memory node in distributed file system
Memory node reads the average retardation of data, the memory node from other each memory nodes and reads data from other each memory nodes
Number and the total data reconstruction reading times of the memory node.
Optionally, described device further include:
Update module, when carrying out data reconstruction for reading data from the preset quantity candidate storage node of selection,
Record reads the delay time of data each time, updates described storage for reading each memory node corresponding record in record sheet
Node reads data from other each memory nodes with the memory node from the average retardation that other each memory nodes read data
Number.
The third aspect, the embodiment of the present invention also propose a kind of electronic equipment, comprising: processor, memory, bus and storage
On a memory and the computer program that can run on a processor；
Wherein, the processor, memory complete mutual communication by the bus；
The processor realizes the above method when executing the computer program.
Fourth aspect, the embodiment of the present invention provide a kind of non-transient computer readable storage medium, the non-transient calculating
It is stored with computer program on machine readable storage medium storing program for executing, which realizes the above method when being executed by processor.
As shown from the above technical solution, a kind of distributed file system correcting and eleting codes data reconstruction provided in an embodiment of the present invention
Memory node selection method and device, by having in the memory node for any memory node in distributed file system
When corrupted data, according to the reading record sheet of memory nodes all in distributed file system, obtain needed for trade-off decision algorithm
All data；According to all data needed for trade-off decision algorithm, the priority weighting of each candidate storage node is obtained, it is described
Each candidate storage node is other each memory nodes in distributed file system other than this memory node；It is alternatively deposited each
The priority weighting of storage node is ranked up, and it is standby that the maximum preset quantity of priority weighting is chosen from each candidate storage node
Memory node is selected, and reads data from the preset quantity candidate storage node of selection and carries out data reconstruction, as a result, compared to existing
Some randomly selects, and can be improved the speed of reading data in data reconstruction, to further improve distributed file system
Performance.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with
Other attached drawings are obtained according to these figures.
Fig. 1 is a kind of distributed file system correcting and eleting codes data reconstruction memory node selection that one embodiment of the invention provides
The flow diagram of method；
Fig. 2 is a kind of distributed file system correcting and eleting codes data reconstruction memory node selection that one embodiment of the invention provides
The structural schematic diagram of device；
Fig. 3 is the entity structure schematic diagram for the electronic equipment that one embodiment of the invention provides.
Specific embodiment
With reference to the accompanying drawing, further description of the specific embodiments of the present invention.Following embodiment is only used for more
Technical solution of the present invention is clearly demonstrated, and not intended to limit the protection scope of the present invention.
Fig. 1 shows a kind of distributed file system correcting and eleting codes data reconstruction memory node of one embodiment of the invention offer
The flow diagram of selection method, as shown in Figure 1, the distributed file system correcting and eleting codes data reconstruction memory node of the present embodiment
Selection method, comprising:
S1, for any memory node in distributed file system, when the memory node has corrupted data, according to point
The reading record sheet of all memory nodes in cloth file system, all data needed for obtaining trade-off decision algorithm.
In a particular application, all data needed for the trade-off decision algorithm may include: the total number of this memory node
The number of data and the performance of other each memory nodes are read from other each memory nodes according to reconstruct reading times, this memory node
Parameter, other described each memory nodes are other each memory nodes in distributed file system other than this memory node.
S2, all data according to needed for the trade-off decision algorithm, obtain the priority weighting of each candidate storage node,
Each candidate storage node is other each memory nodes in distributed file system other than this memory node.
S3, the priority weighting of each candidate storage node is ranked up, is chosen from each candidate storage node excellent
The first maximum preset quantity candidate storage node of grade weight, and number is read from the preset quantity candidate storage node of selection
According to progress data reconstruction.
The distributed file system correcting and eleting codes data reconstruction memory node selection method of the present embodiment, by for distribution
Any memory node in file system is deposited when the memory node has corrupted data according to all in distributed file system
The reading record sheet of node is stored up, all data needed for obtaining trade-off decision algorithm, according to items needed for trade-off decision algorithm
Data, obtain the priority weighting of each candidate storage node, each candidate storage node be in distributed file system in addition to
Other each memory nodes except this memory node, the priority weighting of each candidate storage node is ranked up, from each alternative
The maximum preset quantity candidate storage node of priority weighting is chosen in memory node, and alternative from the preset quantity of selection
Data are read in memory node and carry out data reconstruction, are randomly selected as a result, compared to existing, be can be improved data in data reconstruction
The speed of reading, to further improve the performance of distributed file system.
Further, on the basis of the above embodiments, before the step S1, the present embodiment the method can be with
Include:
Each memory node in distributed file system safeguards a reading record sheet, record the memory node from other
Each memory node reads the average retardation of data, the memory node reads the number of data from other each memory nodes and is somebody's turn to do
The total data reconstruction reading times of memory node.
Further, on the basis of the above embodiments, the step S2 may include:
Each candidate storage section is calculated by the first formula according to all data needed for the trade-off decision algorithm
The priority weighting of point；
First formula are as follows:
wj=xj+sqrt(2ln(n)/nj)
Wherein, wjFor the priority weighting of j-th of memory node in other each memory nodes, j=1 ..., s, s are it
The quantity of his memory node, njFor the number for reading data from j-th of memory node in other each memory nodes, n is originally to deposit
The total data reconstruction reading times of node are stored up, sqrt indicates the square root that counts, xjFor j-th of storage in other each memory nodes
The performance parameter of node, xj=c/yj, yjProlong to read being averaged for data from j-th of memory node in other each memory nodes
Late, c is pre-set constant value.The c parameter to be arranged when being applied in specific system, it is more sensitive to access delay when c is larger, it is former
Any positive number value is taken on then.
The priority weighting of the available each candidate storage node of the present embodiment as a result,.
Further, on the basis of the above embodiments, the trade-off decision algorithm has game theory theoretical foundation, can wrap
It includes: game theory Slot Machine Bandit UCB algorithm etc..
It is understood that original game theory Bandit UCB algorithm magnitude be n times reading in, non-optimal reading
Magnitude be only log n, optimize the speed of reading data in data reconstruction.
It is understood that the present embodiment solves correcting and eleting codes data in distributed file system using the algorithm of game theory
The problem of memory node selects when reconstruct, it can be applied in a variety of distributed file systems with correcting and eleting codes.
Further, on the basis of the above embodiments, the method can also include:
When reading data progress data reconstruction from the preset quantity candidate storage node of selection, record is read each time
The delay time of data, the memory node for updating each memory node corresponding record in reading record sheet are saved from other each storages
The average retardation that point reads data reads the number of data with the memory node from other each memory nodes.
In a particular application, the present embodiment is carrying out data reconstruction every time, when reading data from each candidate storage node, needs
A timestamp is recorded, it should logging timestamp again after this reading.
In a particular application, the reading record sheet can realize into Key (key)-Value (value) form or be directly accessed shape
Formula.
It is understood that the present embodiment can use during data record, data calculate and data are submitted
Key-Value form is directly accessed form, can facilitate subsequent quick retrieval and reading data.
The distributed file system correcting and eleting codes data reconstruction memory node selection method of the present embodiment, prolongs using about reading
Slow and reading times statistical data, the memory node that finally read data is determined in conjunction with trade-off decision algorithm, compared to existing
Some randomly selects, and can be improved the speed of reading data in data reconstruction, to further improve distributed file system
Performance.
Fig. 2 shows a kind of distributed file system correcting and eleting codes data reconstruction memory nodes that one embodiment of the invention provides
The structural schematic diagram of selection device, as shown in Fig. 2, the distributed file system correcting and eleting codes data reconstruction memory node of the present embodiment
Selection device, comprising: first, which obtains module 21, second, obtains module 22 and choose module 23；Wherein:
Described first obtains module 21, for for any memory node in distributed file system, in the storage section
When point has corrupted data, according to the reading record sheet of memory nodes all in distributed file system, trade-off decision algorithm is obtained
Required all data；
Described second obtains module 22, for all data according to needed for the trade-off decision algorithm, obtains each alternative
The priority weighting of memory node, each candidate storage node be distributed file system in other than this memory node
Other each memory nodes；
The selection module 23, for the priority weighting of each candidate storage node to be ranked up, from each alternative
The maximum preset quantity candidate storage node of priority weighting is chosen in memory node, and alternative from the preset quantity of selection
Data are read in memory node carries out data reconstruction.
Specifically, described first module 21 is obtained for any memory node in distributed file system, in the storage
When node has corrupted data, according to the reading record sheet of memory nodes all in distributed file system, obtains trade-off decision and calculate
All data needed for method；Described second obtains all data according to needed for the trade-off decision algorithm of module 22, obtains each
The priority weighting of candidate storage node, each candidate storage node be distributed file system in addition to this memory node it
Other outer each memory nodes；The priority weighting of each candidate storage node is ranked up by the selection module 23, from
The maximum preset quantity candidate storage node of priority weighting is chosen in each candidate storage node, and from the preset quantity of selection
Data are read in a candidate storage node carries out data reconstruction.
In a particular application, all data needed for the trade-off decision algorithm may include: the total number of this memory node
The number of data and the performance of other each memory nodes are read from other each memory nodes according to reconstruct reading times, this memory node
Parameter, other described each memory nodes are other each memory nodes in distributed file system other than this memory node.
The distributed file system correcting and eleting codes data reconstruction memory node selection device of the present embodiment, it is random compared to existing
It chooses, can be improved the speed of reading data in data reconstruction, to further improve the performance of distributed file system.
Further, on the basis of the above embodiments, the present embodiment described device can also include not shown in the figure:
Maintenance module safeguards that a reading record sheet, record should for each memory node in distributed file system
Memory node reads the average retardation of data, the memory node from other each memory nodes and reads data from other each memory nodes
Number and the total data reconstruction reading times of the memory node.
Further, on the basis of the above embodiments, the second acquisition module 22 can be specifically used for
Each candidate storage section is calculated by the first formula according to all data needed for the trade-off decision algorithm
The priority weighting of point；
First formula are as follows:
wj=xj+sqrt(2ln(n)/nj)
Wherein, wjFor the priority weighting of j-th of memory node in other each memory nodes, j=1 ..., s, s are it
The quantity of his memory node, njFor the number for reading data from j-th of memory node in other each memory nodes, n is originally to deposit
The total data reconstruction reading times of node are stored up, sqrt indicates the square root that counts, xjFor j-th of storage in other each memory nodes
The performance parameter of node, xj=c/yj, yjProlong to read being averaged for data from j-th of memory node in other each memory nodes
Late, c is pre-set constant value.The c parameter to be arranged when being applied in specific system, it is more sensitive to access delay when c is larger, it is former
Any positive number value is taken on then.
The priority weighting of the available each candidate storage node of the present embodiment as a result,.
Further, on the basis of the above embodiments, the trade-off decision algorithm has game theory theoretical foundation, can wrap
It includes: game theory Slot Machine Bandit UCB algorithm etc..
It is understood that original game theory Bandit UCB algorithm magnitude be n times reading in, non-optimal reading
Magnitude be only log n, optimize the speed of reading data in data reconstruction.
It is understood that the present embodiment solves correcting and eleting codes data in distributed file system using the algorithm of game theory
The problem of memory node selects when reconstruct, it can be applied in a variety of distributed file systems with correcting and eleting codes.
Further, on the basis of the above embodiments, the present embodiment described device can also include not shown in the figure:
Update module, when carrying out data reconstruction for reading data from the preset quantity candidate storage node of selection,
Record reads the delay time of data each time, updates described storage for reading each memory node corresponding record in record sheet
Node reads data from other each memory nodes with the memory node from the average retardation that other each memory nodes read data
Number.
In a particular application, the present embodiment is carrying out data reconstruction every time, when reading data from each candidate storage node, needs
A timestamp is recorded, it should logging timestamp again after this reading.
In a particular application, the reading record sheet can realize into Key-Value form or be directly accessed form.
It is understood that the present embodiment can use during data record, data calculate and data are submitted
Key-Value form is directly accessed form, can facilitate subsequent quick retrieval and reading data.
The distributed file system correcting and eleting codes data reconstruction memory node selection device of the present embodiment, prolongs using about reading
Slow and reading times statistical data, the memory node that finally read data is determined in conjunction with trade-off decision algorithm, compared to existing
Some randomly selects, and can be improved the speed of reading data in data reconstruction, to further improve distributed file system
Performance.
The distributed file system correcting and eleting codes data reconstruction memory node selection device of the present embodiment, before can be used for execution
The technical solution of embodiment of the method is stated, it is similar that the realization principle and technical effect are similar, and details are not described herein again.
Fig. 3 shows the entity structure schematic diagram of a kind of electronic equipment provided in an embodiment of the present invention, as shown in figure 3, should
Electronic equipment may include: processor 31, memory 32, bus 33 and be stored on memory 32 and can transport on processor 31
Capable computer program；
Wherein, the processor 31, memory 32 complete mutual communication by the bus 33；
The processor 31 realizes method provided by above-mentioned each method embodiment when executing the computer program, such as
It include: for any memory node in distributed file system, when the memory node has corrupted data, according to distributed text
The reading record sheet of all memory nodes in part system, all data needed for obtaining trade-off decision algorithm；According to the selection
All data needed for decision making algorithm, obtains the priority weighting of each candidate storage node, and each candidate storage node is point
Other each memory nodes in cloth file system other than this memory node；By the priority of each candidate storage node
Weight is ranked up, and the maximum preset quantity candidate storage node of priority weighting is chosen from each candidate storage node, and
Data are read from the preset quantity candidate storage node of selection carries out data reconstruction.
The embodiment of the present invention provides a kind of non-transient computer readable storage medium, is stored thereon with computer program, should
Method provided by above-mentioned each method embodiment is realized when computer program is executed by processor, for example, obtain target quotient
The current information of all knights in shop in family, the current information, comprising: for any storage section in distributed file system
Point, according to the reading record sheet of memory nodes all in distributed file system, is obtained when the memory node has corrupted data
All data needed for trade-off decision algorithm；According to all data needed for the trade-off decision algorithm, each candidate storage is obtained
The priority weighting of node, each candidate storage node are other other than this memory node in distributed file system
Each memory node；The priority weighting of each candidate storage node is ranked up, is chosen from each candidate storage node excellent
The first maximum preset quantity candidate storage node of grade weight, and number is read from the preset quantity candidate storage node of selection
According to progress data reconstruction.
It should be understood by those skilled in the art that, embodiments herein can provide as method, apparatus or computer program
Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the application
Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the application, which can be used in one or more,
The computer program implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) produces
The form of product.
The application be referring to according to the method, apparatus of the embodiment of the present application and the flow chart of computer program product and/or
Block diagram describes.It should be understood that each process that can be realized by computer program instructions in flowchart and/or the block diagram and/or
The combination of process and/or box in box and flowchart and/or the block diagram.It can provide these computer program instructions to arrive
General purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices processor to generate one
Machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for realizing flowing
The device/system for the function of being specified in journey figure one process or multiple processes and/or block diagrams one box or multiple boxes.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or
The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one
The step of function of being specified in a box or multiple boxes.
It should be noted that, in this document, relational terms such as first and second and the like are used merely to a reality
Body or operation are distinguished with another entity or operation, are deposited without necessarily requiring or implying between these entities or operation
In any actual relationship or order or sequence.Moreover, the terms "include", "comprise" or its any other variant are intended to
Non-exclusive inclusion, so that the process, method, article or equipment including a series of elements is not only wanted including those
Element, but also including other elements that are not explicitly listed, or further include for this process, method, article or equipment
Intrinsic element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that
There is also other identical elements in process, method, article or equipment including the element.Term " on ", "lower" etc. refer to
The orientation or positional relationship shown is to be based on the orientation or positional relationship shown in the drawings, and is merely for convenience of the description present invention and simplifies
Description, rather than the device or element of indication or suggestion meaning must have a particular orientation, constructed and grasped with specific orientation
Make, therefore is not considered as limiting the invention.Unless otherwise clearly defined and limited, term " installation ", " connected ",
" connection " shall be understood in a broad sense, for example, it may be being fixedly connected, may be a detachable connection, or be integrally connected；It can be
Mechanical connection, is also possible to be electrically connected；It can be directly connected, two can also be can be indirectly connected through an intermediary
Connection inside element.For the ordinary skill in the art, above-mentioned term can be understood at this as the case may be
Concrete meaning in invention.
In specification of the invention, numerous specific details are set forth.Although it is understood that the embodiment of the present invention can
To practice without these specific details.In some instances, well known method, structure and skill is not been shown in detail
Art, so as not to obscure the understanding of this specification.Similarly, it should be understood that disclose in order to simplify the present invention and helps to understand respectively
One or more of a inventive aspect, in the above description of the exemplary embodiment of the present invention, each spy of the invention
Sign is grouped together into a single embodiment, figure, or description thereof sometimes.However, should not be by the method solution of the disclosure
Release is in reflect an intention that i.e. the claimed invention requires more than feature expressly recited in each claim
More features.More precisely, as the following claims reflect, inventive aspect is less than single reality disclosed above
Apply all features of example.Therefore, it then follows thus claims of specific embodiment are expressly incorporated in the specific embodiment,
It is wherein each that the claims themselves are regarded as separate embodiments of the invention.It should be noted that in the absence of conflict, this
The feature in embodiment and embodiment in application can be combined with each other.The invention is not limited to any single aspect,
It is not limited to any single embodiment, is also not limited to any combination and/or displacement of these aspects and/or embodiment.And
And can be used alone each aspect and/or embodiment of the invention or with other one or more aspects and/or its implementation
Example is used in combination.
Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention., rather than its limitations；To the greatest extent
Pipe present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that: its according to
So be possible to modify the technical solutions described in the foregoing embodiments, or to some or all of the technical features into
Row equivalent replacement；And these are modified or replaceed, various embodiments of the present invention technology that it does not separate the essence of the corresponding technical solution
The range of scheme should all cover within the scope of the claims and the description of the invention.
Claims (10)
1. a kind of distributed file system correcting and eleting codes data reconstruction memory node selection method characterized by comprising
For any memory node in distributed file system, when the memory node has corrupted data, according to distributed text
The reading record sheet of all memory nodes in part system, all data needed for obtaining trade-off decision algorithm；
According to all data needed for the trade-off decision algorithm, the priority weighting of each candidate storage node is obtained, it is described each
Candidate storage node is other each memory nodes in distributed file system other than this memory node；
The priority weighting of each candidate storage node is ranked up, chooses priority weighting from each candidate storage node
Maximum preset quantity candidate storage node, and read data from the preset quantity candidate storage node of selection and counted
According to reconstruct.
2. the method according to claim 1, wherein all data needed for the trade-off decision algorithm, comprising:
The total data reconstruction reading times of this memory node, this memory node from other each memory nodes read data number and other
The performance parameter of each memory node, other described each memory nodes be distributed file system in other than this memory node
Other each memory nodes.
3. the method according to claim 1, wherein for any storage section in distributed file system
Point, according to the reading record sheet of memory nodes all in distributed file system, is obtained when the memory node has corrupted data
Before all data needed for trade-off decision algorithm, the method also includes:
Each memory node in distributed file system safeguards a reading record sheet, records the memory node and respectively deposits from other
Store up number and storage that node reads the average retardation of data, the memory node reads data from other each memory nodes
The total data reconstruction reading times of node.
4. the method according to claim 1, wherein the trade-off decision algorithm, comprising: game theory Slot Machine
Bandit UCB algorithm.
5. method according to any of claims 1-4, which is characterized in that the method also includes:
When reading data progress data reconstruction from the preset quantity candidate storage node of selection, record reads data each time
Delay time, update and read the memory node of each memory node corresponding record in record sheet and read from other each memory nodes
The average retardation and the memory node for evidence of fetching read the number of data from other each memory nodes.
6. a kind of distributed file system correcting and eleting codes data reconstruction memory node selection device characterized by comprising
First obtains module, for having data damage in the memory node for any memory node in distributed file system
Bad when, according to the reading record sheet of memory nodes all in distributed file system, items needed for obtaining trade-off decision algorithm
Data；
Second acquisition module obtains each candidate storage node for all data according to needed for the trade-off decision algorithm
Priority weighting, each candidate storage node are other each storages in distributed file system other than this memory node
Node；
Module is chosen, for the priority weighting of each candidate storage node to be ranked up, from each candidate storage node
The maximum preset quantity candidate storage node of priority weighting is chosen, and from the preset quantity candidate storage node of selection
It reads data and carries out data reconstruction.
7. device according to claim 6, which is characterized in that described device further include:
Maintenance module safeguards a reading record sheet for each memory node in distributed file system, records the storage
Node reads the average retardation of data, the memory node from other each memory nodes and reads the secondary of data from other each memory nodes
Several and total memory node data reconstruction reading times.
8. device according to claim 6, which is characterized in that described device further include:
Update module, when carrying out data reconstruction for reading data from the preset quantity candidate storage node of selection, record
The delay time for reading data each time updates described memory node for reading each memory node corresponding record in record sheet
Number of the average retardation of data with the memory node from other each memory nodes reading data is read from other each memory nodes.
9. a kind of electronic equipment characterized by comprising processor, memory, bus and storage on a memory and can located
The computer program run on reason device；
Wherein, the processor, memory complete mutual communication by the bus；
The processor realizes method according to any one of claims 1 to 5 when executing the computer program.
10. a kind of non-transient computer readable storage medium, which is characterized in that in the non-transient computer readable storage medium
It is stored with computer program, which realizes side according to any one of claims 1 to 5 when being executed by processor
Method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810677871.5A CN108958973B (en) | 2018-06-27 | 2018-06-27 | Distributed file system erasure code data reconstruction storage node selection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810677871.5A CN108958973B (en) | 2018-06-27 | 2018-06-27 | Distributed file system erasure code data reconstruction storage node selection method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108958973A true CN108958973A (en) | 2018-12-07 |
| CN108958973B CN108958973B (en) | 2020-07-07 |
Family
ID=64486969
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810677871.5A Active CN108958973B (en) | 2018-06-27 | 2018-06-27 | Distributed file system erasure code data reconstruction storage node selection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108958973B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110515917A (en) * | 2019-08-09 | 2019-11-29 | 苏州浪潮智能科技有限公司 | A kind of method, apparatus and medium of control reconfiguration speed |
| CN112799584A (en) * | 2019-11-13 | 2021-05-14 | 杭州海康威视数字技术股份有限公司 | Data storage method and device |
| CN113672174A (en) * | 2021-08-03 | 2021-11-19 | 中移（杭州）信息技术有限公司 | Data reconstruction method, device, storage medium and device |
| CN118132520A (en) * | 2024-05-08 | 2024-06-04 | 济南浪潮数据技术有限公司 | Storage system file processing method, electronic device, storage medium and program product |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103645861A (en) * | 2013-12-03 | 2014-03-19 | 华中科技大学 | Method for reconstructing failure node in erasure code cluster |
| CN103916483A (en) * | 2014-04-28 | 2014-07-09 | 中国科学院成都生物研究所 | Self-adaptation data storage and reconstruction method for coding redundancy storage system |
| CN107357685A (en) * | 2017-07-11 | 2017-11-17 | 清华大学 | A kind of Tolerate and redundance method and apparatus of data storage |
| CN107689983A (en) * | 2017-07-05 | 2018-02-13 | 上海交通大学 | Cloud storage system and method based on low reparation bandwidth |
-
2018
- 2018-06-27 CN CN201810677871.5A patent/CN108958973B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103645861A (en) * | 2013-12-03 | 2014-03-19 | 华中科技大学 | Method for reconstructing failure node in erasure code cluster |
| CN103916483A (en) * | 2014-04-28 | 2014-07-09 | 中国科学院成都生物研究所 | Self-adaptation data storage and reconstruction method for coding redundancy storage system |
| CN107689983A (en) * | 2017-07-05 | 2018-02-13 | 上海交通大学 | Cloud storage system and method based on low reparation bandwidth |
| CN107357685A (en) * | 2017-07-11 | 2017-11-17 | 清华大学 | A kind of Tolerate and redundance method and apparatus of data storage |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110515917A (en) * | 2019-08-09 | 2019-11-29 | 苏州浪潮智能科技有限公司 | A kind of method, apparatus and medium of control reconfiguration speed |
| CN110515917B (en) * | 2019-08-09 | 2022-12-02 | 苏州浪潮智能科技有限公司 | Method, device and medium for controlling reconstruction speed |
| CN112799584A (en) * | 2019-11-13 | 2021-05-14 | 杭州海康威视数字技术股份有限公司 | Data storage method and device |
| CN112799584B (en) * | 2019-11-13 | 2023-04-07 | 杭州海康威视数字技术股份有限公司 | Data storage method and device |
| CN113672174A (en) * | 2021-08-03 | 2021-11-19 | 中移（杭州）信息技术有限公司 | Data reconstruction method, device, storage medium and device |
| CN113672174B (en) * | 2021-08-03 | 2024-05-07 | 中移（杭州）信息技术有限公司 | Data reconstruction method, device, storage medium and apparatus |
| CN118132520A (en) * | 2024-05-08 | 2024-06-04 | 济南浪潮数据技术有限公司 | Storage system file processing method, electronic device, storage medium and program product |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108958973B (en) | 2020-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11068389B2 (en) | Data resiliency with heterogeneous storage | |
| US11567810B1 (en) | Cost optimized workload placement | |
| US11934260B2 (en) | Problem signature-based corrective measure deployment | |
| US10454498B1 (en) | Fully pipelined hardware engine design for fast and efficient inline lossless data compression | |
| US20190171365A1 (en) | Hybrid data tiering | |
| US11080155B2 (en) | Identifying error types among flash memory | |
| CN108958973A (en) | Distributed file system correcting and eleting codes data reconstruction memory node selection method and device | |
| US11604583B2 (en) | Policy based data tiering | |
| WO2019046635A1 (en) | Encryption management with host-side data reduction | |
| WO2019160689A1 (en) | Storage layer data security | |
| US20240231939A1 (en) | Queueing Storage Operations | |
| CN105468473A (en) | Data migration method and data migration apparatus | |
| WO2022159162A1 (en) | Using data similarity to select segments for garbage collection | |
| US10356150B1 (en) | Automated repartitioning of streaming data | |
| JP6362805B1 (en) | Method for enabling access to past transactions in a blockchain network and nodes for configuring the network | |
| US11847025B2 (en) | Storage system parity based on system characteristics | |
| WO2016118429A1 (en) | System for efficient processing of transaction requests related to an account in a database | |
| Ibrahim et al. | Intelligent data placement mechanism for replicas distribution in cloud storage systems | |
| US8931051B2 (en) | Scalable and highly available clustering for large scale real-time applications | |
| US10515228B2 (en) | Commit and rollback of data streams provided by partially trusted entities | |
| US11392553B1 (en) | Remote data management | |
| CN109582213A (en) | Data reconstruction method and device, data-storage system | |
| WO2015015339A1 (en) | A method for a logging process in a data storage system | |
| WO2019209392A1 (en) | Hybrid data tiering | |
| CN112948279A (en) | Method, apparatus and program product for managing access requests in a storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |