CN103942116B - Method for conducting fault-tolerance processing on data and data processing system - Google Patents
Method for conducting fault-tolerance processing on data and data processing system Download PDFInfo
- Publication number
- CN103942116B CN103942116B CN201410169184.4A CN201410169184A CN103942116B CN 103942116 B CN103942116 B CN 103942116B CN 201410169184 A CN201410169184 A CN 201410169184A CN 103942116 B CN103942116 B CN 103942116B
- Authority
- CN
- China
- Prior art keywords
- data
- fault
- marginal value
- algorithm
- network bandwidth
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Hardware Redundancy (AREA)
Abstract
The invention discloses a method for conducting fault-tolerance processing on data and a data processing system. The method includes the steps of obtaining data; obtaining processing resources of the data processing system, wherein the processing resources are resources which can be used when the data processing system processes the data; selecting a fault-tolerance processing algorithm for conducting fault-tolerance processing on the data according to the processing resources, and conducting fault-tolerance processing on the data according to the selected fault-tolerance processing algorithm. By means of the method, the fault-tolerance processing speed for the data can be increased, and the use ratio of the processing resources in the data processing system can be balanced.
Description
Technical field
The present invention relates to technical field of data processing, more particularly to a kind of method and number that data is carried out with fault-tolerant processing
According to processing system.
Background technology
With the development of computer technology and network technology, data is just increased with the speed of explosion type, and mass data is to depositing
Storage system proposes huge challenge.In order to avoid because of hardware fault, software fault or human operational error, causing storage system
Corrupted data on system or loss, and cause data cannot recover, generally the data in storage system is carried out with fault-tolerant processing,
The ability anti-risk to improve data.
In prior art, the method that carries out fault-tolerant processing to data includes following several ways:
1. copy algorithm, i.e. the complete copy to initial data.By increasing various multi-forms for the data in system
Copy, preserve the data of redundancy, the availability of data can be effectively improved, it is to avoid because network disconnects or mechanical disorder etc.
Factor and the loss of data that causes.Generally, copy amount is more, and the reliability of data is higher, but identical, meeting
Consume substantial amounts of system memory resources, and increase the complexity of data management.
2. correcting and eleting codes algorithm, as a kind of forward error correction technology, is mainly used in avoiding that wraps to lose in network transmission
Lose, storage system is utilized to improve memory reliability.The data that will store in systems is divided into k block, then it is compiled
N data fragmentation obtaining of code simultaneously carries out distribution and stores, then only need to there is the individual available data fragmentation of k ' it is possible to reconstruct former
Beginning data, the space complexity data redundancy of correcting and eleting codes algorithm is relatively low, however it is necessary that a large amount of calculate.
In sum it is known that, correcting and eleting codes algorithm and each have their own merits and demerits of copy algorithm, entered according to correcting and eleting codes algorithm
The reliability of the data after row fault-tolerant processing is higher, but calculates complicated, carries out fault-tolerant processing according to copy algorithm to data, calculates
Simply, but need more memory space.Generally fix in the data handling system of prior art and processed using a kind of data fault-tolerant
Algorithm, and regularly using a kind of data fault-tolerant Processing Algorithm often result in cannot maturely, evenly using system resource, be
System resource utilization is relatively low.
Content of the invention
The invention mainly solves the technical problem of providing a kind of method and device that data is carried out with fault-tolerant processing, can
According to the resource service condition of data handling system, select fault-tolerant processing algorithm, evenly using system resource, improve at data
Reason system resource utilization rate.
For solve above-mentioned technical problem, one aspect of the present invention is: provide a kind of data is carried out fault-tolerant
The method processing, including acquisition data；Obtain the process resource described data being carried out required for fault-tolerant processing；According to described place
Reason resource selection carries out the fault-tolerant processing algorithm of fault-tolerant processing to described data, and according to described selected fault-tolerant processing algorithm
Fault-tolerant processing is carried out to described data；Described process resource includes: idle storage space is than marginal value, current idle memory space
Than, available network bandwidth than marginal value and currently available network bandwidth ratio, wherein, described idle storage space is than for data processing
In system, idle memory space accounts for the ratio in total memory space, and described currently available network bandwidth ratio is for current unappropriated
The network bandwidth accounts for the ratio of overall network bandwidth；Described fault-tolerant processing algorithm includes correcting and eleting codes algorithm；Described according to described process money
Source selects described data is carried out with the fault-tolerant processing algorithm of fault-tolerant processing, and according to described fault-tolerant processing algorithm, described data is entered
The step of row fault-tolerant processing includes: judges whether described current idle memory space ratio is less than described idle storage space than critical
Value, and, whether the described currently available network bandwidth than compares marginal value more than described available network bandwidth；If described current idle
Whether memory space ratio is less than described idle storage space than marginal value, and, whether the described currently available network bandwidth is bigger than
In described available network bandwidth than marginal value, then fault-tolerant processing is carried out to described data according to described correcting and eleting codes algorithm；
Described correcting and eleting codes algorithm includes inner institute code correcting and eleting codes algorithm；Described process resource also includes: node free time faces
Dividing value, data block access frequency marginal value data stripping and slicing quantity marginal value, present node free time, current data block access
Frequency and current data stripping and slicing quantity, wherein, described node free time is that the node in data handling system is in idle shape
The time of state, described data block access frequency is that the data block in data is accessed for frequency, and described data stripping and slicing quantity is number
According to the quantity dividing data block；Described fault-tolerant processing step is carried out to described data according to correcting and eleting codes algorithm also include: judge institute
State whether present node free time is more than node free time marginal value, and, whether described current data block access frequency
Less than data block access frequency marginal value；If described present node free time is more than described node free time marginal value, with
And, described current data block access frequency is less than data block access frequency marginal value, then judge described current data stripping and slicing quantity
Whether it is less than data stripping and slicing quantity marginal value；If described current data stripping and slicing quantity is less than data stripping and slicing quantity marginal value, root
According to described inner institute code correcting and eleting codes algorithm, fault-tolerant processing is carried out to data.
Wherein, described correcting and eleting codes algorithm also includes low density parity check code correcting and eleting codes algorithm；Methods described also includes: if
Described current data stripping and slicing quantity is more than or equal to data stripping and slicing quantity marginal value, then according to described low density parity check code
Algorithm carries out fault-tolerant processing to data.
Wherein, described fault-tolerant processing algorithm includes copy algorithm；Methods described also includes: if the storage of described current idle is empty
Between than more than or equal to described idle storage space than marginal value, and, the described currently available network bandwidth than less than or wait
In described available network bandwidth than marginal value, then fault-tolerant processing is carried out to data according to described copy algorithm.
For solving above-mentioned technical problem, another technical solution used in the present invention is: a kind of data handling system is provided,
Including the first acquisition module, for obtaining data；Second acquisition module, is carried out needed for fault-tolerant processing to described data for obtaining
The process resource wanted；Select processing module, for selecting described data is carried out the appearance of fault-tolerant processing according to described process resource
Wrong Processing Algorithm, and fault-tolerant processing is carried out to described data according to described selected fault-tolerant processing algorithm；Described process resource
Including: idle storage space is than marginal value, current idle memory space ratio, available network bandwidth than marginal value and currently available net
Network bandwidth ratio, wherein, described idle storage space accounts in total memory space than for idle memory space in data handling system
Ratio, the described currently available network bandwidth accounts for the ratio of overall network bandwidth than for the current unappropriated network bandwidth；Described appearance
Wrong Processing Algorithm includes correcting and eleting codes algorithm；Described selection processing module includes: the first judging unit, for judging described current sky
Whether not busy memory space ratio is less than described idle storage space than marginal value, and, whether described currently available network bandwidth ratio
Compare marginal value more than described available network bandwidth；First processing units, for determining current sky in described first judging unit
Not busy memory space than less than idle storage space than marginal value, and, the described currently available network bandwidth is more available than more than described
When the network bandwidth is than marginal value, fault-tolerant processing is carried out to described data according to described correcting and eleting codes algorithm；
Described process resource also includes node free time marginal value, data block access frequency marginal value, data stripping and slicing number
Amount marginal value, present node free time, current data block access frequency and current data stripping and slicing quantity, wherein, described node
Free time is in the time of idle condition for the node in data handling system, and described data block access frequency is in data
Data block is accessed for frequency, and described data stripping and slicing quantity is the quantity of data partitioning data block；Described correcting and eleting codes algorithm includes
In institute's code correcting and eleting codes algorithm；Described first processing units include: the second judging unit, during for judging that described present node is idle
Between whether be more than node free time marginal value, and, whether described current data block access frequency less than data block access frequency
Rate marginal value；3rd judging unit, for determining described present node free time more than section in described second judging unit
Point free time marginal value, and, when described current data block access frequency is less than data block access frequency marginal value, judge institute
State whether current data stripping and slicing quantity is less than data stripping and slicing quantity marginal value；First fault-tolerant processing unit, for the described 3rd
When judging unit determines described current data stripping and slicing quantity less than data stripping and slicing quantity marginal value, entangled according to described inner institute code and delete
Code algorithm carries out fault-tolerant processing to data.
Wherein, described correcting and eleting codes algorithm also includes low density parity check code correcting and eleting codes algorithm；Described first processing units,
Also include: the second fault-tolerant processing unit, it is more than for determining described current data stripping and slicing quantity in described 3rd judging unit
Or when being equal to data stripping and slicing quantity marginal value, according to described low density parity check code correcting and eleting codes algorithm, data is carried out fault-tolerant
Process.
Wherein, described fault-tolerant processing algorithm includes copy algorithm；Described selection processing module also includes second processing unit；
Described second processing unit, for described first judging unit determine described current idle memory space than more than or wait
In described idle storage space than marginal value, and, the described currently available network bandwidth is than less than or equal to described available network
During bandwidth ratio marginal value, fault-tolerant processing is carried out to data according to described copy algorithm.
The invention has the beneficial effects as follows: it is different from the situation of prior art, the present invention is to the number in data handling system
During according to carrying out fault-tolerant processing, obtain the process resource of data handling system, and according to process resource selection, data is carried out fault-tolerant
The fault-tolerant processing algorithm processing, and fault-tolerant processing is carried out to data according to fault-tolerant processing algorithm, in brief, at data
The process resource of reason system, in conjunction with the pluses and minuses of tolerant fail algorithm, selects rational algorithm to carry out fault-tolerant processing to data, can not only
The enough fault-tolerant processing speed improving to data, is capable of the utilization rate of process resource in equilibrium criterion processing system again.
Brief description
Fig. 1 is the structural representation of data handling system first embodiment of the present invention；
Fig. 2 is the structural representation of data handling system second embodiment of the present invention；
Fig. 3 is the flow chart of the method embodiment that the present invention carries out fault-tolerant processing to data；
Fig. 4 is that the present invention carries out a kind of flow process selecting tolerant fail algorithm in the method embodiment of fault-tolerant processing to data
Figure；
Fig. 5 is that the present invention carries out another kind of flow process selecting tolerant fail algorithm in the method embodiment of fault-tolerant processing to data
Figure.
Specific embodiment
The present invention is described in detail with embodiment below in conjunction with the accompanying drawings.
Refer to Fig. 1, as illustrated, data handling system includes the first acquisition module 21, the second acquisition module 22 and place
Reason module 23.In embodiments of the present invention, data handling system can be cluster file system, and data is then file, client
(not shown) is passed through network (not shown) and is sent file to cluster file system, and the file receiving is carried out by cluster file system
Storage, is equivalent to and for file to store high in the clouds (document storage system), if desired during file, then from high in the clouds, file is taken off, non-
Often facilitate.Further, cluster file system can be made up of multiple nodes.
First acquisition module 21 obtains data, and wherein, data can be sent by client, then the first acquisition module 21 passes through
The mode of receiving data obtains data, and data is alternatively stored in the data in the memorizer (not shown) of data handling system,
First acquisition module 21 extracts data from memorizer.
Second acquisition module 22 obtains process resource data being carried out required for fault-tolerant processing, for example: idle storage is empty
Between, available network bandwidth etc..Processing module 23 selects the fault-tolerant processing that data is carried out with fault-tolerant processing to calculate according to process resource
Method, and fault-tolerant processing is carried out to data according to selected fault-tolerant processing algorithm.Data is carried out after fault-tolerant processing, improve data
The ability to ward off risks, more preferably protect the data of user, and the present invention selected according to the process resource of data handling system suitable
Fault-tolerant processing algorithm fault-tolerant processing is carried out to data, can fill part utilize processing system each resource, improve data processing system
The utilization rate of system resource.
Fault-tolerant processing algorithm includes correcting and eleting codes algorithm and copy algorithm, and the process resource of data handling system includes: idle
Memory space than marginal value, current idle memory space ratio, available network bandwidth than marginal value and available network bandwidth ratio, its
In, idle storage space is than marginal value, current idle memory space ratio, available network bandwidth than marginal value and available network bandwidth
Than, wherein, described idle storage space accounts for the ratio in total memory space than for idle memory space in data handling system,
Described available network bandwidth accounts for the ratio of overall network bandwidth, further, idle storage space than for the unappropriated network bandwidth
Pre-set than marginal value than marginal value and available network bandwidth, in embodiments of the present invention, idle storage space ratio
Marginal value and available network bandwidth can incorporate experience into table and pre-set according to the resource being had of processing system than marginal value
, the passing related data that data is carried out with fault-tolerant processing of experience table record data handling system；Overall network bandwidth refers to data
The maximum network bandwidth that processing system can provide, the scale of its data processing system is about it is generally the case that scale is got over
Greatly, the overall network bandwidth that data handling system is provided that is bigger.Current idle memory space ratio refers to that data handling system exists
Under current state, idle memory space accounts for the ratio of total memory space.Currently available network bandwidth ratio refers to data handling system
Under current state, the available network bandwidth accounts for the ratio of overall network bandwidth.
Processing module 23 is selected to include the first judging unit 231, first processing units 232 and second processing unit 233.
First judging unit 231 judge current idle memory space than whether less than idle storage space than marginal value, with
And, whether the currently available network bandwidth is than more than currently available network bandwidth ratio.When the first judging unit 231 determines current sky
Not busy memory space than less than idle storage space than marginal value, and, the currently available network bandwidth is than more than currently available network
During bandwidth ratio marginal value, first processing units 232 carry out fault-tolerant processing according to correcting and eleting codes algorithm to data.
Further, correcting and eleting codes algorithm includes rs (reed solomon, inner institute code) correcting and eleting codes algorithm and ldpc ((low
Density parity check code, low density parity check code) correcting and eleting codes algorithm.It is idle that process resource also includes node
Time critical values, data block access frequency marginal value data stripping and slicing quantity marginal value, present node free time, current data
Block access frequency and current data stripping and slicing quantity, wherein, node free time is that the node in data handling system is in the free time
The time of state, data block access frequency is that the data block in data is accessed for frequency, and data stripping and slicing quantity divides for data
The quantity of data block；Further, node free time marginal value, data block access frequency marginal value data stripping and slicing quantity are faced
Dividing value is set in advance, in embodiments of the present invention, node free time marginal value, data block access frequency marginal value and
Data stripping and slicing quantity marginal value can incorporate experience into what table pre-set according to the resource being had of processing system.Present node
Free time is the free time of the node in data handling system under current state；Current data block access frequency is to work as
Data block in data handling system under front state is accessed for frequency；Current data stripping and slicing quantity is data under current state
Data in processing system is divided into the quantity of data block.
First processing units 232 include the second judging unit 2321, the 3rd judging unit 2322, the first fault-tolerant processing unit
2323 and the second fault-tolerant processing unit 2324.
Second judging unit 2321 judges whether present node free time is more than node free time marginal value, and,
Whether current data block access frequency is less than data block access frequency marginal value.Determine in the second judging unit 2321 and work as prosthomere
Point free time is more than node free time marginal value, and, current data block access frequency is faced less than data block access frequency
During dividing value, the 3rd judging unit 2322 judges whether current data stripping and slicing quantity is less than data stripping and slicing quantity marginal value.The 3rd
When judging unit 2322 determines current data stripping and slicing quantity less than data stripping and slicing quantity marginal value, the first fault-tolerant processing unit
2323 carry out fault-tolerant processing according to rs correcting and eleting codes algorithm to data.Determine current data stripping and slicing number in the 3rd judging unit 2322
When amount is more than or equal to data stripping and slicing quantity marginal value, the second fault-tolerant processing unit 2324 is according to ldpc correcting and eleting codes algorithm logarithm
According to carrying out fault-tolerant processing.
Rs correcting and eleting codes algorithm with respect to ldpc correcting and eleting codes algorithm, its decoding and coding calculation cost all higher, if
When the quantity of deblocking is more, selects rs correcting and eleting codes algorithm, substantial amounts of computing resource will be expended, and substantially reduce data processing system
The calculating disposal ability of system, therefore, when deblocking is more, selects the data of ldpc correcting and eleting codes algorithm to carry out fault-tolerant processing,
The amount of calculation of data handling system can be reduced.
Determine current idle memory space in the first judging unit 231 to face than more than or equal to idle storage space ratio
Dividing value, and, currently available network bandwidth ratio is during less than or equal to currently available network bandwidth ratio, according to copy algorithm to data
Carry out fault-tolerant processing., compared to correcting and eleting codes algorithm, copy algorithm is less to the occupancy of the network bandwidth for copy algorithm, but needs to take
More memory space, as shown in table 1, table 1 is the resource overhead comparison sheet of copy algorithm and correcting and eleting codes algorithm.
Wherein, the size of the data in table 1 is nmb, and data is divided into k according to block, and data block is encoded
After obtain n data fragmentation, the size of each data fragmentation is 1mb, and data fragmentation is carried out distribution storage.In data
The memory space of processing system is more, when the network bandwidth is less, carries out fault-tolerant processing according to copy algorithm to data, at data
The memory space of reason system is less, when the network bandwidth is more, then carries out fault-tolerant processing, Neng Gougeng according to correcting and eleting codes algorithm to data
Balance well the resource of data handling system, improve the resource utilization of data handling system.
Further, after selecting copy algorithm to carry out fault-tolerant processing to data, node free time can also be obtained critical
Value, data block access frequency marginal value data stripping and slicing quantity marginal value, present node free time, current data block access frequency
Rate and current data stripping and slicing quantity.
Present node free time is less than or equal to node free time marginal value, can reduce in data handling system
The usage amount of copy algorithm, and correcting and eleting codes algorithm is used to part data, for example: if some or certain several node is long
Between be in idle condition, can by some data on node are carried out fault-tolerant processing using correcting and eleting codes algorithm, with save deposit
Storage space.
Current data block access frequency is more than or equal to data block access frequency marginal value it is also possible to reduce data processing
The usage amount of the copy algorithm in system, and to part data use correcting and eleting codes algorithm, for example: if some or certain several
Data is in an inactive state for a long time, then can carry out fault-tolerant processing according to correcting and eleting codes algorithm to these data, to save storage
Space.
Refer to Fig. 2, data handling system includes processor 31, memorizer 32 and bus 33, wherein, processor 31 and depositing
Reservoir 32 is all connected with bus 33.
Processor 31 is used for obtaining data；Obtain process resource data being carried out required for fault-tolerant processing；According to process
Resource selection carries out the fault-tolerant processing algorithm of fault-tolerant processing to data, and according to selected fault-tolerant processing algorithm, data is carried out
Fault-tolerant processing.
Wherein, process resource includes: idle storage space is than marginal value, current idle memory space ratio, available network band
Wide ratio marginal value and available network bandwidth ratio, wherein, idle storage space is than marginal value, current idle memory space ratio, available
Than marginal value and available network bandwidth ratio, wherein, described idle storage space ratio is for idle in data handling system for the network bandwidth
Memory space account for ratio in total memory space, described available network bandwidth accounts for overall network band than for the unappropriated network bandwidth
Wide ratio.Fault-tolerant processing algorithm includes correcting and eleting codes algorithm.Processor 31 specifically for judging current idle memory space ratio is
No less than idle storage space than marginal value, and, whether the currently available network bandwidth than being more than described available network bandwidth ratio
Marginal value；If current idle memory space than less than idle storage space than marginal value, and the currently available network bandwidth is than big
In the currently available network bandwidth than marginal value, then fault-tolerant processing is carried out to data according to correcting and eleting codes algorithm.
Correcting and eleting codes algorithm includes inner institute code correcting and eleting codes algorithm.Described process resource also includes: node free time marginal value,
Data block access frequency marginal value data stripping and slicing quantity marginal value, present node free time, current data block access frequency
With current data stripping and slicing quantity, wherein, node free time is in the time of idle condition for the node in data handling system,
Data block access frequency is that the data block in data is accessed for frequency, and data stripping and slicing quantity is the number of data partitioning data block
Amount.Processor 31 is additionally operable to judge whether described present node free time is more than node free time marginal value, and, currently
Whether data block access frequency is less than data block access frequency marginal value；If it is idle that present node free time is more than described node
Time critical values, and, current data block access frequency is less than data block access frequency marginal value, then judge current data stripping and slicing
Whether quantity is less than data stripping and slicing quantity marginal value；If current data stripping and slicing quantity is less than data stripping and slicing quantity marginal value, root
According to inner institute code correcting and eleting codes algorithm, fault-tolerant processing is carried out to data.
Correcting and eleting codes algorithm also includes low density parity check code correcting and eleting codes algorithm.If processor 31 is additionally operable to current data cutting
Number of blocks is more than or equal to data stripping and slicing quantity marginal value, then according to low density parity check code algorithm, data is carried out fault-tolerant
Process.
Fault-tolerant processing algorithm includes copy algorithm.If processor 31 is additionally operable to current idle memory space, ratio is more than or waits
In idle storage space than marginal value, and, the currently available network bandwidth than less than or equal to currently available network bandwidth ratio, then
Fault-tolerant processing is carried out to data according to copy algorithm.
Wherein, above-mentioned processor 31 carried out operation can be stored in memorizer 32 in the form of one section of program, when need into
When row is processed, extract this program, configuration processor from memorizer 31.
In embodiments of the present invention, when fault-tolerant processing is carried out to the data in data handling system, obtain at data
The process resource of reason system, and select data is carried out the fault-tolerant processing algorithm of fault-tolerant processing according to process resource, and according to
Fault-tolerant processing algorithm carries out fault-tolerant processing to data, in brief, according to the process resource of data handling system, in conjunction with fault-tolerant calculation
The pluses and minuses of method, select rational algorithm to carry out fault-tolerant processing to data, can not only improve the fault-tolerant processing speed to data,
It is capable of the utilization rate of process resource in equilibrium criterion processing system again.
The present invention also provides the method embodiment that data is carried out with fault-tolerant processing.As shown in figure 3, method includes:
Step s41: obtain data；
Data handling system can be cluster file system, and data can be file, and the mode obtaining data is 1. reception clients
The data of the transmission at end, 2. extracts data in the memorizer from data handling system.
Step s42: obtain process resource data being carried out required for fault-tolerant processing；
It can be all process resource that data handling system is processed consumed resource to data, for example: data processing system
System is encoded to data and is needed consumption calculations resource when decoding, then computing resource is process resource, after the completion of data processing,
Need to store, then storage resource is process resource.
Step s43: select data is carried out the fault-tolerant processing algorithm of fault-tolerant processing according to process resource, and according to selected
Fault-tolerant processing algorithm fault-tolerant processing is carried out to data；
According to the process resource in data handling system, in conjunction with the pluses and minuses of tolerant fail algorithm, select suitable fault-tolerant processing
Algorithm carries out fault-tolerant processing to data, can not only improve the fault-tolerant processing speed to data, again being capable of equilibrium criterion processing system
The utilization rate of the process resource in system.
Specifically, process resource includes: idle storage space is than marginal value, current idle memory space ratio, available network
Bandwidth ratio marginal value and available network bandwidth ratio, wherein, idle storage space than marginal value, current idle memory space ratio, can
With the network bandwidth than marginal value and available network bandwidth ratio, wherein, idle storage space ratio is for idle in data handling system
Memory space accounts for the ratio in total memory space, and available network bandwidth accounts for the ratio of overall network bandwidth than for the unappropriated network bandwidth
Example, further, idle storage space is according in data handling system than marginal value and available network bandwidth than marginal value
Experience table, pre-sets, and fault-tolerant processing algorithm includes correcting and eleting codes algorithm and secondary algorithm, and step s43 includes step s431, step
S432 and step s433,
Step s431: judge whether current idle memory space ratio is less than idle storage space than marginal value, and, currently
Whether available network bandwidth ratio is more than available network bandwidth than marginal value, if current idle memory space ratio is empty less than idle storage
Between ratio marginal value, and the currently available network bandwidth then enters step s432 than more than the currently available network bandwidth than marginal value, no
Then enter step s433；
Step s432: fault-tolerant processing is carried out to data according to correcting and eleting codes algorithm；
Correcting and eleting codes algorithm includes rs correcting and eleting codes algorithm and ldpc correcting and eleting codes algorithm.Process resource also includes: when node is idle
Between marginal value, data block access frequency marginal value data stripping and slicing quantity marginal value, present node free time, current data block
Access frequency and current data stripping and slicing quantity, wherein, node free time is that the node in data handling system is in idle shape
The time of state, data block access frequency is that the data block in data is accessed for frequency, and data stripping and slicing quantity divides number for data
Quantity according to block.Node free time marginal value, data block access frequency marginal value data stripping and slicing quantity marginal value are according to number
Pre-set according to the experience table of processing system, then step s432 includes step s4321, step s4322, step s4323, step again
Rapid s4324 and step s4325,
Step s4321: judge whether present node free time is more than node free time marginal value, and, current number
Whether it is less than data block access frequency marginal value according to block access frequency, if present node free time faced more than node free time
Dividing value, and, when current data block access frequency is less than data block access frequency marginal value, enter step s4322；
Step s4322: if present node free time be more than node free time marginal value, and, current data block visit
Ask that frequency is less than data block access frequency marginal value, then judge whether current data stripping and slicing quantity is critical less than data stripping and slicing quantity
Value, if judging, current data stripping and slicing quantity is less than data stripping and slicing quantity marginal value, enters step s4323, otherwise enters step
s4324；
Step s4323: fault-tolerant processing is carried out to data according to rs correcting and eleting codes algorithm；
Rs correcting and eleting codes algorithm with respect to ldpc correcting and eleting codes algorithm, its decoding and coding calculation cost all higher, if
When the quantity of deblocking is more, selects rs correcting and eleting codes algorithm, substantial amounts of computing resource will be expended, and substantially reduce data processing system
The calculating disposal ability of system, therefore, when deblocking is more, selects the data of ldpc correcting and eleting codes algorithm to carry out fault-tolerant processing,
The amount of calculation of data handling system can be reduced.
Step s4324: fault-tolerant processing is carried out to data according to ldpc correcting and eleting codes algorithm；
The amount of calculation of ldpc correcting and eleting codes algorithm is less than rs correcting and eleting codes algorithm.
Step s433: fault-tolerant processing is carried out to data according to copy algorithm；
, compared to correcting and eleting codes algorithm, copy algorithm is less to the occupancy of the network bandwidth for copy algorithm, but needs to take more
Memory space, therefore, more in the memory space of data handling system, when the network bandwidth is less, according to copy algorithm to data
Carry out fault-tolerant processing, less in the memory space of data handling system, when the network bandwidth is more, then according to correcting and eleting codes algorithm logarithm
According to carrying out fault-tolerant processing, the resource of data handling system can be better balanced, improve the resource utilization of data handling system.
Further, after selecting copy algorithm to carry out fault-tolerant processing to data, node free time can also be obtained critical
Value, data block access frequency marginal value data stripping and slicing quantity marginal value, present node free time, current data block access frequency
Rate and current data stripping and slicing quantity.
Present node free time is less than or equal to node free time marginal value, can reduce in data handling system
The usage amount of copy algorithm, and correcting and eleting codes algorithm is used to part data, for example: if some or certain several node is long
Between be in idle condition, can by some data on node are carried out fault-tolerant processing using correcting and eleting codes algorithm, with save deposit
Storage space.
Current data block access frequency is more than or equal to data block access frequency marginal value it is also possible to reduce data processing
The usage amount of the copy algorithm in system, and to part data use correcting and eleting codes algorithm, for example: if some or certain several
Data is in an inactive state for a long time, then can carry out fault-tolerant processing according to correcting and eleting codes algorithm to these data, to save storage
Space.
In embodiments of the present invention, when fault-tolerant processing is carried out to the data in data handling system, obtain at data
The process resource of reason system, and select data is carried out the fault-tolerant processing algorithm of fault-tolerant processing according to process resource, and according to
Fault-tolerant processing algorithm carries out fault-tolerant processing to data, in brief, according to the process resource of data handling system, in conjunction with fault-tolerant calculation
The pluses and minuses of method, select rational algorithm to carry out fault-tolerant processing to data, can not only improve the fault-tolerant processing speed to data,
It is capable of the utilization rate of process resource in equilibrium criterion processing system again.
The foregoing is only embodiments of the present invention, not thereby limit the scope of the claims of the present invention, every utilization is originally
Equivalent structure or equivalent flow conversion that description of the invention and accompanying drawing content are made, or directly or indirectly it is used in other correlations
Technical field, is included within the scope of the present invention.
Claims (6)
1. a kind of the method for fault-tolerant processing is carried out it is characterised in that method includes to data:
Obtain data；
Obtain the process resource described data being carried out required for fault-tolerant processing；
Select described data is carried out the fault-tolerant processing algorithm of fault-tolerant processing according to described process resource, and according to described selected
Fault-tolerant processing algorithm fault-tolerant processing is carried out to described data；
Described process resource includes: idle storage space is faced than marginal value, current idle memory space ratio, available network bandwidth ratio
Dividing value and currently available network bandwidth ratio, wherein, described idle storage space is more empty than for idle storage in data handling system
Between account for ratio in total memory space, the described currently available network bandwidth accounts for overall network band than for the current unappropriated network bandwidth
Wide ratio；
Described fault-tolerant processing algorithm includes correcting and eleting codes algorithm；
The described fault-tolerant processing algorithm that according to the selection of described process resource, described data is carried out with fault-tolerant processing, and according to described appearance
The step that wrong Processing Algorithm carries out fault-tolerant processing to described data includes:
Judge whether described current idle memory space ratio is less than described idle storage space than marginal value, and, described current
Whether available network bandwidth than compares marginal value more than described available network bandwidth；
If described current idle memory space than less than described idle storage space than marginal value, and, described currently available net
Network bandwidth ratio is more than described available network bandwidth than marginal value, then carry out fault-tolerant place according to described correcting and eleting codes algorithm to described data
Reason；
Described correcting and eleting codes algorithm includes inner institute code correcting and eleting codes algorithm；
Described process resource also includes: node free time marginal value, data block access frequency marginal value data stripping and slicing quantity
Marginal value, present node free time, current data block access frequency and current data stripping and slicing quantity, wherein, described node is empty
It is in the time of idle condition for the node in data handling system, described data block access frequency is the number in data between idle
It is accessed for frequency according to block, described data stripping and slicing quantity is the quantity of data partitioning data block；
Described fault-tolerant processing step carried out to described data according to correcting and eleting codes algorithm also include:
Judge whether described present node free time is more than node free time marginal value, and, described current data block is visited
Ask whether frequency is less than data block access frequency marginal value；
If described present node free time is more than described node free time marginal value, and, described current data block access
Frequency is less than data block access frequency marginal value, then judge whether described current data stripping and slicing quantity faces less than data stripping and slicing quantity
Dividing value；
If described current data stripping and slicing quantity is less than data stripping and slicing quantity marginal value, according to described inner institute code correcting and eleting codes algorithm pair
Data carries out fault-tolerant processing.
2. method according to claim 1 is it is characterised in that described correcting and eleting codes algorithm also includes low density parity check code
Correcting and eleting codes algorithm；
Methods described also includes:
If described current data stripping and slicing quantity is more than or equal to data stripping and slicing quantity marginal value, according to described low-density parity
Check code algorithm carries out fault-tolerant processing to data.
3. method according to claim 1 is it is characterised in that described fault-tolerant processing algorithm includes copy algorithm；
Methods described also includes:
If described current idle memory space than more than or equal to described idle storage space than marginal value, and, described work as
Front available network bandwidth, than marginal value is then entered to data according to described copy algorithm than less than or equal to described available network bandwidth
Row fault-tolerant processing.
4. a kind of data handling system is it is characterised in that include:
First acquisition module, for obtaining data；
Second acquisition module, for obtaining the process resource described data being carried out required for fault-tolerant processing；
Select processing module, for selecting the fault-tolerant processing that described data is carried out with fault-tolerant processing to calculate according to described process resource
Method, and fault-tolerant processing is carried out to described data according to described selected fault-tolerant processing algorithm；Described process resource includes: idle
Memory space than marginal value, current idle memory space ratio, available network bandwidth than marginal value and currently available network bandwidth ratio,
Wherein, described idle storage space accounts for the ratio in total memory space, institute than for idle memory space in data handling system
State the ratio that the currently available network bandwidth accounts for overall network bandwidth than for the current unappropriated network bandwidth；
Described fault-tolerant processing algorithm includes correcting and eleting codes algorithm；
Described selection processing module includes:
First judging unit, for judging whether described current idle memory space ratio is less than described idle storage space than critical
Value, and, whether the described currently available network bandwidth than compares marginal value more than described available network bandwidth；
First processing units, empty less than idle storage for determining current idle memory space ratio in described first judging unit
Between ratio marginal value, and, described currently available network bandwidth ratio is during more than described available network bandwidth than marginal value, according to described
Correcting and eleting codes algorithm carries out fault-tolerant processing to described data；
Described process resource also includes node free time marginal value, data block access frequency marginal value, data stripping and slicing quantity are faced
Dividing value, present node free time, current data block access frequency and current data stripping and slicing quantity, wherein, described node is idle
Time is in the time of idle condition for the node in data handling system, and described data block access frequency is the data in data
Block is accessed for frequency, and described data stripping and slicing quantity is the quantity of data partitioning data block；
Described correcting and eleting codes algorithm includes inner institute code correcting and eleting codes algorithm；
Described first processing units include:
Second judging unit, for judging whether described present node free time is more than node free time marginal value, and,
Whether described current data block access frequency is less than data block access frequency marginal value；
3rd judging unit, idle more than node for determining described present node free time in described second judging unit
Time critical values, and, when described current data block access frequency is less than data block access frequency marginal value, judge described current
Whether data stripping and slicing quantity is less than data stripping and slicing quantity marginal value；
First fault-tolerant processing unit, is less than data for determining described current data stripping and slicing quantity in described 3rd judging unit
During stripping and slicing quantity marginal value, fault-tolerant processing is carried out to data according to described inner institute code correcting and eleting codes algorithm.
5. system according to claim 4 is it is characterised in that described correcting and eleting codes algorithm also includes low density parity check code
Correcting and eleting codes algorithm；
Described first processing units, also include:
Second fault-tolerant processing unit, for described 3rd judging unit determine described current data stripping and slicing quantity be more than or
During equal to data stripping and slicing quantity marginal value, fault-tolerant place is carried out to data according to described low density parity check code correcting and eleting codes algorithm
Reason.
6. system according to claim 4 it is characterised in that
Described fault-tolerant processing algorithm includes copy algorithm；
Described selection processing module also includes second processing unit；
Described second processing unit, for described first judging unit determine described current idle memory space than more than or
Person is equal to described idle storage space than marginal value, and, the described currently available network bandwidth is more available than less than or equal to described
When the network bandwidth is than marginal value, fault-tolerant processing is carried out to data according to described copy algorithm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410169184.4A CN103942116B (en) | 2014-04-24 | 2014-04-24 | Method for conducting fault-tolerance processing on data and data processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410169184.4A CN103942116B (en) | 2014-04-24 | 2014-04-24 | Method for conducting fault-tolerance processing on data and data processing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103942116A CN103942116A (en) | 2014-07-23 |
| CN103942116B true CN103942116B (en) | 2017-02-01 |
Family
ID=51189790
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410169184.4A Active CN103942116B (en) | 2014-04-24 | 2014-04-24 | Method for conducting fault-tolerance processing on data and data processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103942116B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104281533B (en) * | 2014-09-18 | 2018-03-20 | 深圳市中博科创信息技术有限公司 | A kind of method and device of data storage |
| CN106776146A (en) * | 2016-12-29 | 2017-05-31 | 华为技术有限公司 | A kind of data verification method, apparatus and system |
| CN111858129B (en) * | 2019-04-28 | 2024-02-23 | 深信服科技股份有限公司 | Erasure code read request processing method, system, equipment and computer medium |
| CN110083482B (en) * | 2019-04-30 | 2023-06-02 | 北京金山云网络技术有限公司 | Method and device for performing erasure code processing on file storage system and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102937967A (en) * | 2012-10-11 | 2013-02-20 | 南京中兴新软件有限责任公司 | Data redundancy realization method and device |
| US8504535B1 (en) * | 2010-12-20 | 2013-08-06 | Amazon Technologies, Inc. | Erasure coding and redundant replication |
| CN103631666A (en) * | 2012-08-24 | 2014-03-12 | 中兴通讯股份有限公司 | Data redundancy fault-tolerance adaptation management device, service device, system and method |
| CN103729151A (en) * | 2014-01-16 | 2014-04-16 | 中国人民解放军国防科学技术大学 | Failure data recovery method based on improved erasure codes |
-
2014
- 2014-04-24 CN CN201410169184.4A patent/CN103942116B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8504535B1 (en) * | 2010-12-20 | 2013-08-06 | Amazon Technologies, Inc. | Erasure coding and redundant replication |
| CN103631666A (en) * | 2012-08-24 | 2014-03-12 | 中兴通讯股份有限公司 | Data redundancy fault-tolerance adaptation management device, service device, system and method |
| CN102937967A (en) * | 2012-10-11 | 2013-02-20 | 南京中兴新软件有限责任公司 | Data redundancy realization method and device |
| CN103729151A (en) * | 2014-01-16 | 2014-04-16 | 中国人民解放军国防科学技术大学 | Failure data recovery method based on improved erasure codes |
Non-Patent Citations (2)
| Title |
|---|
| 存储系统中的纠删码研究综述;罗象宏 等;《计算机研究与发展》;20120115(第1期);1-11 * |
| 纠删码的分析与研究;郭春梅 等;《信息安全与技术》;20100901(第7期);38-42 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103942116A (en) | 2014-07-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104272274B (en) | Data processing method and equipment in a kind of distributed file storage system | |
| CN103942116B (en) | Method for conducting fault-tolerance processing on data and data processing system | |
| US20170228300A1 (en) | Faster reconstruction of segments using a dedicated spare memory unit | |
| US9405589B2 (en) | System and method of optimization of in-memory data grid placement | |
| CN107544862A (en) | A kind of data storage reconstructing method and device, memory node based on correcting and eleting codes | |
| KR101781339B1 (en) | Method and device for updating client | |
| US10740198B2 (en) | Parallel partial repair of storage | |
| CN110795120B (en) | Software upgrading method and related device | |
| CN103607424B (en) | Server connection method and server system | |
| CN105205143B (en) | A kind of storage of file and processing method, equipment and system | |
| CN104052576A (en) | Data recovery method based on error correcting codes in cloud storage | |
| CN104301434B (en) | A kind of high-speed communication framework and method based on cluster | |
| US9571299B2 (en) | Method for managing tree structure, information processing system, and medium | |
| US11102322B2 (en) | Data processing method and apparatus, server, and controller | |
| EP2330525A1 (en) | Parallel computing method and computing platform for security and stability analysis of large power grid | |
| CN107346270B (en) | Method and system for real-time computation based radix estimation | |
| CN105554121A (en) | Method and system for realizing load equalization of distributed cache system | |
| CN108762912A (en) | A kind of method and apparatus of container cluster elastic telescopic | |
| CN110502572A (en) | A kind of method and server that data are synchronous | |
| CN109388614A (en) | A kind of method, system and the equipment of catalogue file number quota | |
| CN110874284B (en) | Data processing method and device | |
| CN115426361A (en) | Distributed client packaging method and device, main server and storage medium | |
| CN113347238A (en) | Message partitioning method, system, device and storage medium based on block chain | |
| CN109408597A (en) | A kind of power grid metering big data storage system and its creation method | |
| US20150331752A1 (en) | Method of data storage on cloud data center for reducing processing and storage requirements by engaging user equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20140723 Assignee: Liu Yi Assignor: Shenzhen Zhongbo Kechuang Information Technology Co., Ltd. Contract record no.: 2014440020487 Denomination of invention: Method for conducting fault-tolerance processing on data and data processing system License type: Common License Record date: 20141230 |
|
| LICC | Enforcement, change and cancellation of record of contracts on the licence for exploitation of a patent or utility model | ||
| EC01 | Cancellation of recordation of patent licensing contract |
Assignee: Liu Yi Assignor: Shenzhen Zhongbo Kechuang Information Technology Co., Ltd. Contract record no.: 2014440020487 Date of cancellation: 20161025 |
|
| LICC | Enforcement, change and cancellation of record of contracts on the licence for exploitation of a patent or utility model | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20190902 Address after: 100089 Floor 1-4, No. 2 Building, No. 9 Courtyard, Dijin Road, Haidian District, Beijing Patentee after: Beijing Toyou Feiji Electronics Co., Ltd. Address before: 518057 Room 1402, Feiyada Science and Technology Building, Nanshan District, Shenzhen City, Guangdong Province Patentee before: Shenzhen Zhongbo Kechuang Information Technology Co., Ltd. |
|
| TR01 | Transfer of patent right |