CN103389946B - Go flaking method and system - Google Patents

Abstract

The present invention provides one to remove flaking method, first the method determines the expection read/write bandwidth of each data object accessing operation in access forecasting sequence, the described read/write bandwidth that prefetches is with data object length corresponding to this access operation and access step-length of beating as variable, the matching bandwidth obtained based on matching bandwidth function.Then from the beginning of from access forecasting sequence, certain accesses operation, the read/write bandwidth that prefetches of continuous multiple access operations and each data object accessing operation for exceeding predetermined number is less than bandwidth threshold, and these data objects accessing operation are sequentially written to new continuous print memory space.The method can accurately judge fragmentation data, in the case of little effect current system performance, improves reading performance, and the environment in change has preferable adaptability.

Description

De-fragmentation method and system

technical field

The invention belongs to the field of computer storage, in particular to a method for processing data fragments.

Background technique

Fragmentation means that logically relatively continuous file data is stored in the storage medium, and its storage location becomes relatively scattered, which brings a large number of random read and write operations during data access. For the current mainstream disk-based storage systems, fragmentation will lead to a continuous decline in read/write bandwidth. Therefore, a de-fragmentation method is needed to reduce the impact of fragmentation and reduce random read and write access.

Existing de-fragmentation methods usually input an access prediction sequence, identify fragmented data according to the access prediction sequence, and write the fragmented data into a new storage area, so as to achieve the purpose of de-fragmentation. The access prediction sequence is a prediction of the most likely access sequence in the future. There are many prediction methods, but they can be divided into two types in essence: (1) static prediction, such as prediction according to the logical storage order of data; (2) Dynamic prediction, such as prediction according to the current access sequence.

In the process of de-fragmentation, there are two main problems to be solved: (1) What kind of data object is fragmented data, that is, if the current storage system is accessed according to the access prediction sequence, which data object access will cause read/write bandwidth (2) Which data should be written to the new storage location, so as to reduce the random read and write access in the future and affect the current performance of the system as little as possible. The main problems in the existing de-fragmentation methods are as follows: first, the judgment of fragmented data is not accurate enough to effectively improve performance; second, in the process of de-fragmentation, too much data needs to be moved, Causes a noticeable drop in system performance during operation.

Contents of the invention

Therefore, the object of the present invention is to overcome the above-mentioned defects in the prior art and provide a new method for defragmentation.

The purpose of the present invention is achieved through the following technical solutions:

In one aspect, the present invention provides a defragmentation method, the method comprising:

Step 1) Determine the expected read/write bandwidth of the data object of each access operation in the access prediction sequence, the prefetch read/write bandwidth is based on the length of the data object corresponding to the access operation and the access jitter step as variables, based on the proposed The fitting bandwidth obtained by combining the bandwidth function;

Step 2) Starting from a certain access operation in the access prediction sequence, for more than a predetermined number of consecutive access operations and the prefetch read/write bandwidth of the data object of each access operation is less than the bandwidth threshold, the data of these access operations Objects are written sequentially to the new contiguous storage space.

In the above method, in the step 1), the fitting bandwidth function can be in the storage system that needs to be de-fragmented, with the data object length and the access jitter step as variables, based on different data object lengths and access Benchmark read/write bandwidth data measured under the jitter step, and the bandwidth function expression obtained by the fitting method.

In the above method, the fitted bandwidth function may be obtained through the following steps:

Step a) In the storage system that needs to be de-fragmented, take the data object length x and the access jitter step y as variables, and measure a set of benchmark read/write bandwidth data under different x and y;

Step b) select a main variable from x, y, and the other is a secondary variable;

Step c) find the value of the main variable closest to the main variable;

Step d) finding the value interval of the secondary variable;

Step e) Fitting the read/write bandwidth f(x, y) when x and y are arbitrary values based on the measured reference read/write bandwidth data under different x and y.

In the above method, in the step c), when the main variable is x, find _xi so that the absolute value of Δx=h( _xi )-h(x) is the smallest; when the main variable is y, find y so that _Δy = The absolute value of h(y _i )-h(y) is the smallest;

In the step d), when the secondary variable is y, find y _j such that y∈[y _j , y _j+1 ], and when the secondary variable is x, find x _j such that x ∈ [x _j , x _{j+ 1} ];

In step e), when the secondary variable is y, make $f(x,\mathrm{the\; y})=f_{\mathrm{ij}+1}+\frac{h\left(\mathrm{the\; y}\right)-h\left({\mathrm{the\; y}}_j\right)}{h\left({\mathrm{the\; y}}_{j+1}\right)-{\mathrm{hy}}_j}|f_{\mathrm{ij}}-$ $f_{\mathrm{ij}+1}|+g\left(\mathrm{\Δx}\right),$ Among them, g(Δx) represents a further correction to the fitting result; when the secondary variable is x, make $f(x,\mathrm{the\; y})=f_{\mathrm{ij}}+\frac{h\left(x\right)-h\left(x_j\right)}{h\left(x_{j+1}\right)-h\left(x_j\right)}|f_{\mathrm{ij}}-f_{i+1j}|+g\left(\mathrm{\Δy}\right),$ Among them, g(Δy) represents a further correction to the fitting result.

In the above method, it may also include step 3) starting from a certain access operation in the access prediction sequence, for more than a predetermined number of consecutive access operations and the prefetch read/write bandwidth of the data object of each access operation is greater than the bandwidth threshold, skip these access operations and continue to process the next access operations.

In the above method, step 4) may also be included, for subsequences s _i , ..., s _i that are not processed by step 2) or step 3) in the access prediction sequence and are composed of multiple consecutive access operations _+m , if the overall bandwidth of the subsequence is less than a certain threshold and the assumed continuous bandwidth of the subsequence is greater than a certain threshold, write the data objects of each access operation in the subsequence to a new continuous storage space, and Change the access entry of these data objects to a new location; wherein, the assumed continuous bandwidth represents the bandwidth after the data objects of the access operation in the subsequence are continuously stored; the subsequence s _i , ..., s _i+ Overall bandwidth of _m l _i is the length of the data object of _si , fi is the expected read/write bandwidth of the data object of _{si ,} and m is the number of access operations in the subsequence.

In the above method, the bandwidth threshold can be determined according to the maximum bandwidth value or the continuous bandwidth of the storage system that needs to be de-fragmented, and the continuous bandwidth is determined based on the fitting bandwidth function when the access jitter step is 0 The resulting fitted bandwidth.

In yet another aspect, the present invention also provides a defragmentation system, the system comprising:

A bandwidth determination device, configured to determine the expected read/write bandwidth of the data object of each access operation in the access prediction sequence, the prefetch read/write bandwidth is based on the length of the data object corresponding to the access operation and the access jitter step as variables , the fitted bandwidth obtained based on the fitted bandwidth function;

The defragmentation device is used to start from a certain access operation in the access prediction sequence, and for a plurality of consecutive access operations exceeding a predetermined number and the prefetch read/write bandwidth of the data object of each access operation is less than the bandwidth threshold, these access The manipulated data objects are sequentially written to the new continuous storage space.

In the above system, the fitting bandwidth function may be based on benchmarks measured under different data object lengths and access jitter steps in a storage system that needs to be de-fragmented, with data object lengths and access jitter steps as variables Read/write bandwidth data, the bandwidth function expression obtained by the fitting method.

In the above system, the defragmentation device can also be used to prefetch the read/write bandwidth of the data object for multiple consecutive access operations exceeding a predetermined number and for each access operation starting from a certain access operation in the access prediction sequence is greater than the bandwidth threshold, skip these access operations, and continue to process the next access operations.

In the above system, the defragmentation device can also be used for data objects whose prefetch read/write bandwidths are greater than or less than the bandwidth threshold for data objects that do not meet the predetermined number of consecutive multiple access operations. If the subsequence s _i , ..., s _i+m of the subsequence is less than a certain threshold and the assumed continuous bandwidth of the subsequence is greater than a certain threshold, the subsequence of each access operation in the subsequence Write the data objects into a new continuous storage space, and change the access entries of these data objects to new locations; wherein, the assumed continuous bandwidth means that if the data objects of the access operations in the subsequence are stored continuously, the bandwidth ;The overall bandwidth of the subsequence s _i ,..., s _i+m l _i is the length of the data object of _si , fi is the expected read/write bandwidth of the data object of _{si ,} and m is the number of access operations in the subsequence.

Compared with the prior art, the de-fragmentation method provided by the present invention can more accurately judge the fragmented data, improve the reading performance with less impact on the current system performance, and have better performance in changing environments. adaptability.

Description of drawings

Embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

FIG. 1 is a schematic flow diagram of a de-fragmentation method according to an embodiment of the present invention;

Fig. 2 is a schematic flowchart of a defragmentation method according to yet another embodiment of the present invention.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The access prediction sequence is actually a series of access operations, and the read/write bandwidth is mainly related to the size of the data object accessed by each access operation and the change distance of the access position before or after accessing the data object (or it can also be called access Jumping step size) related. Usually, access to data objects with smaller read/write bandwidth in the predicted sequence has a higher degree of fragmentation, but considering that read/write bandwidth is a posteriori indicator, it cannot be obtained in advance. Therefore, in the embodiment of the present invention, the read/write bandwidth fitting method is adopted to calculate the expected read/write bandwidth of the data object accessed by each access operation in the access prediction sequence (hereinafter sometimes referred to as access The fitting bandwidth corresponding to the operation), and judge the degree of fragmentation based on this index.

Fig. 1 shows a schematic flowchart of a defragmentation method according to an embodiment of the present invention. The method includes: step 1) determining the expected read/write bandwidth of the data object for each access operation in the access prediction sequence; step 2) starting from a certain access operation in the access prediction sequence, if more than a predetermined number of consecutive access The prefetch read/write bandwidth of the operated data objects is less than the bandwidth threshold, and these access operated data objects are sequentially written to the new continuous storage space.

Referring to FIG. 1 , more specifically, in step 1), the expected read/write bandwidth of the data object for each access operation in the access prediction sequence is determined. The prefetching read/write bandwidth is a fitting bandwidth obtained based on a fitting bandwidth function using the length of the data object corresponding to the access operation and the access jitter step as variables. According to an embodiment of the present invention, the step 1) may include the following steps:

Step 11) Receive the access operations in the access prediction sequence one by one, and store them in the queue S: <s ₁ , . . . , s _n >. The information stored in the queue S includes, but is not limited to, the starting position o _i of the data object accessed by each access operation and the length l _i of the data object obtained through a certain method (such as metadata access, index search, etc.) . The access prediction sequence stored in the queue S may be a directly obtained prediction sequence (static prediction or dynamic prediction). But it can also be _{a preprocessed} prediction sequence (such as removing repeated prediction sequences) _; If the same data object is accessed, the access operation _sk will not be added to the queue S.

Step 12) Calculate the fitting bandwidth f _i corresponding to each access operation _si in the queue S.

As mentioned above, the fitting bandwidth is actually a bandwidth obtained based on fitting a bandwidth function using the length of the data object corresponding to the access operation and the access jitter step as variables. The fitting bandwidth function is based on a set of measured benchmark read/writes under different data object lengths and access jitter steps in a storage system that needs to be de-fragmented, with the data object length and access jitter step as variables Bandwidth data, the bandwidth function expression obtained by the fitting method. In one embodiment of the present invention, the expression of fitting bandwidth function is obtained by following steps:

Step a) In the storage system that needs to be de-fragmented, take the data object length x and the access jitter step y as variables, and measure a set of benchmark read/write bandwidth data under different x and y, as shown in Table 1. Among them, x _i , y _j represent different values of x and y. After actual measurement, the corresponding read/write bandwidth is f _ij . Wait.

Table 1

parameter y ₀ y ₁ ... y _n x ₀ f ₀₀ f ₀₁ ... f _{on x1} f ₁₀ f ₁₁ ... f _1n ... ... ... ... ... x _m f _m0 f _m2 ... _f

Step b) Select a main variable from x, y, and the other is a secondary variable.

Step c) Find the value of the main variable closest to the main variable (for example, one of x ₀ ~ x _m ), that is, when the main variable is x, find x _i so that Δx=h( _xi )-h( The absolute value of x) is the smallest, when the main variable is y, find y _i to make the absolute value of Δy=h(y _i )-h(y) the smallest, where the function h(x) includes but not limited to linear functions (such as h( x)=x), logarithmic function (such as h(x)=log ₂ x), etc.

Step d) Find the value interval of the secondary variable, that is, when the secondary variable is y, find y _j such that y∈[y _j , y _j+1 ], and when the secondary variable is x, find x _j such that x∈[x _j , x _j+1 ].

Step e) Fit the read/write bandwidth f(x, y) based on the reference data when x and y are arbitrary values: take a reasonable value proportionally within the bandwidth interval corresponding to the value interval of the secondary variable as the fitting As a result, when the secondary variable is y, make $f(x,\mathrm{the\; y})=f_{\mathrm{ij}+1}+\frac{h\left(\mathrm{the\; y}\right)-h\left({\mathrm{the\; y}}_j\right)}{h\left({\mathrm{the\; y}}_{j+1}\right)-{\mathrm{hy}}_j)}|f_{\mathrm{ij}}-f_{\mathrm{ij}+1}|+g\left(\mathrm{\&Delta;x}\right),$ Among them, g(Δx) represents a further correction to the fitting result. When the secondary variable is x, let $f(x,\mathrm{the\; y})=f_{\mathrm{ij}}+\frac{h\left(x\right)-h\left(x_j\right)}{h\left(x_{j+1}\right)-h\left(x_j\right)}|f_{\mathrm{ij}}-f_{i+1j}|+g\left(\mathrm{\&Delta;y}\right),$ Among them, g(Δy) represents a further correction to the fitting result.

Through the steps a)-e) above, the expression of the fitting bandwidth function is obtained, which includes two variable data object length x and access jitter step size y. The length of the data object can be a physical length in bytes, or a logical length in a user-defined unit, and the access jitter step refers to the change distance of the access position before or after accessing the data object. It can be a physical distance in bytes or a logical distance in custom units.

For example, the measured reference data can be read bandwidth data, and the data object length x and access jitter step can be taken as the value of the integer power of 2 from small to large, that is, x _i =2 ⁱ , y _j =2 ^j , according to x _i , y _j read a large file from the beginning to the end for multiple times, and take the average value as f _ij . For each _xi, it is also necessary to measure the benchmark bandwidth data when sequential reading (ie, the access jitter step size y=0), that is, the bandwidth data when there is no fragmentation (it can be called continuous bandwidth). Here, the primary variable can be selected as access data size x, and the secondary variable is jumping step size y. Then find the value closest to the primary variable (a value from x ₀ to x _m ) and the value interval of the secondary variable. Let h(x)=log ₂ x, Δx=i-log ₂ x, then Δx is the smallest when i=[log ₂ x+0.5]. make , the value interval of the secondary variable is [2 ^j , 2 ^j+1 ], and when y=0, the interval is [0, 1]. Let g(Δx)=Δx×(f _ij -f _ij+1 ), then as mentioned above, the read bandwidth fitting function can be obtained $f(x,\mathrm{the\; y})=f_{\mathrm{ij}}+1+\frac{h\left(\mathrm{the\; y}\right)-h\left({\mathrm{the\; y}}_j\right)}{h\left({\mathrm{the\; y}}_{j+1}\right)-{\mathrm{hy}}_j}|f_{\mathrm{ij}}-f_{\mathrm{ij}+1}|+$ $g\left(\mathrm{\&Delta;x}\right).$

Then, according to the obtained fitting bandwidth function, the fitting bandwidth f _i =f(l _i , |o _i -o _i-1 -l _i-1 |) corresponding to each access operation s _i in the queue S is calculated. It can be seen that the fitting bandwidth is actually based on the length of the data object (for example, l _i ) and the change distance of the access position before or after accessing the object (for example, |o _i -o _i-1 -l _i-1 |) , a numerical value obtained through the above calculation of the fitted bandwidth function. When, for example, the reception of the access prediction sequence is completed, the upper limit of the delay is reached (the threshold set to avoid excessive delay caused by defragmentation), or the receiving queue S is full, etc., it starts to go to step 2) and continues to execute.

Continuing to refer to FIG. 1, in step 2) starting from a certain access operation in the access prediction sequence, if the prefetch read/write bandwidth of data objects exceeding a predetermined number of consecutive multiple access operations is less than the bandwidth threshold, these access operations The data objects are sequentially written to the new contiguous storage space. In this way, the method chooses to defragment those data objects through the fitted bandwidth calculated above to minimize the negative impact on the current write bandwidth. If the prefetch read/write bandwidth of the data object exceeding the predetermined number of consecutive multiple access operations is greater than the bandwidth threshold, these access operations are skipped, and the next access operation is continued to be processed.

Wherein, the bandwidth threshold may be a static threshold, which may be determined according to the maximum bandwidth value of the storage system that needs to be de-fragmented. For example, the bandwidth threshold can be set as a percentage of the maximum bandwidth of the storage system, such as more than 60% of the maximum bandwidth of the storage system; when the bandwidth threshold can also be a dynamic threshold, it can be determined according to the continuous bandwidth, which is obtained from the fitting function The calculated result is the fitting bandwidth f(x, 0) when the access jitter step is 0. For example, the bandwidth threshold may be set as a percentage of the continuous bandwidth, such as more than 70% of the continuous bandwidth.

In order to more effectively reduce the system fragmentation program and avoid the impact of write operations on the overall performance of the system, in another embodiment, in step 2) the access prediction sequence is divided into three types of continuous subsequences according to three conditions, and each type Subsequences are treated differently. As mentioned above, the received access prediction sequence is stored in the queue S: <s ₁ , ..., s _n >, and the receiving queue S is scanned from the beginning to the end, and the access prediction sequence is divided according to the following three conditions: Multiple subsequences:

Condition 1: s _i ,..., s _i+m satisfy f _i+1 ≥T _fi+1 ,..., f _i+m ≥T _fi+m , m≥T _m , f _i+m+1 <T _fi+m+1 or i+m=n; here indicates the fitting of multiple consecutive access operations exceeding a certain number (ie T _m ) starting from a certain access operation (such as s _i ) in the access prediction sequence The bandwidth is greater than the bandwidth threshold, which means that the fragmentation of the data objects of these access operations is low, and such subsequences can be ignored. At this time, the data objects corresponding to s _i ,..., s i ₊ m do not need to be written to a new continuous storage location, and no processing is required for s _i ,..., s _i+m , and continue to identify the next subsequence . Wherein, T _fi represents the bandwidth threshold of the fitting bandwidth for _si , and T _m is a preset integer threshold. As mentioned above, the bandwidth threshold T _fi can be a static threshold, such as can be set to more than 60% of the maximum bandwidth of the storage system that needs to be defragmented; it can also be a dynamic threshold, such as can be set to be configurable It is more than 70% of the continuous bandwidth, which refers to the fitting bandwidth f(x, 0) when the access jitter step size is 0.

Condition 2: s _i ..., s _i+m satisfy f _i <T _fi , ..., f _i+m <T _fi+m , m≥T′ _m , f _i+m+1 ≥T _{fi+ m+1} or i+m=n; here it indicates that starting from a certain access operation (such as s _i ) in the access prediction sequence, the fitting bandwidth of multiple consecutive access operations exceeding a certain number (such as T′ _m ) is equal to It is less than the bandwidth threshold, which means that the data objects of these access operations are highly fragmented, and such subsequences need to be de-fragmented. At this time, the data objects corresponding to s _i , . . . , s _i+m need to be written into new continuous storage. Wherein, T _fi represents the bandwidth threshold of the fitting bandwidth for _si , and T′ _m is a preset integer threshold.

For the subsequences s _i , ..., s _i+m satisfying condition 2, the data objects corresponding to s _i , ..., s _i +m can be sequentially written into a new continuous storage space, and The access entries for these data objects are changed to the new location. Wherein, the written data object can be read from a local storage location, and can also be obtained from other devices through the network.

Condition 3, except for conditions 1 and 2. For example, s _i , ..., s _i+m do not satisfy condition 1 or condition 2, while i+m=n, or s _i+m+1 , ..., s _i+m+k satisfy condition 1 or Condition 2.

For subsequences that do not satisfy the division conditions of (1) and (2), such as s _i , ..., s _i+m , calculate the overall bandwidth of the sequence, that is, the overall bandwidth generated by executing all access operations in the sequence, Rather than the bandwidth of an access operation. If the overall bandwidth of the sequence is greater than a certain threshold (for example, threshold a), it means that the data objects accessed by the sequence are not highly fragmented, and the sequence does not need to be processed. On the contrary, if the overall bandwidth is less than the threshold a, calculate the assumed continuous bandwidth of the sequence, and the assumed continuous bandwidth refers to the bandwidth after the data objects accessed in the sequence are stored continuously. If the overall bandwidth is less than the threshold a and it is assumed that the continuous bandwidth is greater than a certain threshold (such as threshold b), it means that if the data objects corresponding to these sequences are written to the continuous storage area, the bandwidth will be effectively improved, so the data objects corresponding to these sequences can be Corresponding processing is performed, that is, the data objects corresponding to the subsequences s _i , . . . , s _i+m are written into a new continuous storage space. If the overall bandwidth is less than the threshold a and the continuous bandwidth is assumed to be less than the threshold b, it means that the bandwidth cannot be improved even if the data objects corresponding to these sequences are written into the continuous storage area, so the sequence is not processed. Wherein, the setting of the threshold a and the threshold b may refer to the setting of the bandwidth threshold mentioned above.

where the overall bandwidth and assumed continuous bandwidth are calculated as follows:

The overall bandwidth value of s _i ,..., s _i+m $f=\frac{l_i+\&CenterDot;\&CenterDot;\&CenterDot;+l_{i+m}}{\frac{l_i}{f_i}+\&CenterDot;\&CenterDot;\&CenterDot;+\frac{l_{i+m}}{f_{i+m}}};$

Obtain the starting position o of the continuous storage space that can be written into the corresponding data object of s _i ,..., s _i+m and calculate the assumed continuous bandwidth f'=f(l _i +...+l _i+m ,|oo' _i -1-l _i-1 |), if f<T _f and f′>T′ _f , write the data objects corresponding to s _i ,..., s _i+m to the new continuous storage space in sequence , and change the access entries of these data objects to the new location. Among them, o' _i-1 and l _i-1 are the latest starting position (may have been written to a new position) and size of the data object corresponding to the prediction sequence s _i-1 , and the setting of T _f and T' _f can be Referring to the bandwidth threshold setting described above, the written data object can be read from the local storage location, or obtained from other devices through the network.

When the processing of the queue S is completed and the queue S is emptied, another access prediction sequence is continued to be received.

In order to better understand the above-mentioned defragmentation method, a more detailed example of the defragmentation process will be described below in conjunction with Table 2 and FIG. 2 . Table 2 shows a specific access prediction sequence to be received in the de-fragmentation process, which includes the sequence number i of the access operation in the access prediction sequence, the length of the corresponding data object l _i degree and the starting position o _i of the data object .

Table 2

serial number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 l _i 3 2 5 6 4 3 3 5 4 3 2 3 3 3 o _i 10 20 30 50 3 10 7 35 40 13 16 20 7 twenty three

Among them, the length of the receiving queue S is set to 12. In the initial state, assuming that the last storage access location is 0, the specific execution process is as follows:

1. Execute step 201, receive access operations 1 to 14 in the access prediction sequence in sequence, and go to step 202 each time an access operation is received.

2. Execute step 202 to determine whether it is a repeated access operation; if yes, go to step 201; if not, go to step 203.

Specifically, if the currently received access operation is the same as the data object accessed by the received access operation in the queue, ignore the currently received access operation and go to step 201 to avoid repeated calculation of the access bandwidth of the same data object, for example, When access operations 6 and 13 are received, the data objects accessed by access operations 6 and 13 are the same as those of 1 and 7 (the starting position is the same as the data length).

3. Step 203 is executed to calculate the fitting bandwidth of the access operation.

Specifically, obtain the jitter step of the data object accessed by the currently received access operation relative to the data object at the end of the queue S, input the size of the data object and the obtained jitter step into the fitting bandwidth function obtained above, and input the data Information such as the object size, the starting position of the data object, and the calculated fitting bandwidth are recorded at the end of the queue. For example, for access operation 1, the size of the data object is 3, and the jump step is 10 (jumping from the last access position 0 to The starting position of the data object is 10), the fitting bandwidth is f ₁ =f(3,10), in the access operation 2, the size of the data object is 2, and the jitter step is 7 (jumping from the end position 13 of the data object of the access operation 1 To the starting position of the data object 20), the fitting bandwidth f ₂ =f(2,7).

4. Execute step 204 to determine whether the processing queue condition is satisfied. For example, when the queue is full or reaches the upper limit of delay (avoiding excessive delay caused by de-fragmentation), process the queue from beginning to end, otherwise continue to receive the access prediction sequence.

Steps 201-204 are executed repeatedly until the access prediction sequence is received, or the queue is full to meet the processing conditions, the state of the queue S is shown in Table 3 (visits 6 and 13 are repeated accesses, and reception is ignored):

table 3

queue item s ₁ s ₂ s ₃ s ₄ s ₅ s ₆ l _i 3 2 5 6 4 3 o _i 10 20 30 50 3 7 f _i f(3,10) f(2,7) f(5,8) f(6, 15) f(4, 53) f(3,0) queue item s ₇ s ₈ s ₉ s ₁₀ s ₁₁ s ₁₂ l _i 5 4 3 2 3 3 o _i 35 40 12 15 20 twenty three f _i f(5, 25) f(4,0) f(3, 32) f(2,0) f(3,3) f(3,0)

5. Execute step 205, scan the queue S and divide the subsequences according to the above conditions.

Specifically, scan the queue items s ₁ ~ s ₁₂ from the beginning to the end, and divide the subsequences according to three conditions: Condition 1, s _i ,..., s _i+m satisfy f _i+1 ≥ T _fi+1 , .. ., f _i+m ≥T _fi+m , m≥T _m , f _i+m+1 <T _fi+m+1 or m=12; condition 2, s _i+1 ..., s _i+m Satisfy f _i <T _fi ,..., f _i+m <T _fi+m , m≥T′m, f _i+m+1 ≥T _fi+m+1 or m=12; condition 3, except condition 1 and the rest of condition 2 (condition 1 or condition 2 is not satisfied, while m=12 or s _m+1 , ..., s _m+k satisfy condition 1 or condition 2), wherein, T _fi =0.75×f (l _i , 0), T _m , T′ _m take the integer 4. After scanning, the comparison results are shown in Table 4, in which, the fitting bandwidth of s ₁ to s ₅ is small, satisfying condition 2, and the fitting bandwidth of s ₆ to s ₈ is different in size, satisfying condition 3, s ₉ to s ₁₂ The fitting bandwidth is large (except s ₉ ), and condition 1 is satisfied, and the subsequence division results are marked in the queue accordingly.

Table 4

6. Execute step 206, determine that the subsequence meets the condition; if condition 1 is met, go to step 209; if condition 2 is satisfied, perform step 208; if condition 3 is satisfied, perform step 207.

Specifically, according to the marking result of step 205 , s ₁ to s ₅ satisfy condition 2, and go to step 208 .

7. Execute step 208 to write the data objects of s ₁ to s ₅ into the new storage space

Specifically, find and write the starting position of the continuous storage space where the data objects of s ₁ to s ₅ can be written, assuming that the starting position 100 is found in this example, write the data objects of s ₁ to s ₅ respectively Go to the storage spaces of 100-102, 103-104, 105-109, 110-115, 116-119, and change the access entries of these data objects to new storage locations.

8. Execute step 209, remove s ₁ ~ s ₅ from the queue (point the queue head pointer to s6)

9. Execute step 210 (the queue is not empty at this time, go to step 206)

10. Step 206 is executed to determine that the subsequence satisfies the condition.

Specifically, according to the marking result of step 205 , s ₆ -s ₈ satisfy condition 3, and go to step 207 .

11. Execute step 207 to determine whether the subsequences s ₆ -s ₈ meet the writing conditions.

Specifically, obtain the storage space starting position o of the data objects that can be written into s ₆ ~ s ₈ , and calculate the overall bandwidth The calculation assumes that the write bandwidth f'=f(l ₆ +l ₇ +l ₈ , |oo' ₅ -l ₅ |), after calculation, o=120 (the end position of s ₅ ), o' ₅ =116 (s5 new write position of the data object), l ₅ =4, then f′=f(12,0), threshold T _f =T′ _f =0.75×f(l ₆ +l ₇ +l ₈ ,0)=0.75 ×f(12,0) (threshold T _f =T′ _f =0.75×f(l _i + . . . +l _i+m ,0) corresponding to s _i ˜s i ₊ m . At this time, it is satisfied that f<T _f and f′>T′ _f , go to 208 .

12. Execute step 208, write the data objects of s ₆ to s ₈ into the new continuous storage space

Specifically, write the data objects of s ₆ to s ₈ into the storage spaces of 120 to 122, 123 to 127, and 128 to 131 respectively in sequence, and change the access entries of these data objects to new storage locations.

13. Execute step 209, remove subsequences s ₆ ~ s ₈ from the queue (point the queue head pointer to s ₉ )

14. Execute step 210 (the queue is not empty at this time, go to 206)

15. Execute step 206 to determine whether the subsequence satisfies the condition

Specifically, according to the marking result of step 205 , s ₉ -s ₁₂ satisfy condition 1, and go to 209 .

16. Execute step 209, remove s ₉ ~ s ₁₂ from the queue (point the queue head pointer to s1)

17. Execute step 210 (the queue is empty at this time, go to 201)

The inventor also tested the above-mentioned method by using a backup load in a real environment in a content-finding storage system. The test results show that with the increase of data volume, the read bandwidth increases by 12% to 60%, and the data redundancy is controlled at 1% to 2%, which basically eliminates the trend of continuous decline in read bandwidth; in the same system, using The test results of the two-week data synchronization load show that the read bandwidth is increased by 5 to 8 times, avoiding the severe drop in read bandwidth caused by data fragmentation.

Although the present invention has been described in terms of preferred embodiments, the present invention is not limited to the embodiments described herein, and various changes and changes are included without departing from the scope of the present invention.

Claims (7)

1. go a flaking method, described method to include:

Step 1) determine and access the expection read/write bandwidth of each data object accessing operation in forecasting sequence, described expection reads/ Writing bandwidth is with data object length corresponding to this access operation and access step-length of beating as variable, based on matching bandwidth function The matching bandwidth obtained, described matching bandwidth function is in the storage system needing fragmentation, with data object length and Access step-length of beating is variable, based in different pieces of information object length with access the benchmark read/write bandwidth measured under step-length of beating Data, the bandwidth function expression obtained by approximating method；

Step 2) from accessing in forecasting sequence from the beginning of certain access operation, for exceeding continuous multiple access operations of predetermined number And the expection read/write bandwidth of each data object accessing operation is less than bandwidth threshold, and these are accessed the data object of operation Sequentially it is written to new continuous print memory space；

Wherein, described matching bandwidth function obtains through the following steps:

Step a) is in the storage system needing fragmentation, with data object length x and access step-length y of beating as variable, surveys Measure the benchmark read/write bandwidth data under one group of difference x and y；

Step b) is a selected master variable from x, y, and another is time variable；

Step c) find with master variable closest to master variable value；

Step d) finds the interval of time variable；

Step e) is based on the benchmark read/write bandwidth data matching x under measured different x and y, and y is read/write during any value Bandwidth f (x, y)；

Wherein, in described step c), when master variable is x, find x_iMake Δ x=h (x_i) absolute value of-h (x) is minimum；Work as main transformer When amount is for y, find y_iMake Δ y=h (y_i) absolute value of-h (y) is minimum；

In described step d), when secondary variable is y, find y_jMake y ∈ [y_j,y_j+1], when secondary variable is x, find x_jMake x ∈ [x_j,x_j+1]；

In step e), when secondary variable is y, make Wherein, g Δ x represents the further correction to fitting result；When secondary variable is x, makeWherein, (Δ y) represents to enter fitting result g One step correction.

Method the most according to claim 1, also includes step 3) from accessing in forecasting sequence from the beginning of certain access operation, The expection read/write bandwidth of continuous multiple access operations and each data object accessing operation for exceeding predetermined number is more than Bandwidth threshold, then skip these and access operation, continues with ensuing access and operates.

Method the most according to claim 2, also includes step 4), for accessing in forecasting sequence not by step 2) or step 3) subsequence s that process, that be made up of continuous multiple access operations_i,…,s_i+mIf the overall bandwidth of this subsequence is less than certain The hypothesis continuous bandwidth of individual threshold value and this subsequence is more than certain threshold value, then by this subsequence, each accesses the data pair operated As being written to new continuous print memory space, and the access entrance of these data objects is changed to new position；Wherein, described Assume that continuous bandwidth represents if the bandwidth after being deposited continuously by the data object accessing operation in this subsequence；Subsequence s_i,…,s_i+mOverall bandwidthl_iFor s_iThe length of data object, f_iFor s_iThe expection of data object Read/write bandwidth, m is the number accessing operation in this subsequence.

Method the most according to claim 2, described bandwidth threshold goes the maximum belt of the storage system of fragmentation as required Width values or continuous bandwidth determine, described continuous bandwidth is to beat in the case of step-length is 0 in access, based on matching bandwidth letter The matching bandwidth counted and obtain.

5. go a fragmentation system, described system to include:

Bandwidth determines device, accesses the expection read/write bandwidth of each data object accessing operation in forecasting sequence for determining, Described expection read/write bandwidth is with data object length corresponding to this access operation and access step-length of beating as variable, based on plan Crossed belt width function and the matching bandwidth that obtains, described matching bandwidth function is in the storage system needing fragmentation, with number It is variable according to object length and access step-length of beating, based in different pieces of information object length with access the base measured under step-length of beating Quasi-read/write bandwidth data, the bandwidth function expression obtained by approximating method；

Remove crumb units, for from accessing in forecasting sequence from the beginning of certain access operation, for exceeding the continuous many of predetermined number These, less than bandwidth threshold, are accessed operation by the individual expection read/write bandwidth of operation and each data object accessing operation that accesses Data object be sequentially written to new continuous print memory space；

Wherein, described matching bandwidth function obtains through the following steps:

Step a) is in the storage system needing fragmentation, with data object length x and access step-length y of beating as variable, surveys Measure the benchmark read/write bandwidth data under one group of difference x and y；

Step b) is a selected master variable from x, y, and another is time variable；

Step c) find with master variable closest to master variable value；

Step d) finds the interval of time variable；

Step e) is based on the benchmark read/write bandwidth data matching x under measured different x and y, and y is read/write during any value Bandwidth f (x, y)；

Wherein, in described step c), when master variable is x, find x_iMake Δ x=h (x_i) absolute value of-h (x) is minimum；Work as main transformer When amount is for y, find y_iMake Δ y=h (y_i) absolute value of-h (y) is minimum；

In described step d), when secondary variable is y, find y_jMake y ∈ [y_j,y_j+1], when secondary variable is x, find x_jMake x ∈ [x_j,x_j+1]；

In step e), when secondary variable is y, make Wherein, g Δ x represents the further correction to fitting result；When secondary variable is x, makeWherein, (Δ y) represents to enter fitting result g One step correction.

System the most according to claim 5, described in go crumb units to be additionally operable to from accessing forecasting sequence certain accesses behaviour Work starts, for exceeding continuous multiple access operations and the expection read/write of each data object accessing operation of predetermined number Band is wider than bandwidth threshold, then skip these and access operation, continues with ensuing access and operates.

System the most according to claim 6, described in go crumb units to be additionally operable to the company for being unsatisfactory for exceeding predetermined number The expection read/write bandwidth of continuous multiple data object accessing operation both greater than or less than bandwidth threshold, by continuous multiple access The subsequence s that operation is constituted_i,…,s_i+mIf the overall bandwidth of this subsequence is less than certain threshold value and the hypothesis of this subsequence Continuous bandwidth is more than certain threshold value, then each data object accessing operation in this subsequence is written to new continuous print storage Space, and the access entrance of these data objects is changed to new position；Wherein, described hypothesis continuous bandwidth represents if will This subsequence accesses the bandwidth after the data object of operation is deposited continuously；Subsequence s_i,…,s_i+mOverall bandwidthl_iFor s_iThe length of data object, f_iFor s_iThe expection read/write bandwidth of data object, m is this sub-sequence Row access the number of operation.