CN1996226A - I/O requests area forecasting method based on sequence-degree clustering algorithm and time series - Google Patents

Abstract

The invention relates to a I/O zone pre selection method based on continuity gathering and time sequence. It predicts the densely I/O visit zone during leisure time to make pre selection for the zone based on the partial features of the request. It is effective, reliable in finding densely requested zones, and it uses ARMA time sequence module to predict future possible visit zones and timing. Under the same testing environment, compared with existing RAID system, it can read gathered request and predict pre selection gathering I/O zone. It is practical and effective in finding gathered requested visit zone with accuracy and improved storage system performance.

Description

A kind of based on continuation degree cluster and seasonal effect in time series I/O area forecasting method

Technical field

The present invention relates to field of storage, specifically be meant a kind of based on continuation degree cluster and seasonal effect in time series I/O area forecasting method.

Background technology

Though the performance of parallel I/raising storage system that O system (disk array) can be bigger, because the inherent shortcoming of disk (tracking of long period postpones and rotational latency) still exists bigger performance gap between them.Buffer memory can be good at remedying performance gap between them as a basic realization technology, but the capacity of simple increase buffer memory has not been a method of effectively dealing with problems, prefetching technique then is by prediction request of data in the future, data block in the memory device was got in the buffer memory before using, request can be hit in buffer memory to reduce the dead time of CPU, and eliminate contention to disk or passage by prediction cache miss, overlapping I/O technology, reduce the delay of disk access, thereby improve the performance of system.Simultaneously there is stronger correlativity in the visit of data in the storage system, as when visiting certain file, will inevitably visit the catalogue of this file.System can utilize the correlativity of data access to improve the precision of looking ahead, and it is congested to reduce buffer memory pollution and passage.But in existing file system, be in the storage system of mechanical floor and do not know the semantic information that any I/O visits, the semanteme that therefore can not make full use of the I/O visit next data that constantly will visit of looking ahead can only utilize the characteristics such as locality, sequential access and cyclic access of better simply mode such as I/O visit to realize simple prediction.Simultaneously we know in the intensive application of I/O, the I/O visit exists paroxysmal feature, can visit the less moment (system is constantly idle) disposable all data of the intensive moment of next read request of looking ahead at I/O and improve the performance of system, large capacity cache also makes this method become possibility in the existing storage system.Therefore how to utilize original information to predict that next moment read request close quarters is the problem that needs solution.

Summary of the invention

The object of the present invention is to provide a kind of based on continuation degree cluster and seasonal effect in time series I/O area forecasting method, use this method can improve the I/O performance of storage subsystem, practical and find the zone of intensive read request visit efficiently and look ahead accurately, and then improve the performance of storage system.

Provided by the invention a kind of based on continuation degree cluster and seasonal effect in time series I/O area forecasting method, its step comprises:

(1) asks the intensive moment at I/O, the read request in the I/O stream is sorted according to physical block address, form an object chained list;

(2) all physical addresss are linked to each other or overlapping object is merged into an object, its length is each object length sum, calculates the continuation degree of all objects by following rule:

If the length of object p is 1,, then establish its continuation degree S if its left side or right side do not have continuous object _Io(p)=a ₁If there is a continuous object on its left side or right side, then establish its continuation degree S _Io(p)=a ₂If all there is continuous object both sides, then establish its continuation degree S _Io(p)=a ₃, wherein, a ₃＞a ₂＞a ₁

(3) the mark continuation degree is greater than threshold values H _oObject as kernel object, wherein, threshold values H _oCalculate according to formula (1), wherein k is the quantity of object:

$H_o\≥\frac{\underset{i=1}{\overset{k}{\mathrm{\Σ}}}{S}_{\mathrm{io}}\left(p_i\right)}{k},i=1,\mathrm{\Λ},k---\left(1\right)$

(4) all objects that can reach or link to each other according to following rule searching and each kernel object, it is constructed cluster:

(A1) be the center with kernel object o, the left and right sides is the d-neighborhood that the zone of radius is called o apart from d; If D is an object set, p ∈ D, o ∈ D, if o is kernel object, p then claims p directly can reach about d from o in the d-of o neighborhood;

(A2) if there is an object chain p ₁, p ₂..., p _np _i∈ D, 1≤i≤n-1, n are the number of object in the object chain, p _I+1Be from p _iSet out, directly can reach about d, then p _nBe from p ₁Set out and to reach about d;

(A3) in object set D, o ∈ D, p ∈ D, q ∈ D if having p and q all is from o, about reaching of d, claims that then p, q are linking to each other about d;

(A4) all objects that will satisfy following condition are constructed clusters:

(I)  p, q: if p ∈ C and q are about the reaching of d, then q ∈ C from p;

(II)  p, q ∈ C:p, q are about the linking to each other of d, then p ∈ C;

(5) from each bunch that forms, select continuation degree more than or equal to a bunch threshold value H _cBunch, form effective result bunch, wherein bunch threshold value H _cCalculate according to formula (2):

$H_c=\frac{1}{2}[a_{3}l-2(a_3-a_2)]---\left(2\right)$

Wherein l is a bunch shared space length;

(6) repeating step (1) is to (5), until a plurality of effective result who obtains a plurality of time periods bunch; Otherwise change step (7) over to;

(7) from effective result bunch, obtain access region information, and constitute access time information with the time of a plurality of intensive I/O visits, form the time series in read request zone and the time series T of access time respectively _{C (t)}, effectively result's bunch central point and bunch radius formation access region, the center point value of establishing effective result bunch is P _{BA (t)}, corresponding effectively result's bunch bunch radius is R _(t)

(8) judge whether access region information and access time information satisfy the stationary time series that the ARMA time series is a zero-mean, if enter step (10), otherwise enter step (9);

(9) sequence of non-stationary being carried out tranquilization handles;

(10) time series constructs the ARMA time series predicting model stably, promptly

P _BA(t)-φ ₁P _BA(t-1)-Λ-φ _kP _BA(t-p)＝a _t-θ ₁a _t-1-Λ-θ _qa _t-q (5)

R _(t)-φ ₁R _(t-1)-Λ-φ _kR _(t-p)＝a _t-θ ₁a _t-1-Λ-θ _qa _t-q (6)

T _c(t)-φ ₁T _c(t-1)-Λ-φ _kT _c(t-p)＝a _t-θ ₁a _t-1-Λ-θ _qa _t-q (7)

Wherein, φ _k, θ _qBe undetermined coefficient, a _tBe error coefficient;

(11) nearest I/O is asked the intensive moment and before a plurality of I/O ask the P that the intensive moment obtains _{BA (t)}, R _(t)And T _{C (t)}Value is brought in formula (5), (6) and (7), predicts that next I/O asks the intensive moment to look ahead regional and the time value of looking ahead;

(12) data predicted is reduced processing, form the predicted value of the time of looking ahead after the reduction and the predicted value of the area information of looking ahead;

(13) judge whether system is idle, if idle, looks ahead, and enters step (14) then, otherwise repeating step (13);

(14) with the data of looking ahead and actual read access data contrast, if when the error of the two surpasses the threshold value of defined, enter step (10), re-construct forecast model, otherwise enter step (15);

(15) finish up to system works repeating step (1)-(15).

The present invention starts with from two aspects: the one, adopt clustering algorithm based on continuation degree, and it can find the zone of intensive read request efficiently, reliably; Next is the zone and the visit moment of utilizing ARMA (Auto-Regressive andMoving Average) time series models to predict that following intensive read request may be visited.Under same test environment, the storage system that adopts the inventive method and existing RAID system have been carried out contrast test, show at the load testing that utilizes 3 disk volumes: the cluster read request that the clustering algorithm based on continuation degree that the present invention proposes can be correct; While is based on the intensive I/O zone of accurately looking ahead of the performance prediction algorithm of AMRA seasonal effect in time series forecast model.In a word, the present invention proposes a kind of can be practical and find the zone of intensive read request visit efficiently based on continuation degree cluster and seasonal effect in time series I/O area forecasting method, and the zone that may visit of the intensive read request of looking ahead accurately, bigger raising performance of storage system.

Description of drawings

Fig. 1 is the process flow diagram of the inventive method;

Fig. 2 is that seasonal difference is judged synoptic diagram;

Fig. 3 is the cluster result synoptic diagram;

Fig. 4 is the prefetch hit rate synoptic diagram.

Embodiment

The present invention is the feature that request has locality according to I/O, and the idle moment is predicted the zone that intensive I/O visits in system, and is looked ahead in the zone of prediction.

As shown in Figure 1, the step of the inventive method comprises:

(1) utilizes the defined clustering algorithm of the present invention, the I/O of storage system is analyzed, find the zone of intensive read request based on continuation degree.For ease of describing, read request is referred to as object.Its detailed process is:

(1) asks the intensive moment at I/O, the read request in the I/O stream is sorted according to physical block address, form an object chained list.

(2) all physical addresss are linked to each other or overlapping object is merged into an object, its length is each object length sum, calculates the continuation degree of all objects by following rule:

Continuation degree s _Io(p) the continuous degree of indicated object is directly proportional with its length.Concerning an object p, if its length is 1, its left side or right side do not have continuous object, and continuation degree is a ₁There is a continuous object on its left side or right side, and continuation degree is a ₂All there is continuous object both sides, and continuation degree is a ₃, a ₃＞a ₂＞a ₁Object is in a ₃State, its storage efficiency is best; Request is in a ₁State, its storage efficiency is the poorest.Therefore according to top setting, if the length of an object is l ' and l '＞1, its continuation degree is a ₃L '-2 (a ₃-a ₂).

(3) the mark continuation degree is greater than threshold values H _oObject as kernel object.

If the continuation degree of an object is more than or equal to threshold values H _o, this object is called kernel object.Kernel object threshold values H _oValue be to ask with I/O that the average continuation degree of object is a standard in the intensive moment, that is:

$H_o\≥\frac{\underset{i=1}{\overset{k}{\mathrm{\Σ}}}{S}_{\mathrm{io}}\left(p_i\right)}{k},i=1,\mathrm{\Λ},k---\left(1\right)$

Wherein k is the quantity of object.

(4) find all object structure clusters that can reach or link to each other with each kernel object:

(A1) be the center with kernel object o, the left and right sides is the d-neighborhood that the zone of radius is called o apart from d, and the value of d is generally half of kernel object length.If D is an object set, p ∈ D, o ∈ D, if o is kernel object, p then claims p directly can reach about d from o in the d-of o neighborhood.The continuation degree sum of object is called the continuation degree of o in the d-neighborhood in the neighborhood.

(A2) if there is an object chain p ₁, p ₂... p _np _i∈ D (1≤i≤n-1), n is the number of object in the object chain; p _I+1Be from p _iSet out, directly can reach about d, then p _nBe from p ₁Set out and to reach about d.

(A3) in object set D, o ∈ D, p ∈ D, q ∈ D if having p and q all is from o, about reaching of d, claims that then p, q are linking to each other about d.

(A4) all objects that satisfy following condition are constructed clusters.

(I)  p, q: if p ∈ C and q are about the reaching of d, then q ∈ C from p.

(II)  p, q ∈ C:p, q are about the linking to each other of d, then p ∈ C.

(5) from each bunch that forms, select continuation degree more than or equal to threshold value H _cBunch, form effective result bunch.Bunch threshold value H _cBe made as:

$H_c=\frac{1}{2}[a_{3}l-2(a_3-a_2)]---\left(2\right)$

Wherein l is a bunch shared space length.

(6) if operation first, repeating step (1) is to (5), until a plurality of effective result who obtains a plurality of time periods bunch; Otherwise change step (7) over to.Effectively result's bunch quantity determines that according to the needs of following forecast model the determined quantity of the present invention is generally 10-15.

(2) after having obtained formed a plurality of effective results of a plurality of intensive I/O access time sections bunch, adopt the ARMA time series predicting model to dope I/O zone that next intensive I/O visit will visit constantly and disposable it is looked ahead.

The detailed process that adopts the ARMA time series predicting model to predict is:

(7) from effective result bunch, obtain access region information, and constitute access time information with the time of a plurality of intensive I/O visits, form the time series in read request zone and the time series T of access time respectively _{C (t)}P wherein _{BA (t)}Represent effective result's bunch of different time sections center point value, R _(t)Be corresponding effectively result's bunch bunch radius, effectively result's bunch a central point and a bunch radius have constituted access region.

(8) judge whether access region information and access time information satisfy the stationary time series that the ARMA time series is a zero-mean, if enter step (10), otherwise enter step (9).

Criterion is to see whether the decay of the autocorrelation function of sequence and partial correlation function is slow, promptly

$p=\frac{1}{J}\underset{k=0}{\overset{J}{\mathrm{\Σ}}}\frac{\left|P_k\right|-\left|P_{k+1}\right|}{\left|P_k\right|}---\left(3\right)$

P wherein _kThe auto-correlation function value of expression sequence, J represent that last surpasses the sequence number of the auto-correlation function value in self-confident interval.When rate of change less than 15% the time, think that these time series right and wrong are stably.

(9) sequence of non-stationary being carried out tranquilization handles: promptly the sequence of non-stationary is carried out difference or zero-mean processing, up to forming stationary sequence.

Calculate the exponent number d of difference according to equation (4) _a:

$d_a=[1g\underset{n=1}{\overset{N}{\mathrm{\Σ}}}X{\left(n\right)}^2/21\mathrm{gN}]---\left(4\right)$

Wherein X (n) represents a plurality of samples of sequence.If it is self-confident interval to still have part to surpass through differentiated seasonal effect in time series auto-correlation function value, we think that this sequence has seasonality so.

In order to find the seasonal feature of sequence, the difference method that the present invention adopts.As shown in Figure 2, form a time series with surpassing the pairing lag coefficient of self-confident interval value in the autocorrelation function, then this sequence is carried out first order difference, as finding that certain value frequency of occurrences in differentiated sequence surpasses threshold value (being generally 50%), we just can think that this value is the seasonal cycle number so.If after first order difference, do not find its seasonal characteristics, then adopt two jumps to divide also by that analogy, up to finding its seasonal characteristics value.

(10) time series constructs the ARMA time series predicting model stably, promptly

P _BA(t)-φ ₁P _BA(t-1)-Λ-φ _kP _BA(t-p)＝a _t-θ ₁a _t-1-Λ-θ _qa _t-q (5)

R _(t)-φ ₁R _(t-1)-Λ-φ _kR _(t-p)＝a _t-θ ₁a _t-1-Λ-θ _qa _t-q (6)

T _c(t)-φ ₁T _c(t-1)-Λ-φ _kT _c(t-p)＝a _t-θ ₁α _t-1-Λ-θ _qa _t-q (7)

ARMA time series parameter evaluation method by routine just can be obtained undetermined coefficient φ _k, θ _qWith error coefficient a _t

(11) nearest I/O is asked the intensive moment and before a plurality of I/O ask the P that the intensive moment obtains _{BA (t)}, R _(t)And T _{C (t)}Value is brought in formula (5), (6) and (7), then can dope the look ahead zone and the time value of looking ahead of next intensive moment of I/O read request.

(12) data predicted is reduced processing, form the predicted value of the time of looking ahead after the reduction and the predicted value of the area information of looking ahead: if legacy data has carried out difference, data predicted need add by the data of difference so; If former data sequence has been carried out the zero-mean processing, data predicted need add the average of legacy data.

Handle if before setting up arma modeling, data are carried out tranquilization, and the predicted data that obtains also is the data after handling at tranquilization, therefore will obtains real actual prediction data, need reduce data predicted.

(13) judge whether system is idle, if idle, looks ahead, and enters step (14) then, otherwise repeating step (13).

Through forming the time of looking ahead and the area information of looking ahead after the reduction of data, when look ahead by the time control of looking ahead, it constantly must be constantly idle in system looking ahead, and judges that the step of the actual time of looking ahead is:

(B1) suppose that certain I/O request is last intensive I/O request, and

No I/O request in time, it is constantly idle to think that then system now is in, and utilizes formula (8) to calculate the value of σ;

$10\≤\mathrm{\σ}\≤\frac{\stackrel{\‾}{H_n}-{\mathrm{\σ}}_H-\stackrel{\‾}{t_p}-m\stackrel{\‾}{t_t}}{\stackrel{\‾}{T_{\mathrm{io}}}}---\left(8\right)$

Wherein:

$\stackrel{\‾}{H_n}=\frac{\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{H}_i}{n_a}---\left(9\right)$

N wherein _aBe the time interval of nearest 10 cluster areas, H _iBe each cluster areas time interval, average time interval is

, σ _HBe the mean square deviation of average time interval, the average intensive I/O time interval is

The average location of disk or array postpones (comprising that rotational latency adds that tracking postpones), the time of a data block of transmission is

, m is the data block quantity of looking ahead.

(B2) establish t ₁, t ₂Being respectively is the moment and last moment, the then t the earliest that looks ahead ₁=10T _Io, $t_2=\stackrel{\‾}{H_n}-{\mathrm{\σ}}_H-\stackrel{\‾}{t_p}-m\stackrel{\‾}{t_t},$ Looking ahead of prediction is T constantly _c, actual looking ahead is T constantly _pAccording to the following method T constantly that obtains looking ahead _p, at T _pConstantly look ahead:

(I) if t ₂〉=T _c〉=t ₁, T then _p=T _c

(II) if T _c≤ t ₁, T then _p=t ₁

(III) if T _c〉=t ₂, T then _p=t ₂

(14),, re-construct forecast model, otherwise enter step (15) if when the error of the two surpasses the threshold value of defined (as 50%), enter step (10) with the data of looking ahead and actual read access data contrast.

(15) finish up to system works repeating step (1)-(14).

Example:

In the test of this invention, utilize one section trace file of Hewlett-Packard Corporation, what its write down is 3 disk volumes (reel number is 21,23 and 35) situation of client-access server under the typical working environment, time span is 191.12 hours, and 230370 request numbers are arranged.To a ₁, a ₂And a ₃, the present invention adopts fuzzy quantitative methods, and its assignment is respectively 1,2 and 3.

Fig. 3 shows be volume 21 time spans be 191.12 hours I/O request point and volume poly-in the center result.As can be seen from the figure, what obtain by the algorithm based on continuation degree is 5 effective bunch, i.e. C0-C4.In whole observation process, the request number of actual cluster is 45523, accounts for 91.16% of whole process request number.

What Fig. 4 then reflected is cluster areas prefetch hit rate process in the whole test process.As can be seen, its prefetch hit rate is a process of constantly adjusting.When looking ahead 1089 cluster areas, produced 316 adjustings, account for 29.02% of total prediction number of times, its ratio is still lower.In whole test, 145548 requests of having looked ahead account for 95.37% of total request number.Its average hit rate is 61.12%.

Claims (3)

1, a kind of based on continuation degree cluster and seasonal effect in time series I/O area forecasting method, its step comprises:

(1) asks the intensive moment at I/O, the read request in the I/O stream is sorted according to physical block address, form an object chained list;

(2) all physical addresss are linked to each other or overlapping object is merged into an object, its length is each object length sum, calculates the continuation degree of all objects by following rule:

If the length of object p is 1,, then establish its continuation degree S if its left side or right side do not have continuous object _Io(p)=a ₁If there is a continuous object on its left side or right side, then establish its continuation degree S _Io(p)=a ₂If all there is continuous object both sides, then establish its continuation degree S _Io(p)=a ₃, wherein, a ₃＞a ₂＞a ₁

(3) the mark continuation degree is greater than threshold values H _oObject as kernel object, wherein, threshold values H _oCalculate according to formula (1), wherein k is the quantity of object:

$H_o\≥\frac{\underset{i=1}{\overset{k}{\mathrm{\Σ}}}{S}_{\mathrm{io}}\left(p_i\right)}{k},i=1,\mathrm{\Λ},k---\left(1\right)$

(4) all objects that can reach or link to each other according to following rule searching and each kernel object, it is constructed cluster:

(A1) be the center with kernel object o, the left and right sides is the d-neighborhood that the zone of radius is called o apart from d; If D is an object set, p ∈ D, o ∈ D, if o is kernel object, p then claims p directly can reach about d from o in the d-of o neighborhood;

(A2) if there is an object chain p ₁, p ₂..., p _np _i∈ D, 1≤i≤n-1, n are the number of object in the object chain, p _I+1Be from p _iSet out, directly can reach about d, then p _nBe from p ₁Set out and to reach about d;

(A3) in object set D, o ∈ D, p ∈ D, q ∈ D if having p and q all is from o, about reaching of d, claims that then p, q are linking to each other about d;

(A4) all objects that will satisfy following condition are constructed clusters:

(I)  _{P, q}: if p ∈ C and q are about the reaching of d, then q ∈ C from p;

(II)  _{P, q}∈ C:p, q are about the linking to each other of d, then p ∈ C;

(5) from each bunch that forms, select continuation degree more than or equal to a bunch threshold value H _cBunch, form effective result bunch, wherein bunch threshold value H _cCalculate according to formula (2):

$H_c=\frac{1}{2}[a_{3}l-2(a_3-a_2)]---\left(2\right)$

Wherein l is a bunch shared space length;

(6) repeating step (1) is to (5), until a plurality of effective result who obtains a plurality of time periods bunch; Otherwise change step (7) over to;

(7) from effective result bunch, obtain access region information, and constitute access time information with the time of a plurality of intensive I/O visits, form the time series in read request zone and the time series T of access time respectively _{C (t)}, effectively result's bunch central point and bunch radius formation access region, the center point value of establishing effective result bunch is P _{BA (t)}, corresponding effectively result's bunch bunch radius is R _(t)

(8) judge whether access region information and access time information satisfy the stationary time series that the ARMA time series is a zero-mean, if enter step (10), otherwise enter step (9);

(9) sequence of non-stationary being carried out tranquilization handles;

(10) time series constructs the ARMA time series predicting model stably, promptly

P _BA(t)-φ ₁P _BA(t-1)-Λ-φ _kP _BA(t-p)＝a _t-θ ₁a _t-1-Λ-θ _qa _t-q (5)

R _(t)-φ ₁R _(t-1)-Λ-φ _kR _(t-p)＝a _t-θ ₁a _t-1-Λ-θ _qa _t-q (6)

T _c(t)-φ ₁T _c(t-1)-Λ-φ _kT _c(t-p)＝a _t-θ ₁a _t-1-Λ-θ _qa _t-q (7)

Wherein, φ _k, θ _qBe undetermined coefficient, a _tBe error coefficient;

(11) nearest I/O is asked the intensive moment and before a plurality of I/O ask the P that the intensive moment obtains _{BA (t)}, R _(t)And T _{C (t)}Value is brought in formula (5), (6) and (7), predicts that next I/O asks the intensive moment to look ahead regional and the time value of looking ahead;

(12) data predicted is reduced processing, form the predicted value of the time of looking ahead after the reduction and the predicted value of the area information of looking ahead;

(13) judge whether system is idle, if idle, looks ahead, and enters step (14) then, otherwise repeating step (13);

(14) with the data of looking ahead and actual read access data contrast, if when the error of the two surpasses the threshold value of defined, enter step (10), re-construct forecast model, otherwise enter step (15);

(15) finish up to system works repeating step (1)-(15).

2, method according to claim 1 is characterized in that: in the step (4), d is half of kernel object length.

3, method according to claim 1 is characterized in that: step (13) is looked ahead the time according to following process judgement:

(B1) suppose that certain I/O request is last intensive I/O request, and No I/O request in time, it is constantly idle to think that then system now is in, and utilizes formula (8) to calculate the value of σ;

$10\≤\mathrm{\σ}\≤\frac{\stackrel{\‾}{H_n}-{\mathrm{\σ}}_H-\stackrel{\‾}{t_p}-m\stackrel{\‾}{t_t}}{\stackrel{\‾}{T_{\mathrm{io}}}}---\left(8\right)$

Wherein:

$\stackrel{\‾}{H_n}=\frac{\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{H}_i}{n_a}---\left(9\right)$

N wherein _aBe the time interval of nearest 10 cluster areas, H _iBe each cluster areas time interval, average time interval is σ _HBe the mean square deviation of average time interval, the average intensive I/O time interval is The average location of disk or array postpones

The time of a data block of transmission is

M is the data block quantity of looking ahead;

(B2) establish t ₁, t ₂Being respectively is the moment and last moment, the then t the earliest that looks ahead ₁=10T _Io, $t_2=\stackrel{\‾}{H_n}-{\mathrm{\σ}}_H-\stackrel{\‾}{t_p}-m\stackrel{\‾}{t_t},$ Looking ahead of prediction is T constantly _c, actual looking ahead is T constantly _pAccording to the following method T constantly that obtains looking ahead _p, at T _pConstantly look ahead:

(I) if t ₂〉=T _c〉=t ₁, T then _p=T _c

(II) if T _c≤ t ₁, T then _p=t ₁

(III) if T _c〉=t ₂, T then _p=t ₂