CN102033718B

CN102033718B - Extensible quick stream detection method

Info

Publication number: CN102033718B
Application number: CN 201010598517
Authority: CN
Inventors: 许建卫; 李麟; 刘新春
Original assignee: Dawning Information Industry Co Ltd
Current assignee: Dawning Network Technology Co ltd
Priority date: 2010-12-17
Filing date: 2010-12-17
Publication date: 2013-06-19
Anticipated expiration: 2030-12-17
Also published as: CN102033718A

Abstract

The invention provides an extensible quick stream detection method, which comprises the processes of current stream update, outdated stream deletion, new stream detection and new stream initialization. The method can realize a stream detection mechanism under O (N) time overhead, and can simultaneously maintain the information of N streams. On the aspect of space overhead, the bitmap information can be expressed by using a 64bit integer, so for a detection mechanism for maintaining the N streams, the occupation of a memory is N stream-vec-t, and the occupation rate is low.

Description

A kind of extendible quick stream detection method

Technical field

The present invention relates to performance of storage system and optimize the field, be specifically related to a kind of extendible quick stream detection method.

Background technology

Disk system is owing to adopting mechanical type rotating equipment, so performance boost is always very slow.With respect to based on other machine elements such as the CPU of integrated circuit and internal memories, disk unit and their performance gap enlarge year by year, have become the serious bottleneck of limiting computer entire system performance.But have the problem of the aspects such as price, life-span due to present new memory device, disk unit is still occupied an leading position in storage system.

The performance of disk unit is larger with the difference fluctuation of access pattern, be that the sequential access bandwidth is higher, the disk of common SATA interface can reach the above actual measurement bandwidth of 80MB/s in the market, and the Disk bandwidth of SAS interface can reach the above actual measurement bandwidth of 120MB/s.But small grain size (＜4KB) immediately under access module, all need the movement of disk arm due to each data access, so access bandwidth is very low, disk 10 bandwidth of the application of some completely randoms only have the 1MB left and right.Along with the increase of access granularity, 10 bandwidth of disk system increase gradually.When access granularity is increased to a certain degree (＞4MB), whether the performance of disk system becomes no longer responsive at random for access module.

Cause the more random reason of disk access pattern to be divided into two kinds, a kind of is that the access module of using self is exactly stochastic distribution, such as use the website, online thing is exerted process database etc.In this application, process or number of threads are few, but the data access patterns of each process or thread has larger randomness.Another kind is that the access module of using self is comprised of a plurality of processes or thread, and the access sequence of each process or thread is order, but due to the stack of a plurality of sequences with interweave, has formed random 10 access on the whole.Typical use as the Streaming Media classes such as IPTV are used, it is characterized in that in this case be a kind of pseudorandom at random in fact, it is comprised of the streaming access of a plurality of orders.Stream is defined as follows:

S-1Ai, A2 ... one Ai_i-Ai+TJ

The address sequence A that fixing access interval T is namely arranged.Stream S of common composition.

Because disk system is still higher for the random access performance of coarsegrain, therefore access for streaming, can adopt stream to detect first-class method of looking ahead, increase the access granularity of each disk, data in each stream are accessed with coarsegrain in advance, be buffered in afterwards in a certain high-speed equipment, such as internal memory.When 10 actual access arrive,, just can directly access from high-speed equipment Already in high-speed equipment due to data.When the data of buffer memory in high-speed equipment have exhausted or have been about to exhaust, again from disk unit with larger granularity access and buffer memory.Under this mechanism, can will be converted into the random access of coarsegrain to the small grain size random access of disk, thereby improve the overall performance of disk system.

In above-mentioned steps, the detection of stream is the basis of whole process.Therefore, the stream that how to detect fast and effectively in deep and remote access sequence just becomes key.

Summary of the invention

Make discovery from observation, the T value that flows in practical application is 1 substantially, and namely the address is continuous.Mix for the continuous multithread in address in the present invention and use, proposed a kind of extendible Rapid Flow detection technique.Can expand the time and the space expense that refer to the stream detection algorithm that proposes in the present invention is 0 (N), and wherein N is the number of stream to be detected, and namely the number of detection overhead and stream is linear scaling and increases.

A kind of extendible quick stream detection method, step is as follows:

A, when new address stream request is arranged, judge whether bitmap corresponding to current index effective;

If A1 effectively judges whether to equate with current flow vector address, equal to the address renewal of current flow vector, current flow vector age zero clearing; Unequally increase the current flow vector age;

Increase the current flow vector age if A2 is invalid;

B, current flow vector age of judgement, if the age surpasses invalid should stream of threshold value, do not increase the index value and search number of times if surpass;

C, judge that whether the index value surpasses the value of cutting off from, the index value makes zero if surpass, abandons this stream and continues next stream; Whether search number of times over threshold value if be no more than judgement;

Surpass threshold value if D searches number of times, newly flow initialization; If do not surpass, return to A and again search.

The first optimal technical scheme of the present invention is: described step D newly flows initialization procedure and is:

D1, judge whether the corresponding bitmap of current index is effective, if invalid respective streams vector address+1, the age sets to 0; If effectively increase the index value;

If D2 index value is greater than threshold value, the index value returns 0, then returns to D1; If be not more than, return to D1, again detect and return to whether again detect the corresponding bitmap of current inclex effective.

The second optimal technical scheme of the present invention is: described index is flow index, points to up-to-date active stream.

The third optimization technique ten thousand cases of the present invention are: described flow vector comprises address information, needed lock when age information and this flow vector of access.

The present invention can realize a stream testing mechanism under 0 (N) time overhead, and can safeguard simultaneously the information of N stream.Space expense aspect, message bit pattern can represent with 64 integers, like this for a testing mechanism of safeguarding N stream, internal memory take size for N stream_vec_t, occupancy is very low.

Description of drawings

Fig. 1 is the stream testing mechanism

Fig. 2 upgrades and invalidation mechanism for stream

Fig. 3 is the initialization of new stream

Embodiment

For given address access sequence A, (i=1---n), use stream testing mechanism as shown in Figure 1 to detect A, the stream information of middle existence in the stream testing mechanism.

As shown in Figure 1, this testing mechanism comprises that current flow index index, message bit pattern bitmap and one flow to scale vec, and message bit pattern is with to flow to scale length identical.Wherein flow index is used to refer to up-to-date active stream, and according to the principle of locality of routine access, the probability that current active stream may be accessed again is very large, therefore uses flow index to keep a record, to accelerate the stream search procedure.Each bitmap in message bit pattern points to one that flows in scale, whether effectively is used for identifying corresponding flow vector.Item in each vector table is a structure, and it has described the essential information of a stream.

Each flow vector comprises three territory: address information addr, needed lock lock when age information age Qi accesses this vector.Wherein recorded the next data of expection in this stream in address information, how long not accessed age information has recorded this stream has.Because certain stream information may be accessed, be upgraded to a plurality of processes or thread simultaneously, so need to realize mutual exclusion with lock.

For a stream, mainly comprise three kinds of basic operations: existing stream upgrades, out-of-date stream is deleted and new stream detecting.Have the stream renewal now and comprise two aspects, for the situation that hit in existing stream new address, need to upgrade and hit the expectating address information of stream and the age information of this stream.For the stream that does not hit, to increase its aging age, make the long-term no stream can be identified.The deletion of out-of-date stream is to surpass the stream of threshold value for the age, it is set for invalid, in order in time abdicate corresponding vector table position, for other new stream.New stream detecting is for the current address, if all oneself detection streams all do not hit, just it is thought the beginning of a new stream, need to do to it initial work of new stream.

Fig. 2 has provided the deletion flow process of existing stream renewal and out-of-date stream.Therefrom can find out, existing stream renewal, the deletion of out-of-date stream and new these three kinds of functions of stream detecting can all realize in once circulating.Namely detect successively each element in stream, if hit, directly scheduler and age information.If do not hit, only upgrade age information.If the age surpasses threshold value, it directly is set is invalid stream.In testing process, with the station location marker at invalid stream place out.Fig. 3 has provided the initialization mechanism of new stream.If all existing stream does not all hit, in the position of the invalid stream of mark, new address information is upgraded into, and age information is set.

Claims

1. extendible quick stream detection method, it is characterized in that: step is as follows:

If A1 effectively judges whether to equate with current flow vector address, equal to the address renewal of current flow vector, current flow vector age zero clearing; The unequal current flow vector address of increasing;

Increase the current flow vector age if A2 is invalid;

B, judgement current flow vector address if the age surpasses invalid current flow vector of threshold value, does not increase index value and searches number of times if surpass;

Whether C, index value surpass threshold value, and the index value makes zero if surpass, abandons current flow vector and continues next stream;

Whether search number of times over threshold value if be no more than judgement;

Surpass threshold value if D searches number of times, newly flow initialization; If do not surpass, return to A and again search;

Described step D newly flows initialization procedure:

D1: judge whether the corresponding bitmap of current index is effective, if invalid respective streams vector address adds 1, the flow vector age sets to 0; If effectively the index value adds 1;

D2: if the index value greater than threshold value, the index value returns 0, then returns to D1; If be not more than, return to D1, whether effectively again detect the corresponding bitmap of current index;

The described bitmap of each in message bit pattern points to one that flows in scale, whether effectively is used for identifying corresponding flow vector;

Described index is flow index, points to up-to-date active stream;

Described flow vector comprises address information, needed lock when age information and the current flow vector of access;

The next data of expecting in the current flow vector of described address information recording; How long not accessed described age information records current flow vector has; During the current flow vector of described access, needed lock is in order to realize mutual exclusion when certain stream information is accessed, upgraded to a plurality of processes or thread simultaneously.