CN110209350A - It is a kind of to mix in storage architecture HPC system using the dynamic dispatching method of I/O request - Google Patents
It is a kind of to mix in storage architecture HPC system using the dynamic dispatching method of I/O request Download PDFInfo
- Publication number
- CN110209350A CN110209350A CN201910386909.8A CN201910386909A CN110209350A CN 110209350 A CN110209350 A CN 110209350A CN 201910386909 A CN201910386909 A CN 201910386909A CN 110209350 A CN110209350 A CN 110209350A
- Authority
- CN
- China
- Prior art keywords
- application
- request
- hdd
- random
- ssd
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0685—Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses the dynamic dispatching methods in a kind of mixing storage architecture HPC system using I/O request, belong to High Performance Computing field.The present invention indicates the access module feature of application using random degree, dynamic dispatching I/O request, the SSD insensitive to random degree is written in the random more application of selection, the HDD sensitive to random degree is written in the random lesser application of degree, HDD handles the request of continuous mode as far as possible, SSD handles the request of stochastic model as far as possible, to mitigate I/O interference problem.It is distributed metastable phenomenon according to the I/O resource contention Time Bandwidth of discovery, is proposed for the calculation method of bandwidth allocation, can be used for predicting storing the dynamic load of equipment.The random degree of connected applications and can two parameters of obtainable bandwidth in storage equipment, utilize application access mode characteristic and different storage device load characteristic, rational management I/O request, preceding completion is run between ensureing application at runtime, system performance is improved simultaneously, to ensure application service quality.
Description
Technical field
The invention belongs to High Performance Computing field, more particularly, to being answered in a kind of mixing storage architecture HPC system
The dynamic dispatching method requested with I/O.
Background technique
Multi-tenant multi-load scene is in cloud HPC (High Performance Computing, high-performance calculation) system
It is increasingly common, it means that the storage resource in system is shared between more and more different users.So in object
It manages in the limited situation of resource, is most important using service quality guarantee is provided with tenant.Service quality guarantee refers to protecting
Card application performance be able to maintain that one reach an agreement on reasonable threshold value within the scope of, no matter the access request of other application arrival speed
How rate, the random degree of request and succession and the read-write ratio of request etc. change, and all do not interfere with another application
Service performance.The also scale calculated constantly expands, and needs more and more storage resources, while limited storage resource needs
Serve more applications.When it is multiple application simultaneously access storage service when, they will compete I/O resource, this will lead to I/O gather
The wide degradation of crossed belt.These have the I/O request of different access mode to mix, and current main storage is situated between
Matter --- mechanical hard disk (Hard Disk Drive, HDD) can effectively handle the request of continuous mode, but random in processing
When the request of mode, I/O can be caused to interfere because frequent positioning leads to serious magnetic head tracking expense.
In order to solve the serious degradation problem of random request bring, many solutions are suggested, wherein using new
Type storage medium becomes preferable selection.Since solid state hard disk (Solid State Drive, SSD) is had excellent performance and to random
Insensitive characteristic is requested, and as resident storage, every GB cost is far below DRAM, therefore SSD is passed through frequently as most common
High-speed processing apparatus.And in HPC system, due to there is great storage demand, and the price of SSD is relatively high, so with
SSD replaces HDD to be still unpractical completely.Therefore the mixing storage architecture of SSD and HDD is just at a preferable selection.
However, in current mixing storage architecture, mostly application competition I/O still very original for the scheduling of application I/O
Resource can still occur I/O interference, cause application service quality that cannot ensure, performance of storage system it is low.
Summary of the invention
In view of the drawbacks of the prior art, it is an object of the invention to solve the I/O scheduling of prior art mixing storage architecture
The technical problem that application service quality cannot ensure, performance of storage system is low.
To achieve the above object, in a first aspect, the embodiment of the invention provides answer in a kind of mixing storage architecture HPC system
The dynamic dispatching method requested with I/O, method includes the following steps:
S1. all I/O request for the file system layer for reaching HPC system is grouped, the request from same application
It is assigned to same group；
S2. judge whether N=1 is true, if so, this is write direct HDD using corresponding request；Otherwise, judge 1 < N≤
N0It is whether true, if so, calculating the random degree of each application and sorting according to random degree, by the smallest application of random degree
HDD is written in corresponding request, and SSD is written in the corresponding request of others application；Otherwise, it calculates the random degree of each application and presses
It sorts according to random degree, SSD is written into the corresponding request of random more M application, remaining N-M application is corresponding
Request write-in HDD；
S3. if there is operation is added in the application newly to arrive, calculate and compare the B of the applicationi-SAnd Bi-HDIf Bi-SSD≤
Bi-HDD, HDD is written into using corresponding request in this, if Bi-SSD> Bi-HDD, further compare the application and operate in HDD
The random maximum application of degree, by the corresponding request write-in SSD of degree the greater random in the two, by degree random in the two compared with
HDD is written in the corresponding request of small person；
Wherein, Bi-SSD、Bi-HDDRespectively application operate in SSD may distribute acquisition bandwidth, using operate in HDD can
The bandwidth obtained, M B can be distributedi-SSDStart to be less than Bi-HDDCritical value, N be parallel number of applications, N0For parallel application
Quantity preset threshold.
Specifically, the calculation method of the random degree of application is as follows: being ranked up to the offset of I/O request；If I/O is asked
Offset is asked to sort the distance between latter two request equal to request size, it is believed that the two requests are continuously that random factor is then
Value is 0, otherwise it is assumed that the two requests are random requests, then value is 1 to random factor；Random journey is calculated with following formula
Spend R:
Wherein, R_Factor is random factor, and K is this using the total distance number in corresponding all requests.
Specifically, parallel number of applications preset threshold N0Value range is [3,10].
Specifically, concurrently application may distribute the bandwidth B of acquisitioniCalculation formula is as follows:
Wherein, BiIndicate that i-th of application operates in storage equipment and may distribute the bandwidth of acquisition, B is indicated under storage equipment
Aggregate bandwidth, PiIndicate the proportionality coefficient of i-th of application request size, QiIndicate i-th of application request size, n indicates parallel
Number of applications.
Specifically, the aggregate bandwidth B under HDDHDDCalculation formula is as follows:
Wherein, BpeakFor the peak bandwidth of HDD under no I/O disturbed condition,To operate in HDD using random degree R's
Mean value.
Specifically, the aggregate bandwidth B under SSDSSDFor the peak bandwidth of solid state hard disk.
Second aspect, the embodiment of the invention provides a kind of computer readable storage medium, the computer-readable storage mediums
Computer program is stored in matter, which realizes that mixing described in above-mentioned first aspect stores when being executed by processor
Using the dynamic dispatching method of I/O request in framework HPC system.
In general, through the invention it is contemplated above technical scheme is compared with the prior art, have below beneficial to effect
Fruit:
1. the present invention indicates the access module feature of application using random degree, convenient for combining storage device characteristics, dynamic
Dispatch I/O request.The SSD insensitive to random degree is written in the random more application of selection, and random degree is lesser to answer
With the HDD sensitive to random degree is written, HDD handles the request of continuous mode as far as possible, and SSD handles stochastic model as far as possible
Request, to alleviate I/O interference problem.
2. the present invention is distributed metastable phenomenon according to the I/O resource contention Time Bandwidth of discovery, it is proposed for bandwidth point
The calculation method matched can be used for predicting storing the dynamic load of equipment, and in order to dynamic sensing load, it is suitable to facilitate selection
Store equipment scheduling I/O request.
3. the random degree of connected applications of the present invention and in storage equipment can two parameters of obtainable bandwidth, using answering
With access module characteristic and different storage device load characteristic, rational management I/O request ensures preceding completion between application at runtime
Operation, while system performance is improved, to ensure application service quality.
Detailed description of the invention
Fig. 1 is the dynamic dispatching in a kind of mixing storage architecture HPC system provided in an embodiment of the present invention using I/O request
Method flow diagram；
Fig. 2 is the dynamic dispatching schematic diagram that the embodiment of the present invention one provides；
Fig. 3 is dynamic dispatching schematic diagram provided by Embodiment 2 of the present invention；
Fig. 4 is the dynamic dispatching schematic diagram that the embodiment of the present invention three provides；
Fig. 5 (a) is the experimental result schematic diagram of IOR1 provided in an embodiment of the present invention；
Fig. 5 (b) is the experimental result schematic diagram of IOR2 provided in an embodiment of the present invention.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawings and embodiments, right
The present invention is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and
It is not used in the restriction present invention.
As shown in Figure 1, using the dynamic dispatching method of I/O request, this method packet in a kind of mixing storage architecture HPC system
Include following steps:
S1. all I/O request for the file system layer for reaching HPC system is grouped, the request from same application
It is assigned to same group；
S2. judge whether N=1 is true, if so, this is write direct HDD using corresponding request；Otherwise, judge 1 < N≤
N0It is whether true, if so, calculating the random degree of each application and sorting according to random degree, by the smallest application of random degree
HDD is written in corresponding request, and SSD is written in the corresponding request of others application；Otherwise, it calculates the random degree of each application and presses
It sorts according to random degree, SSD is written into the corresponding request of random more M application, remaining N-M application is corresponding
Request write-in HDD；
S3. if there is operation is added in the application newly to arrive, calculate and compare the B of the applicationi-SSDAnd Bi-HDIf Bi-SSD≤
Bi-HD, HDD is written into using corresponding request in this, if Bi-SSD> Bi-HDD, further compare the application and operate in HDD with
SSD is written in the corresponding request of degree the greater random in the two by the maximum application of machine degree, and degree random in the two is smaller
HDD is written in the corresponding request of person；
Wherein, Bi-SSD、Bi-HDDRespectively application operate in SSD may distribute acquisition bandwidth, using operate in HDD can
The bandwidth obtained, M B can be distributedi-SSDStart to be less than Bi-HCritical value, N be parallel number of applications, N0For parallel application
Quantity preset threshold.
Step S1. is grouped all I/O request for the file system layer for reaching HPC system, from same application
Request is assigned to same group.
One application may correspond to multiple requests, set to guarantee that the request of the same application is written into same storage
It is standby, all I/O request for the file system layer for reaching HPC system is grouped, the request from same application is assigned to together
One group.
Step S2. judges whether N=1 is true, if so, this is write direct HDD using corresponding request；Otherwise, judge 1
<N≤N0It is whether true, if so, calculate the random degree of each application and sort according to random degree, random degree is the smallest
HDD is written using corresponding request, SSD is written in the corresponding request of others application；Otherwise, the random degree of each application is calculated
And sort according to random degree, SSD, remaining N-M application pair is written into the corresponding request of random more M application
The request write-in HDD answered.
HDD is that random I/O request is very sensitive for access module, may result in frequent magnetic head tracking, thus
Performance is caused to decline.SSD is insensitive for stochastic model, can efficiently handle these requests.So using random degree table
The feature of existing application access mode, so that the suitable storage equipment of selection is respectively written into these requests.
According to it is initial when Parallel application quantity N it is different, be divided into the dynamic dispatching that three kinds of situations carry out I/O request.
This is write direct HDD using corresponding request by situation 1:N=1.
Situation 2:1 < N≤N0, it calculates the random degree of each application and sorts according to random degree, random degree is minimum
Using corresponding request be written HDD, others application corresponding request write-in SSD.
Situation 3:N > N0, it calculates the random degree of each application and sorts according to random degree, it will random more M
A to apply corresponding request that SSD is written, HDD, M B is written in the corresponding request of remaining N-M applicationi-SSDStart to be less than Bi-HDD
Critical value, Bi-SSD、Bi-HDDRespectively application operates in SSD and may distribute the bandwidth of acquisition, may divide using operating in HDD
Bandwidth with acquisition.
At this time if operation M are applied in SSD, N-M are applied in HDD, Bi-SSDJust start to be less than Bi-HDD, then M is to face
Dividing value.N0Value range be [3,10], preferably 5.
The calculation method of the random degree of application is as follows: firstly, being ranked up to the offset of I/O request.Random factor
(Random Factor) depends on I/O request and deviates the distance between latter two request of sorting.If it is big that distance is equal to request
It is small, it is believed that the two requests are that continuously, then value is 0 to random factor.Otherwise it is assumed that the two requests are random requests, with
Then value is 1 to the machine factor.Finally, random degree can be calculated with following formula.
Wherein, R_Factor is random factor, and K is this using the total distance number in corresponding all requests.
Operation is added if there is the application newly to arrive in step S3., calculates and compare the B of the applicationi-SSAnd Bi-HDDIf Bi-SS
≤Bi-HDD, HDD is written into using corresponding request in this, if Bi-SSD> Bi-HDD, further compare the application and operate in HDD
The maximum application of random degree, SSD is written into the corresponding request of degree the greater random in the two, by degree random in the two
HDD is written in the corresponding request of smaller.
Many experiments are run, reality is combined in different storage device to different access mode, applying for size of request
It tests, analyzes experimental result.It is learnt according to the bandwidth distribution observation concurrently run to mostly application, when I/O request passes through file system
When layer, the service that they are obtained will comply with relatively-stationary distribution, and the rule distribution almost divided equally is presented in bandwidth allocation.This meaning
Within the given duration, can be evaluated whether to obtain the processing capacity that equipment is stored in competitive environment, i.e. processing is how many special
Fixed I/O request.
Using this phenomenon, the present invention proposes a kind of calculate concurrently using the bandwidth B that may distribute acquisitioni, for characterizing
The loading condition of equipment is stored in the process of running.
Wherein, BiIndicate that i-th of application operates in storage equipment and may distribute the bandwidth of acquisition, B is indicated under storage equipment
Aggregate bandwidth, PiIndicate the proportionality coefficient of i-th of application request size, QiIndicate i-th of application request size, n indicates parallel
Number of applications.According to the bandwidth distribution almost divided equally, PiValue is 1.QiValue 64K, 128K, 256K etc..
For example, one is 64k, and one is 128k, then requesting size is answering for 64k if there are two application while running
The bandwidth of 1/3B is obtained with request, request size is that the application request of 128k obtains the bandwidth of 2/3B.
It is found according to experiment, in HDD, the random degree of I/O bandwidth and application request is in the relationship of approximately linear, with
The increase of the random degree of application request, I/O bandwidth are gradually reduced.So can according to mechanical hard disk mixing storage architecture in,
The random degree of peak bandwidth and application request without HDD under I/O disturbed condition, calculates in mechanical hard disk in the storage architecture
Under concurrently run using total bandwidth B obtainedHDD, so as to further calculate out while operating in an application in HDD
The bandwidth B obtained may be distributedi-HDD。
It is as follows to calculate HDD aggregate bandwidth formula:
Wherein, BpeakFor the peak bandwidth of HDD under no I/O disturbed condition,To operate in HDD using random degree
Mean value.
It is insensitive for random request due to solid-state hard disk SSD self-characteristic (almost 0 access delay), it is believed that mixed
Close the aggregate bandwidth B of solid state hard disk under storage architectureSSDIt is same so as to further calculate out for the peak bandwidth of solid state hard disk
When operate in an application in SSD and may distribute the bandwidth B of acquisitioni-SSD。
As shown in Fig. 2, only one in embodiment one applies isolated operation in mixing storage architecture HPC system, and do not have
There is other application while running, using not interfered by I/O, the end of run time of the application can be met by write-in HDD,
So HDD is write direct in selection when one application of operation.
As shown in figure 3, in embodiment two two application simultaneously run, calculate first the two apply random degree, so
Compare the random degree of the two applications afterwards.The random lesser application of degree is written into HDD, another is then written into SSD.
There is N number of application to start simultaneously at operation in embodiment three, calculates application first and operate in SSD and HDD and may distribute
Obtain bandwidth Bi-SSDAnd Bi-HDD；Then a critical value is found, at this time if operation M are applied in SSD, N-M are applied
HDD, Bi-SSDJust start to be less than Bi-HDD；Secondly, calculating the random degree of N number of application, and it is ranked up, M is a random before selecting
More application operates in SSD, and remaining N-M application then operates in HDD.As shown in figure 4, original N number of application
It is currently running, continuous at the same time to have new application that operation is added again, calculating a apply of N+1 first may in SSD and HDD
The bandwidth B being assigned toi-SAnd Bi-HDDIf Bi-SSDLess than Bi-HDD, illustrate that being not suitable for again more operations N+1 in SSD at this time answers
With, if selection operates in SSD, considerable bandwidth will not be not only obtained, the application operated in SSD can be also tied down,
So the N+1 application is write direct in HDD；If Bi-SSDGreater than Bi-HDD, the random degree of the N+1 application is calculated, than
Compared with the random degree of the maximum application of random degree run in the N+1 application and HDD, if the N+1 apply it is random
Degree in HDD less than or equal to what is applied, then the application originally run in HDD will be redirected to SSD, and N+1 exclusive
HDD, on the contrary it will be in the N+1 application write-in SSD.
Mean that limited physical resource is shared by more and more users in cloud HPC environment, the present invention does not chase after blindly
Peak performance is sought, but distributes I/O resource according to application demand to ensure the service quality of application, similar to pressing in cloud computing
It needs to service.The present invention defines the demand of application using the end of run time, and the end of run time refers to applying and individually transport
Runing time when row is interference-free.Each application has the end of run time, when implemented between be more than end of run when, just very
It is difficult to guarantee the service quality of application.So at its best, each application can oneself the end of run time it
Before be finished.Meanwhile application the end of run time and handling capacity can be used as the judgment basis of service quality guarantee.Cause
This, the invention proposes a kind of Dynamic Scheduling Strategies, and being able to satisfy for each application is ensured using the characteristic of different storage device
The end of run time, to improve bandwidth availability ratio and provide high I/O performance.
The present invention carries out being configured with 10 nodes in the cluster of confirmatory experiment, wherein and 8 nodes are as calculate node, and 2
Node is as I/O node, wherein the memory size of calculate node is configured to 64GB, and configures the mechanical disk of 300GB, I/O section
The memory size of point is configured to 8GB, and configures the solid state hard disk of 300GB, is connected to the network between node using Infiniband, remaining
It configures identical.In each calculate node, the MPICH compiled by ROMIO is mounted with to realize Parallel I/O, and DDL-QoS
Realization in used MPI-IO to handle MPICH, while OrangeFS (original file system) is mounted on two I/O nodes
With DDL-QoS (being configured with dynamic dispatching method proposed by the present invention).In an experiment, main benchmark to be used is IOR, this is main
Flow a kind of parallel file system commonly testing tool.Experiment simultaneously operation two IOR examples, according between two examples when
Between be spaced different, carried out five groups of experiments respectively.Time interval between example is 0,9,18,27,36 second.Each IOR example
The data of 8GB are written, each I/O request size is 512KB, and default process number is 32.
Compare Fig. 5 (a) and Fig. 5 (b) it is found that the performance of IOR1 improves 2.5 in DDL-QoS when time interval is 0s
Times, the performance of IOR2 improves 3.79 times.This is because the I/O between two IOR examples interferes journey when time interval is 0
Degree is maximum, leads to performance degradation.The DDL-QoS that the present invention designs can analyze the access module of each application program,
IOR1 is the access module of Segmented-Contiguous a kind of, and good performance still can be obtained on HDD, so
IOR1 is still written into HDD.When IOR2 is in the access module of Segmented-Random, need to be written into asking at random
Insensitive SSD is sought, to obtain maximum performance improvement.
When the time, which is spaced apart, begins to increase, the IOR1 of first operation can during this period of time monopolize the I/O money of server
Source.When parallel operation is added in IOR2, it starts to generate I/O interference.Therefore, in original OrangeFS, the performance of IOR1
It is gradually increased with the increase of time interval.In experiment of the invention, when time interval is 36 seconds, in OrangeFS
The performance of IOR1 maximizes, this is identical as the performance in DDL-QoS, because absolutely not I/O is interfered IOR1 during operation.
In IOR2, time interval is bigger, and the I/O interference generated between IOR1 and IOR2 is smaller.Equally, with the increasing of time interval
Add, the performance of IOR2 can also improve in OrangeFS.However, in DDL-QoS, the performance of IOR2 with time interval increase
And it reduces.This is because needing to be written SSD can be reduced to avoid the part that I/O is interfered with the reduction of I/O annoyance level.It writes
The data volume for entering HDD also increased, finally suitable with the performance for being separately written HDD.
More than, the only preferable specific embodiment of the application, but the protection scope of the application is not limited thereto, and it is any
Within the technical scope of the present application, any changes or substitutions that can be easily thought of by those familiar with the art, all answers
Cover within the scope of protection of this application.Therefore, the protection scope of the application should be subject to the protection scope in claims.
Claims (7)
1. using the dynamic dispatching method of I/O request in a kind of mixing storage architecture HPC system, which is characterized in that this method packet
Include following steps:
S1. all I/O request for the file system layer for reaching HPC system is grouped, the request from same application is divided
To same group；
S2. judge whether N=1 is true, if so, this is write direct HDD using corresponding request；Otherwise, judge 1 < N≤N0It is
No establishment, if so, calculate the random degree of each application and sort according to random degree, the smallest application of random degree is corresponding
Request be written HDD, others application corresponding request write-in SSD；Otherwise, calculate the random degree of each application and according to
SSD is written in the corresponding request of random more M application by the sequence of machine degree, and remaining N-M is applied corresponding request
HDD is written；
S3. if there is operation is added in the application newly to arrive, calculate and compare the B of the applicationi-SSDAnd Bi-HDDIf Bi-SSD≤Bi-HDD,
HDD is written into using corresponding request in this, if Bi-SSD> Bi-HDD, further compare the application and operate in the random journey in HDD
Maximum application is spent, SSD is written into the corresponding request of degree the greater random in the two, by degree smaller couple random in the two
The request write-in HDD answered；
Wherein, Bi-SSD、Bi-HDDRespectively application operates in SSD and may distribute the bandwidth of acquisition, may distribute using operating in HDD
The bandwidth of acquisition, M Bi-SSDStart to be less than Bi-HDDCritical value, N be parallel number of applications, N0It is pre- for parallel number of applications
If threshold value.
2. dynamic dispatching method as described in claim 1, which is characterized in that the calculation method of the random degree of application is as follows:
The offset of I/O request is ranked up；If the distance between latter two request of I/O request offset sequence is equal to request size,
Think the two requests be it is continuous, then value is 0 to random factor, otherwise it is assumed that the two requests are random requests, it is random because
Sub then value is 1；Random degree R is calculated with following formula:
Wherein, R_Factor is random factor, and K is this using the total distance number in corresponding all requests.
3. dynamic dispatching method as claimed in claim 1 or 2, which is characterized in that parallel number of applications preset threshold N0Value
Range is [3,10].
4. dynamic dispatching method as described in any one of claims 1 to 3, which is characterized in that concurrently application may distribute acquisition
Bandwidth BiCalculation formula is as follows:
Wherein, BiIndicate that i-th of application operates in storage equipment and may distribute the bandwidth of acquisition, B indicates the polymerization under storage equipment
Bandwidth, PiIndicate the proportionality coefficient of i-th of application request size, QiIndicate i-th of application request size, n indicates Parallel application number
Amount.
5. dynamic dispatching method as claimed in claim 4, which is characterized in that the aggregate bandwidth B under HDDHDDCalculation formula is such as
Under:
Wherein, BpeakFor the peak bandwidth of HDD under no I/O disturbed condition,To operate in HDD using the equal of random degree R
Value.
6. dynamic dispatching method as claimed in claim 4, which is characterized in that the aggregate bandwidth B under SSDSSDFor solid state hard disk
Peak bandwidth.
7. a kind of computer readable storage medium, which is characterized in that be stored with computer on the computer readable storage medium
Program realizes such as mixing storage architecture as claimed in any one of claims 1 to 6 when the computer program is executed by processor
Using the dynamic dispatching method of I/O request in HPC system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910386909.8A CN110209350B (en) | 2019-05-10 | 2019-05-10 | Dynamic scheduling method for application I/O (input/output) request in HPC (high performance computing) system of hybrid storage architecture |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910386909.8A CN110209350B (en) | 2019-05-10 | 2019-05-10 | Dynamic scheduling method for application I/O (input/output) request in HPC (high performance computing) system of hybrid storage architecture |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110209350A true CN110209350A (en) | 2019-09-06 |
| CN110209350B CN110209350B (en) | 2020-07-10 |
Family
ID=67785638
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910386909.8A Active CN110209350B (en) | 2019-05-10 | 2019-05-10 | Dynamic scheduling method for application I/O (input/output) request in HPC (high performance computing) system of hybrid storage architecture |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110209350B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111679813A (en) * | 2020-08-11 | 2020-09-18 | 南京云联数科科技有限公司 | Method for information processing, electronic device, and storage medium |
| CN114691698A (en) * | 2022-04-24 | 2022-07-01 | 北京梦蓝杉科技有限公司 | Data processing system and method for computer system |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130159597A1 (en) * | 2011-12-19 | 2013-06-20 | Electronics And Telecommunications Research Institute | Hybrid storage device and method of operating the same |
| CN104216988A (en) * | 2014-09-04 | 2014-12-17 | 天津大学 | SSD (Solid State Disk) and HDD(Hard Driver Disk)hybrid storage method for distributed big data |
| CN104376094A (en) * | 2014-11-24 | 2015-02-25 | 浪潮电子信息产业股份有限公司 | File hierarchical storage method and system taking visit randomness into consideration |
| US20160179683A1 (en) * | 2014-12-23 | 2016-06-23 | Prophetstor Data Services, Inc. | Ssd caching system for hybrid storage |
| US20160299697A1 (en) * | 2015-04-08 | 2016-10-13 | Prophetstor Data Services, Inc. | Workload-aware i/o scheduler in software-defined hybrid storage system |
| CN106469193A (en) * | 2016-08-30 | 2017-03-01 | 北京航空航天大学 | Multi load metadata I/O service quality performance support method and system |
| US20170060472A1 (en) * | 2015-08-31 | 2017-03-02 | Linkedin Corporation | Transparent hybrid data storage |
| CN107015763A (en) * | 2017-03-03 | 2017-08-04 | 北京中存超为科技有限公司 | Mix SSD management methods and device in storage system |
| CN107193487A (en) * | 2017-04-26 | 2017-09-22 | 华中科技大学 | A kind of random access recognition methods and system |
| CN107844269A (en) * | 2017-10-17 | 2018-03-27 | 华中科技大学 | A kind of layering mixing storage system and method based on uniformity Hash |
| US20180307413A1 (en) * | 2013-11-27 | 2018-10-25 | Alibaba Group Holding Limited | Control of storage of data in a hybrid storage system |
| CN109144411A (en) * | 2018-07-24 | 2019-01-04 | 中国电子科技集团公司第三十八研究所 | Data center's hybrid magnetic disc array and its data dynamic migration strategy |
| CN109240611A (en) * | 2018-08-28 | 2019-01-18 | 郑州云海信息技术有限公司 | The cold and hot data hierarchy method of small documents, small documents data access method and its device |
| CN109697033A (en) * | 2018-12-19 | 2019-04-30 | 中国人民解放军国防科技大学 | Tile record disk sensing storage caching method and system |
-
2019
- 2019-05-10 CN CN201910386909.8A patent/CN110209350B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130159597A1 (en) * | 2011-12-19 | 2013-06-20 | Electronics And Telecommunications Research Institute | Hybrid storage device and method of operating the same |
| US20180307413A1 (en) * | 2013-11-27 | 2018-10-25 | Alibaba Group Holding Limited | Control of storage of data in a hybrid storage system |
| CN104216988A (en) * | 2014-09-04 | 2014-12-17 | 天津大学 | SSD (Solid State Disk) and HDD(Hard Driver Disk)hybrid storage method for distributed big data |
| CN104376094A (en) * | 2014-11-24 | 2015-02-25 | 浪潮电子信息产业股份有限公司 | File hierarchical storage method and system taking visit randomness into consideration |
| US20160179683A1 (en) * | 2014-12-23 | 2016-06-23 | Prophetstor Data Services, Inc. | Ssd caching system for hybrid storage |
| US20160299697A1 (en) * | 2015-04-08 | 2016-10-13 | Prophetstor Data Services, Inc. | Workload-aware i/o scheduler in software-defined hybrid storage system |
| US20170060472A1 (en) * | 2015-08-31 | 2017-03-02 | Linkedin Corporation | Transparent hybrid data storage |
| CN106469193A (en) * | 2016-08-30 | 2017-03-01 | 北京航空航天大学 | Multi load metadata I/O service quality performance support method and system |
| CN107015763A (en) * | 2017-03-03 | 2017-08-04 | 北京中存超为科技有限公司 | Mix SSD management methods and device in storage system |
| CN107193487A (en) * | 2017-04-26 | 2017-09-22 | 华中科技大学 | A kind of random access recognition methods and system |
| CN107844269A (en) * | 2017-10-17 | 2018-03-27 | 华中科技大学 | A kind of layering mixing storage system and method based on uniformity Hash |
| CN109144411A (en) * | 2018-07-24 | 2019-01-04 | 中国电子科技集团公司第三十八研究所 | Data center's hybrid magnetic disc array and its data dynamic migration strategy |
| CN109240611A (en) * | 2018-08-28 | 2019-01-18 | 郑州云海信息技术有限公司 | The cold and hot data hierarchy method of small documents, small documents data access method and its device |
| CN109697033A (en) * | 2018-12-19 | 2019-04-30 | 中国人民解放军国防科技大学 | Tile record disk sensing storage caching method and system |
Non-Patent Citations (3)
| Title |
|---|
| MING LI等: "《SSDUP:An Efficient SSD Write Buffer Using Pipeline》", 《2016 IEEE INTERNATIONAL CONFERENCE ON CLUSTER COMPUTING》 * |
| SHI XUANHUA: "《https://dl.acm.org/doi/pdf/10.1145/3079079.3079087?download=true》", 16 June 2017 * |
| 祝青: "混合存储系统研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111679813A (en) * | 2020-08-11 | 2020-09-18 | 南京云联数科科技有限公司 | Method for information processing, electronic device, and storage medium |
| CN111679813B (en) * | 2020-08-11 | 2020-11-06 | 南京云联数科科技有限公司 | Method for information processing, electronic device, and storage medium |
| CN114691698A (en) * | 2022-04-24 | 2022-07-01 | 北京梦蓝杉科技有限公司 | Data processing system and method for computer system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110209350B (en) | 2020-07-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105205014B (en) | A kind of date storage method and device | |
| US11556541B2 (en) | Data query method, apparatus and device | |
| CN107124472A (en) | Load-balancing method and device, computer-readable recording medium | |
| US7418568B2 (en) | Memory management technique | |
| US11093410B2 (en) | Cache management method, storage system and computer program product | |
| CN106406987A (en) | Task execution method and apparatus in cluster | |
| CN103327072A (en) | Method for cluster load balance and system thereof | |
| CN107526542A (en) | Object storage device and its operating method | |
| US20190332328A1 (en) | Storage Controller and IO Request Processing Method | |
| CN106569892A (en) | Resource scheduling method and device | |
| CN104102646B (en) | The method, apparatus and system of data processing | |
| CN110209350A (en) | It is a kind of to mix in storage architecture HPC system using the dynamic dispatching method of I/O request | |
| CN110636388A (en) | Service request distribution method, system, electronic equipment and storage medium | |
| WO2021259246A1 (en) | Resource scheduling method and apparatus, electronic device, and computer-readable storage medium | |
| WO2021258512A1 (en) | Data aggregation processing apparatus and method, and storage medium | |
| US20190149478A1 (en) | Systems and methods for allocating shared resources in multi-tenant environments | |
| CN109756429A (en) | Bandwidth allocation methods and equipment | |
| CN109788013B (en) | Method, device and equipment for distributing operation resources in distributed system | |
| US11194619B2 (en) | Information processing system and non-transitory computer readable medium storing program for multitenant service | |
| CN107800744A (en) | A kind of service request retransmission method, apparatus and system | |
| WO2018205890A1 (en) | Task assignment method and system of distributed system, computer readable storage medium and computer device therefor | |
| Gao et al. | A load-aware data migration scheme for distributed surveillance video processing with hybrid storage architecture | |
| CN112667392B (en) | Cloud computing resource allocation method and device, computer equipment and storage medium | |
| US20130151794A1 (en) | Memory controller and memory control method | |
| US9894670B1 (en) | Implementing adaptive resource allocation for network devices |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |