CN108900626A - Date storage method, apparatus and system under a kind of cloud environment - Google Patents
Date storage method, apparatus and system under a kind of cloud environment Download PDFInfo
- Publication number
- CN108900626A CN108900626A CN201810792783.XA CN201810792783A CN108900626A CN 108900626 A CN108900626 A CN 108900626A CN 201810792783 A CN201810792783 A CN 201810792783A CN 108900626 A CN108900626 A CN 108900626A
- Authority
- CN
- China
- Prior art keywords
- back end
- load factor
- memory space
- data
- residual memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
Abstract
This application discloses date storage methods under a kind of cloud environment, including：Receive the request for the application storing data that client is sent；According to the request of application storing data from the data of the residual memory spaces of all back end and I/O load factor, the residual memory space and the first I/O load factor of i-th of back end are inquired；I-th of back end is the back end that residual memory space in the residual memory space of all back end comes i-th bit；The residual memory space of i-th of back end is back to the client, so that data block is sent to i-th of back end according to the residual memory space of i-th of back end by client；Compare whether the first I/O load factor is less than given threshold；If the first I/O load factor is less than given threshold, the data block that store instruction sends for i-th of back end storage client is sent to i-th of back end.The residual memory space and I/O load factor for considering back end simultaneously in client storage file, promote the performance of storage system.
Description
Technical field
The invention belongs to cloud storage fields, and in particular to date storage method, apparatus and system under a kind of cloud environment.
Background technique
In recent years, as the infrastructures such as processor and network obtain rapid development, data volume to be treated is fast
While speed increases, huge challenge also proposed to the performance of mass storage system (MSS).In current field of storage, in particular for
The change of the virtualization of trend of the times, memory technology can be described with " overturning ", wherein most important is exactly distributed storage
The introduction of technology.
HDFS is exactly to solve mass data storage and the distributed storage technology that designs.It is counted using more replication policies
According to redundancy and load balancing, each copy be a block, the data of storage using block as unit, duplication more parts of random storages collecting
Multiple nodes of group.HDFS has the advantages such as high fault-tolerant, high extension, low cost.
The selection of HDFS memory node have randomness, this will cause the storage device processes of superior performance task it is few and
The case where low storage equipment of performance works overloadingly, influences the performance of global storage system.Data block is in cluster simultaneously
In be unevenly distributed, in load balancing, HDFS can randomly choose the data block in back end, it is possible to by access frequency
High data block focuses on a few a back end, will lead to the overload of the back end, so that storage system
It can substantially reduce.
Summary of the invention
The application provides date storage method under a kind of cloud environment, to solve the performance of storage system under existing cloud environment
Lower problem.In addition the application provides data storage device under a kind of cloud environment, data-storage system under a kind of cloud environment.
The application provides date storage method under a kind of cloud environment, including：
Receive the request for the application storing data that client is sent；
According to the request of the application storing data from the residual memory space of all back end and the number of I/O load factor
In, the residual memory space and the first I/O load factor of i-th of back end are inquired；I-th of back end is described
Residual memory space comes the back end of i-th bit in all back end；
The residual memory space of i-th of back end is back to the client, for the client according to
Data block is sent to i-th of back end by the residual memory space of i-th of back end；
Compare whether the first I/O load factor is less than given threshold；
If the first I/O load factor is less than the given threshold, store instruction is sent to i-th of back end
So that i-th of back end stores the data block that the client is sent.
Optionally, the first I/O load factor P of i-th of back endiIt is calculated by the following formula and obtains：
Pi=ω1Oi+ω2Ri+ω3ti；
Wherein OiIndicate the actual read or write speed of i-th of back end after normalizing；RiIt indicates described the after normalization
The actual handling capacity of i back end；tiI-th of back end actual waiting time is indicated after normalizing；ω1、ω2、
ω3Respectively read or write speed, handling capacity and the weighted value of waiting time, and ∑ ωi(i=1,2,3)=1.
Optionally, the actual read or write speed O of i-th of back end after the normalizationiIt is calculated by the following formula
It obtains：
WhereinIiFor the actual read-write of i-th of back end
Speed；
The actual handling capacity R of i-th of back end after the normalizationiIt is calculated by the following formula and obtains：
WhereinHiIt is actual for i-th of back end
Handling capacity；
The actual waiting time t of i-th of back end after the normalizationiIt is calculated by the following formula and obtains：
WhereinTiFor the actual waiting of i-th of back end
Time.
Optionally, the given threshold Q, which is calculated by the following formula, obtains：
Optionally, whether the first I/O load factor is less than after given threshold step, further includes：
If the first I/O load factor is greater than or equal to the given threshold, i is carried out plus 1 is handled, is continued to execute described
According to the request of the application storing data from the data of the residual memory spaces of all back end and I/O load factor, look into
The step of asking out the residual memory space and the first I/O load factor of i-th of back end.
Optionally, it if the first I/O load factor is less than the given threshold, is sent to i-th of back end
Store instruction is for further including after i-th of back end stores data block step that the client sends：
The request for obtaining the 2nd I/O load factor is sent to i-th of back end；
It receives calculated i-th of the back end of i-th of back end and stores institute after the data block
State the 2nd I/O load factor；
Compare whether the 2nd I/O load factor is less than the given threshold；
If the 2nd I/O load factor is less than the given threshold, is sent to i-th of back end and obtain described the
The request of the data of the residual memory space and I/O load factor of i back end；
Receive the residual memory space and I/O load factor of i-th of back end that i-th of back end returns
Data；
The residual memory space of i-th of back end and the data of I/O load factor are updated, the reception client is returned
The step of holding the request of the application storing data sent.
Optionally, whether the 2nd I/O load factor is less than after the given threshold step, further includes：
If the 2nd I/O load factor is greater than or equal to the given threshold, recalled to i-th of back end transmission
Instruction recalls the data block for i-th of back end；
I is carried out plus 1 is handled, continues to execute the residue according to the request for applying for storing data from all back end
In memory space and I/O load factor, the step of inquiring the residual memory space and the first I/O load factor of i-th of back end.
The application also provides data storage device under a kind of cloud environment, including：
Receiving module, the request of the application storing data for receiving client transmission；
Enquiry module, for according to the request of the application storing data from the residual memory spaces of all back end and
In the data of I/O load factor, the residual memory space and the first I/O load factor of i-th of back end are inquired；Described i-th
Back end is the back end that residual memory space comes i-th bit in all back end；
Sending module, for the residual memory space of i-th of back end to be back to the client, for institute
It states client and data block is sent to by i-th of back end according to the residual memory space of i-th of back end；
Whether the first comparison module is less than given threshold for the first I/O load factor；
Memory module, if being less than the given threshold for the first I/O load factor, to i-th of data section
Point sends the store instruction data block that the client is sent so that i-th of back end stores.
Optionally, the enquiry module, if being also used to the first I/O load factor is greater than or equal to the given threshold,
Then i is carried out plus 1 processing, is continued to run in the enquiry module according to the request of the application storing data from all data sections
In the residual memory space and I/O load factor of point, the residual memory space and the first I/O load of i-th of back end are inquired
The part of rate.
Optionally, described device further includes：Second comparison module and update module；
Whether second comparison module is less than the given threshold for the 2nd I/O load factor；
The update module, for updating the residual memory space of i-th of back end and the number of I/O load factor
According to, return it is described receive client send application storing data request the step of；
The sending module is also used to send the request for obtaining the 2nd I/O load factor to i-th of back end；If
The 2nd I/O load factor is less than the given threshold, sends to i-th of back end and obtains i-th of data section
The request of the data of the residual memory space and I/O load factor of point；
The receiving module is also used to receive calculated i-th of back end storage of i-th of back end
The 2nd I/O load factor after the data block；Receive i-th of data section that i-th of back end returns
The residual memory space of point and the data of I/O load factor.
Optionally, described device further includes：Recall module；
It is described to recall module, if being greater than or equal to the given threshold for the 2nd I/O load factor, to described i-th
A back end transmission recalls instruction so that i-th of back end recalls the data block；
The enquiry module, is also used to carry out i plus 1 is handled, and continues to run in the enquiry module according to the application
The request of storing data inquires i-th of back end from the residual memory space and I/O load factor of all back end
The part of residual memory space and the first I/O load factor.
The application also provides data-storage system under a kind of cloud environment, including：The clothes of metadata described in claim 8-11
Business device, client and back end；
The client, for sending the request of application storing data to the meta data server；Receive first number
According to the residual memory space for i-th of back end that server is sent；According to the residual memory space of i-th of back end
Data block is sent to i-th of back end；
The back end, the data block sent for receiving the client；The meta data server is received to send
Store instruction store the data block that the client is sent.
The embodiment of the present application is directed to existing distributed storage technology under cloud environment and improves, and makes the selection of back end
With specific aim, make to store that equipment task distribution is balanced, be not in the storage device processes of superior performance task is few and property
The case where storage equipment that can be low works overloadingly selects the back end storage that back end remaining space is big, is promoted whole
The performance of storage system.The load factor for controlling each back end simultaneously, is no more than the load factor of different back end
Given threshold is not in the excessively high situation of a few a back end I/O load factors, reduces the file access time, improves storage
The operational efficiency of system.
Detailed description of the invention
Fig. 1 is the flow chart of date storage method under a kind of cloud environment that the application first embodiment provides；
Fig. 2 is the flow chart of date storage method under a kind of cloud environment that the application second embodiment provides；
Fig. 3 is the structural schematic diagram of data storage device under a kind of cloud environment that the application 3rd embodiment provides；
Fig. 4 is the structural schematic diagram of data storage device under a kind of cloud environment that the application fourth embodiment provides；
Fig. 5 is the structural schematic diagram of data-storage system under a kind of cloud environment that the 5th embodiment of the application provides.
Specific embodiment
Technical solution in order to enable those skilled in the art to better understand the present invention, with reference to the accompanying drawing and specific embodiment party
Present invention is further described in detail for formula.
The application is completed jointly under the cooperation of client, meta data server and back end, and the application provides one
Date storage method, apparatus and system under kind cloud environment.Below in conjunction with embodiment provided by the present application attached drawing one by one into
Row is described in detail.
Date storage method is as follows under a kind of cloud environment that the application first embodiment provides：
The executing subject of the embodiment of the present application is meta data server, as shown in Figure 1, it illustrates the embodiment of the present application to mention
The flow chart of date storage method, includes the following steps under a kind of cloud environment supplied.
Step S101 receives the request for the application storing data that client is sent.
Step S102, according to the request of the application storing data from the residual memory space and I/O of all back end
In the data of load factor, the residual memory space and the first I/O load factor of i-th of back end are inquired；I-th of data
Node is the back end that residual memory space in the residual memory space of all back end comes i-th bit.
The residual memory space of i-th of back end is back to the client, for the visitor by step S103
Data block is sent to i-th of back end according to the residual memory space of i-th of back end by family end.
Step S104, whether the first I/O load factor is less than given threshold, if so, thening follow the steps S105；If
It is no, then follow the steps S106.
Step S105 is sent out if the first I/O load factor is less than the given threshold to i-th of back end
The store instruction data block that the client is sent so that i-th of back end stores is sent, process terminates.
Step S106, carries out i plus 1 is handled, and continues to execute step S102.
In current field of storage, in particular for the virtualization of trend of the times, the change of memory technology can use " top
Cover " it describes, wherein most important is exactly the introduction of distributed storage technology.And cloud storage is in cloud computing (cloud
Computing) conceptive extension and the new concept that developed, are a kind of emerging Network storage technologies, refer to logical
The functions such as cluster application, network technology or distributed file system are crossed, by various types of storage equipment a large amount of in network
Collaborative work is gathered by application software, it is common that the system of data storage and business access function is externally provided.When cloud meter
When calculation system operations and the core of processing are the storage and management of mass data, just need to configure in cloud computing system and largely deposit
Equipment is stored up, then cloud computing system is transformed into as a cloud storage system, stores and manages with data so cloud storage is one
Reason is the cloud computing system of core.In simple terms, cloud storage is exactly to be put into storage resources on cloud for a kind of emerging of people's access
Scheme.User at any time, Anywhere can be connected on cloud through any web-enabled device and easily access number
According to.
The embodiment of the present application is directed to existing distributed storage technology under cloud environment and improves, and makes the selection of back end
With specific aim, make to store that equipment task distribution is balanced, be not in the storage device processes of superior performance task is few and property
The case where storage equipment that can be low works overloadingly selects the back end storage that back end remaining space is big, is promoted whole
The performance of storage system.The load factor for controlling each back end simultaneously, is no more than the load factor of different back end
Given threshold is not in the excessively high situation of a few a back end I/O load factors, reduces the file access time, improves storage
The operational efficiency of system.
Date storage method is as follows under a kind of cloud environment that the application second embodiment provides：
The executing subject of the embodiment of the present application is meta data server, as shown in Fig. 2, it illustrates the embodiment of the present application to mention
The flow chart of date storage method, includes the following steps under a kind of cloud environment supplied.
Step S201 receives the request for the application storing data that client is sent.
Fig. 5 is please referred to, data-storage system (distributed storage architecture) is made of three parts under cloud environment：Client 2,
Meta data server 1 and back end 3 (data server).Client 2 be responsible for send read-write requests, cache file metadata and
File data.Meta data server is responsible for the acquisition and storage of load information, overall load information regulation and processing client
Request, be the core component of whole system.Cloud storage only has a kind of this server of meta data server, can store any class
The data of type.Back end 3 is responsible for storage file data, manages and maintains the load information of itself automatically, and main includes monitoring
The residual memory space and I/O load factor of itself, and in real time to meta data server feedback-related information.
This step is used to receive the request of the application storing data of client transmission, and meta data server 1 receives client
After the request for holding the application storing data sent, corresponding data are inquired according to the request of this application storing data.
Step S202, according to the request of the application storing data from the residual memory space and I/O of all back end
In the data of load factor, the residual memory space and the first I/O load factor of i-th of back end are inquired；I-th of data
Node is the back end that residual memory space in the residual memory space of all back end comes i-th bit.
It include memory space collection module, I/O information collection module, load information management module in meta data server 1.
It include memory space monitoring module, I/O load monitoring module in back end 3.It is collected in the memory space of meta data server 1
Pre-existing storage space information in module records the residual memory space and storage space utilization of all back end.It deposits
Each back end in spatial information is stored up according to the arrangement from big to small of residual memory space.When the storage in back end 3 is empty
Between monitoring module discovery current data node residual memory space variation when, memory space collection module by obtain current data
The newest storage space information of node updates the residual memory space and storage space utilization of current data node.
According to the request of the application storing data from the storage space information in the memory space collection module, look into
Ask out the residual memory space of i-th of back end.Each back end in storage space information is according to residual memory space
It arranges from big to small, i-th of back end is the residual memory space row in the residual memory space of all back end
In the back end of i-th bit, for example, i is 1, i.e., the maximum back end of residual memory space in all back end.Accordingly
, i+1 back end is that the residual memory space in the residual memory space of all back end comes i+1 position
Back end, for example, i is 1, i+1 2, i.e., residual memory space comes deputy back end in all back end.
The pre-existing I/O load information in the I/O information collection module of meta data server 1, i.e., all back end
I/O load factor.I/O information collection module periodically sends I/O information request to back end 3, the I/ in back end 3
The I/O load factor of O load monitoring module monitors current data node feeds back the newest I/O load factor of current data node
To meta data server 1, the I/O load information of current data node is updated.
According to the request of the application storing data from the I/O load information in the I/O information collection module, inquiry
First I/O load factor of i-th of back end out.Before first I/O load factor refers to that i-th of back end receives data block
I/O load factor, correspondingly, the 2nd I/O load factor refers to that i-th of back end receives the I/O load factor after data block.
This step is used for the memory space pre-existing from meta data server according to the request of the application storing data
In information and I/O load information, the residual memory space and the first I/O load factor of i-th of back end are inquired, in case carrying out
Judge in next step, thus storing data.
Preferably, the first I/O load factor P of i-th of back endiIt is calculated by the following formula and obtains：
Pi=ω1Oi+ω2Ri+ω3ti；
Wherein OiIndicate the actual read or write speed of i-th of back end after normalizing；RiIt indicates described the after normalization
The actual handling capacity of i back end；tiI-th of back end actual waiting time is indicated after normalizing；ω1、ω2、
ω3Respectively read or write speed, handling capacity and the weighted value of waiting time, and ∑ ωi(i=1,2,3)=1.
Preferably, the actual read or write speed O of i-th of back end after the normalizationiIt is calculated by the following formula
It obtains：
WhereinEqual to the arithmetic average of the actual read or write speed of n back end.S is equal to the variance of the read or write speed of i-th of back end.IiFor i-th of back end
Actual read or write speed；
The actual handling capacity R of i-th of back end after the normalizationiIt is calculated by the following formula and obtains：
WhereinEqual to the arithmetic average of the actual handling capacity of n back end.S' is equal to the variance of the handling capacity of i-th of back end.HiFor i-th of back end
Actual handling capacity；
The actual waiting time t of i-th of back end after the normalizationiIt is calculated by the following formula and obtains：
WhereinEqual to the arithmetic average of the actual waiting time of n back end.S " is equal to the variance of the waiting time of i-th of back end.TiFor i-th of data section
The point actual waiting time.
The influence factor of the I/O load factor of back end mainly includes that read or write speed per second, handling capacity and data carry out
It the waiting time etc. of read-write operation, is standardized respectively for factors above, each back end metering is unified, obtains normalizing
The actual read or write speed I of i-th of back end after changei, handling capacity Ri, waiting time Ti.So that Ii、Ri、TiValue be both less than
1, convenient for calculating.ω1、ω2、ω3Respectively read or write speed, handling capacity and the weighted value of waiting time, weighted value is bigger, shows
Influence of the influence factor to the I/O load factor of i-th of back end is bigger, ∑ ωi(i=1,2,3)=1.
Preferably, the given threshold Q, which is calculated by the following formula, obtains：
The given threshold Q of I/O load factor is preset, given threshold Q is equal to the calculation of the I/O load factor of n back end
Number average value.For judging Q and PiSize, by judging Q and PiSize, from which further follow that whether storing data.
The residual memory space of i-th of back end is back to the client, for the visitor by step S203
Data block is sent to i-th of back end according to the residual memory space of i-th of back end by family end.
After the residual memory space for inquiring i-th of back end, meta data server sends it to client, visitor
Data block is sent to i-th of back end according to the residual memory space of i-th of the back end inquired by family end.For example,
Data block is sent to the 1st back end, the i.e. maximum back end of residual memory space by client.
Step S204, whether the first I/O load factor is less than given threshold, if so, thening follow the steps S205；If
It is no, then follow the steps S213.
Step S205 sends store instruction to i-th of back end for described in i-th of back end storage
The data block that client is sent.
Client sends data block to i-th back end, at the same comparison query to the first I/O load factor whether be less than
Given threshold.If so, Pi<Q shows that the I/O load factor of i-th of back end does not reach given threshold, i-th of data
The I/O load factor of node is smaller, can store new data block.Store instruction is sent to i-th of back end for institute
It states i-th of back end and stores the data block that the client is sent.
If it is not, Pi>Q, then the data block cannot be placed on this back end, which is stored in residual memory space
The back end that next back end (i+1) in information, i.e. residual memory space are only second to i-th of back end, until
The P of some back endi<Q.I is carried out plus 1 processing, according to the request of the application storing data from all back end
In the data of residual memory space and I/O load factor, the residual memory space and the first I/O of i+1 back end are inquired
Load factor.Subsequent step is successively executed according to the step of i-th of back end based on i+1 back end, until finding one
A back end meets Pi<Until the condition of Q.
Step S206 sends the request for obtaining the 2nd I/O load factor to i-th of back end.
Step S207 receives calculated i-th of the back end of i-th of back end and stores the data block
The 2nd I/O load factor later.
After i-th of back end stores new data block, I/O load factor can be changed, in order to make each back end
I/O load reach balanced, also need to count the I/O load factor after back end storing data block, at this time i-th of number
I/O load factor according to node is P'.The I/O information collection module of meta data server 1 is sent to i-th back end obtains the
The I/O of the request of two I/O load factors, current i-th of the back end of I/O load monitoring module monitors of i-th of back end is negative
The newest 2nd I/O load factor of current i-th of back end is fed back to meta data server 1 by load rate.
Whether step S208, the 2nd I/O load factor are less than the given threshold if so, thening follow the steps
S209；If it is not, thening follow the steps S212.
Step S209, sent to i-th of back end the residual memory space for obtaining i-th of back end and
The request of the data of I/O load factor；
Step S210, receive i-th of back end that i-th of back end returns residual memory space and
The data of I/O load factor；
Step S211 updates the residual memory space of i-th of back end and the data of I/O load factor, continues to hold
Row step S102, process terminate.
If Pi<When Q, which is placed on i-th of back end, counts the I/O load factor of i-th of back end at this time
For P'.
If P'<When Q, which can store in i-th of back end.The I/O information of meta data server 1 is collected
Module sends the request for obtaining I/O load factor, the I/O load monitoring module monitors of i-th of back end to i-th of back end
The newest I/O load factor of current i-th of back end is fed back to metadata by the I/O load factor of current i-th of back end
Server 1.
The memory space collection module of meta data server 1 sends to i-th of back end and obtains i-th of back end
The request of residual memory space, when the memory space monitoring module discovery current data node residue in i-th of back end is deposited
When storing up spatial variations, the current newest residual memory space of i-th of back end is fed back into meta data server 1.More Singapore dollar
The data of the residual memory space of current i-th of back end and I/O load factor, process terminate, then return in data server 1
The step of returning the request for receiving the application storing data that client is sent, carries out the storage of next data block.
It is described so that i-th of back end recalls to recall instruction to i-th of back end transmission by step S212
Data block.
Step S213, carries out i plus 1 is handled, and continues to execute step S202.
Due to current existing network each core with portfolio raising, amount of access and data traffic it is quick
Increase, processing capacity and calculating intensity also correspondingly increase, so that single server apparatus can not undertake at all.In this feelings
Under condition, if throwing away existing equipment does a large amount of HardwareUpgring, the waste of existing resource will be caused in this way, and if face again
When facing the promotion of portfolio next time, this will lead to again the great number cost input of HardwareUpgring again or even performance is brilliant again
Equipment can not meet current traffic growth the needs of.
A kind of cheap effectively transparent method being derived for this situation is to extend conventional network equipment and service
The bandwidth of device, the technology for increasing handling capacity, Strengthens network data-handling capacity, the flexibility for improving network and availability are exactly negative
Carry balanced (Load Balance).
Load balance also referred to as loads shared, refers to and carries out dynamic adjustment to the loading condition in system, to eliminate as far as possible or
Each unbalanced phenomenon of node load in reduction system.Concrete methods of realizing is that overloading on node for task is transferred to other gently
It carries on node, the load balance of each node of system is realized as far as possible, to improve the handling capacity of system.Load is shared to be conducive to unite
The various resources in managing distributing type system are raised, convenient for expanding the processing capacity of system using shared information and its service mechanism.
In the present embodiment, if Pi<When Q, which is placed on i-th of back end, counts i-th of back end at this time
I/O load factor be P'.
If P'>When Q, shows to be added the I/O overload of i-th of back end after the data block, will affect overall performance.
Therefore instruction is recalled so that i-th of back end recalls the data block, by the number to i-th of back end transmission
It is stored in next back end (i+1) in residual memory space information according to block, i.e. residual memory space is only second to i-th of number
According to the back end of node, until the P' of some back end<Q, to reach load balance.
I is carried out plus 1 processing, according to the request of the application storing data from the residual memory spaces of all back end
In the data of I/O load factor, the residual memory space and the first I/O load factor of i+1 back end are inquired.Subsequent step
Suddenly it is successively executed according to the step of i-th of back end based on i+1 back end, is met until finding a back end
P'<Until the condition of Q.Can guarantee the I/O load factor of the back end of each storing data block in given threshold hereinafter,
Be not in the excessively high situation of the I/O load factor of back end, improve the load balancing of system.
After customer data storage is completed, the client of variation with to(for) the frequency of data access, the I/O of back end
Load factor can change, and can carry out dynamic load leveling according to the method described above at this time, balance the I/O access of each back end
Amount.If certain a period of time some back end I/O load factor it is excessively high, then by one or several visits in the back end
The high data block of the amount of asking is recalled from the back end, carries out load balancing process.The data block recalled is placed in metadata to deposit
In residual memory space information table i+1 back end in reservoir, then compare the first I/O load factor P of the back endi
The relationship of the 2nd I/O load factor P' and threshold value Q after being put into this back end with the relationship of threshold value Q and the data block, if all
Smaller than threshold value Q, then data block can be put into the back end, until I/O load factor is less than threshold value Q, load balancing process knot
Beam.
The embodiment of the present application is directed to existing distributed storage technology under cloud environment and improves, in client's storing data
Initial stage carries out load balancing, while considering the memory space and I/O loading condition of back end, first selection memory node space
Whether big back end attempts storage, then consider the I/O load factor of current back end than set given threshold
Small, if meeting situation simultaneously, user data block is stored in current data node.After client stores, for storage file
Rate of people logging in change, back end will appear the excessively high situation of I/O load factor, at this time that fractional load rate is excessively high data
Block is dynamically dispatched to low-load back end, to guarantee the overall performance of storage system.
Make the selection of back end that there is specific aim, storage equipment task distribution is balanced, selects back end remaining space
Big back end storage, promotes the performance of global storage system.Meanwhile selecting the process of data block that also there is specific aim, number
When excessively high according to the I/O load factor of node, the data block of part I/O load factor is dynamically dispatched to low I/O load factor data section
Point improves the load balancing of system, reduces the file access time, improves the operational efficiency of system.
It should be noted that each step provided by the embodiments of the present application can carry out sequence change as needed, can also root
It is the range that the application is protected according to needing to delete part steps.Above-mentioned second embodiment is only a kind of example, be relative to
First embodiment protection scope lesser one is embodiment.
Data storage device is as follows under a kind of cloud environment that the application 3rd embodiment provides：
In above-mentioned first embodiment, date storage method under a kind of cloud environment, corresponding, this Shen are provided
It please additionally provide data storage device under a kind of cloud environment.
Since Installation practice is substantially similar to embodiment of the method, so describing fairly simple, relevant part please join
The corresponding explanation of date storage method embodiment under the cloud environment of above-mentioned offer is provided.Installation practice described below is only
It is schematically, to be illustrated with reference to the accompanying drawing.
As shown in figure 3, it illustrates the structural schematic diagram of data storage device under cloud environment provided by the embodiments of the present application,
It comprises the following modules.
Receiving module 11, the request of the application storing data for receiving client transmission；
Enquiry module 12, for the residual memory space according to the request for applying for storing data from all back end
In the data of I/O load factor, the residual memory space and the first I/O load factor of i-th of back end are inquired；Described i-th
A back end is the back end that residual memory space comes i-th bit in all back end；
Sending module 13, for the residual memory space of i-th of back end to be back to the client, for
Data block is sent to i-th of back end according to the residual memory space of i-th of back end by the client；
Whether the first comparison module 14 is less than given threshold for the first I/O load factor；
Memory module 15, if being less than the given threshold for the first I/O load factor, to i-th of data
Node sends the store instruction data block that the client is sent so that i-th of back end stores.
Data storage device is as follows under a kind of cloud environment that the application fourth embodiment provides：
Optionally, as shown in figure 4, it illustrates the structures of data storage device under cloud environment provided by the embodiments of the present application
Schematic diagram, the embodiment of the present application are illustrated on the basis of above-mentioned 3rd embodiment.
The enquiry module 12, if being also used to the first I/O load factor more than or equal to the given threshold, to i
It carries out plus 1 processing, continues to run in the enquiry module according to the request of the application storing data from all back end
In residual memory space and I/O load factor, the residual memory space and the first I/O load factor of i-th of back end are inquired
Part.
Optionally, as shown in figure 4, data storage device under the cloud environment, further includes：Second comparison module 16 and update
Module 17；
Whether second comparison module 16 is less than the given threshold for the 2nd I/O load factor；
The update module 17, for updating the residual memory space of i-th of back end and the number of I/O load factor
According to, return it is described receive client send application storing data request the step of；
The sending module 13 is also used to send the request for obtaining the 2nd I/O load factor to i-th of back end；
If the 2nd I/O load factor is less than the given threshold, is sent to i-th of back end and obtain i-th of data
The request of the data of the residual memory space and I/O load factor of node；
The receiving module 11 is also used to receive calculated i-th of the back end of i-th of back end and deposits
Store up the 2nd I/O load factor after the data block；Receive i-th of data that i-th of back end returns
The residual memory space of node and the data of I/O load factor.
Optionally, as shown in figure 4, data storage device under the cloud environment, further includes：Recall module 18；
It is described to recall module 18, if being greater than or equal to the given threshold for the 2nd I/O load factor, to described the
I back end transmission recalls instruction so that i-th of back end recalls the data block；
The enquiry module 12, is also used to carry out i plus 1 is handled, and continues to run in the enquiry module according to the Shen
Please storing data request from the residual memory space and I/O load factor of all back end, inquire i-th of back end
Residual memory space and the first I/O load factor part.
It should be noted that each module provided by the embodiments of the present application can carry out sequence change as needed, can also root
It is the range that the application is protected according to needing to delete part of module.Above-mentioned fourth embodiment is only a kind of example, be relative to
3rd embodiment protection scope lesser one is embodiment.
Data-storage system is as follows under a kind of cloud environment that the 5th embodiment of the application provides：
In the above-described embodiment, date storage method and device under a kind of cloud environment are provided, it is corresponding,
Present invention also provides data-storage systems under a kind of cloud environment, are illustrated with reference to the accompanying drawing.
As shown in figure 5, the structural schematic diagram of data-storage system under cloud environment provided by the embodiments of the present application is shown, institute
The system of stating includes：Meta data server 1, client 2, back end 3.
Wherein, number under the cloud environment that above-mentioned 3rd embodiment or fourth embodiment provide can be used in meta data server 1
According to storage device, it is not repeated to describe herein.
Wherein, the client 2, for sending the request of application storing data to the meta data server；Receive institute
State the residual memory space of i-th of back end of meta data server transmission；It is deposited according to the residue of i-th of back end
It stores up space and data block is sent to i-th of back end；
Wherein, the back end 3, the data block sent for receiving the client；Receive the Metadata Service
The store instruction that device is sent stores the data block that the client is sent.
It is understood that the principle that embodiment of above is intended to be merely illustrative of the present and the exemplary implementation that uses
Mode, however the present invention is not limited thereto.For those skilled in the art, essence of the invention is not being departed from
In the case where mind and essence, various changes and modifications can be made therein, these variations and modifications are also considered as protection scope of the present invention.
Claims (12)
1. date storage method under a kind of cloud environment, which is characterized in that including：
Receive the request for the application storing data that client is sent；
According to the request of the application storing data from the residual memory space of all back end and the data of I/O load factor
In, inquire the residual memory space and the first I/O load factor of i-th of back end；I-th of back end is the institute
There is residual memory space in back end to come the back end of i-th bit；
The residual memory space of i-th of back end is back to the client, so that the client is according to
Data block is sent to i-th of back end by the residual memory space of i-th of back end；
Compare whether the first I/O load factor is less than given threshold；
If the first I/O load factor be less than the given threshold, to i-th of back end send store instruction for
I-th of back end stores the data block that the client is sent.
2. date storage method under cloud environment according to claim 1, which is characterized in that i-th of back end
The first I/O load factor PiIt is calculated by the following formula and obtains：
Pi=ω1Oi+ω2Ri+ω3ti；
Wherein OiIndicate the actual read or write speed of i-th of back end after normalizing；RiIt indicates after normalizing described i-th
The actual handling capacity of back end；tiI-th of back end actual waiting time is indicated after normalizing；ω1、ω2、ω3
Respectively read or write speed, handling capacity and the weighted value of waiting time, and Σ ωi=(i=1,2,3)=1.
3. date storage method under cloud environment according to claim 2, which is characterized in that described i-th after the normalization
A actual read or write speed O of back endiIt is calculated by the following formula and obtains：
WhereinIiFor the actual read or write speed of i-th of back end；
The actual handling capacity R of i-th of back end after the normalizationiIt is calculated by the following formula and obtains：
WhereinHiIt handles up for i-th of back end is actual
Amount；
The actual waiting time t of i-th of back end after the normalizationiIt is calculated by the following formula and obtains：
WhereinTiWhen waiting actual for i-th of back end
Between.
4. date storage method under cloud environment according to claim 2, which is characterized in that the given threshold Q by with
Lower formula is calculated：
5. date storage method under cloud environment according to claim 1, which is characterized in that the first I/O
Whether load factor is less than after given threshold step, further includes：
If the first I/O load factor is greater than or equal to the given threshold, i is carried out plus 1 is handled, continues to execute the basis
The request of the application storing data is inquired from the data of the residual memory spaces of all back end and I/O load factor
The step of residual memory space and the first I/O load factor of i-th of back end.
6. date storage method under cloud environment according to claim 5, which is characterized in that if the first I/O load factor
Less than the given threshold, then store instruction is sent so that i-th of back end stores institute to i-th of back end
After the data block step for stating client transmission, further include：
The request for obtaining the 2nd I/O load factor is sent to i-th of back end；
Receive calculated i-th of the back end of i-th of back end store after the data block described
Two I/O load factors；
Compare whether the 2nd I/O load factor is less than the given threshold；
If the 2nd I/O load factor is less than the given threshold, sends and obtain described i-th to i-th of back end
The request of the data of the residual memory space and I/O load factor of back end；
Receive the residual memory space for i-th of back end that i-th of back end returns and the number of I/O load factor
According to；
The residual memory space of i-th of back end and the data of I/O load factor are updated, the reception client hair is returned
The step of request for the application storing data sent.
7. date storage method under cloud environment according to claim 6, which is characterized in that the 2nd I/O
Whether load factor is less than after the given threshold step, further includes：
If the 2nd I/O load factor is greater than or equal to the given threshold, instruction is recalled to i-th of back end transmission
So that i-th of back end recalls the data block；
I is carried out plus 1 is handled, continues to execute and is stored according to the request of the application storing data from the residue of all back end
In space and I/O load factor, the step of inquiring the residual memory space and the first I/O load factor of i-th of back end.
8. data storage device under a kind of cloud environment, which is characterized in that including：
Receiving module, the request of the application storing data for receiving client transmission；
Enquiry module, for the residual memory space and I/O according to the request for applying for storing data from all back end
In the data of load factor, the residual memory space and the first I/O load factor of i-th of back end are inquired；I-th of data
Node is the back end that residual memory space comes i-th bit in all back end；
Sending module, for the residual memory space of i-th of back end to be back to the client, for the visitor
Data block is sent to i-th of back end according to the residual memory space of i-th of back end by family end；
Whether the first comparison module is less than given threshold for the first I/O load factor；
Memory module is sent out if being less than the given threshold for the first I/O load factor to i-th of back end
Send the store instruction data block that the client is sent so that i-th of back end stores.
9. data storage device under cloud environment according to claim 8, which is characterized in that the enquiry module is also used to
If the first I/O load factor is greater than or equal to the given threshold, i is carried out plus 1 is handled, continues to run the inquiry mould
In block according to the request of the application storing data from the residual memory space and I/O load factor of all back end, inquiry
The part of the residual memory space of i-th back end and the first I/O load factor out.
10. data storage device under cloud environment according to claim 8, which is characterized in that described device further includes：The
Two comparison modules and update module；
Whether second comparison module is less than the given threshold for the 2nd I/O load factor；
The update module is returned for updating the residual memory space of i-th of back end and the data of I/O load factor
The step of returning the request for receiving the application storing data that client is sent；
The sending module is also used to send the request for obtaining the 2nd I/O load factor to i-th of back end；If described
2nd I/O load factor is less than the given threshold, sends to i-th of back end and obtains i-th of back end
The request of the data of residual memory space and I/O load factor；
The receiving module is also used to receive described in calculated i-th of back end storage of i-th of back end
The 2nd I/O load factor after data block；Receive i-th of back end that i-th of back end returns
The data of residual memory space and I/O load factor.
11. data storage device under cloud environment according to claim 8, which is characterized in that described device further includes：It adjusts
Module out；
It is described to recall module, if being greater than or equal to the given threshold for the 2nd I/O load factor, to i-th of number
Instruction is recalled according to node transmission so that i-th of back end recalls the data block；
The enquiry module, is also used to carry out i plus 1 is handled, and continues to run in the enquiry module and is stored according to the application
The request of data inquires the residue of i-th of back end from the residual memory space and I/O load factor of all back end
The part of memory space and the first I/O load factor.
12. data-storage system under a kind of cloud environment, which is characterized in that including：Metadata Service described in claim 8-11
Device, client and back end；
The client, for sending the request of application storing data to the meta data server；Receive the metadata clothes
The residual memory space for i-th of back end that business device is sent；It will be counted according to the residual memory space of i-th of back end
I-th of back end is sent to according to block；
The back end, the data block sent for receiving the client；Receive depositing for the meta data server transmission
Storage instruction stores the data block that the client is sent.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810792783.XA CN108900626B (en) | 2018-07-18 | 2018-07-18 | Data storage method, device and system in cloud environment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810792783.XA CN108900626B (en) | 2018-07-18 | 2018-07-18 | Data storage method, device and system in cloud environment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108900626A true CN108900626A (en) | 2018-11-27 |
| CN108900626B CN108900626B (en) | 2021-11-19 |
Family
ID=64350820
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810792783.XA Active CN108900626B (en) | 2018-07-18 | 2018-07-18 | Data storage method, device and system in cloud environment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108900626B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110071964A (en) * | 2019-03-26 | 2019-07-30 | 罗克佳华科技集团股份有限公司 | File synchronisation method, device, file sharing network, file are total to system and storage medium |
| CN110158430A (en) * | 2019-05-08 | 2019-08-23 | 中铁北京工程局集团有限公司 | A kind of automatic plucking laminating machine in bridge concrete face |
| CN111124316A (en) * | 2019-12-30 | 2020-05-08 | 青岛海尔科技有限公司 | Method and device for sharing storage space and computer readable storage medium |
| CN113396566A (en) * | 2019-04-16 | 2021-09-14 | 阿里巴巴集团控股有限公司 | Resource allocation based on comprehensive I/O monitoring in distributed storage system |
| CN114745563A (en) * | 2022-04-11 | 2022-07-12 | 中国联合网络通信集团有限公司 | Method, device and system for selecting edge computing node to process live broadcast task |
| CN114827180A (en) * | 2022-06-22 | 2022-07-29 | 蒲惠智造科技股份有限公司 | Distribution method of cloud data distributed storage |
| CN116339622A (en) * | 2023-02-20 | 2023-06-27 | 深圳市数存科技有限公司 | Data compression system and method based on block level |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080225714A1 (en) * | 2007-03-12 | 2008-09-18 | Telefonaktiebolaget Lm Ericsson (Publ) | Dynamic load balancing |
| CN101710339A (en) * | 2009-11-20 | 2010-05-19 | 中国科学院计算技术研究所 | Method and system for controlling data storage in cluster file system and method for creating file |
| CN105760227A (en) * | 2016-02-04 | 2016-07-13 | 中国联合网络通信集团有限公司 | Method and system for resource scheduling in cloud environment |
| US20170220382A1 (en) * | 2016-01-28 | 2017-08-03 | Ca, Inc. | Weight adjusted dynamic task propagation |
| CN107436813A (en) * | 2017-08-03 | 2017-12-05 | 郑州云海信息技术有限公司 | A kind of method and system of meta data server dynamic load leveling |
| CN107948293A (en) * | 2017-11-29 | 2018-04-20 | 重庆邮电大学 | One kind is based on MongoDB load balance optimization system and methods |
| CN108196791A (en) * | 2017-12-29 | 2018-06-22 | 北京奇虎科技有限公司 | Data access method and device based on multiple storage devices |
| CN108200151A (en) * | 2017-12-29 | 2018-06-22 | 创新科存储技术（深圳）有限公司 | ISCSI Target load-balancing methods and device in a kind of distributed memory system |
| CN108287666A (en) * | 2018-01-16 | 2018-07-17 | 中国人民公安大学 | Date storage method and device for cloud storage environment |
-
2018
- 2018-07-18 CN CN201810792783.XA patent/CN108900626B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080225714A1 (en) * | 2007-03-12 | 2008-09-18 | Telefonaktiebolaget Lm Ericsson (Publ) | Dynamic load balancing |
| CN101710339A (en) * | 2009-11-20 | 2010-05-19 | 中国科学院计算技术研究所 | Method and system for controlling data storage in cluster file system and method for creating file |
| US20170220382A1 (en) * | 2016-01-28 | 2017-08-03 | Ca, Inc. | Weight adjusted dynamic task propagation |
| CN105760227A (en) * | 2016-02-04 | 2016-07-13 | 中国联合网络通信集团有限公司 | Method and system for resource scheduling in cloud environment |
| CN107436813A (en) * | 2017-08-03 | 2017-12-05 | 郑州云海信息技术有限公司 | A kind of method and system of meta data server dynamic load leveling |
| CN107948293A (en) * | 2017-11-29 | 2018-04-20 | 重庆邮电大学 | One kind is based on MongoDB load balance optimization system and methods |
| CN108196791A (en) * | 2017-12-29 | 2018-06-22 | 北京奇虎科技有限公司 | Data access method and device based on multiple storage devices |
| CN108200151A (en) * | 2017-12-29 | 2018-06-22 | 创新科存储技术（深圳）有限公司 | ISCSI Target load-balancing methods and device in a kind of distributed memory system |
| CN108287666A (en) * | 2018-01-16 | 2018-07-17 | 中国人民公安大学 | Date storage method and device for cloud storage environment |
Non-Patent Citations (1)
| Title |
|---|
| 熊安萍 等: "基于对象存储的负载均衡存储策略", 《计算机工程与设计》 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110071964A (en) * | 2019-03-26 | 2019-07-30 | 罗克佳华科技集团股份有限公司 | File synchronisation method, device, file sharing network, file are total to system and storage medium |
| CN113396566A (en) * | 2019-04-16 | 2021-09-14 | 阿里巴巴集团控股有限公司 | Resource allocation based on comprehensive I/O monitoring in distributed storage system |
| CN110158430A (en) * | 2019-05-08 | 2019-08-23 | 中铁北京工程局集团有限公司 | A kind of automatic plucking laminating machine in bridge concrete face |
| CN111124316A (en) * | 2019-12-30 | 2020-05-08 | 青岛海尔科技有限公司 | Method and device for sharing storage space and computer readable storage medium |
| CN111124316B (en) * | 2019-12-30 | 2023-12-19 | 青岛海尔科技有限公司 | Storage space sharing method and device and computer readable storage medium |
| CN114745563A (en) * | 2022-04-11 | 2022-07-12 | 中国联合网络通信集团有限公司 | Method, device and system for selecting edge computing node to process live broadcast task |
| CN114745563B (en) * | 2022-04-11 | 2024-01-30 | 中国联合网络通信集团有限公司 | Method, device and system for processing live broadcast task by selecting edge computing node |
| CN114827180A (en) * | 2022-06-22 | 2022-07-29 | 蒲惠智造科技股份有限公司 | Distribution method of cloud data distributed storage |
| CN116339622A (en) * | 2023-02-20 | 2023-06-27 | 深圳市数存科技有限公司 | Data compression system and method based on block level |
| CN116339622B (en) * | 2023-02-20 | 2023-11-14 | 深圳市数存科技有限公司 | Data compression system and method based on block level |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108900626B (en) | 2021-11-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108900626A (en) | Date storage method, apparatus and system under a kind of cloud environment | |
| US10466899B2 (en) | Selecting controllers based on affinity between access devices and storage segments | |
| JP5484471B2 (en) | Storage-side storage request management | |
| CN101370030B (en) | Resource load stabilization method based on contents duplication | |
| CN104050042B (en) | The resource allocation methods and device of ETL operations | |
| CN108182105B (en) | Local dynamic migration method and control system based on Docker container technology | |
| JP5744707B2 (en) | Computer-implemented method, computer program, and system for memory usage query governor (memory usage query governor) | |
| CN106233276A (en) | The coordination access control of network-accessible block storage device | |
| CN1921449B (en) | Stable, minimal skew resource flow control method and system | |
| CN106161120A (en) | The distributed meta-data management method of dynamic equalization load | |
| CN103607424B (en) | Server connection method and server system | |
| CN103152393A (en) | Charging method and charging system for cloud computing | |
| US10158709B1 (en) | Identifying data store requests for asynchronous processing | |
| CN110383764A (en) | The system and method for usage history data processing event in serverless backup system | |
| CN108183947A (en) | Distributed caching method and system | |
| CN101963969B (en) | Method and database server for realizing load balancing in Oracle RAC (Real Application Cluster) system | |
| CN105683928B (en) | For the method for data cache policies, server and memory devices | |
| CN109085999A (en) | data processing method and processing system | |
| CN111737168A (en) | Cache system, cache processing method, device, equipment and medium | |
| CN109981702A (en) | A kind of file memory method and system | |
| CN108804242A (en) | A kind of data counts De-weight method, system, server and storage medium | |
| Gill et al. | Dynamic cost-aware re-replication and rebalancing strategy in cloud system | |
| CN106331160A (en) | Data migration method and system | |
| CN110309229A (en) | The data processing method and distributed system of distributed system | |
| Weissman et al. | The service grid: supporting scalable heterogeneous services in wide-area networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |