CN109085999A - data processing method and processing system - Google Patents

data processing method and processing system Download PDF

Info

Publication number
CN109085999A
CN109085999A CN201810626541.3A CN201810626541A CN109085999A CN 109085999 A CN109085999 A CN 109085999A CN 201810626541 A CN201810626541 A CN 201810626541A CN 109085999 A CN109085999 A CN 109085999A
Authority
CN
China
Prior art keywords
storage
data
storage cluster
stored
cluster
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201810626541.3A
Other languages
Chinese (zh)
Other versions
CN109085999B (en
Inventor
陈绍元
王士铨
王进锋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to CN201810626541.3A priority Critical patent/CN109085999B/en
Publication of CN109085999A publication Critical patent/CN109085999A/en
Application granted granted Critical
Publication of CN109085999B publication Critical patent/CN109085999B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0629—Configuration or reconfiguration of storage systems
    • G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/0644—Management of space entities, e.g. partitions, extents, pools
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0674—Disk device
    • G06F3/0676—Magnetic disk device

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the present application discloses a kind of data processing method and processing system.The embodiment of the present application method includes: that processing system determines that having occupied memory space is greater than the first storage cluster for storing threshold value first from storage cluster set, wherein, the first physical volume in first storage cluster and between logical volume have the first mapping relations, first physical address of corresponding first physical volume of the logical address that first mapping relations are used to indicate logical volume, since the residual memory space of the first storage cluster is less, after-treatment system therefore will determine the second storage cluster for having occupied memory space and being less than storage threshold value from storage cluster set, and further establish the second mapping relations in logical volume and the second storage cluster between the second physical volume, second physical address of corresponding second physical volume of the logical address that second mapping relations are used to indicate logical volume.

Description

Data processing method and processing system
Technical field
This application involves the communications field more particularly to a kind of data processing methods and processing system.
Background technique
Traditional network store system stores all data using the storage equipment concentrated, but due to storage its object of equipment The limitation of reason composition (such as disc driver quantity, the number of servers connected, memory size and controller performance), cannot Meet the needs of Mass storage application.
For this purpose, generalling use the storage mode of each node into storage cluster by data distribution at present to improve storage system The memory capacity of support, wherein storage cluster is that the memory space in multiple physical volumes (such as disk or hard disk) is aggregated into one A storage pool that can provide clients with unified access interface, client can be accessed and be utilized storage by the access interface Memory space on cluster.
However, the number of nodes in storage cluster can not unlimited extension, i.e. individually storage due to the limitation of network bandwidth The capacity of cluster has the upper limit, for terminal, it is possible to which data to be stored occur, all storage is completed not yet, and is deposited The full situation of accumulation capacity will lead to storage service at this time and interrupt because of the deficiency of memory space.
Summary of the invention
The embodiment of the present application provides a kind of data processing method and processing system, in logical volume and the first storage cluster All there are mapping relations between the second physical volume in first physical volume and the second storage cluster, when the sky of the first storage cluster Between it is insufficient when, data to be stored can be stored in the second storage cluster according to the second mapping relations, avoid storage service Stopping.
In view of this, the application first aspect provides a kind of method of data processing, may include:
Processing system determines first from storage cluster set has occupied the first storage that memory space is greater than storage threshold value Cluster, wherein the first physical volume in the first storage cluster and between logical volume have the first mapping relations, this first mapping close System is used to indicate the first physical address of corresponding first physical volume of logical address of logical volume, due to remaining for the first storage cluster Remaining memory space is less, thus after-treatment system will be determined from storage cluster set occupied memory space be less than storage threshold Second storage cluster of value, and the second mapping further established in logical volume and the second storage cluster between the second physical volume is closed System, the second physical address of corresponding second physical volume of the logical address which is used to indicate logical volume.
It should be noted that storage cluster is that the memory space in multiple physical volumes (such as disk or hard disk) is aggregated into one A storage pool that can provide clients with unified access interface, client can be accessed and be utilized storage by the access interface Memory space on cluster.Physical volume for storing data, including movable memory equipment and non-removable storage device, example Such as: disk.Logical volume is by Storage Virtualization, and storage is no longer limited by the size of physical disk, can polymerize multiple disks or Disk partition is at a logical volume.
That is, logical volume can establish mapping relations between one or more physical volumes, specifically, which is closed System can indicate the physical address of physical volume corresponding to the logical address of logical volume, wherein logical address is application program institute The address used, can be the position of the disk provided by the logical block number (LBN) of data, and physical address be then by disk cylinder, Address determined by the physical locations such as head, section.For example, logical address can be (inclined with segment base (sector address) and section bias internal amount Move address) it indicates, the section where segment base determines data occupy the position of disk, and section bias internal amount determines the data in the section Position, variation of the logical address Jing Guo addressing system or calculate available physical address.
It is understood that since logical volume can be corresponding with multiple physical volumes, logical volume and the can kept On the basis of the first mapping relations between one physical volume, the second mapping between logical volume and the second physical volume is further established All there are mapping relations between relationship, i.e. logical volume and the first physical volume and the second physical volume.
In the application embodiment, for terminal, due to only having logical volume to be visible, i.e., terminal only needs to know The logical address of data storage, then having first to reflect between the first physical volume in logical volume and the first storage cluster It penetrates on the basis of relationship, and establishes the second mapping relations between the second physical volume in logical volume and the second storage cluster, It is thus achieved that the dilatation to logical volume.
Optionally, in some possible embodiments,
Processing system will also receive the data storage request of terminal transmission, include wait store in the data storage request The instruction information of data indicates in information and includes the logical address of logical volume, and later, processing system will compare number to be stored According to the size between occupied memory space and the first storage cluster residual memory space, if the residue of the first storage cluster is deposited It stores up space and is less than the memory space that data to be stored occupies, then processing system can be incited somebody to action according to established second mapping relations Data to be stored is stored in the second physical volume of the second storage cluster.
It is understood that if the residual memory space of the first storage cluster is greater than the storage sky that data to be stored occupies Between, then data to be stored is then stored in the first of the first storage cluster according to established first mapping relations by processing system In physical volume.
The first physical volume and the second storage cluster in the application embodiment, in logical volume and the first storage cluster In the second physical volume between all there are mapping relations, when the insufficient space of the first storage cluster, data to be stored can be with It is stored in the second storage cluster according to the second mapping relations, avoids storage service and interrupted because of the deficiency of memory space.
Optionally, in some possible embodiments, if the residual memory space of the first storage cluster is less than wait store The memory space that data occupy, data to be stored is stored in the second physical volume of the second storage cluster by processing system, can be with Include:
If the first storage cluster does not have residual memory space, i.e. the first storage cluster has reached storage cap, then entirely The data to be stored in portion requires in the second storage cluster of deposit, and specifically, processing system is patrolled according to what data to be stored occupied The third physical address of the second physical volume in the second storage cluster can be determined by collecting address and the second mapping relations, wherein third Memory space corresponding to physical address is the part memory space of memory space corresponding to the second physical address, and then is handled The third physical address is added in the first data storage request and obtains the second data storage request by system, and by this second number It is sent to the second storage cluster according to storage request, being used to indicate the second storage cluster will number be stored according to the third physical address According to being stored in the second physical volume.
It should be noted that if the occupied space of the second storage cluster also surpasses in the storing process of data to be stored Storage threshold value has been crossed, then processing system will determine third storage cluster, and has established third in logical volume and third storage cluster Third mapping relations between physical volume realize the further dilatation to logical volume.
By the above-mentioned means, providing a kind of processing system general in the case where the first storage cluster reaches storage cap Data to be stored is stored in the specific implementation of the second storage cluster, when the insufficient space of the first storage cluster, wait store Data can be stored in the second storage cluster according to the second mapping relations, avoid deficiency of the storage service because of memory space And it interrupts.
Optionally, in some possible embodiments, data to be stored is stored in the second storage cluster by processing system The second physical volume in, may include:
If the first storage cluster still has residual memory space, i.e. the first storage cluster is also not up to storage cap, but The residual memory space of first storage cluster is less than the memory space that data to be stored occupies, then, data to be stored is still wanted The first storage cluster of preferential deposit, another part data after the first storage cluster reaches storage cap, in data to be stored It is restored again into the second storage cluster, specifically, data to be stored can be split as the first data and the second data, and the first data occupy Memory space be less than or equal to the first storage cluster residual memory space, processing system first according to the first mapping relations determine The 4th physical address corresponding with the logical address that the first data occupy, wherein memory space corresponding to the 4th physical address For the part memory space of memory space corresponding to the first physical address, the 4th physical address is added to the first number later Third data storage request is obtained according to storing in request, and the third data storage request is sent to the first storage cluster, is used The first data are stored in by the first physical volume according to the 4th physical address in the first storage cluster of instruction, are stored to the first data After the completion, processing system determines the logical address the corresponding 5th occupied with the second data according to the second mapping relations physically Location, wherein memory space corresponding to the 5th physical address is the part storage of memory space corresponding to the second physical address 5th physical address is added in the first data storage request obtains the 4th data storage request later by space, and should 4th data storage request is sent to the second storage cluster, is used to indicate the second storage cluster according to the 5th physical address for Two data are stored in the second physical volume.
By the above-mentioned means, data to be stored will be preferential in the case where the first storage cluster is also not up to storage cap It is stored in the first storage cluster, after the first storage cluster reaches storage cap, the remainder of data to be stored is restored again into second Storage cluster can make full use of existing available memory space in this way.
Optionally, in some possible embodiments,
When data to be stored has been stored in the second physical volume, processing system can be obtained from the second storage cluster and deposited Store up storage address of the data in the second physical volume, wherein also include in the instruction information carried in the first data storage request There is at least one Data Identification of data to be stored, later, processing system will be established between the Data Identification and storage address Third mapping relations, the third mapping relations with can identifying storage of the corresponding data in the second physical volume with designation date Location.
Optionally, in some possible embodiments,
Processing system will also receive the first data access request of terminal transmission, wherein in first data access request Data Identification comprising data to be visited, processing system can determine deposit corresponding with the Data Identification according to third mapping relations Address is stored up, the storage address is added in first data access request obtains the second data access request later, and should Second data access request is sent to the second storage cluster, is used to indicate the second storage cluster and treats the progress of access data accordingly Processing, such as: delete data or modification data.
It should be noted that data corresponding to Data Identification, which are possible to a part, is stored in the first storage cluster, it is another Part is stored in the second storage cluster, then processing system can find the data in the first storage cluster according to Data Identification The first storage address and the data the second storage cluster the second storage address, and then processing system by first storage ground Location is added in data access request and forwards the request to the first storage cluster, and the second storage address data that are added to are visited It asks in request and the request to the second storage cluster, the first storage cluster and the second storage cluster is forwarded to be asked respectively according to what is received It asks and performs corresponding processing to being stored in local data.
In the embodiment of the present application, due to only having logical volume to be visible for terminal, i.e., terminal does not perceive specific use In the storage cluster of storing data, deposited then the data to be visited being stored in the first storage cluster just no longer need to migrate to second Accumulation reduces the workload of Data Migration.
It optionally, in some possible embodiments, should before determining the second storage cluster in storage cluster set Method further include:
Processing system can each storage cluster into storage cluster send inquiry request, be specifically used for inquiring each storage and collect The occupation rate of group, later, itself current occupation rate is fed back to processing system by each storage cluster, wherein the occupation rate can be with Indicate that storage cluster has accounted for the percentage of total memory space with memory space, and then processing system is to the occupation rate of each storage cluster Judged, the storage cluster that occupation rate is greater than preset threshold is the first storage cluster.
Optionally, in some possible embodiments, determine that the second storage cluster includes: from storage cluster set
Processing system determines that occupation rate is less than the storage cluster of preset threshold from each storage cluster occupation rate got For the second storage cluster, wherein if the storage cluster for thering are multiple occupation rates to be less than preset threshold, then processing system can will be each Storage cluster is arranged according to the sequence of occupation rate from big to small or from small to large, and then processing system can be according to the sequence Preferentially select the smallest storage cluster of occupation rate as the second storage cluster.
In the embodiment of the present application, the occupation rate of each storage cluster in the available storage cluster set of processing system, first It determines that occupation rate is greater than the first storage cluster of preset threshold, determines that occupation rate is less than the second storage collection of preset threshold again later Group, by the above-mentioned means, for present solution provides the specific implementation sides of a kind of determination the first storage cluster and the second storage cluster Formula.
The embodiment of the present application second aspect provides a kind of processing system, which can functionally be divided into expansion Unit, storage unit and indexing units are filled, specifically, expansion unit can be coordinator (coordinator), storage unit It can be storage gateway (storage gateway), indexing units can be index (index), be introduced separately below:
Expansion unit can inquire the memory space that each storage cluster has occupied in storage cluster set, and by query result It is stored in indexing units, if expansion unit determines that the memory space of occupancy of the first storage cluster is greater than storage threshold value, then Expansion unit will determine the second storage cluster for having occupied memory space and being less than storage threshold value, wherein logical volume and the first storage There are the first mapping relations, further, expansion unit will also establish logical volume and the second storage between first physical volume of cluster The second mapping relations between second physical volume of cluster, and the second mapping relations are stored in indexing units.
Storage unit can receive the first data storage request of terminal initiation, in first data storage request comprising to The instruction information of storing data, the Data Identification of logical address and data to be stored in the instruction information comprising logical volume, If the residual memory space of the first storage cluster is less than the memory space that data to be stored occupies, then storage unit can inquire The second mapping relations in indexing units, and data to be stored are stored in the second storage cluster according to second mapping relations The second physical volume in, and then the storage address of the available data being stored in the second physical volume of storage unit, storage Unit also by the third mapping relations between the storage address for establishing the Data Identification and the data of the data and is stored in index In unit；In addition, storage unit can also receive the first data access request of terminal initiation, in first data access request Data Identification comprising data to be visited, storage unit query indexing units determine storage address corresponding with the Data Identification, And then the second data access request for carrying the storage address can be forwarded to the second storage cluster by storage unit.
Optionally, in some possible embodiments,
After storage unit receives data storage request, the occupied memory space of data to be stored and the first storage will be compared Size between cluster residual memory space, when the residual memory space of the first storage cluster is greater than depositing for data to be stored occupancy When storing up space, then data to be stored can be stored in the first storage collection according to established first mapping relations by storage unit In the first physical volume of group.
Optionally, in some possible embodiments,
When storage unit determines that the first storage cluster does not have residual memory space, storage unit is according to data to be stored The logical address of occupancy and the second mapping relations can determine the third physical address of the second physical volume in the second storage cluster, the Memory space corresponding to three physical address is the part memory space of memory space corresponding to the second physical address, Jin Ercun The third physical address is added in the first data storage request and obtains the second data storage request by storage unit, and by this second Data storage request is sent to the second storage cluster, and being used to indicate the second storage cluster will be wait store according to the third physical address Data are stored in the second physical volume.
In addition, expansion unit will also be true when the memory space of occupancy of the second storage cluster has also exceeded storage threshold value Determine third storage cluster, and establishes the third mapping relations in logical volume and third storage cluster between third physical volume.
Optionally, in some possible embodiments,
When still there is the first storage cluster the residual memory space of residual memory space and the first storage cluster to be less than wait deposit When storing up the memory space that data occupy, data to be stored can be split as the first data and the second data by storage unit, The memory space that first data occupy is less than or equal to the residual memory space of the first storage cluster, and storage unit is according to the first mapping Relationship determines the 4th physical address corresponding with the logical address that the first data occupy, and storage corresponding to the 4th physical address is empty Between for memory space corresponding to the first physical address part memory space, the 4th physical address is added to by storage unit Third data storage request is obtained in first data storage request, and the third data storage request is sent to the first storage collection Group, is used to indicate the first storage cluster according to the 4th physical address and the first data is stored in the first physical volume, when the first number When completing according to storage, storage unit determines the 5th object corresponding with the logical address that the second data occupy according to the second mapping relations Address is managed, memory space corresponding to the 5th physical address is that the part of memory space corresponding to the second physical address stores sky Between, the 5th physical address is added in the first data storage request and obtains the 4th data storage request by storage unit, and will 4th data storage request is sent to the second storage cluster, and being used to indicate the second storage cluster will according to the 5th physical address Second data are stored in the second physical volume.
Optionally, in some possible embodiments,
It, can and the Data Identification pair determining according to third mapping relations after storage unit receives the first data access request The storage address answered, which is added to by storage unit obtains the second data access in first data access request and asks It asks, and second data access request is sent to the second storage cluster, be used to indicate the second storage cluster and treat access data It performs corresponding processing, such as: delete data or modification data.
It should be noted that data corresponding to Data Identification, which are possible to a part, is stored in the first storage cluster, it is another Part is stored in the second storage cluster, then storage unit can find the data in the first storage cluster according to Data Identification The first storage address and the data in the second storage address of the second storage cluster, storage unit folds the first storage address It is added in data access request and forwards the request to the first storage cluster, and the second storage address data access that is added to is asked In asking and forward the request to the second storage cluster.
Optionally, in some possible embodiments,
Expansion unit can each storage cluster into storage cluster send inquiry request, be specifically used for inquiring each storage and collect The occupation rate of group, expansion unit judge the occupation rate of each storage cluster, and determine that occupation rate is greater than depositing for preset threshold Accumulation is the first storage cluster.
Optionally, in some possible embodiments,
Expansion unit determines that occupation rate is less than the storage cluster of preset threshold from each storage cluster occupation rate got For the second storage cluster, if the storage cluster for thering are multiple occupation rates to be less than preset threshold, then expansion unit can be by each storage Cluster is arranged according to the sequence of occupation rate from big to small or from small to large, and then expansion unit can be preferential according to the sequence Select the smallest storage cluster of occupation rate as the second storage cluster.
It is understood that expansion unit, storage unit and indexing units typically operate in mutually independent equipment, For example, expansion unit, storage unit and indexing units can be run in different physical machines or expansion unit, storage Unit and indexing units are separately operable in different virtual machines (virtual machine, VM) or container (container) On.
In addition, expansion unit, storage unit or indexing units can also be run in the same equipment, for example, expanding single Member with storage unit runs on the same physical machine, virtual machine or container or storage unit and indexing units run on it is same Physical machine, virtual machine or container or expansion unit and indexing units run on same physical machine, virtual machine or container, and or Person's expansion unit, storage unit and indexing units all run on same physical machine, virtual machine or container, do not limit herein specifically It is fixed.
Optionally, expansion unit, storage unit and indexing units can also run on one or more in physical machine and include In the chip for having processor and input/output interface.
Specifically, expansion unit may further be subdivided into the first determination unit, the second determination unit, the first foundation list Member, second acquisition unit, judging unit and third determination unit.
Optionally, in some possible embodiments,
First determination unit, for determining the first storage cluster from storage cluster set, in the first storage cluster the There are the first mapping relations, the logical address that the first mapping relations are used to indicate logical volume is corresponding between one physical volume and logical volume The first physical volume the first physical address；
Second determination unit, for when the memory space of occupancy of the first storage cluster be greater than storage threshold value when, from storage It is determined in cluster set and has occupied the second storage cluster that memory space is less than storage threshold value；
First establishing unit is closed for establishing the second mapping in logical volume and the second storage cluster between the second physical volume System, the second mapping relations are used to indicate the second physical address of corresponding second physical volume of logical address of logical volume.
Optionally, in some possible embodiments,
Second acquisition unit, for obtaining the occupation rate of each storage cluster from storage cluster set；
Judging unit, for judging whether the occupation rate of the first storage cluster is greater than preset threshold；
Third determination unit, for determining the first storage collection when the occupation rate of the first storage cluster is greater than preset threshold The memory space of occupancy of group is greater than storage threshold value.
Optionally, in some possible embodiments,
Second determination unit is the second storage cluster specifically for the storage cluster for determining that occupation rate is less than preset threshold.
Specifically, storage unit can be further subdivided into receiving unit, requesting processing, first acquisition unit and Second establishes unit.
Optionally, in some possible embodiments,
Receiving unit includes data to be stored in the first data storage request for receiving the first data storage request Indicate that information, instruction information include the logical address of logical volume；
Requesting processing, for handling the first data storage request, when the residue storage of the first storage cluster When space is less than the memory space that data to be stored occupies, data to be stored is stored in the second physical volume of the second storage cluster In.
Optionally, in some possible embodiments,
Requesting processing is specifically used for:
It, will be corresponding with the logical address that data to be stored occupies when the first storage cluster does not have residual memory space Third physical address, which is added in the first data storage request, obtains the second data storage request, corresponding to third physical address Memory space is the part memory space of memory space corresponding to the second physical address；And second is sent to the second storage cluster Data storage request, the second data storage request are used to indicate the second storage cluster according to third physical address for data to be stored It is stored in the second physical volume.
Optionally, in some possible embodiments,
Requesting processing is specifically used for:
Data to be stored is split as the first data and the second data, the memory space that the first data occupy is less than or equal to the The residual memory space of one storage cluster；Corresponding 4th physical address of the logical address occupied with the first data is added to Obtain third data storage request in one data storage request, memory space corresponding to the 4th physical address is first physically The part memory space of memory space corresponding to location；And third data storage request, third number are sent to the first storage cluster The first storage cluster is used to indicate according to storage request, and the first data are stored in by the first physical volume according to the 4th physical address；It will be with Corresponding 5th physical address of logical address that second data occupy, which is added in the first data storage request, obtains the 4th data Storage is requested, and memory space corresponding to the 5th physical address is the part storage of memory space corresponding to the second physical address Space；And the 4th data storage request is sent to the second storage cluster, the 4th data storage request is used to indicate the second storage collection Second data are stored in the second physical volume according to the 5th physical address by group.
Optionally, in some possible embodiments,
First acquisition unit, for being obtained from the second storage cluster when data to be stored has been stored in the second physical volume Take storage address of the storing data in the second physical volume；
Second establishes unit, the third mapping relations for establishing between Data Identification and storage address, and third mapping is closed System is used to indicate storage address of the corresponding data of Data Identification in the second physical volume.
Optionally, in some possible embodiments,
Receiving unit is also used to:
The first data access request is received, includes the Data Identification of data to be visited in the first data access request；
Requesting processing is also used to:
Storage address corresponding with Data Identification is determined according to third mapping relations；Storage address is added to the first data The second data access request is obtained in access request；And the second data access request is sent to the second storage cluster.
The embodiment of the present application third aspect provides a kind of processing system, may include:
Processor, memory and input/output interface, the processor, the memory are connect with the input/output interface； The memory, for storing program code；The processor executes the application first party when calling the program code in the memory The step of processing system that face or first aspect any embodiment provide executes.
The embodiment of the present application fourth aspect provides a kind of storage medium, including instruction, when run on a computer, uses The program designed by processing system in the above-mentioned first aspect of execution.
The 5th aspect of the embodiment of the present application provides a kind of computer program product comprising instruction, when it is transported on computers When row, so that computer executes the method as described in the application first aspect any optional embodiment.
As can be seen from the above technical solutions, the embodiment of the present application has the advantage that
In the embodiment of the present application, processing system determines that having occupied memory space is greater than storage first from storage cluster set First storage cluster of threshold value, wherein the first physical volume in the first storage cluster and between logical volume have first mapping close System, the first physical address of corresponding first physical volume of the logical address which is used to indicate logical volume, due to The residual memory space of first storage cluster is less, thus after-treatment system will determine to have occupied from storage cluster set and deposit The second storage cluster that space is less than storage threshold value is stored up, and further establishes the second physical volume in logical volume and the second storage cluster Between the second mapping relations, the of corresponding second physical volume of the logical address which is used to indicate logical volume Two physical address.By the above-mentioned means, due to only having logical volume to be visible, i.e., terminal only needs to know number for terminal According to the logical address of storage, and in the first physical volume and the second storage cluster in logical volume and the first storage cluster All there are mapping relations between second physical volume, when the insufficient space of the first storage cluster, data to be stored can basis Second mapping relations are stored in the second storage cluster, are avoided storage service and are interrupted because of the deficiency of memory space.
Detailed description of the invention
Fig. 1 is the network architecture diagram of data processing method in the embodiment of the present application；
Fig. 2 is a kind of embodiment schematic diagram of data processing method in the embodiment of the present application；
Fig. 3 is another embodiment schematic diagram of data processing method in the embodiment of the present application；
Fig. 4 is another embodiment schematic diagram of data processing method in the embodiment of the present application；
Fig. 5 is a kind of embodiment schematic diagram of processing system in the embodiment of the present application；
Fig. 6 is another embodiment schematic diagram of processing system in the embodiment of the present application；
Fig. 7 is the structural schematic diagram of processing system in the embodiment of the present application.
Specific embodiment
The embodiment of the present application provides a kind of data processing method, the first physical volume in logical volume and the first storage cluster And all there are mapping relations between second the second physical volume in storage cluster, when the insufficient space of the first storage cluster, Data to be stored can be stored in the second storage cluster according to the second mapping relations, avoid the stopping of storage service.
The description and claims of this application and term " first ", " second ", " third ", " in above-mentioned attached drawing The (if present)s such as four " are to be used to distinguish similar objects, without being used to describe a particular order or precedence order.It should manage The data that solution uses in this way are interchangeable under appropriate circumstances, so that embodiments herein described herein for example can be to remove Sequence other than those of illustrating or describe herein is implemented.In addition, term " includes " and " having " and theirs is any Deformation, it is intended that cover it is non-exclusive include, for example, containing the process, method of a series of steps or units, system, production Product or equipment those of are not necessarily limited to be clearly listed step or unit, but may include be not clearly listed or for this A little process, methods, the other step or units of product or equipment inherently.
The application is mainly used in data storage network, specifically can be cloud storage network system, the concept of cloud storage Similar with cloud computing, it refers to through functions such as cluster application, grid or distributed file systems, a large amount of various in network Different types of storage equipment gathers collaborative work by application software, common externally to provide data storage and business access One system of function, guarantees the safety of data, and save memory space.
Object stores the core as cloud storage, and the storage basic unit of object storage system is object, and each object is The synthesis of data and data property set, data attribute can be configured according to the demand of application, including data distribution and Service quality etc..The core design thought of object storage is virtualization, is exactly the physical storage locations file particularly, Such as catalogue, disk etc., volume (bucket) is virtually turned to, file is virtually turned to object.For application layer, logarithm is simplified According to access visit, shield the isomerism and complexity of bottom memory technology.
Below referring to Fig. 1, Fig. 1 is the embodiment of the present application data storage network configuration diagram, including one or more Terminal 101, one or more processing systems 102 and one or more storage equipment 103, wherein terminal 101, processing system Data interaction can be realized by wired or wireless network communication mode between 102 and storage equipment 103.
The client call can be passed through in terminal 101 with the client 1011 (such as Dropbox) of operation data storage class Relevant interface (such as S3 interface or REST interface) access process system 102, specifically, client 1011 can be according to processing The domain name or IP address of system 102 search out processing system 102 and initiate to request.
Storage equipment 103 is used to store the data that terminal 101 needs to store, for example, storage equipment 103 can be object and deposit Object storage device (Object Storage Device, OSD) in storage system, each OSD is a smart machine, tool There are oneself storage medium, processor, memory and network system etc., it is responsible to manage the object being locally stored.Wherein, storage is set Standby 103 may include one or more storage clusters (storage cluster), and each storage cluster can be by one or more Physical volume (bucket) composition, has also set up logical volume, each logical volume can correspond to one or more objects on storage cluster Reason volume, it is to be understood that storage cluster is that the memory space in multiple physical volumes (such as disk or hard disk) is aggregated into one The storage pool of unified access interface can be provided clients with, client can be accessed and be utilized storage to collect by the access interface Memory space on group, for example, storage cluster may include the physical volume (such as hard disk) that 5 memory spaces are 1G, then should Storage cluster can provide the memory space of 5G for client.
It is understood that physical volume is for storing data, including movable memory equipment and non-removable storage device, Such as: hard disk.Logical volume is by Storage Virtualization, and storage is no longer limited by the size of physical disk, polymerize multiple disks or magnetic Disk divides a logical volume into, is created that the logical volume come when data are not written in user, can not have to real allocated Amount of physical memory, but when arrived write-in data, dynamically distribute amount of physical memory.Logical volume is by multiple logics point District's groups at set, the logical partition in logical volume is continuous, but two corresponding physical extents of continuous logical partition It may be discontinuous.Such as: two corresponding physical extents of continuous logical partition are on different disks.Logical partition, which refers to, to be reflected The logic unit being mapped on physical extent, a logical partition can correspond to one or more physical extents.Physical extent is handle Physical volume is divided into what continuous, equal-sized storage cell obtained.For example, the logical volume that memory space is 30G is corresponding with The physical volume that three memory spaces are 10G, for the client 1011 in terminal 101, only logical volume is visible, i.e., objective The memory space of 30G can be shown on family end 1011, but data are specifically stored in the physical volume terminal of which 10G memory space 101 do not perceive.
Processing system 102 is mainly used for management and stores the storage cluster in equipment 103 and do accordingly to the request of terminal 101 Processing.For example, processing system 102 can be the meta data server in object storage system, metadata is provided for terminal 101 (metadata).Metadata is the information for describing data attribute, and metadata mainly includes the storage position of object in the application It sets.
Specifically, in the embodiment of the present application, processing system 102 can functionally divide into coordinator (coordinator) 1021, storage gateway (storage gateway) 1022 and index (index) 1023, below with reference to figure The function of 1 pair of three above component is introduced:
Coordinator 1021 can periodically inquire the residual capacity of each storage cluster in storage equipment 103.Specifically, it assists Device 1021 is adjusted to send inquiry instruction to each storage cluster, each storage cluster stores empty to the occupancy that coordinator 1021 feeds back itself Between, the data that each storage cluster is fed back can be stored in index 1023 by coordinator 1021, if the first storage cluster 1031 It has occupied memory space and has been greater than storage threshold value, then coordinator 1021 will acquire a new storage cluster, for example, coordinator The second storage cluster 1032 that memory space is less than storage threshold value is occupied in 1021 selection storage equipment 103, due to currently patrolling Volume volume logical address and the first storage cluster 1031 in physical volume physical address between there are mapping relations, patrolled according to existing The mapping relations between volume and physical volume are collected, terminal 101 needs the data stored that can only be stored in the first storage cluster 1031 In physical volume, therefore, the mapping that coordinator 1021 will also establish existing logic and be rolled onto physical volume in the second storage cluster 1032, association Adjust device 1021 that can be stored in the mapping relations in index 1023.By the above-mentioned means, coordinator 1021 is existing logic Volume establishes new mapping, i.e. mapping in logical volume and the second storage cluster 1032 between physical volume, it is thus achieved that patrolling Collect the dilatation of volume.
After coordinator 1021 completes dilatation to logical volume, storage gateway 1022 can be to the request of the initiation of terminal 101 It is handled.For example, terminal 101 initiates the request of storing data by client 1011 to storage gateway 1022, in the request Instruction information comprising data to be stored, the data of logical address and data to be stored in the instruction information comprising logical volume Mark, when the first storage cluster 1031 is when no longer there is residual capacity to carry out storing data, storage gateway 1022 is further inquired Index 1023 simultaneously determines the mapping in logical volume and the second storage cluster 1032 between physical volume, and then stores 1022 basis of gateway Data to be stored is stored in the physical volume of the second storage cluster 1032 by the mapping, and storage gateway 1022 is available later should Data are stored in the storage address in physical volume, in addition store Data Identification and the data that gateway 1022 will also establish the data Storage address between mapping relations, and by the mapping relations be recorded in index 1023 in, it is to be understood that instruction information It is known information for terminal.
When terminal 101 accesses to stored data by client 1011, gateway 1022 is stored according to terminal The Data Identification search index 1023 carried in 101 access requests sent, so that it is determined that client 1011 needs the number accessed According to the address in the storage cluster and the be stored in physical volume of the specific data at place.It is understood that each storage collection Group has corresponding port (endpoint), which can be forwarded to by storage gateway 1022 stores the data Port corresponding to storage cluster, and then the address that can be stored according to data in physical volume is realized to storing data Access.
Optionally, processing system 102 can also provide value-added service to stored data, for example, data smoothing migrates, Strange land duplication, data encryption, data compression, data deduplication etc..
It should be noted that generally non-relational database (the not only structured query of index 1023 Language, NoSQL), in addition it is also possible to be traditional relevant database, specifically herein without limitation.
It is understood that storage gateway 1022, coordinator 1021 and index 1023 typically operate in it is mutually independent In equipment, for example, storage gateway 1022, coordinator 1021 and index 1023 can be run in different physical machines, or Storage gateway 1022, coordinator 1021 and index 1023 be separately operable in different virtual machine (virtual machine, VM) or On person's container (container).
In addition, storage gateway 1022, coordinator 1021 or index 1023 can also be run in the same equipment, for example, Storage gateway 1022 and coordinator 1021 run on the same physical machine, virtual machine or container, or storage gateway 1022 and rope Draw 1023 run on same physical machine, virtual machine or container or coordinator 1021 and index 1023 run on same physical machine, Virtual machine or container, or storage gateway 1022, coordinator 1021 and index 1023 all run on same physical machine, virtual machine Or container, specifically herein without limitation.
Optionally, can also be run in physical machine by storing gateway 1022, coordinator 1021 and index 1023 by one or more It is a include processor and input/output interface chip in.
It should be noted that VM refer to by software simulate with complete hardware system function, and operate in one every From the complete computer in environment.By the new virtual mirror image of the existing operating system of generation, it has true virtual system The real duplicate function of windows system, into after virtual system, all operations are all completely new independent virtual at this It is carried out inside system, can save data with independently installed runs software, possess the independent table of oneself, it will not be to real system System generates any influence, and has the type operating system that can flexibly switch between existing system and virtual image.
Container is a kind of software sandbox, is a kind of security mechanism, is the isolation environment that active program provides, The resource that usual strict control program therein can access.
Optionally, terminal 101 may include in a variety of manners existing user equipment, car networking equipment, wearable device, Internet of things equipment or intelligent robot equipment etc., such as: mobile phone, tablet computer, smartwatch, car-mounted terminal, intelligent water The equipment such as table, intelligent electric meter.
Optionally, it can be counted by preset agreement between terminal 101, processing system 102 and storage equipment 103 According to transmission, for example, transmission control protocol (transmission control protocol, TCP), network protocol (internet Protocol, IP) etc..
For ease of understanding, figure is please referred to the dilation process of logical volume being introduced in this programme further below 2, one embodiment of the application data processing method includes:
201, processing system sends inquiry request to storage equipment.
In the embodiment of the present application, processing system periodically (such as once a day) can send inquiry to storage equipment and ask It asks, which is specifically used for the occupation rate of each storage cluster in inquiry storage equipment, for example, each storage cluster is with depositing Storage space accounts for the percentage of total memory space, or, the occupation rate of each storage cluster can also be expressed as each storage collection Group's residual memory space accounts for the percentage of total memory space.
202, storage equipment sends inquiry response to processing system.
Storage equipment obtains the occupation rate of local each storage cluster according to the inquiry request that processing system is sent, and will inquiry As a result processing system is fed back to.Processing system can save the query result received, also, system per treatment receives The new feedback of equipment is stored, the query result locally saved can be all updated.
203, processing system, which determines, has occupied the first storage cluster that memory space is greater than storage threshold value.
Processing system can know that each storage cluster occupies in storage equipment according to the query result of storage equipment feedback Rate, it should be noted that each storage cluster is previously provided with corresponding storage threshold value.For example, the storage threshold value can To be the threshold value of storage cluster itself occupation rate, then, if the occupation rate of storage cluster itself is more than that threshold value is assured that this Storage cluster is the first storage cluster.In addition to this, if the occupation rate of storage cluster itself is close but and be less than threshold value and (such as account for There is rate also poor 5% to reach threshold value), it can also determine that the storage cluster is the first storage cluster, specifically herein without limitation.
In addition, the threshold value of occupation rate corresponding to each storage cluster itself may be the same or different.In the application The quantity of first storage cluster can be one, be also possible to it is multiple, specifically herein without limitation.
It is understood that between physical volume and existing logical volume in the first storage cluster, there are mapping relations.
204, processing system, which determines, has occupied the second storage cluster that memory space is less than storage threshold value.
Processing system will determine that having occupied memory space is less than storage threshold value after the first storage cluster has been determined in next step The second storage cluster.Specifically, processing system can according to storage equipment feedback newest query result from occupied storage Space, which is less than in the storage cluster of storage threshold value, selects the storage cluster of occupation rate minimum (i.e. residual memory space is most) preferential As the second storage cluster.
Optionally, processing system can also determine the second storage cluster according to preset sequence, this is ordered as setting in advance The sequencing of each storage cluster arrangement in the storage cluster set set, then, preferential storage cluster is ranked in the sequence i.e. For the second storage cluster.
205, processing system establishes the mapping in logical volume and the second storage cluster between physical volume.
It is understood that processing system, which needs to establish logic, is rolled onto the mapping of the second storage cluster to complete to logical volume Dilatation.Specifically, multiple physical volumes can have been divided in the second storage cluster, then processing system can establish logic respectively Mapping in volume and the second storage cluster between each physical volume, in addition, newly-built mapping can be stored in this by processing system In the map listing on ground.
It is understood that logical volume is by Storage Virtualization, multiple disks or disk partition can aggregate into one and patrol Volume volume, logical volume is to the visible volume of client, wherein the physical volume in logical volume and the first storage cluster in this programme it Between equally exist mapping relations.
In the embodiment of the present application, the occupation rate of each storage cluster in processing system inquiry storage equipment, and determine occupation rate Greater than the first storage cluster of storage threshold value, then, processing system selects occupation rate to be less than the second storage cluster for storing threshold value, In turn, establish the mapping in existing logical volume and the second storage cluster between physical volume, by the above-mentioned means, existing logical volume with There are mapping relations between the physical volume of first storage cluster, client needs the data stored that can be first stored in the first storage In the physical volume of cluster, when the memory space inadequate of the first storage cluster, processing system has just selected memory space abundance Second storage cluster, and the mapping in existing logical volume and the second storage cluster between physical volume is created, it is thus achieved that existing There is the dilatation of logical volume, it should be noted that the mapping between logical volume and each physical volume is independent from each other, and logical volume can With corresponding one or more physical volumes, and each physical volume can only correspond to a unique logical volume, for example, patrolling in this programme Collecting volume is logical volume A, and there are mapping relations between logical volume A and the physical volume of the first storage cluster, and in the second storage cluster Part physical volume (such as accounting for the physical volume in 30% space of the second storage cluster) and logical volume B between there are mapping relations, then, In order to realize the dilatation to logical volume A, it is necessary to establish logical volume A and (such as be accounted for another part physical volume in the second storage cluster Second storage cluster residue 70% memory space physical volume) between mapping relations.
The dilation process of the logical volume in this programme is introduced above, in the following, after the completion of to dilatation in this programme, eventually The process for being stored and being accessed to data is held to be introduced, referring to Fig. 3, one embodiment of the application data processing method Include:
301, terminal sends data storage request to processing system.
In the embodiment of the present application, the client run in terminal can be by hypertext transfer protocol (hyperText Transfer protocol, HTTP) with processing system establish connection, for example, client is with the domain name or IP address of processing system The storage request of data is sent as endpoint, processing system can monitor the data storage that client is sent in endpoint Request.
It should be noted that including the instruction information of data to be stored in data storage request, wherein instruction information is specific It may include the Data Identification of data to be stored and the logical address to the visible logical volume of terminal, it is to be understood that refer to Show that information is known information for terminal.
302, processing system obtains the physical address of physical volume in the second storage cluster.
Processing system handles the data storage request received, specifically, due to including to patrol in data storage request The logical address of volume is collected, processing system can determine second according to the mapping in logical volume and the second storage cluster between physical volume The physical address of physical volume in storage cluster.
303, the data storage request for carrying physical address is sent to storage equipment by processing system.
The data storage that is added to of the physical address of physical volume is asked in the second storage cluster that processing system can will acquire In asking, and then forward the request to the endpoint of the second storage cluster in storage equipment.
304, storage equipment stores data in the physical volume of the second storage cluster.
The second storage cluster in storage equipment receives the data storage request of processing system forwarding, then, the second storage Cluster can store data in local physical volume according to the physical address carried in request.
It should be noted that after the mapping in logical volume and the second storage cluster between physical volume is established, if first deposits Accumulation still has remaining memory space (although occupation rate of such as the first storage cluster is more than threshold value but there are also available spaces), excellent Selection of land, data to be stored will preferentially be stored in the first storage cluster, reach storage cap to the first storage cluster, further again will be to The remainder of storing data is stored in the second storage cluster；Specifically, data to be stored can be split as the first data and second Data, wherein the memory space that the first data occupy is less than or equal to the residual memory space of the first storage cluster, processing system The physics of physical volume in the first storage cluster can be determined according to the mapping in logical volume and the first storage cluster between physical volume Address, and the physical address of physical volume in the first storage cluster is added in data storage request, and then forward the request extremely The endpoint of first storage cluster, the first storage cluster can be stored data according to the physical address carried in the request In local physical volume, then the second data are just stored in accordingly in the physical volume of the second storage cluster, specific steps and step Rapid 302 to 304 is similar, and details are not described herein again.
305, the storage address of data is sent to processing system by storage equipment.
After data are stored in the physical volume of the second storage cluster, storage equipment will record the storage address of the data, later The storage address can be sent to processing system by storage equipment, and be saved by processing system, and specifically, storage address can The uniform resource locator (uniform resource locator, URL) of the data is thought, in addition, processing system will also note Record the mapping relations between Data Identification and the address data memory.
306, terminal sends the access request of data to processing system.
After the data storage that terminal needs to store is completed, subsequent terminal also needs to visit stored data It asks.So firstly, terminal specifically can carry number to the access request that processing system sends data in the request According to mark, it is to be understood that terminal sends data to processing system in the sending method of data access request and step 301 The mode for storing request is similar, and details are not described herein again.
307, the storage address of processing system inquiry data.
After processing system receives the data access request of terminal transmission, it can be looked into according to the Data Identification carried in the request Look for the corresponding storage address of data.
308, the data access request for carrying storage address is sent to storage equipment by processing system.
The address data memory that processing system will acquire is added in the access request of data, and then the request is forwarded The endpoint of the second storage cluster into storage equipment.
309, storage equipment handles data.
The second storage cluster in storage equipment can be according to the data access request received to being stored in local data It performs corresponding processing, for example, terminal is modified, deleted or sent data to data.
It should be noted that may be all stored in original first storage cluster and the second storage cluster newly determined Terminal needs the data stored, and equally also record has reflecting between physical volume in logical volume and the first storage cluster to processing system It penetrates and storage address of the data in the first storage cluster, therefore, the data stored in the first storage cluster of terminal access Process and step 306 are to 309 similar.In addition, data corresponding to Data Identification, which are possible to a part, is stored in the first storage collection Group, another part is stored in the second storage cluster, then processing system can find the data first according to Data Identification The first storage address and the data of storage cluster are in the second storage address of the second storage cluster, and then processing system is by One storage address is added in data access request and forwards the request to the endpoint of the first storage cluster, and processing system is also Second storage address is added in data access request and the request is forwarded to deposit to the endpoint of the second storage cluster, first Accumulation and the second storage cluster are performed corresponding processing according to the request received to local data are stored in respectively.
In the embodiment of the present application, since in the dilation process of above-mentioned logical volume, processing system establishes logic and is rolled onto The mapping of physical volume in two storage clusters.Therefore, it after processing system receives the data storage request that terminal is sent, can indicate to deposit Terminal is needed the data stored to be stored in the physical volume of the second storage cluster by storage equipment according to the mapping, has been stored in data Cheng Hou, processing system can recorde the storage address of the data, when terminal is needed to stored data access, in terminal For the client of operation, only logical volume is visible, and the first storage cluster and the second storage cluster all with logical volume it Between there are mapping relations, therefore, the data in the first storage cluster just no longer need to migrate to the second storage cluster, reduce data The workload of migration.
For ease of understanding, the method for data processing in the application is retouched in detail with a concrete application scene below It states, referring to Fig. 4, Fig. 4 is a flow diagram of data processing method in the application application scenarios, specifically:
401-402, coordinator periodically send inquiry request to storage equipment, are specifically used for each in inquiry storage equipment The occupation rate of storage cluster.The occupation rate of each storage cluster of equipment query is stored, and obtained query result is added to inquiry Coordinator is sent in response.Specifically, which can be as shown in table 1 below, it is to be understood that each storage cluster It is all previously provided with corresponding storage condition, for example, each storage cluster is provided with the threshold value of itself occupation rate, Ke Yili It solves, the threshold value of itself occupation rate set by each storage cluster is not necessarily invariable, according to the actual situation Difference, can accordingly adjust the threshold value of occupation rate corresponding to each storage cluster.
Table 1
Storage cluster Occupancy Threshold value
A 73% 70%
B 50% 80%
C 87% 90%
D 30% 80%
E 10% 80%
Optionally, the query result received can be stored in index by coordinator, also, coordinator receives inquiry every time After response, the query result in index can be updated.
403, coordinator determines that occupation rate is greater than the first storage cluster of preset threshold according to query result, is exemplified by Table 1, It can determine that storage cluster A is the first storage cluster, (such as occupation rate is also if the occupation rate of storage cluster is close to itself threshold value Having 5% is more than threshold value) it can be considered as and be unsatisfactory for storage condition, then can also determine that storage cluster C is the first storage cluster.
It is understood that default storage condition corresponding to each storage cluster itself may be the same or different, The quantity of the first storage cluster can be one in the application, be also possible to it is multiple, specifically herein without limitation.
404, coordinator determines that occupation rate is less than the second storage cluster of preset threshold, and specifically, coordinator can foundation Query result selects occupation rate minimum (i.e. residual memory space is most) from the storage cluster that occupation rate is less than preset threshold Storage cluster is preferentially used as the second storage cluster, is exemplified by Table 1, it can be seen that the occupation rate of storage cluster E is minimum, then just Storage cluster E be can choose as the second storage cluster.
Optionally, coordinator can also determine the second storage cluster according to preset sequence, this is ordered as presetting Storage cluster set in occupation rate be less than preset threshold each storage cluster arrange sequencing, be exemplified by Table 1, storage collection The occupation rate of group B, storage cluster D and storage cluster E are less than preset threshold, then the sequencing of these three storage clusters arrangement It preferential, storage cluster D can preferentially will be determined according to this sequence coordinator secondly, be storage cluster E again for storage cluster B Storage cluster B is the second storage cluster, it is to be understood that the sequence can be a kind of artificial pre-set arrangement mode, Or arranged according to the sequencing that storage cluster generates, specifically sortord is subject to practical application, herein not It limits.
Optionally, coordinator can also determine the second storage collection according to the residual memory space size of each storage cluster Group preferentially selects the maximum storage cluster of residual memory space as the second storage cluster, guarantee the second storage collection as far as possible Group has enough memory space storage data to be stored, is exemplified by Table 1, it is assumed that itself capacity of storage cluster D is greater than storage collection Itself capacity of group E, although the occupation rate of storage cluster E is lower, but the residual memory space of storage cluster D is bigger, at this point, It can preferentially select storage cluster D for the second storage cluster.
405, coordinator establishes the mapping in logical volume and the second storage cluster between physical volume, it is to be understood that the One or more physical volumes can be divided on two storage clusters, coordinator will establish respectively logic and be rolled onto each physical volume All there are mapping relations in mapping, i.e. each of logical volume and the second storage cluster physical volume.
Specifically, which can be as shown in table 2 below, it is assumed that has logical volume A and logical volume B, for customer end A, patrol Volume volume A be visible, for customer end B, logical volume B is visible, logical volume A respectively with physical volume 1, physical volume 2 and object There is mapping between reason volume 3, there is mapping between logical volume B and physical volume 4.
Table 2
It should be noted that coordinator can also establish physical volume and reflecting between the endpoint of storage cluster where it Relationship is penetrated, which can be as shown in table 3 below, and physical volume 1 is established on storage cluster M, and physical volume 4 is established to be collected in storage On group N, specifically, the form of the endpoint of storage cluster can be the domain name or IP of storage cluster M, for example, physical volume 1 is right Should there are the domain name or IP of storage cluster M, physical volume 4 is corresponding with the domain name or IP of storage cluster N.
It should be noted that if coordinator monitors the occupation rate of the second storage cluster during data storage Greater than preset threshold, then coordinator will determine third storage cluster, and logical volume and third storage cluster physical volume are established Between mapping relations, the further dilatation to logical volume, the description class of detailed process and step 401 to 405 are realized with this Seemingly.
Table 3
Physical volume The port of storage cluster
Physical volume 1 The domain name or IP of storage cluster M
Physical volume 4 The domain name or IP of storage cluster N
406, the map information of generation can be stored in index by coordinator, in addition, physical volume and storage cluster port Between mapping relations be stored in index, for example, the table 2 of the example above and table 3 are stored in index by coordinator.
407, terminal sends the storage request of data to storage gateway, wherein includes instruction information in the request, specifically Ground, the storage for running on domain name or IP address as endpoint transmission data of the client of terminal to store gateway are requested.
Optionally, which can be transmitted based on HTTP1.1, and following letter can also be specifically included in request Breath:
The domain name or port numbers for storing gateway, for example, host:endpoint.storagegateway.com；
The date or time sent is requested, for example, 2010 08:12:31GMT of date:Tue, 15 Nov；
The certificate of authority of HTTP authorization, for example, Authorization:authorization string.
408-409, storage gateway send inquiry request to index, for rolling up between physical volume from query logic in index Mapping, specifically, storage gateway can be inquired from index according to the logical address of logical volume in instruction information and obtain this and patrol Mapping in volume address and the second storage cluster between the physical address of each physical volume, for example, in table 2 logical volume A respectively with object Mapping between reason volume 1, physical volume 2 and physical volume 3.
410, the data storage request for carrying physical address is sent to storage equipment by storage gateway, specifically, storage The physical address of physical volume is added in data storage request in the second storage cluster that gateway can will acquire, and stores Gateway can determine the endpoint of the corresponding storage cluster of physical volume by search index, for example, physical volume is corresponding in table 3 The domain name or IP of storage cluster M, and then store gateway and forward the request to the second storage cluster in storage equipment endpoint。
411, the second storage cluster stored in equipment stores data in physical volume according to physical address, needs to illustrate If dividing in the second storage cluster has multiple physical volumes, then needing successively to select physical volume in a certain order Storing data, for example, the idle capacity of 3 these three physical volumes of physical volume 1, physical volume 2 and physical volume is different, the free time holds Amount from physical volume 1, physical volume 2, physical volume 3 is more to followed successively by less, then the physical volume 1 that can preferentially select idle capacity most Storing data, 2 storing data of reselection physical volume after being filled with to physical volume 1, and so on.
412-413, data are stored in after physical volume, and the storage address of data can be sent to storage net by storage equipment It closes, and is recorded the storage address in the index by storage gateway, the unified resource which specifically can be data is fixed Position symbol (uniform resource locator, URL), it is to be understood that can be looked into the index according to the mark of data Find the corresponding storage address of the data.
Specifically, the storage address of data can be as shown in table 4 below in index, Data Identification can by logical volume name and The mode of data name combination indicates.For example, data 1 and the corresponding storage address of data 2 are the first URL and second URL, the corresponding physical location of data 3 are the 3rd URL.
Table 4
Data Identification Storage address
Logical volume A/ data 1 First URL
Logical volume A/ data 2 2nd URL
Logical volume B/ data 3 3rd URL
414, after the physical volume that data are stored in the second storage cluster, if desired terminal accesses the data, and terminal is wanted The access request of data is sent to storage gateway, wherein include Data Identification in the request, specifically, run on terminal Client using store gateway domain name or IP address as endpoint send data access request.
Optionally, which can be transmitted based on HTTP1.1, and following letter can also be specifically included in request Breath:
The domain name or port numbers for storing gateway, for example, host:endpoint.storagegateway.com；
The date or time sent is requested, for example, 2010 08:12:31GMT of date:Tue, 15 Nov；
The certificate of authority of HTTP authorization, for example, Authorization:authorization string；
The range of request data entity, for example, range:bytes=500-999.
After 415-416, storage gateway receive the data access request of terminal, it can be determined according to Data Identification search index The storage address of data.
417, the data access request for carrying storage address is sent to storage equipment by storage gateway, specifically, storage The storage address that gateway can will acquire is added in data access request, and storing gateway can be true by search index Determine the endpoint of the corresponding storage cluster of physical volume, for example, in table 3 the corresponding storage cluster M of physical volume domain name or IP, into And store the endpoint that gateway forwards the request to the second storage cluster in storage equipment.
418, storage equipment data are handled according to data access request, for example, modify to data, delete or Person sends data to terminal.
In the embodiment of the present application, there are mapping relations between existing logical volume and the first storage cluster, and current first deposits The memory space inadequate of accumulation then processing system has selected the second storage cluster of memory space abundance, and has created existing There is the mapping in logical volume and the second storage cluster between physical volume, it is thus achieved that the dilatation to existing logical volume, in addition, right For the client run in terminal, only logical volume is visible, and the first storage cluster and the second storage cluster all with patrol There are mapping relations between volume volume, and therefore, the data in the first storage cluster just no longer need to migrate to the second storage cluster, reduce The workload of Data Migration.
The data processing method in the embodiment of the present application is described above, below to the place in the embodiment of the present application Reason system is described:
Referring to Fig. 5, one embodiment of processing system includes expansion unit 501, storage unit in the embodiment of the present application 502 and indexing units 503, specifically, expansion unit can be coordinator, and storage unit can be storage gateway, Index List Member can be rope, be introduced separately below:
Expansion unit 501, for determining the first storage cluster from storage cluster set, in the first storage cluster first There are the first mapping relations, the logical address that the first mapping relations are used to indicate logical volume is corresponding between physical volume and logical volume First physical address of the first physical volume；When the memory space of occupancy of the first storage cluster is greater than storage threshold value, from storage Determine that the second storage cluster, the memory space of occupancy of the second storage cluster are less than storage threshold value in cluster set；And it establishes and patrols The second mapping relations in volume and the second storage cluster between the second physical volume are collected, the second mapping relations are used to indicate logical volume Second physical address of corresponding second physical volume of logical address；
Indexing units 503, for storing the first mapping relations and the second mapping relations of the foundation of expansion unit 501.
Optionally, in some possible embodiments,
Expansion unit 501 can each storage cluster into storage cluster send inquiry request, respectively deposited specifically for inquiry The occupation rate of accumulation, expansion unit judge the occupation rate of each storage cluster, and determine that occupation rate is greater than preset threshold Storage cluster be the first storage cluster.
Optionally, in some possible embodiments,
Expansion unit 501 determines that occupation rate is less than the storage collection of preset threshold from each storage cluster occupation rate got Group is the second storage cluster, if the storage cluster for having multiple occupation rates to be less than preset threshold, then expansion unit 501 can will be each Storage cluster is arranged according to the sequence of occupation rate from big to small or from small to large, and then expansion unit 501 can be according to this Sequence preferentially selects the smallest storage cluster of occupation rate as the second storage cluster.
Storage unit 502 includes data to be stored in the first data storage request for receiving the first data storage request Instruction information, instruction information include logical volume logical address；When the residual memory space of the first storage cluster is less than wait deposit When storing up the memory space that data occupy, then data to be stored is stored in the second physical volume of the second storage cluster.
Optionally, in some possible embodiments, storage unit 502 are specifically used for:
The first data storage request is received, includes the instruction information of data to be stored, instruction in the first data storage request Information includes the logical address of logical volume；When the residual memory space of the first storage cluster is less than the storage that data to be stored occupies When space, data to be stored is stored in the second physical volume of the second storage cluster.
Optionally, in some possible embodiments, storage unit 502 are specifically used for:
It, will be corresponding with the logical address that data to be stored occupies when the first storage cluster does not have residual memory space Third physical address, which is added in the first data storage request, obtains the second data storage request, corresponding to third physical address Memory space is the part memory space of memory space corresponding to the second physical address；And second is sent to the second storage cluster Data storage request, the second data storage request are used to indicate the second storage cluster according to third physical address for data to be stored It is stored in the second physical volume.
Optionally, in some possible embodiments, storage unit 502 are specifically used for:
Data to be stored is split as the first data and the second data, the memory space that the first data occupy is less than or equal to the The residual memory space of one storage cluster；Corresponding 4th physical address of the logical address occupied with the first data is added to Obtain third data storage request in one data storage request, memory space corresponding to the 4th physical address is first physically The part memory space of memory space corresponding to location；And third data storage request, third number are sent to the first storage cluster The first storage cluster is used to indicate according to storage request, and the first data are stored in by the first physical volume according to the 4th physical address；It will be with Corresponding 5th physical address of logical address that second data occupy, which is added in the first data storage request, obtains the 4th data Storage is requested, and memory space corresponding to the 5th physical address is the part storage of memory space corresponding to the second physical address Space；And the 4th data storage request is sent to the second storage cluster, the 4th data storage request is used to indicate the second storage collection Second data are stored in the second physical volume according to the 5th physical address by group.
Optionally, in some possible embodiments, storage unit 502 are also used to:
When data to be stored has been stored in the second physical volume, from the second storage cluster obtain storing data second Storage address in physical volume；The third mapping relations between Data Identification and storage address are established, third mapping relations are used for Designation date identifies storage address of the corresponding data in the second physical volume.
Indexing units 503 are also used to store the third mapping relations of the foundation of storage unit 502.
Optionally, in some possible embodiments, storage unit 502 are also used to:
The first data access request is received, includes the Data Identification of data to be visited in the first data access request；According to Third mapping relations determine storage address corresponding with Data Identification；Storage address is added in the first data access request and is obtained To the second data access request；And the second data access request is sent to the second storage cluster.
In the embodiment of the present application, expansion unit 501, storage unit 502 and indexing units 503 can specifically execute Fig. 2, The movement of all or part performed by processing system in Fig. 3 or embodiment illustrated in fig. 4, specific details are not described herein again.
Specifically, the expansion unit in above-described embodiment can be further subdivided into the first determination unit, the second determining list Member, first establishing unit, second acquisition unit, judging unit and third determination unit；Storage unit can be segmented further Unit is established for receiving unit, requesting processing, first acquisition unit and second.
It is described below with reference to above-mentioned each unit:
Referring to Fig. 6, another embodiment of processing system includes: in the embodiment of the present application
First determination unit 601, for determining the first storage cluster from storage cluster set, in the first storage cluster There are the first mapping relations, the first mapping relations are used to indicate the logical address pair of logical volume between first physical volume and logical volume First physical address of the first physical volume answered；
Second determination unit 602, for when the memory space of occupancy of the first storage cluster is greater than storage threshold value, from depositing Determine that the second storage cluster, the memory space of occupancy of the second storage cluster are less than storage threshold value in accumulation set；
First establishing unit 603, second for establishing in logical volume and the second storage cluster between the second physical volume reflects Relationship is penetrated, the second mapping relations are used to indicate the second physical address of corresponding second physical volume of logical address of logical volume.
In the embodiment of the present application, the first determination unit 601 determines the first storage cluster from storage cluster set, wherein The first physical volume in first storage cluster and there are the first mapping relations between logical volume, which is used to indicate First physical address of corresponding first physical volume of the logical address of logical volume, if the occupancy memory space of the first storage cluster Greater than storage threshold value, then the second determination unit 602 determines that having occupied memory space is less than storage threshold from storage cluster set Second storage cluster of value, and then first establishing unit 603 is established in logical volume and the second storage cluster between the second physical volume The second mapping relations, the second object of corresponding second physical volume of the logical address which is used to indicate logical volume Manage address.By the above-mentioned means, due to only having logical volume to be visible, i.e., terminal only needs to know that data are deposited for terminal The logical address of storage, and second in the first physical volume and the second storage cluster in logical volume and the first storage cluster All there are mapping relations between physical volume, when the insufficient space of the first storage cluster, data to be stored can be according to second Mapping relations are stored in the second storage cluster, are avoided storage service and are interrupted because of memory space inadequate.
Optionally, in some possible embodiments,
Receiving unit 604 includes data to be stored in the first data storage request for receiving the first data storage request Instruction information, instruction information include logical volume logical address；
Requesting processing 605 is less than what data to be stored occupied for the residual memory space when the first storage cluster When memory space, data to be stored is stored in the second physical volume of the second storage cluster.
Optionally, in some possible embodiments, requesting processing 605 are specifically used for:
It, will be corresponding with the logical address that data to be stored occupies when the first storage cluster does not have residual memory space Third physical address, which is added in the first data storage request, obtains the second data storage request, corresponding to third physical address Memory space is the part memory space of memory space corresponding to the second physical address；And second is sent to the second storage cluster Data storage request, the second data storage request are used to indicate the second storage cluster according to third physical address for data to be stored It is stored in the second physical volume.
Optionally, in some possible embodiments, requesting processing 605 are specifically used for:
Data to be stored is split as the first data and the second data, the memory space that the first data occupy is less than or equal to the The residual memory space of one storage cluster；Corresponding 4th physical address of the logical address occupied with the first data is added to Obtain third data storage request in one data storage request, memory space corresponding to the 4th physical address is first physically The part memory space of memory space corresponding to location；And third data storage request, third number are sent to the first storage cluster The first storage cluster is used to indicate according to storage request, and the first data are stored in by the first physical volume according to the 4th physical address；It will be with Corresponding 5th physical address of logical address that second data occupy, which is added in the first data storage request, obtains the 4th data Storage is requested, and memory space corresponding to the 5th physical address is the part storage of memory space corresponding to the second physical address Space；And the 4th data storage request is sent to the second storage cluster, the 4th data storage request is used to indicate the second storage collection Second data are stored in the second physical volume according to the 5th physical address by group.
Optionally, in some possible embodiments,
First acquisition unit 606, for when data to be stored has been stored in the second physical volume, from the second storage cluster Obtain storage address of the storing data in the second physical volume；
Second establishes unit 607, and the third mapping relations for establishing between Data Identification and storage address simultaneously store, the Three mapping relations are used to indicate storage address corresponding to the storing data being stored in the second physical volume.
Optionally, in some possible embodiments, the receiving unit 604, is also used to:
The first data access request is received, includes the Data Identification of data to be visited in the first data access request；
Requesting processing 605, is also used to:
Storage address corresponding with Data Identification is determined according to third mapping relations；Storage address is added to the first data The second data access request is obtained in access request；And the second data access request is sent to the second storage cluster.
Optionally, in some possible embodiments,
Second acquisition unit 608, for obtaining the occupation rate of each storage cluster from storage cluster set；
Judging unit 609, for judging whether the occupation rate of the first storage cluster is greater than preset threshold；
Third determination unit 610, for determining the first storage when the occupation rate of the first storage cluster is greater than preset threshold The memory space of occupancy of cluster is greater than storage threshold value.
Optionally, in some possible embodiments, second determination unit 602, is specifically used for:
The storage cluster for determining that occupation rate is less than preset threshold is the second storage cluster.
In the embodiment of the present application, each unit module in processing system can specifically execute to be implemented shown in Fig. 2, Fig. 3 or Fig. 4 The movement of all or part performed by processing system in example, specific details are not described herein again.
The processing system in the embodiment of the present application is described from the angle of modular functionality entity above, below from The angle of hardware handles is applied the processing system in example to the application and is described:
Referring to Fig. 7, the processing system 700 can generate bigger difference because configuration or performance are different, may include One or more central processing units (central processing units, CPU) 722 (for example, one or more Processor) and memory 732, the storage medium 730 (such as one of one or more storage application programs 742 or data 744 A or more than one mass memory unit).Wherein, memory 732 and storage medium 730 can be of short duration storage or persistently deposit Storage.The program for being stored in storage medium 730 may include one or more modules (diagram does not mark), and each module can be with Including being operated to the series of instructions in processing system.Further, central processing unit 722 can be set to and storage medium 730 communications execute the series of instructions operation in storage medium 730 in processing system 700.
Processing system 700 can also include one or more power supplys 726, one or more wired or wireless nets Network interface 750, one or more input/output interfaces 756, and/or, one or more operating systems 741, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM etc..
Above-described embodiment Fig. 2 specific steps as performed by processing system into Fig. 4 can be based on the processing shown in Fig. 7 System structure.
It is apparent to those skilled in the art that for convenience and simplicity of description, the system of foregoing description, The specific work process of device and unit, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.
In several embodiments provided herein, it should be understood that disclosed system, device and method can be with It realizes by another way.For example, the apparatus embodiments described above are merely exemplary, for example, the unit It divides, only a kind of logical function partition, there may be another division manner in actual implementation, such as multiple units or components It can be combined or can be integrated into another system, or some features can be ignored or not executed.Another point, it is shown or The mutual coupling, direct-coupling or communication connection discussed can be through some interfaces, the indirect coupling of device or unit It closes or communicates to connect, can be electrical property, mechanical or other forms.
The unit as illustrated by the separation member may or may not be physically separated, aobvious as unit The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple In network unit.It can select some or all of unit therein according to the actual needs to realize the mesh of this embodiment scheme 's.
It, can also be in addition, each functional unit in each embodiment of the application can integrate in one processing unit It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list Member both can take the form of hardware realization, can also realize in the form of software functional units.
If the integrated unit is realized in the form of SFU software functional unit and sells or use as independent product When, it can store in a computer readable storage medium.Based on this understanding, the technical solution of the application is substantially The all or part of the part that contributes to existing technology or the technical solution can be in the form of software products in other words It embodies, which is stored in a storage medium, including some instructions are used so that a computer Equipment (can be personal computer, server or other network equipments etc.) executes the application Fig. 2 each implementation into Fig. 6 The all or part of the steps of example the method.And storage medium above-mentioned include: USB flash disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic or disk etc. it is various It can store the medium of program code.
The above, above embodiments are only to illustrate the technical solution of the application, rather than its limitations；Although referring to before Embodiment is stated the application is described in detail, those skilled in the art should understand that: it still can be to preceding Technical solution documented by each embodiment is stated to modify or equivalent replacement of some of the technical features；And these It modifies or replaces, the range of each embodiment technical solution of the application that it does not separate the essence of the corresponding technical solution.

Claims (18)

1. a kind of data processing method characterized by comprising
The first storage cluster is determined from storage cluster set, wherein the first physical volume in first storage cluster with patrol Collecting has the first mapping relations, the corresponding institute of the logical address that first mapping relations are used to indicate the logical volume between volume State the first physical address of the first physical volume；
If the memory space of occupancy of first storage cluster is greater than storage threshold value, determined from the storage cluster set Second storage cluster, wherein the memory space of occupancy of second storage cluster is less than the storage threshold value；
Establish the second mapping relations in the logical volume and second storage cluster between the second physical volume, wherein described Second mapping relations are used to indicate the second physical address of corresponding second physical volume of logical address of the logical volume.
2. the method according to claim 1, wherein the method also includes:
The first data storage request is received, includes the instruction information of data to be stored in first data storage request, it is described Indicate that information includes the logical address of the logical volume；If the residual memory space of first storage cluster is less than described wait deposit The memory space that data occupy is stored up, then is stored in the data to be stored in the second physical volume of second storage cluster.
3. if according to the method described in claim 2, the it is characterized in that, residual memory space of first storage cluster Less than the memory space that the data to be stored occupies, then the data to be stored is stored in the of second storage cluster In two physical volumes, comprising:
If first storage cluster does not have residual memory space, according to first data storage request and with described the Two mapping relations generate the second data storage request, wherein carry and the number to be stored in second data storage request According to the corresponding third physical address of the logical address of occupancy；
Second data storage request is sent to second storage cluster, wherein second data storage request is used for Indicate that the data to be stored is stored in second physical volume according to the third physical address by second storage cluster.
4. according to the method described in claim 2, it is characterized in that, described be stored in described second for the data to be stored and deposit Include: in second physical volume of accumulation
The data to be stored is split as the first data and the second data, the memory space that first data occupy is less than etc. In the residual memory space of first storage cluster；
Third data storage request is generated according to first data storage request and first mapping relations, wherein described The 4th physical address corresponding with the logical address that first data occupy is carried in third data storage request；
The third data storage request is sent to first storage cluster, wherein the third data storage request is used for Indicate that first data are stored in first physical volume according to the 4th physical address by first storage cluster；
The 4th data storage request is generated according to first data storage request and second mapping relations, wherein described The 5th physical address corresponding with the logical address that second data occupy is carried in 4th data storage request；
The 4th data storage request is sent to second storage cluster, wherein the 4th data storage request is used for Indicate that second data are stored in second physical volume according to the 5th physical address by second storage cluster.
5. according to the method described in claim 2, it is characterized in that, the instruction information also include the data to be stored extremely A few Data Identification, each Data Identification correspond at least one storage address, the method also includes:
When the data to be stored has been stored in second physical volume, number has been stored from second storage cluster acquisition According to the storage address in second physical volume；
The third mapping relations established between the Data Identification and the storage address simultaneously store, and the third mapping relations are used In storage address of the corresponding data of the instruction Data Identification in second physical volume.
6. according to the method described in claim 5, it is characterized in that, the method also includes:
The first data access request is received, includes the Data Identification of data to be visited in first data access request；
Storage address corresponding with the Data Identification is determined according to the third mapping relations；
According to the storage address and first data access request, the second data access request is generated, wherein described the The storage address is carried in two data access requests；
Second data access request is sent to second storage cluster.
7. method according to any one of claim 1 to 6, which is characterized in that determined from the storage cluster set Before second storage cluster, the method also includes:
The occupation rate of each storage cluster is obtained from the storage cluster set；
Judge whether the occupation rate of first storage cluster is greater than preset threshold, if so, determining first storage cluster The memory space of occupancy be greater than the storage threshold value.
8. the method according to the description of claim 7 is characterized in that determining the second storage cluster from the storage cluster set Include:
The storage cluster for determining that occupation rate is less than the preset threshold is second storage cluster.
9. a kind of processing system, which is characterized in that including expansion unit and indexing units；
The expansion unit, for determining the first storage cluster from storage cluster set, wherein in first storage cluster The first physical volume and logical volume between there are the first mapping relations, first mapping relations are used to indicate the logical volume First physical address of corresponding first physical volume of logical address；In the occupancy memory space of first storage cluster When greater than storage threshold value, the second storage cluster, the occupancy of second storage cluster are determined from the storage cluster set Memory space is less than the storage threshold value；And it establishes in the logical volume and second storage cluster between the second physical volume Second mapping relations, corresponding second physical volume of the logical address that second mapping relations are used to indicate the logical volume The second physical address；
The indexing units, for storing first mapping relations and second mapping relations.
10. processing system according to claim 9, which is characterized in that the processing system further includes storage unit；
The storage unit includes number to be stored in first data storage request for receiving the first data storage request According to instruction information, it is described instruction information include the logical volume logical address；When the residue of first storage cluster is deposited When storing up space less than the memory space that the data to be stored occupies, then the data to be stored is stored in second storage In second physical volume of cluster.
11. processing system according to claim 10, which is characterized in that the storage unit is specifically used for:
When first storage cluster do not have residual memory space when, then according to first data storage request and with it is described Second mapping relations generate the second data storage request, carry in second data storage request and account for the data to be stored The corresponding third physical address of logical address；
Second data storage request is sent to second storage cluster, second data storage request is used to indicate institute It states the second storage cluster and the data to be stored is stored in by second physical volume according to the third physical address.
12. processing system according to claim 10, which is characterized in that the storage unit is specifically used for:
The data to be stored is split as the first data and the second data, the memory space that first data occupy is less than etc. In the residual memory space of first storage cluster；
Third data storage request, the third number are generated according to first data storage request and first mapping relations According to carrying the 4th physical address corresponding with the logical address that first data occupy in storage request；
The third data storage request is sent to first storage cluster, the third data storage request is used to indicate institute It states the first storage cluster and first data is stored in by first physical volume according to the 4th physical address；
The 4th data storage request, the 4th number are generated according to first data storage request and second mapping relations According to carrying the 5th physical address corresponding with the logical address that second data occupy in storage request；
The 4th data storage request is sent to second storage cluster, the 4th data storage request is used to indicate institute It states the second storage cluster and second data is stored in by second physical volume according to the 5th physical address.
13. processing system according to claim 10, which is characterized in that the storage unit is also used to,
When the data to be stored has been stored in second physical volume, number has been stored from second storage cluster acquisition According to the storage address in second physical volume；
The third mapping relations established between the Data Identification and the storage address simultaneously store, and the third mapping relations are used In storage address of the corresponding data of the instruction Data Identification in second physical volume；
The indexing units are also used to, and store the third mapping relations.
14. processing system according to claim 13, which is characterized in that the storage unit is also used to,
The first data access request is received, includes the Data Identification of data to be visited in first data access request；
Storage address corresponding with the Data Identification is determined according to the third mapping relations；
According to the storage address and first data access request, the second data access request, second number are generated According to carrying the storage address in access request；
Second data access request is sent to second storage cluster.
15. the processing system according to claim 9 to 14, which is characterized in that the expansion unit is specifically used for, from described The occupation rate of each storage cluster is obtained in storage cluster set；
Judge whether the occupation rate of first storage cluster is greater than preset threshold；
When the occupation rate of first storage cluster is greater than the preset threshold, the occupancy of first storage cluster is determined Memory space is greater than storage threshold value.
16. processing system according to claim 15, which is characterized in that the expansion unit is also used to,
The storage cluster for determining that occupation rate is less than the preset threshold is second storage cluster.
17. a kind of processing system characterized by comprising
Memory, for storing program；
Processor, for executing the described program of the memory storage, when described program is performed, the processor is used for Execute such as step described in any one of claims 1-8.
18. a kind of computer readable storage medium, including instruction, when described instruction is run on computers, so that computer Execute the method as described in any one of claim 1-8.
CN201810626541.3A 2018-06-15 2018-06-15 Data processing method and processing system Active CN109085999B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810626541.3A CN109085999B (en) 2018-06-15 2018-06-15 Data processing method and processing system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810626541.3A CN109085999B (en) 2018-06-15 2018-06-15 Data processing method and processing system

Publications (2)

Publication Number Publication Date
CN109085999A true CN109085999A (en) 2018-12-25
CN109085999B CN109085999B (en) 2022-04-22

Family

ID=64839717

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810626541.3A Active CN109085999B (en) 2018-06-15 2018-06-15 Data processing method and processing system

Country Status (1)

Country Link
CN (1) CN109085999B (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109976669A (en) * 2019-03-15 2019-07-05 百度在线网络技术（北京）有限公司 A kind of edge storage method, device and storage medium
CN110968446A (en) * 2019-11-24 2020-04-07 苏州浪潮智能科技有限公司 Metadata repairing method, device and system and computer readable storage medium
CN111124291A (en) * 2019-12-09 2020-05-08 北京金山云网络技术有限公司 Data storage processing method and device of distributed storage system and electronic equipment
CN111610936A (en) * 2020-05-25 2020-09-01 广州市百果园信息技术有限公司 Object storage platform, object aggregation method and device and server
CN112905122A (en) * 2021-02-20 2021-06-04 炬芯科技股份有限公司 Data storage method and device
CN113194158A (en) * 2021-04-13 2021-07-30 杭州迪普科技股份有限公司 Information storage method, device, equipment and computer readable storage medium
WO2021190232A1 (en) * 2020-03-25 2021-09-30 华为技术有限公司 Storage system, data processing method and apparatus, node, and storage medium
WO2022002010A1 (en) * 2020-07-02 2022-01-06 华为技术有限公司 Method for using intermediate device to process data, computer system, and intermediate device
CN114285797A (en) * 2021-12-30 2022-04-05 北京天融信网络安全技术有限公司 Method and device for processing IP address and storage medium

Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030023811A1 (en) * 2001-07-27 2003-01-30 Chang-Soo Kim Method for managing logical volume in order to support dynamic online resizing and software raid
CN101446924A (en) * 2008-12-16 2009-06-03 成都市华为赛门铁克科技有限公司 Method and system for storing and obtaining data
CN101840308A (en) * 2009-10-28 2010-09-22 创新科存储技术有限公司 Hierarchical memory system and logical volume management method thereof
CN101986655A (en) * 2010-10-21 2011-03-16 浪潮(北京)电子信息产业有限公司 Storage network and data reading and writing method thereof
CN103226518A (en) * 2012-01-31 2013-07-31 国际商业机器公司 Method and device for performing volume expansion in storage management system
CN103544045A (en) * 2013-10-16 2014-01-29 南京大学镇江高新技术研究院 HDFS-based virtual machine image storage system and construction method thereof
CN105528302A (en) * 2015-12-03 2016-04-27 Tcl集团股份有限公司 Logical volume-based method and system for dynamically managing disk
CN106681669A (en) * 2017-01-25 2017-05-17 郑州云海信息技术有限公司 Method, device and system for virtual disk capacity expansion
CN107111653A (en) * 2015-02-25 2017-08-29 华为技术有限公司 The query optimization that Installed System Memory suitable for parallel database system is loaded
CN107423301A (en) * 2016-05-24 2017-12-01 华为技术有限公司 A kind of method of data processing, relevant device and storage system
CN107436725A (en) * 2016-05-25 2017-12-05 杭州海康威视数字技术股份有限公司 A kind of data are write, read method, apparatus and distributed objects storage cluster
US20180067673A1 (en) * 2016-03-15 2018-03-08 International Business Machines Corporation Storage capacity allocation using distributed spare space
CN108073363A (en) * 2017-12-28 2018-05-25 深圳市得微电子有限责任公司 Date storage method, storage device and computer readable storage medium
CN108111628A (en) * 2018-01-18 2018-06-01 吉浦斯信息咨询（深圳）有限公司 A kind of dynamic capacity-expanding storage method and system

Patent Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030023811A1 (en) * 2001-07-27 2003-01-30 Chang-Soo Kim Method for managing logical volume in order to support dynamic online resizing and software raid
CN101446924A (en) * 2008-12-16 2009-06-03 成都市华为赛门铁克科技有限公司 Method and system for storing and obtaining data
CN101840308A (en) * 2009-10-28 2010-09-22 创新科存储技术有限公司 Hierarchical memory system and logical volume management method thereof
CN101986655A (en) * 2010-10-21 2011-03-16 浪潮(北京)电子信息产业有限公司 Storage network and data reading and writing method thereof
CN103226518A (en) * 2012-01-31 2013-07-31 国际商业机器公司 Method and device for performing volume expansion in storage management system
CN103544045A (en) * 2013-10-16 2014-01-29 南京大学镇江高新技术研究院 HDFS-based virtual machine image storage system and construction method thereof
CN107111653A (en) * 2015-02-25 2017-08-29 华为技术有限公司 The query optimization that Installed System Memory suitable for parallel database system is loaded
CN105528302A (en) * 2015-12-03 2016-04-27 Tcl集团股份有限公司 Logical volume-based method and system for dynamically managing disk
US20180067673A1 (en) * 2016-03-15 2018-03-08 International Business Machines Corporation Storage capacity allocation using distributed spare space
CN107423301A (en) * 2016-05-24 2017-12-01 华为技术有限公司 A kind of method of data processing, relevant device and storage system
CN107436725A (en) * 2016-05-25 2017-12-05 杭州海康威视数字技术股份有限公司 A kind of data are write, read method, apparatus and distributed objects storage cluster
CN106681669A (en) * 2017-01-25 2017-05-17 郑州云海信息技术有限公司 Method, device and system for virtual disk capacity expansion
CN108073363A (en) * 2017-12-28 2018-05-25 深圳市得微电子有限责任公司 Date storage method, storage device and computer readable storage medium
CN108111628A (en) * 2018-01-18 2018-06-01 吉浦斯信息咨询（深圳）有限公司 A kind of dynamic capacity-expanding storage method and system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
不足: "不足", 《HTTPS://WWW.CNBLOGS.COM/OLD-SCHOOL/P/7722675.HTML》 *

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109976669A (en) * 2019-03-15 2019-07-05 百度在线网络技术（北京）有限公司 A kind of edge storage method, device and storage medium
CN110968446A (en) * 2019-11-24 2020-04-07 苏州浪潮智能科技有限公司 Metadata repairing method, device and system and computer readable storage medium
CN110968446B (en) * 2019-11-24 2022-08-12 苏州浪潮智能科技有限公司 Metadata repairing method, device and system and computer readable storage medium
CN111124291A (en) * 2019-12-09 2020-05-08 北京金山云网络技术有限公司 Data storage processing method and device of distributed storage system and electronic equipment
CN111124291B (en) * 2019-12-09 2023-05-30 北京金山云网络技术有限公司 Data storage processing method and device of distributed storage system and electronic equipment
WO2021190232A1 (en) * 2020-03-25 2021-09-30 华为技术有限公司 Storage system, data processing method and apparatus, node, and storage medium
CN111610936A (en) * 2020-05-25 2020-09-01 广州市百果园信息技术有限公司 Object storage platform, object aggregation method and device and server
WO2021238408A1 (en) * 2020-05-25 2021-12-02 百果园技术(新加坡)有限公司 Object storage platform, object aggregation method and apparatus, and server
CN111610936B (en) * 2020-05-25 2023-04-14 广州市百果园信息技术有限公司 Object storage platform, object aggregation method and device and server
WO2022002010A1 (en) * 2020-07-02 2022-01-06 华为技术有限公司 Method for using intermediate device to process data, computer system, and intermediate device
CN112905122A (en) * 2021-02-20 2021-06-04 炬芯科技股份有限公司 Data storage method and device
CN112905122B (en) * 2021-02-20 2024-04-09 炬芯科技股份有限公司 Method and device for storing data
CN113194158A (en) * 2021-04-13 2021-07-30 杭州迪普科技股份有限公司 Information storage method, device, equipment and computer readable storage medium
CN114285797A (en) * 2021-12-30 2022-04-05 北京天融信网络安全技术有限公司 Method and device for processing IP address and storage medium
CN114285797B (en) * 2021-12-30 2024-04-19 北京天融信网络安全技术有限公司 Processing method, device and storage medium of IP address

Also Published As

Publication number Publication date
CN109085999B (en) 2022-04-22

Similar Documents

Publication Publication Date Title
CN109085999A (en) data processing method and processing system
US10466899B2 (en) Selecting controllers based on affinity between access devices and storage segments
CN106487850B (en) The methods, devices and systems of mirror image are obtained under a kind of cloud environment
CN103152393B (en) A kind of charging method of cloud computing and charge system
CN102971724B (en) The method and apparatus relevant with the management based on modular virtual resource in data center environment
CN103905572B (en) The processing method and processing device of domain name mapping request
CN111541760B (en) Complex task allocation method based on server-free mist computing system architecture
CN107357896A (en) Expansion method, device, system and the data base cluster system of data-base cluster
CN108900626B (en) Data storage method, device and system in cloud environment
CN104980494B (en) A kind of cloud storage download shared platform and method with local cache
CN111666131A (en) Load balancing distribution method and device, computer equipment and storage medium
CN109561054A (en) A kind of data transmission method, controller and access device
CN109951543A (en) A kind of data search method of CDN node, device and the network equipment
CN110149377A (en) A kind of video service node resource allocation methods, system, device and storage medium
CN110688213A (en) Resource management method and system based on edge calculation and electronic equipment
Hsieh et al. The incremental load balance cloud algorithm by using dynamic data deployment
CN112019577B (en) Exclusive cloud storage implementation method and device, computing equipment and computer storage medium
CN110008029B (en) ceph metadata cluster directory distribution method, system, device and readable storage medium
CN109995890A (en) A kind of method and server managing network address translation NAT gateway
US11138215B2 (en) Method and system for implementing parallel database queries
Petrovska et al. Features of the distribution of computing resources in cloud systems
KR102289100B1 (en) Container-based cluster construction method and cluster device for big data analysis
US9432476B1 (en) Proxy data storage system monitoring aggregator for a geographically-distributed environment
CN106775942B (en) Cloud application-oriented solid-state disk cache management system and method
CN112738247B (en) Cloud computing resource distribution system and method based on multi-layer space scheduling

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant