WO2019001020A1 - 基于分布式系统的存储空间整理方法、装置及系统 - Google Patents

基于分布式系统的存储空间整理方法、装置及系统 Download PDF

Info

Publication number
WO2019001020A1
WO2019001020A1 PCT/CN2018/079241 CN2018079241W WO2019001020A1 WO 2019001020 A1 WO2019001020 A1 WO 2019001020A1 CN 2018079241 W CN2018079241 W CN 2018079241W WO 2019001020 A1 WO2019001020 A1 WO 2019001020A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
idle
storage space
storage
data units
Prior art date
Application number
PCT/CN2018/079241
Other languages
English (en)
French (fr)
Inventor
李丹
Original Assignee
北京奇虎科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京奇虎科技有限公司 filed Critical 北京奇虎科技有限公司
Publication of WO2019001020A1 publication Critical patent/WO2019001020A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/0644—Management of space entities, e.g. partitions, extents, pools
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Definitions

  • the present disclosure relates to the field of computer technologies, and in particular, to a storage space sorting method, apparatus, and system based on a distributed system.
  • the inventors have found that at least the following problems exist in the prior art: as the amount of data of the stored data increases, a large amount of redundant fragmented storage space is accumulated in the storage space. Due to the large number of these accumulated fragmented storage spaces, the total storage space occupied by them will become very large. Moreover, when the data is permanently stored, the stored data is multiplied with time, which makes the above problem more prominent.
  • there is no effective way in the prior art to effectively utilize the fragmented storage space thereby not only causing serious waste in the use of the storage space, but also reducing the allocation and management efficiency of the storage space. It also requires a lot of money to maintain or develop new storage space, which brings a lot of inconvenience to the storage of Internet data.
  • the present disclosure has been made in order to provide a distributed system-based storage space sorting method, apparatus and system that overcomes the above problems or at least partially solves the above problems.
  • a storage space sorting method based on a distributed system includes at least a plurality of distributed data fragmentation clusters, and the method includes: separately acquiring, for each data fragmentation cluster, information corresponding to a data storage state of the data fragmentation cluster; The result of the analysis determines the storage capacity and/or location distribution of the free storage space included in the data fragmentation cluster; and sorts the free storage space included in the data fragmentation cluster according to the storage capacity and/or location distribution of the free storage space; And updating information corresponding to the data storage state of the data fragmentation cluster according to the collation result.
  • a storage space sorting apparatus based on a distributed system.
  • the distributed system includes at least a plurality of distributed data fragmentation clusters, and the device includes: an analysis module, configured to acquire and analyze data storage status of the data fragmentation cluster for each data fragmentation cluster respectively.
  • the determining module is adapted to determine, according to the analysis result, a storage capacity and/or a location distribution of the free storage space included in the data fragmentation cluster; the collating module is adapted to be based on the storage capacity and/or the location distribution of the free storage space And arranging the free storage space included in the data fragmentation cluster; and updating the module, and is adapted to update information corresponding to the data storage state of the data fragmentation cluster according to the collation result.
  • a distributed system comprising any of the above-described storage space organizing devices, and a plurality of distributed data fragmentation clusters.
  • an electronic device includes: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface complete communication with each other through a communication bus;
  • the memory is configured to store at least one executable instruction, and the executable instruction causes the processor to perform the operation corresponding to the storage system finishing method based on the distributed system.
  • a computer program comprising:
  • Computer readable code when the computer readable code is run on a computing device, causes the computing device to perform the operations corresponding to the above described distributed system based storage space grooming method.
  • a non-transitory computer readable storage medium having stored therein at least one executable instruction that causes a processor to perform execution as described above The operation corresponding to the storage space sorting method of the distributed system.
  • the storage space when the storage space is sorted, firstly, information corresponding to the data storage state of each data fragment cluster is separately obtained and analyzed, and then analyzed according to the analysis. As a result, the storage capacity and/or location distribution of the free storage space included in the data fragmentation cluster is determined, and the free storage space included in the data fragmentation cluster is sorted according to the determination result, and finally, the data is updated according to the collation result.
  • the information corresponding to the slice storage state when the storage space is sorted, firstly, information corresponding to the data storage state of each data fragment cluster is separately obtained and analyzed, and then analyzed according to the analysis. As a result, the storage capacity and/or location distribution of the free storage space included in the data fragmentation cluster is determined, and the free storage space included in the data fragmentation cluster is sorted according to the determination result, and finally, the data is updated according to the collation result.
  • the information corresponding to the slice storage state when the storage space is sorted, firstly, information corresponding to the data storage state of each data fragment
  • the solution in the present disclosure can solve the problem that in the distributed system, due to the inability to effectively integrate and utilize the fragmented storage space, a large amount of available storage space is wasted, and the storage space is allocated and managed.
  • the present disclosure can effectively integrate and allocate the fragmented storage space, improve the utilization of the storage space and the distribution efficiency, and realize the effective management of the storage space.
  • FIG. 1 is a schematic flowchart diagram of a storage system finishing method based on a distributed system according to an embodiment of the present disclosure
  • FIG. 2a is a schematic structural diagram of a distributed system provided by Embodiment 2 of the present disclosure.
  • FIG. 2b is a schematic diagram of data nodes included in each data fragmentation cluster in a distributed system according to Embodiment 2 of the present disclosure
  • FIG. 3 is a schematic flowchart diagram of a storage system finishing method based on a distributed system according to Embodiment 2 of the present disclosure
  • FIG. 4 is a structural block diagram of a storage system arranging apparatus based on a distributed system according to Embodiment 3 of the present disclosure
  • FIG. 5 is a structural block diagram of a distributed system according to Embodiment 4 of the present disclosure.
  • FIG. 6 is a schematic structural diagram of an electronic device according to Embodiment 6 of the present disclosure.
  • the present disclosure provides a method, device and system for storing storage space based on a distributed system, which can solve a large number of available storage spaces in a distributed system due to the inability to effectively integrate and utilize fragmented storage space. It is wasted, and the allocation and management of the storage space are inconvenient.
  • the present disclosure can effectively integrate and allocate the fragmented storage space, improve the utilization rate and distribution efficiency of the storage space, and realize the storage space. Effective management.
  • FIG. 1 is a schematic flowchart diagram of a storage system finishing method based on a distributed system according to an embodiment of the present disclosure. As shown in Figure 1, the method includes the following steps:
  • Step S110 Acquire and analyze information corresponding to the data storage state of the data fragmentation cluster for each data fragmentation cluster.
  • the distributed system in this embodiment includes at least a plurality of distributed data fragment clusters.
  • the data to be stored may be divided into multiple fragments, and each fragment is separately stored on a different data fragmentation cluster. Therefore, the specific number of the data fragmentation clusters may be set or adjusted according to the amount of data to be stored, which is not limited in this disclosure.
  • the information acquired in step S110 may include metadata information.
  • Metadata is mainly used to record attribute information of stored data, and the specific content of the metadata may include multiple types.
  • metadata is used to record the specific content, storage location, data size, and the like of each piece of stored data.
  • Metadata can be used to implement historical data query, resource search, file recording and other functions. Among them, the above information recorded by metadata is collectively referred to as metadata information.
  • the metadata information corresponding to the data storage state of the data fragmentation cluster is first acquired, and then the metadata information is analyzed to determine that the data fragmentation cluster includes Information such as free storage space, so that the related information in the above determined free storage space is further obtained in the subsequent step (corresponding to step S120).
  • the free storage space may include: a storage space that is stored empty, and a storage space that is provided with an invalid flag and/or a delete flag.
  • a storage space that can write new data or can modify data can be regarded as an idle storage space.
  • the specific manner of obtaining and analyzing the metadata information may be set by a person skilled in the art according to actual conditions, and the disclosure does not limit this.
  • Step S120 Determine, according to the analysis result, a storage capacity and/or a location distribution of the free storage space included in the data fragmentation cluster.
  • step S110 information such as storage capacity, and/or location distribution corresponding to the free storage space included in each data fragmentation cluster is determined.
  • the storage capacity corresponding to the free storage space information such as the total storage capacity, the remaining storage capacity, the used storage capacity, and the like of the free storage space corresponding to the free storage space may be determined; and the location of the free storage space is determined.
  • the specific information of the storage capacity and/or the location distribution of the free storage space may be determined by a person skilled in the art according to actual conditions. The disclosure does not need to determine the storage capacity of the free storage space and the specific information of the location distribution. limit.
  • Step S130 Organize the free storage space included in the data fragmentation cluster according to the storage capacity and/or the location distribution of the free storage space.
  • the corresponding free storage space is sorted according to the information such as the storage capacity and/or the location distribution of the free storage space determined in step S120, and the plurality of fragmented free storage spaces are integrated into one large storage space, thereby A plurality of fragmented free storage spaces that cannot be utilized can be effectively utilized to improve storage space utilization and allocation efficiency.
  • a large idle space may be obtained by integrating multiple consecutive free storage spaces.
  • Storage space for a plurality of non-contiguous free storage spaces, the non-contiguous storage space can be converted into continuous idle storage by transferring storage data in non-free storage spaces between multiple non-contiguous free storage spaces.
  • Space and integrate the converted continuous free storage space according to the method of integrating multiple consecutive free storage spaces, and so on.
  • the specific manner of arranging the free storage space according to the storage capacity and/or the location distribution of the free storage space may be set by a person skilled in the art, which is not limited in the disclosure.
  • Step S140 Update information corresponding to the data storage state of the data fragmentation cluster according to the collation result.
  • the information includes metadata information
  • the metadata information can record the data storage state of the data fragment cluster to be allocated and managed for each data fragment cluster.
  • the collation result in step S130 is updated to the metadata information corresponding to the data storage state of each fragment cluster, so as to maintain the data storage state information of the data fragmentation cluster and the corresponding metadata information.
  • the consistency of the recorded corresponding information, at the same time, the updated metadata information can be further used to manage the collation results to ensure the accuracy of subsequent management of the storage space.
  • the distributed system includes at least a plurality of distributed data fragment clusters, and when sorting the storage space, first acquiring and analyzing each of the storage spaces respectively Information corresponding to the data storage state of the data fragmentation cluster, and then determining the storage capacity and/or location distribution of the free storage space included in the data fragmentation cluster according to the analysis result, and including the data segmentation cluster according to the determination result
  • the free storage space is sorted, and finally the information corresponding to the storage state of the data fragment cluster is updated according to the collation result.
  • the solution in the present disclosure can solve the problem that in a distributed system, due to the inability to effectively integrate and utilize the fragmented storage space, a large amount of available storage space is wasted, and at the same time, the storage space is allocated and managed.
  • the present disclosure can effectively integrate and allocate the fragmented storage space, improve the utilization rate and distribution efficiency of the storage space, and realize effective management for the storage space.
  • the distributed system in this embodiment includes at least a plurality of distributed data fragment clusters.
  • FIG. 2a shows a schematic structural diagram of a distributed system 200 in the present disclosure
  • FIG. 2b shows a schematic diagram of data nodes included in each data fragmentation cluster in the distributed system 200.
  • the distributed system 200 includes a routing node 210, a configuration node (also called a config node) 220, and a plurality of data fragmentation clusters 230 connected to the routing node 210.
  • the distributed system 200 shown in FIG. 2a includes a routing node 210 and a config node 220.
  • FIG. 2b shows a schematic diagram of data nodes contained in each data fragmentation cluster in distributed system 200.
  • each data fragmentation cluster further includes three data nodes, which are an M node (master node), an S node (slave node), and an A node (voting node) shown in FIG. 2b.
  • the data stored in the S node is mirror data of the M node data, and is equivalent to the backup data of the M node.
  • the S node and the M node are exchanged through the A node, thereby ensuring the security and accuracy of the stored data.
  • the routing node 210 is configured with a corresponding routing rule, and according to the routing rule, the masked fragment structure can be implemented to obtain complete data. Specifically, in a distributed system, a complete data is segmented to form a plurality of data fragments, and each data fragment is stored in a corresponding data fragment cluster (storing in different data fragment clusters) The data is different. Correspondingly, multiple data fragments are stored in multiple data fragment clusters.
  • the routing node is provided with a corresponding routing rule for implementing the above-mentioned reading and writing process for data.
  • the implementation of the foregoing routing rule generally includes two types, one is a range rule, and the other is a hash rule.
  • the scope rule refers to storing data fragments within a certain range into corresponding data fragment clusters. For example, there are a total of 30 data fragments, numbered 1, 2, 3...30, and there are three data fragment clusters, namely cluster 1, cluster 2, and cluster 3.
  • the data fragment corresponding to the above numbers 1-10 is correspondingly stored in the cluster 1.
  • the data fragments numbered 11-20 are correspondingly stored in the cluster 2, and the number is 21-
  • the data fragment of 30 is correspondingly stored to the cluster 3.
  • Another rule is a hash rule (also called a random rule), that is, by performing hash calculation on each data slice, and determining a data slice cluster corresponding to each data slice according to the hash calculation result.
  • a hash rule also called a random rule
  • the purpose is to be able to read the complete data when using the business, to ensure the integrity of data writing and reading.
  • the config node stores metadata information for recording information such as location information and change logs of each data fragment cluster.
  • metadata information for recording information such as location information and change logs of each data fragment cluster.
  • the distributed system mechanism also includes an equalization mechanism (or a balance mechanism).
  • the equalization mechanism is used to implement: when the amount of data stored in the plurality of data fragment clusters is large, the stored data in the data fragment cluster with the relatively large amount of stored data is transferred to the stored data according to the equalization mechanism. A relatively small number of data fragmentation clusters balance the amount of data stored in each data fragmentation cluster. In the data balancing operation, the data is transferred, and the storage space after the data transfer is set with a corresponding mark to indicate that the storage space can be written into the new data.
  • the implementation of the equalization mechanism further includes, for example, when new data is to be inserted in the stored original data, if the amount of data of the newly inserted data is large, the original data is also migrated.
  • the piece of data includes an identifier (ID) and a name (name).
  • ID identifier
  • name name
  • allocating storage space for the piece of data only a storage space sufficient to store the above one ID and one name is required. can. For example, if the remaining storage space in the data shard cluster 1 is just or slightly larger than the occupied space of the data including one ID and one name, the data is stored in the data shard cluster 1.
  • a resume information needs to be inserted for the piece of data, and the resume information occupies a large storage space, and the storage space for storing the piece of data is insufficient to store the resume information, and the corresponding information will include an ID and The data of a name is migrated. For example, it is determined by correlation calculation that the remaining storage space in the data fragment cluster 2 is sufficient to store the data after the insertion of the resume information, and the data is migrated from the data fragment cluster 1 to the data fragment cluster 2 In the data fragmentation cluster 2, the operation of inserting resume information into the data is performed at the same time.
  • information about the migration operation of each piece of data (for example, information of the originally stored data fragmentation cluster information, data size, data location, etc., and the currently stored data fragmentation cluster information, data) Information such as size, data location, etc.) is updated synchronously to the config node.
  • the above is a brief introduction to the distributed system in this embodiment.
  • the advantage of the distributed system is that the infinite expansion of the storage space is realized, that is, the number of data fragmentation clusters in FIG. 2 can be increased indefinitely, and the size of the storage space can be expanded indefinitely.
  • FIG. 3 is a schematic flowchart diagram of a distributed system-based storage space sorting method according to Embodiment 2 of the present disclosure. As shown in FIG. 3, the method includes the following steps:
  • Step S310 Acquire and analyze metadata information corresponding to the data storage state of the data fragmentation cluster for each data fragmentation cluster.
  • the distributed system includes at least a plurality of distributed data fragmentation clusters.
  • the data to be stored may be divided into multiple fragments, and each fragment is separately stored on a different data fragmentation cluster. Therefore, the specific number of the data fragmentation clusters may be set or adjusted according to the amount of stored data, which is not limited in this disclosure.
  • Metadata is mainly used to record attribute information of stored data, and the specific content of the metadata may include multiple types. For example, metadata is used to record the specific content, storage location, data size, and the like of each piece of stored data. Metadata can be used to implement historical data query, resource search, file record and other functions. Among them, the above information recorded by metadata is collectively referred to as metadata information.
  • the distributed system since the distributed system includes a configuration server (ie, the config node described above) for storing metadata information corresponding to the data storage state of each data fragmentation cluster,
  • a configuration server ie, the config node described above
  • metadata information corresponding to the data storage state of the data fragmentation cluster is obtained by the above configuration server.
  • metadata information corresponding to the data storage state of the data fragmentation cluster is obtained from the config node, and then the metadata information is analyzed, and the metadata information is searched for inclusion.
  • the storage space of the idle tag, and the found storage space is determined as the free storage space.
  • the idle mark may include: an invalid flag and/or a delete flag generated due to a data equalization operation, a data update operation, and/or a data deletion operation.
  • the storage space occupied by the data is deleted by corresponding setting, and the data in the storage space set with the deletion mark can be altered to indicate that the data is deleted.
  • the storage space can be multiplexed (that is, the storage space can be rewritten into new data).
  • the data may also generate the above-mentioned similar storage space that can be multiplexed after performing related operations such as data equalization operation and data update operation, and the storage space is also correspondingly set to delete the mark and/or invalid mark to indicate This storage space can be multiplexed. Therefore, when determining the free storage space, if it is determined that the metadata information corresponding to the data storage state of the data fragment cluster includes the idle flag, the storage space corresponding to the metadata may be determined as the free storage space.
  • each data fragmentation cluster includes a plurality of data units for storing data, and correspondingly, a storage capacity of a free storage space included in each data fragmentation cluster and/or Or the location distribution is specifically: the storage capacity of each idle data unit in the data fragmentation cluster, and/or the location distribution of each idle data unit.
  • the division granularity of the data unit can be flexibly set by a person skilled in the art.
  • the above data unit may further include at least one of the following: a data block, a data partition, and a data sub-partition. Wherein, one data block includes at least one data partition, and one data partition includes at least one data sub-partition.
  • the size of each data block is generally 64 MB.
  • the size of each data block can also be adjusted according to the size of the storage file. For example, a file is too large, and the capacity of one data block is insufficient to store one file. , the capacity of the data block can be adjusted from the default 64MB to 128MB, 256MB or more capacity to achieve storage of the corresponding file.
  • the size of the data block can be adjusted according to actual conditions by a person skilled in the art, and the disclosure does not limit this.
  • the data partition, or the data sub-partition is set and adjusted in a similar manner to the data block, and will not be described here. A person skilled in the art can determine the specific granularity of the data unit according to actual needs. For example, if only the large free space is to be collated, each data block can be regarded as the smallest data unit; for example, if it is needed for the fine free space By sorting, each data sub-partition can be used as the smallest data unit.
  • Step S320 Determine, according to the analysis result, a storage capacity and/or a location distribution of the free storage space included in the data fragmentation cluster.
  • the storage capacity and/or location distribution information of the free storage space is further determined according to the metadata information corresponding to the free storage space.
  • each data fragmentation cluster includes a plurality of data units for storing data.
  • the determining the storage capacity of the free storage space included in the data fragmentation cluster may be: determining a storage capacity of each idle data unit in the data fragmentation cluster; determining a location distribution of the idle storage space included in the data fragmentation cluster. It may be: determining the position distribution of each idle data unit.
  • the specific process may be: determining a plurality of idle data units that are adjacent and continuous in position according to the location distribution of each idle data unit. Specifically, in an optional solution, if there are no non-idle data units between every two adjacent idle data units, the plurality of idle data units are determined to be adjacent to each other. And a plurality of consecutive idle data units.
  • the amount of stored data in the non-idle data unit is less than a preset Threshold, performing a data migration operation on all the data stored in the non-idle data unit, and determining the plurality of idle data units and the non-idle data unit after performing the data migration operation as adjacent and continuous multiple idle locations Data unit.
  • the specific value of the preset threshold may be set by a person skilled in the art according to actual conditions, and the disclosure does not limit this.
  • the above-described data transfer operation storage unit is provided with an idle flag, so the storage unit can be regarded as a free storage unit.
  • the determining manners of the plurality of idle data units that are adjacent to each other and in the continuation include, but are not limited to, the foregoing two implementation manners.
  • a person skilled in the art may also determine the foregoing determining manner according to actual conditions. It is set as other implementation manners, and the present disclosure does not limit its specific determination manner as long as it can determine a plurality of idle data units that are adjacent and continuous in position.
  • the above-mentioned free storage unit can be either a data block, a data partition or a data sub-partition, depending on the collation granularity of the present disclosure.
  • Step S330 arranging for a plurality of idle data units adjacent to each other and consecutively.
  • this step can be implemented.
  • the scheme directly combines a plurality of idle data units adjacent to each other and successively into one larger storage space (data units to be allocated), thereby realizing direct and rapid merging of multiple idle data units. In this manner, only a plurality of idle data units adjacent to each other and consecutively merged into one data unit to be allocated, specifically, a plurality of idle data units adjacent to each other and consecutively merged into one overall space, and A token to be assigned is assigned to the holistic space.
  • the to-be-allocated tag is used to indicate that the holistic space is an idle, unallocated space. That is, the whole space has not been divided into several specific size data units. For example, it is assumed that the size of the overall space obtained by merging a plurality of adjacent idle data units is 900M, and the 900M space corresponds to the label to be allocated, so that in the subsequent process, the 900M space can be used by those skilled in the art. It is divided into several data units, and the size of each data unit is set according to the size of the data to be stored. It can be seen that the data unit to be allocated has greater flexibility in the subsequent process, and can be flexibly divided into at least one equal or unequal data unit according to storage requirements.
  • multiple idle data units that are adjacent in position and consecutive are recovered and then reallocated as one or more data units to be used.
  • the storage capacity of each data unit to be used is set, and a corresponding to-be-used flag is set for each data unit to be used.
  • a plurality of idle data units adjacent to each other are successively recovered, and then the collected plurality of free data units are integrated according to the size of the stored data to obtain a storage space corresponding to the size of the stored data, so that the integration is performed.
  • the subsequent idle data unit can be directly used; or, the collected plurality of idle data units can be integrated according to the capacity of the data block, the data partition or the data sub-partition set in step S320, and the plurality of idle data units are integrated.
  • Multiple data blocks, data partitions or data sub-partitions, etc., and corresponding tokens to be used are set to indicate that the data unit can be used directly.
  • the size of the overall space obtained by retrieving a plurality of adjacent free and consecutive data units is 600 MB.
  • the 600 MB storage space is further divided into three sets of data units of a set size, for example, each The data units are 100MB, 200MB, and 300MB, respectively, and the tokens to be used are set for each data unit. Therefore, in the solution, the number of data units and the capacity of each data unit are set for the storage space after the recovery, so that when the data is allocated and stored subsequently, the storage space to be used may be directly used.
  • Storage capacity stores data.
  • the above two schemes may be used alone or in combination.
  • the process may further be: contiguous and continuous to the location according to the sum of the storage capacities of the plurality of idle data units adjacent to each other and consecutively located. Multiple idle data units are collated.
  • a storage capacity threshold may be set, and a specific finishing scheme is determined according to a relationship between a storage capacity sum of a plurality of idle data units adjacent to and consecutive to the storage capacity threshold. For example, when the sum of the storage capacities of the plurality of idle data units is greater than the storage capacity threshold, a plurality of idle data units adjacent to and consecutive to the foregoing locations are merged.
  • the sum of the storage capacities of the plurality of idle data units adjacent to each other and consecutively compared with the storage capacity of the adjacent non-idle storage spaces may be compared, if the sum of the storage capacities is greater than the storage capacity of the adjacent non-idle storage spaces.
  • the storage capacity transfers the data in the non-idle storage space, converts the non-idle storage space into a free storage space, and then integrates the adjacent and consecutive plurality of idle data units in the converted position, etc. Wait.
  • those skilled in the art may also set the collation manner for arranging a plurality of idle data units adjacent to each other in a position according to an actual situation as other implementation manners than the above enumerated manners, as long as the positions can be adjacent and continuous.
  • the plurality of idle data units are effectively integrated, and the disclosure does not limit the manner in which it is organized.
  • Step S340 Update the metadata information corresponding to the data storage state of the data fragment cluster according to the collation result.
  • the metadata information can record the data storage state of the data fragmentation cluster for allocation and management for each data fragmentation cluster.
  • the metadata information corresponding to the data storage state of the data fragment cluster stored in the configuration server that is, the metadata information in the update config node
  • the data storage state information of the fragment cluster is consistent with the corresponding information recorded in the corresponding metadata information, and the updated metadata information can be further used to manage the collation result to ensure subsequent management for the storage space. The accuracy.
  • the metadata information of the storage state corresponding to the plurality of distributed data fragment clusters included in the distributed system can be obtained, and the metadata information is analyzed by analyzing the metadata information. Determining a corresponding free storage space in each data fragmentation cluster, and analyzing the storage capacity and/or location distribution of the free storage space to sort the free storage space included in the data fragmentation cluster, and fragmenting
  • the free storage space that cannot be utilized is integrated into a large and available storage space, which improves the utilization and allocation efficiency of the storage space, and also realizes effective management for the storage space.
  • the solution in this embodiment can further divide the free storage space in a more detailed manner, so that the accuracy of the fragmented free storage space and the utilization of the fragmented free storage space are higher.
  • FIG. 4 is a structural block diagram of a distributed system-based storage space organizing apparatus according to Embodiment 3 of the present disclosure. As shown in FIG. 4, the apparatus includes an analysis module 41, a determination module 42, a collation module 43, and an update module 44.
  • the analysis module 41 is adapted to acquire and analyze information corresponding to the data storage state of the data fragmentation cluster for each data fragmentation cluster.
  • the distributed system includes at least a plurality of distributed data fragmentation clusters.
  • the data to be stored may be divided into multiple fragments, and each fragment is separately stored on a different data fragmentation cluster. Therefore, the specific number of the data fragmentation clusters may be set or adjusted according to the amount of stored data, which is not limited in this disclosure.
  • the information acquired by the analysis module 41 may include metadata information.
  • Metadata is mainly used to record attribute information of stored data, and the specific content of the metadata may include multiple types.
  • metadata is used to record the specific content, storage location, data size, and the like of each piece of stored data.
  • Metadata can be used to implement historical data query, resource search, file record and other functions. Among them, the above information recorded by metadata is collectively referred to as metadata information.
  • a configuration server for storing metadata information corresponding to a data storage state of each data fragmentation cluster may also be set, and therefore, obtaining and fragmenting each data
  • the analysis module 41 For each data fragmentation cluster, the analysis module 41 first acquires metadata information corresponding to the data storage state of the data fragmentation cluster, and then analyzes the metadata information, and searches for the storage containing the idle token from the metadata information. Space, and determine the storage space found as free storage space.
  • the idle mark may include: an invalid flag and/or a delete flag generated due to a data equalization operation, a data update operation, and/or a data deletion operation.
  • the storage space occupied by the data is deleted by corresponding setting, and the data in the storage space set with the deletion mark can be altered to indicate that the data is deleted.
  • the storage space can be multiplexed (that is, the storage space can be rewritten into new data).
  • the data may also generate the above-mentioned similar storage space that can be multiplexed after performing related operations such as data equalization operation and data update operation, and the storage space is also correspondingly set to delete the mark and/or invalid mark to indicate This storage space can be multiplexed. Therefore, when determining the free storage space, if it is determined that the metadata information corresponding to the data storage state of the data fragment cluster includes the idle flag, the storage space corresponding to the metadata may be determined as the free storage space.
  • the determining module 42 is adapted to determine a storage capacity and/or a location distribution of the free storage space included in the data fragmentation cluster based on the analysis result.
  • the determining module 42 acquires the storage capacity and/or the location distribution information of the free storage space from the metadata information corresponding to the free storage space according to the free storage space determined by the analysis module 41.
  • Each of the data fragment clusters may also be divided into a plurality of data units for storing data.
  • the determining the storage capacity of the free storage space included in the data fragmentation cluster may be: determining a storage capacity of each idle data unit in the data fragmentation cluster; and determining the free storage space included in the data fragmentation cluster.
  • the location distribution may specifically be: determining a location distribution of each idle data unit.
  • each data fragmentation cluster includes a plurality of data units for storing data, and correspondingly, a free storage space included in each data fragmentation cluster.
  • the storage capacity and/or location distribution is specifically: the storage capacity of each idle data unit in the data fragmentation cluster, and/or the location distribution of each idle data unit.
  • the division granularity of the data unit can be flexibly set by a person skilled in the art. For example, for each of the above data units, its storage space can also be further subdivided.
  • the foregoing data unit may further include at least one of the following: a data block, a data partition, and a data sub-partition.
  • one data block includes at least one data partition
  • one data partition includes at least one data sub-partition.
  • the determining module 42 may further determine the storage capacity of the data block, the data partition, or the data sub-partition in each of the idle data units in the data fragment cluster when determining the storage capacity and/or the location distribution of the free storage space, and / or the location distribution of data blocks, data partitions, or data sub-partitions in each idle data unit.
  • the size of each data block is generally 64 MB. In a specific implementation, the size of each data block can also be adjusted according to the size of the storage file. For example, a file is too large, and the capacity of one data block is insufficient to store one file.
  • the capacity of the data block can be adjusted from the default 64MB to 128MB, 256MB or more capacity to achieve storage of the corresponding file.
  • the size of the data block can be adjusted according to actual conditions by a person skilled in the art, and the disclosure does not limit this.
  • the data partition, or the data sub-partition is set and adjusted in a similar manner to the data block, and will not be described here.
  • a person skilled in the art can determine the specific granularity of the data unit according to actual needs. For example, if only the large free space is to be collated, each data block can be regarded as the smallest data unit; for example, if it is needed for the fine free space By sorting, each data sub-partition can be used as the smallest data unit.
  • the specific process may be: determining a plurality of idle data units that are adjacent and continuous in position according to the location distribution of each idle data unit. Specifically, in an optional solution, if there are no non-idle data units between every two adjacent idle data units among the plurality of idle data units, the determining module 42 determines the plurality of idle data units as A plurality of idle data units that are adjacent and continuous in position.
  • the determining module 42 performs a data migration operation on all the data stored in the non-idle data unit, and determines the plurality of idle data units and the non-idle data unit after performing the data migration operation as being adjacent and continuous.
  • the specific value of the preset threshold may be set by a person skilled in the art according to actual conditions, and the disclosure does not limit this.
  • the above-described data transfer operation storage unit is provided with an idle flag, so the storage unit can be regarded as a free storage unit.
  • the determining module 42 may be determined by a person skilled in the art according to actual conditions.
  • the determining manner is set to other implementation manners, and the present disclosure does not limit the determining manner of the determining module 42 as long as it can determine a plurality of idle data units that are adjacent and continuous in position.
  • the above-mentioned free storage unit can be either a data block, a data partition or a data sub-partition, depending on the collation granularity of the present disclosure.
  • the collating module 43 is adapted to organize the free storage space included in the data fragmentation cluster according to the storage capacity and/or the location distribution of the free storage space.
  • the manner in which the sorting module 43 sorts the free storage space may be various.
  • the collating module 43 may combine a plurality of idle data units that are adjacent to each other and are consecutively into one data unit to be allocated, and set a corresponding to-be-allocated label for the to-be-allocated data unit.
  • the scheme directly combines a plurality of idle data units adjacent to each other and successively into one larger storage space (data units to be allocated), thereby realizing direct and rapid merging of multiple idle data units.
  • the to-be-allocated tag is used to indicate that the holistic space is an idle, unallocated space. That is, the whole space has not been divided into several specific size data units. For example, it is assumed that the size of the overall space obtained by merging a plurality of adjacent idle data units is 900 MB, and the 900 MB space corresponds to the label to be allocated, so that in the subsequent process, the person skilled in the art can use the 900 MB space.
  • each data unit is set according to the size of the data to be stored. It can be seen that the data unit to be allocated has greater flexibility in the subsequent process, and can be flexibly divided into at least one equal or unequal data unit according to storage requirements.
  • the collation module 43 reclaims the plurality of idle data units that are adjacent and consecutive in position and reassigns them into one or more data units to be used.
  • the storage capacity of each data unit to be used is set, and a corresponding to-be-used flag is set for each data unit to be used.
  • the collating module 43 first reclaims a plurality of idle data units that are adjacent to each other and consecutively, and then integrates the collected plurality of idle data units according to the size of the stored data to obtain a storage space corresponding to the size of the stored data.
  • the integrated idle data unit can be directly used; or, the collected plurality of idle data units can be integrated according to the capacity of the data block, the data partition or the data sub-partition, and the plurality of idle data units can be integrated into multiple Data blocks, data partitions or data sub-partitions, etc., and set corresponding to-be-used tags to indicate that the data unit can be used directly.
  • the size of the overall space obtained by retrieving a plurality of adjacent free and consecutive data units is 600 MB.
  • the 600 MB storage space is further divided into three sets of data units of a set size, for example, each The data units are 100MB, 200MB, and 300MB, respectively, and the tokens to be used are set for each data unit.
  • the number of data units and the capacity of each data unit are set for the reclaimed storage space. Therefore, when the data is allocated and stored subsequently, the storage space of each storage space to be used may be directly stored. Capacity stores data.
  • the above two schemes may be used alone or in combination.
  • the process may further be: the sum of the storage capacities of the plurality of idle data units adjacent to each other and consecutively Adjacent and consecutive multiple idle data units are collated.
  • a storage capacity threshold may be set, and a specific finishing scheme is determined according to a relationship between a storage capacity sum of a plurality of idle data units adjacent to and consecutive to the storage capacity threshold. For example, when the sum of the storage capacities of the plurality of idle data units is greater than the storage capacity threshold, a plurality of idle data units adjacent to and consecutive to the foregoing locations are merged.
  • the sum of the storage capacities of the plurality of idle data units adjacent to each other and consecutively compared with the storage capacity of the adjacent non-idle storage spaces may be compared, if the sum of the storage capacities is greater than the storage capacity of the adjacent non-idle storage spaces.
  • the storage capacity integrates a plurality of idle data units that are adjacent to each other and consecutively; or transfers data in the non-idle storage space, converts the non-idle storage space into a free storage space, and then converts the converted storage space into Multiple idle data units that are adjacent and continuous in position are integrated, and so on.
  • collation mode of the collation module 43 can also set to other implementation manners than the above-mentioned enumerated manners according to actual conditions, as long as the adjacent data units that are adjacent and continuous can be effectively integrated.
  • the specific arrangement of the sorting module 43 is not limited.
  • the update module 44 is adapted to update information corresponding to the data storage state of the data shard cluster based on the collation results.
  • the information may include metadata information capable of recording a data storage state of the data fragmentation cluster for allocation and management for each data fragmentation cluster.
  • the update module 44 updates the collation result of the collation module 43 to the metadata information corresponding to the data storage state of the corresponding data fragmentation cluster, so as to maintain the data storage state information of the data fragmentation cluster corresponding thereto.
  • the consistency of the corresponding information recorded in the metadata information, and the updated metadata information can be further used to manage the collation result to ensure the accuracy of subsequent management for the storage space.
  • the update process corresponding to the update module 44 is: As a result, the metadata information corresponding to the data storage state of the data fragment cluster stored in the above configuration server is updated.
  • the distributed system includes at least a plurality of distributed data fragment clusters.
  • the analysis module 41 firstly acquires and analyzes each of the storage spaces.
  • the data storage status of the data fragmentation clusters is corresponding to the information, and then the determination module 42 determines the storage capacity and/or location distribution of the free storage space included in the data fragmentation cluster according to the analysis result, and determines according to the determination module 43
  • the update module 44 updates the information corresponding to the storage state of the data fragmentation cluster according to the collation result.
  • the solution in the present disclosure can solve the problem that in the distributed system, due to the inability to effectively integrate and utilize the fragmented storage space, a large amount of available storage space is wasted, and the storage space is allocated and managed. Inconvenience caused.
  • the disclosure can effectively integrate and allocate the fragmented storage space, improve the utilization rate and distribution efficiency of the storage space, and realize effective management for the storage space.
  • the solution in this embodiment can further divide the free storage space in a more detailed manner, so that the accuracy of the fragmented free storage space and the utilization of the fragmented free storage space are higher.
  • FIG. 5 shows a block diagram of a structure of a distributed system 500 provided in accordance with an embodiment 4 of the present disclosure.
  • the system includes: a storage system finishing device 400 based on a distributed system and a plurality of distributed data fragmentation clusters 510.
  • the storage space organization apparatus 400 based on the distributed system includes an analysis module 41, a determination module 42, a sorting module 43, and an update module 44.
  • a fifth embodiment of the present application provides a non-transitory computer readable storage medium storing at least one executable instruction, the computer executable instruction being executable in any of the foregoing method embodiments Storage space sorting method based on distributed system.
  • FIG. 6 is a schematic structural diagram of an electronic device according to Embodiment 6 of the present disclosure, and the specific embodiment of the present disclosure does not limit the specific implementation of the electronic device.
  • the electronic device can include a processor 602, a communications interface 604, a memory 606, and a communications bus 608.
  • Processor 602, communication interface 604, and memory 606 complete communication with one another via communication bus 608.
  • the communication interface 604 is configured to communicate with network elements of other devices, such as clients or other servers.
  • the processor 602 is configured to execute the program 610. Specifically, the related steps in the foregoing storage system-based storage space organization method may be performed.
  • program 610 can include program code, the program code including computer operating instructions.
  • the processor 602 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present disclosure.
  • the one or more processors included in the electronic device may be the same type of processor, such as one or more CPUs; or may be different types of processors, such as one or more CPUs and one or more ASICs.
  • the memory 606 is configured to store the program 610.
  • Memory 606 may include high speed RAM memory and may also include non-volatile memory, such as at least one disk memory.
  • the program 610 may be specifically configured to: the processor 602: perform, respectively, for each data fragmentation cluster, acquire and analyze metadata information corresponding to a data storage state of the data fragmentation cluster; and determine the data segment according to the analysis result. Storage capacity and/or location distribution of the free storage space included in the slice cluster; sorting the free storage space included in the data slice cluster according to the storage capacity and/or location distribution of the free storage space; updating according to the finishing result Metadata information corresponding to the data storage state of the data fragmentation cluster.
  • each data fragmentation cluster includes multiple data units for storing data
  • the storage capacity and/or location distribution of the free storage space included in each data fragmentation cluster is specifically : storage capacity of each idle data unit in the data fragmentation cluster, and/or location distribution of each idle data unit;
  • the program 610 further causes the processor 602 to perform operations of determining a plurality of idle data units that are adjacent and continuous in position according to a position distribution of the respective idle data units, collating the plurality of idle data units adjacent and consecutive to the position.
  • the program 610 further causes the processor 602 to perform the following operations: if there are no non-idle data units between every two adjacent idle data units among the plurality of idle data units,
  • the plurality of idle data units are determined as a plurality of idle data units that are adjacent in position and consecutive; or, if there are multiple idle data units, at least two adjacent free data units have non-idle data units, and the non- If the amount of stored data in the idle data unit is less than a preset threshold, performing a data migration operation on all data stored in the non-idle data unit, and performing the data migration operation on the plurality of idle data units
  • the idle data unit is determined to be a plurality of idle data units that are adjacent in position and continuous.
  • the program 610 further causes the processor 602 to perform the operations of merging the plurality of idle data units that are adjacent and consecutive in the location into one data unit to be allocated; wherein, the data to be allocated is The unit sets a corresponding to-be-allocated tag; or, the plurality of idle data units adjacent to and consecutive to the location are recovered and re-allocated into one or more data units to be used; wherein, the storage capacity of each data unit to be used is set. And set the corresponding to-be-used flag for each data unit to be used.
  • the program 610 further causes the processor 602 to perform an operation of contiguous and continuous to the location based on a sum of storage capacities of the plurality of idle data units adjacent to and consecutive to the location. An idle data unit is organized.
  • the program 610 further causes the processor 602 to: search for the storage space containing the idle tag from the metadata information, and determine the found storage space as a free storage space;
  • the idle flag includes an invalid flag and/or a delete flag generated due to a data equalization operation, a data update operation, and/or a data deletion operation.
  • the data unit includes at least one of: a data block, a data partition, and a data sub-partition; wherein, one data block includes at least one data partition, and one data partition includes at least one Data subpartition.
  • the distributed system further includes: a configuration server configured to store metadata information corresponding to a data storage state of each data fragmentation cluster;
  • Program 610 further causes processor 602 to perform the following operations:
  • the metadata information corresponding to the data storage state of the data fragment cluster stored in the configuration server is updated according to the collation result.
  • modules in the devices of the embodiments can be adaptively changed and placed in one or more devices different from the embodiment.
  • the modules or units or components of the embodiments may be combined into one module or unit or component, and further they may be divided into a plurality of sub-modules or sub-units or sub-components.
  • any combination of the features disclosed in the specification, including the accompanying claims, the abstract and the drawings, and any methods so disclosed, or All processes or units of the device are combined.
  • Each feature disclosed in this specification (including the accompanying claims, the abstract and the drawings) may be replaced by alternative features that provide the same, equivalent or similar purpose.
  • Various component embodiments of the present disclosure may be implemented in hardware, or in a software module running on one or more processors, or in a combination thereof.
  • a microprocessor or digital signal processor may be used in practice to implement some of some or all of the components of a distributed system based storage space grooming device in accordance with embodiments of the present disclosure or All features.
  • the invention can also be implemented as a device or device program (e.g., a computer program and a computer program product) for performing some or all of the methods described herein.
  • Such a program implementing the invention may be stored on a computer readable medium or may be in the form of one or more signals. Such signals may be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种基于分布式系统的存储空间整理方法、装置及系统。其中，分布式系统至少包括多个分布式的数据分片集群，该方法包括：分别针对每个数据分片集群，获取并分析与该数据分片集群的数据存储状态相对应的信息(S110)；根据分析结果确定该数据分片集群中包含的空闲存储空间的存储容量和/或位置分布(S120)；根据空闲存储空间的存储容量和/或位置分布对该数据分片集群中包含的空闲存储空间进行整理(S130)；根据整理结果更新与该数据分片集群的数据存储状态相对应的信息(S140)。

Description

基于分布式系统的存储空间整理方法、装置及系统
相关申请的交叉参考
本申请要求于2017年6月30日提交中国专利局、申请号为201710555164.4、名称为“基于分布式系统的存储空间整理方法、装置及系统”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本公开涉及计算机技术领域，具体涉及一种基于分布式系统的存储空间整理方法、装置及系统。
背景技术
在互联网业务中，随着业务数据量的不断增加，需要大量的存储空间来存储互联网业务中的相关数据。由于分布式存储系统的存储空间能够无限扩展，因此在存储互联网的业务数据时得到了广泛利用。在互联网业务中，由于经常需要对数据进行插入、更新、删除等相关操作，因此在互联网业务的分布式存储过程中，也会对应地对存储的数据进行诸如插入、删除、拆分以及迁移等相关操作，这些操作容易导致分布式存储系统的存储空间碎片化。
在实现本公开实施例的过程中，发明人发现现有技术中至少存在如下问题：随着存储数据的数据量增多，存储空间中会累积起大量冗余的碎片化存储空间。这些累积的碎片化存储空间由于数量巨大，导致其占用的总存储空间会变得非常大。并且，在对数据进行永久性存储时，存储数据会随着时间的增加而成倍增加，从而使上述问题更加凸显。然而，现有技术中尚没有一种有效的方式能够针对这些碎片化存储空间进行有效利用，因而不仅在存储空间的使用上造成了严重浪费，同时还会降低存储空间的分配及管理效率，另外，还需要花费大量经费去维护或者开发新的存储空间，给互联网数据的存储带来了诸多不便。
发明内容
鉴于上述问题，提出了本公开以便提供一种克服上述问题或者至少部分地解决上述问题的基于分布式系统的存储空间整理方法、装置及系统。
根据本公开的一个方面，提供了一种基于分布式系统的存储空间整理方法。其中，上述分布式系统至少包括多个分布式的数据分片集群，该方法包括：分别针对每个数据分片集群，获取并分析与该数据分片集群的数据存储状态相对应的信息；根据分析结果确定该数据分片集群中包含的空闲存储空间的存储容量和/或位置分布；根据空闲存储空间的存储容量和/或位置分布对该数据分片集群中包含的空闲存储空间进行整理；以及根据整理结果更新与该数据分片集群的数据存储状态相对应的信息。
根据本公开的另一方面，提供了一种基于分布式系统的存储空间整理装置。其中，上述分布式系统至少包括多个分布式的数据分片集群，该装置包括：分析模块，适于分别针对每个数据分片集群，获取并分析与该数据分片集群的数据存储状态相对应的信息；确定模块，适于根据分析结果确定该数据分片集群中包含的空闲存储空间的存储容量和/或位置分布；整理模块，适于根据空闲存储空间的存储容量和/或位置分布对该数据分片集群中包含的空闲存储空间进行整理；以及更新模块，适于根据整理结果更新与该数据分片集群的数据存储状态相对应的信息。
根据本公开的又一方面，提供了一种分布式系统，包括上述任一存储空间整理装置，以及多个分布式的数据分片集群。
根据本公开的又一方面，提供了一种电子设备，包括：处理器、存储器、通信接口和通信总线，处理器、存储器和通信接口通过通信总线完成相互间的通信；
存储器用于存放至少一可执行指令，可执行指令使处理器执行上述基于分布式系统的存储空间整理方法对应的操作。
根据本公开的又一方面，提供了一种计算机程序，包括：
计算机可读代码，当计算机可读代码在计算设备上运行时，导致计算设备执行上述基于分布式系统的存储空间整理方法对应的操作。
根据本公开的再一方面，提供了一种非易失性计算机可读存储介质，非易失性计算机可读存储介质中存储有至少一可执行指令，可执行指令使处理器执行如上述基于分布式系统的存储空间整理方法对应的操作。
在本公开提供的基于分布式系统的存储空间整理的方法、装置及系统中，对存储空间进行整理时，首先分别获取并分析每个数据分片集群的数据存储状态对应的信息，然后根据分析结果确定该数据分片集群中包含的空闲存储空间的存储容量和/或位置分布，并根据确定结果对该数据分片集群中包含的空闲存储空间进行整理，最后根据整理结果更新与该数据分片集群存储状态相对应的信息。由此可见，本公开中的方案能够解决在分布式系统中，由于无法对碎片化的存储空间进行有效整合和利用，导致大量可利用的存储空间被浪费，并且给存储空间的分配和管理都造成不便的问题，本公开能够对碎片化的存储空间进行有效整合和分配，提升了存储空间的利用率及分配效率，同时也实现了针对存储空间的有效管理。
上述说明仅是本公开技术方案的概述，为了能够更清楚了解本公开的技术手段，而可依照说明书的内容予以实施，并且为了让本公开的上述和其它目的、特征和优点能够更明显易懂，以下特举本公开的具体实施方式。
附图概述
通过阅读下文优选实施方式的详细描述，各种其他的优点和益处对于本领域普通技术人员将变得清楚明了。附图仅用于示出优选实施方式的目的，而并不认为是对本公开的限制。而且在整个附图中，用相同的参考符号表示相同的部件。在附图中：
图1示出了根据本公开实施例一提供的基于分布式系统的存储空间整理方法的流程示意图；
图2a示出了本公开实施例二提供的分布式系统的结构示意图；
图2b示出了本公开实施例二提供的分布式系统中每个数据分片集群包含的数据节点的示意图；
图3示出了根据本公开实施例二提供的基于分布式系统的存储空间整理方法的流程示意图；
图4示出了根据本公开实施例三提供的基于分布式系统的存储空间整理装置的结构框图；
图5示出了根据本公开实施例四提供的分布式系统的结构框图；
图6示出了根据本公开实施例六提供的一种电子设备的结构示意图。
本公开的较佳实施方式
下面将参照附图更详细地描述本公开的示例性实施例。虽然附图中显示了本公开的示例性实施例，然而应当理解，可以以各种形式实现本公开而不应被这里阐述的实施例所限制。相反，提供这些实施例是为了能够更透彻地理解本公开，并且能够将本公开的范围完整的传达给本领域的技术人员。
本公开提供了一种基于分布式系统的存储空间整理的方法、装置及系统，能够解决在分布式系统中，由于无法对碎片化的存储空间进行有效整合和利用，导致大量可利用的存储空间被浪费，并且给存储空间的分配和管理都造成不便的问题，本公开能够对碎片化的存储空间进行有效整合和分配，提升了存储空间的利用率及分配效率，同时也实现了针对存储空间的有效管理。
实施例一
图1示出了根据本公开实施例一提供的基于分布式系统的存储空间整理方法的流程示意图。如图1所示，该方法包括以下步骤：
步骤S110：分别针对每个数据分片集群，获取并分析与该数据分片集群的数据存储状态相对应的信息。
其中，本实施例中的分布式系统至少包括多个分布式的数据分片集群。在分布式系统中，为了提升存储容量，可以将待存储的数据切分为多个分片，将各个分片分别存储在不同的数据分片集群上。因此，数据分片集群的具体数量可以根据待存储的数据量进行设定或调整，本公开对此不做限定。
步骤S110中所获取的信息可以包括元数据信息。元数据主要用于记录已存储的数据的属性信息，元数据的具体内容可以包括多种。例如，元数据用于记录各条存储数据的具体内容、存储位置、数据大小等。利用元数据能 够实现历史数据查询、资源查找、文件记录等多种功能。其中，通过元数据记录的上述信息统称为元数据信息。
在本实施例中，针对每个数据分片集群，首先获取与该数据分片集群的数据存储状态相对应的元数据信息，然后对该元数据信息进行分析，确定该数据分片集群中包含的空闲存储空间等信息，以便后续步骤(对应步骤S120)中进一步获取上述确定的空闲存储空间中的相关信息。其中，空闲存储空间可以包括：存储为空的存储空间、以及设置有无效标记和/或删除标记的存储空间。具体实施中，可以写入新数据或者可以涂改数据的存储空间都可以看作是空闲的存储空间。
在这里，获取以及分析元数据信息的具体方式可以由本领域技术人员根据实际情况进行设置，本公开对此不作限制。
步骤S120：根据分析结果确定该数据分片集群中包含的空闲存储空间的存储容量和/或位置分布。
在本步骤中，根据步骤S110中的分析结果，确定与每个数据分片集群中包含的空闲存储空间对应的存储容量，和/或位置分布等信息。其中，在确定与上述空闲存储空间对应的存储容量时，可以确定与上述空闲存储空间对应的空闲存储空间总存储容量、剩余存储容量、已用存储容量等信息；在确定上述空闲存储空间的位置分布时，可以确定多个空闲存储空间之间是否相邻或连续、非连续的多个空闲存储空间与其中的非空闲存储空间的相对位置信息等等。具体实施中，空闲存储空间的存储容量和/或位置分布的具体信息可以由本领域技术人员根据实际情况进行对应确定，本公开对需要确定的空闲存储空间的存储容量、以及位置分布的具体信息不作限制。
步骤S130：根据空闲存储空间的存储容量和/或位置分布对该数据分片集群中包含的空闲存储空间进行整理。
在本步骤中，根据步骤S120中确定的空闲存储空间存储容量和/或位置分布等信息对对应的空闲存储空间进行整理，将多个零碎的空闲存储空间整合为一个较大的存储空间，从而使多个不能被利用的零碎的空闲存储空间可以得到有效利用，提升存储空间的利用率及分配效率。
其中，针对空闲存储空间进行整理的方式可以有多种，例如，在位置分 布上，对于多个连续的空闲存储空间，可以通过将多个连续的空闲存储空间进行整合来获取一个较大的空闲存储空间；对多个非连续的空闲存储空间，可以通过转移间隔在多个非连续的空闲存储空间之间的非空闲存储空间中的存储数据，将非连续的存储空间转化为连续的空闲存储空间，并按照整合多个连续的空闲存储空间的方法对转化后的连续的空闲存储空间进行整合，等等。具体实施中，根据空闲存储空间的存储容量和/或位置分布对空闲存储空间进行整理的具体方式可以由本领域技术人员进行设置，本公开对此不作限制。
步骤S140：根据整理结果更新与该数据分片集群的数据存储状态相对应的信息。
具体地，信息包括元数据信息，元数据信息能够记录数据分片集群的数据存储状态，以便针对每个数据分片集群进行分配和管理。在本步骤中，将步骤S130中的整理结果更新到与每个分片集群的数据存储状态相对应的元数据信息中，以便保持数据分片集群的数据存储状态信息与其对应的元数据信息中记录的对应信息的一致性，同时，经过更新的元数据信息还能够进一步用于对整理结果进行管理，确保后续针对存储空间进行管理时的准确性。
由此可见，在本公开提供的基于分布式系统的存储空间整理的方法中，分布式系统至少包括多个分布式的数据分片集群，在对存储空间进行整理时，首先分别获取并分析每个数据分片集群的数据存储状态相对应的信息，然后根据分析结果确定该数据分片集群中包含的空闲存储空间的存储容量和/或位置分布，并根据确定结果对数据分片集群中包含的空闲存储空间进行整理，最后根据整理结果更新与该数据分片集群存储状态相对应的信息。本公开中的方案能够解决在分布式系统中，由于无法对碎片化的存储空间进行有效整合和利用，导致大量可利用的存储空间被浪费，同时给存储空间的分配和管理都造成不便的问题，本公开能够对碎片化的存储空间进行有效整合和分配，提升了存储空间的利用率及分配效率，同时也实现了针对存储空间的有效管理。
实施例二
为了便于理解，在详细介绍本实施例的实现方案之前，先介绍本实施例中的分布式系统。其中，本实施例中的分布式系统至少包括多个分布式的数据分片集群。如图2a和图2b所示，图2a示出了本公开中的分布式系统200的结构示意图；图2b示出了分布式系统200中每个数据分片集群包含的数据节点的示意图。
具体地，分布式系统200包括路由节点210、配置节点(也叫config节点)220、以及与路由节点210相连接的多个数据分片集群230。其中，图2a所示的分布式系统200包含一个路由节点210、以及一个config节点220。但是具体实施中，为了提升灾备能力，路由节点210、以及config节点220分别可以有多个。图2b示出了分布式系统200中每个数据分片集群包含的数据节点的示意图。具体地，每个数据分片集群中进一步包括三个数据节点，分别为图2b所示的M节点(主节点)、S节点(从节点)以及A节点(投票节点)。其中，S节点中存储的数据是M节点数据的镜像数据，相当于M节点的备份数据。当M节点的数据出现故障时，通过A节点使S节点与M节点进行调换，以此来保证存储数据的安全与准确。
其中，路由节点210中设置有对应的路由规则，按照该路由规则能够实现屏蔽分片结构，获取完整数据的目的。具体地，由于在分布式系统中，一个完整的数据被进行切分，形成多个数据分片，每个数据分片被存储在对应的数据分片集群中(不同的数据分片集群中存储的数据是不同的)，对应地，多个数据分片则被存储在多个数据分片集群中。但是，在互联网业务使用中，由于读取或者写入的数据必须保证是一个完整的数据，而不是分散在多个数据分片集群中的多个数据分片，因此在分布式系统中，对于与一个完整数据对应的多个数据分片，需要设定一定的规则去读取该完整数据分散在多个数据分片集群中的数据分片，使读取出的数据分片能够组成对应的完整数据；同样地，在向分布式系统中的多个数据分片集群中写入数据分片时，也需要设置一定的规则，以保证写入的多个数据分片在被读取时能够组成对应的完整数据。因此路由节点中设置有对应的路由规则，用于实现上述针对数据的读取以及写入过程。具体实施中，上述路由规则的实现一般包括两种，一种为范围规则，一种为哈希规则。其中，范围规则是指将一定范围内的数据分 片存储到对应的数据分片集群中。例如，一共有30个数据分片，编号分别为1、2、3…30，一共有三个数据分片集群，分别为集群1、集群2以及集群3。在使用范围规则时，其具体为：将上述编号为1-10的数据分片对应存储到集群1、将上述编号为11-20的数据分片对应存储到集群2、将上述编号为21-30的数据分片对应存储到集群3。另一种规则为哈希规则(也可以叫做随机规则)，即通过对每个数据分片进行哈希计算，并根据哈希计算结果确定与每个数据分片对应的数据分片集群。在这里，无论上述那种路由规则，其目的都是为了能够在使用业务时能够读取到完整的数据，保证数据写入与读取的完整性。
config节点中存储有记录各个数据分片集群的位置信息、变更日志等信息的元数据信息。当数据分片集群中的数据出现变化(例如对存储的数据进行插入、删除、拆分以及迁移等相关操作)时，对应的数据变化被记录在config节点中。
由于分布式系统中包含多个数据分片集群，为了使存储的数据被均匀分配在多个数据分片集群，分布式系统的机制中还包括均衡机制(或者叫balance机制)。该均衡机制用于实现：当多个数据分片集群中存储的数据量相差较大时，则将存储的数据量相对较大的数据分片集群中的存储数据按照均衡机制转移到存储的数据量相对较小的数据分片集群，使各个数据分片集群中存储的数据量实现均衡。数据均衡操作中会对数据进行数据转移，进行数据转移后的存储空间会被设置上对应的标记，以表明该存储空间可以被写入新的数据。另外，均衡机制的实现还包括：例如，当存储的原始数据中要插入新的数据时，若新插入数据的数据量较大，则对该原始数据也进行迁移操作。例如，若存储有一条数据，该条数据中包括一个标识(ID)和一个名称(name)，在为该条数据分配存储空间时，只需要分配足够存储上述一个ID和一个name的存储空间即可。例如数据分片集群1中的剩余存储空间刚好或者略大于上述包含一个ID和一个name的数据的占用空间，则将该条数据存储在数据分片集群1中。但是，在后续存储数据的过程中，针对该条数据需要插入一个简历信息，该简历信息占用的存储空间较大，存储该条数据的存储空间不够存储上述简历信息，则对应将包含一个ID和一个name的数 据进行迁移，例如，通过相关计算确定数据分片集群2中的剩余存储空间足够存储插入简历信息后的数据，则将该条数据从数据分片集群1迁移到数据分片集群2中，同时在数据分片集群2中对这条数据进行插入简历信息的操作。同时，在进行上述迁移操作之后，将每条数据的迁移操作的相关信息(例如，原存储的数据分片集群信息、数据大小、数据位置等信息，以及现存储的数据分片集群信息、数据大小、数据位置等信息)同步更新到config节点中。
以上是针对本实施例中分布式系统的简单介绍。分布式系统的优点在于实现了存储空间的无限扩容，即图2中的数据分片集群的数量可以无限增加，存储空间的大小可以无限扩展。
下面来详细介绍本实施例中的方案。其中，本实施例中的方法执行主体可以为分布式系统中的路由节点，或者，也可以为分布式系统中的各个数据分片集群。图3示出了根据本公开实施例二提供的基于分布式系统的存储空间整理方法的流程示意图。如图3所示，该方法包括以下步骤：
步骤S310：分别针对每个数据分片集群，获取并分析与该数据分片集群的数据存储状态相对应的元数据信息。
具体地，分布式系统至少包括多个分布式的数据分片集群。在分布式系统中，为了提升存储容量，可以将待存储的数据切分为多个分片，将各个分片分别存储在不同的数据分片集群上。因此，数据分片集群的具体数量可以根据存储的数据量进行设定或调整，本公开对此不作限定。元数据主要用于记录已存储的数据的属性信息，元数据的具体内容可以包括多种。例如，元数据用于记录各条存储数据的具体内容、存储位置、数据大小等。利用元数据能够实现历史数据查询、资源查找、文件记录等多种功能。其中，通过元数据记录的上述信息统称为元数据信息。
其中，在本实施例中，由于分布式系统中包括用于存储与各个数据分片集群的数据存储状态相对应的元数据信息的配置服务器(即上述的config节点)，因此，在获取与每个数据分片集群的数据存储状态相对应的元数据信息时，通过上述配置服务器来获取。具体地，针对每个数据分片集群，首先从config节点中获取与该数据分片集群的数据存储状态相对应的元数据信 息，然后对该元数据信息进行分析，从元数据信息中查找包含空闲标记的存储空间，并将查找到的存储空间确定为空闲存储空间。其中，上述空闲标记可以包括：因数据均衡操作、数据更新操作和/或数据删除操作而产生的无效标记和/或删除标记。具体地，在分布式系统中，当一条数据被删除后，该数据所占用的存储空间会被对应设置删除标记，设置有删除标记的存储空间中的数据可以被涂改，以表明删除数据后的存储空间能够进行复用(即该存储空间可以被重新写入新的数据)。同样地，数据在进行数据均衡操作、数据更新操作等相关操作后也会产生上述类似的可以被复用的存储空间，同时该存储空间也会被对应设置删除标记和/或无效标记，以表明该存储空间可以被进行复用。因此，在确定空闲存储空间时，只要确定与该数据分片集群的数据存储状态相对应的元数据信息中包含有空闲标记，与该元数据对应的存储空间则可以确定为空闲存储空间。
具体实施中，为了便于实现数据的存储和管理，每个数据分片集群包括多个用于存储数据的数据单元，相应地，每个数据分片集群中包含的空闲存储空间的存储容量和/或位置分布具体为：该数据分片集群中的各个空闲数据单元的存储容量，和/或各个空闲数据单元的位置分布。其中，数据单元的划分粒度可由本领域技术人员灵活设定。例如，上述数据单元还可以包括以下中的至少一种：数据块、数据分区、以及数据子分区。其中，一个数据块包括至少一个数据分区，一个数据分区包括至少一个数据子分区。其中，每个数据块的容量大小一般为64MB，具体实施中，每个数据块的大小还可以根据存储文件的大小进行调整，例如某个文件太大了，一个数据块的容量不够存储一个文件，则可以将该数据块的容量由默认的64MB调整为128MB、256MB或更大容量，以实现对对应文件的存储。在这里，上述数据块的容量大小可以由本领域技术人员根据实际情况进行调整，本公开对此不作限制。同样地，数据分区、或者数据子分区的设置以及调整方式与数据块类似，此处不再赘述。本领域技术人员可根据实际需求确定数据单元的具体粒度，例如，若只需针对大块的空闲空间进行整理，可以将各个数据块作为最小的数据单元；又如，若需要针对细碎的空闲空间进行整理，可以将各个数据子分区作为最小的数据单元。
步骤S320：根据分析结果确定该数据分片集群中包含的空闲存储空间的存储容量和/或位置分布。
具体地，在本实施例中，根据步骤S310中确定的空闲存储空间，进一步根据与该空闲存储空间对应的元数据信息确定该空闲存储空间的存储容量和/或位置分布信息。
其中，由于每个数据分片集群包括多个用于存储数据的数据单元。则确定数据分片集群中包含的空闲存储空间的存储容量具体可以为：确定该数据分片集群中的各个空闲数据单元的存储容量；确定数据分片集群中包含的空闲存储空间的位置分布具体可以为：确定各个空闲数据单元的位置分布。
具体地，在确定各个空闲数据单元的位置分布时，其具体过程可以为：根据各个空闲数据单元的位置分布，确定位置相邻且连续的多个空闲数据单元。具体地，在一种可选的方案中，若多个空闲数据单元中，每两个相邻的空闲数据单元之间没有非空闲数据单元，则将上述多个空闲数据单元确定为位置相邻且连续的多个空闲数据单元。
在另一种可选的方案中，若多个空闲数据单元中，至少两个相邻的空闲数据单元之间具有非空闲数据单元，且上述非空闲数据单元中的已存储数据量小于预设阈值，则针对上述非空闲数据单元中已存储的全部数据执行数据迁移操作，并将上述多个空闲数据单元以及执行数据迁移操作后的非空闲数据单元确定为位置相邻且连续的多个空闲数据单元。其中，上述预设阈值的具体数值可以由本领域技术人员根据实际情况进行设置，本公开对此不作限制。上述经过数据迁移操作的存储单元被设置有空闲标记，因此该存储单元可以看作是空闲存储单元。
在这里，要说明的是，位置相邻且连续的多个空闲数据单元的确定方式包括但不限于上述两种实现方式，在具体实施中，本领域技术人员还可以根据实际情况将上述确定方式设置为其它实现方式，只要能够确定位置相邻且连续的多个空闲数据单元，本公开对其具体的确定方式不作限制。
本领域技术人员能够理解的是，上述的空闲存储单元既可以是数据块、也可以是数据分区或者数据子分区，具体取决于本公开的整理粒度。
步骤S330：针对位置相邻且连续的多个空闲数据单元进行整理。
具体地，本步骤实现的方式可以有多种。其中，在一种可选的方案中，可以将位置相邻且连续的多个空闲数据单元合并为一个待分配数据单元，并为该待分配数据单元设置对应的待分配标记。本方案直接将位置相邻且连续的多个空闲数据单元合并为一个更大的存储空间(待分配数据单元)，实现了对多个空闲数据单元直接且快速的合并。在该种方式中，只是将位置相邻且连续的多个空闲数据单元合并为一个待分配数据单元，具体地，将位置相邻且连续的多个空闲数据单元合并为一个整体性空间，并为该整体性空间赋予待分配标记。该待分配标记用于表明：该整体性空间为空闲的、尚未分配的空间。即：尚未将该整体性空间划分为若干个具体大小的数据单元。例如，假设位置相邻且连续的多个空闲数据单元合并后得到的整体性空间的大小为900M，该900M空间对应于待分配标记，因而在后续过程中，本领域技术人员可以将这900M空间划分为若干个数据单元，并根据待存储的数据大小分别设定各个数据单元的大小。由此可见，待分配的数据单元在后续过程中的灵活性较大，可根据存储需要灵活划分为至少一个相等或不等的数据单元。
在另一种可选的方案中，将位置相邻且连续的多个空闲数据单元回收后重新分配为一个或多个待使用数据单元。其中，设定各个待使用数据单元的存储容量，并为各个待使用数据单元设置对应的待使用标记。在本方案中，首先将位置相邻且连续的多个空闲数据单元进行回收，然后根据存储数据的大小对回收的多个空闲数据单元进行整合，获取与存储数据大小对应的存储空间，使整合后的空闲数据单元能够被直接使用；或者，也可以根据步骤S320中设置的数据块、数据分区或者数据子分区的容量大小对回收的多个空闲数据单元进行整合，将多个空闲数据单元整合成多个数据块、数据分区或者数据子分区等等，并为其设置对应的待使用标记，以表明该数据单元能够被直接使用。例如，假设位置相邻且连续的多个空闲数据单元回收后得到的整体性空间的大小为600MB，进一步地，将该600MB存储空间进一步划分为三个设定大小的数据单元，例如，每个数据单元分别为100MB、200MB和300MB，并分别为每个数据单元设置待使用标记。由此可见，在本方案中，针对回收后的存储空间设定了数据单元的数量及每个数据单元的容量，因此在后续对数据进行分配存储时，可以直接根据各个待使用的存储空间的存储 容量对数据进行存储。在这里，要说明的是，上述两种方案可以单独使用，也可以结合使用。
可选地，在针对位置相邻且连续的多个空闲数据单元进行整理时，其过程还可以为：根据位置相邻且连续的多个空闲数据单元的存储容量总和，对位置相邻且连续的多个空闲数据单元进行整理。具体实施中，例如，可以设置一个存储容量阈值，根据位置相邻且连续的多个空闲数据单元的存储容量总和与该存储容量阈值之间的关系确定具体的整理方案。例如，当上述多个空闲数据单元的存储容量总和大于上述存储容量阈值时，对上述位置相邻且连续的多个空闲数据单元进行合并。或者，还可以将位置相邻且连续的多个空闲数据单元的存储容量总和与相邻的非空闲的存储空间的存储容量进行比较，若上述存储容量总和大于相邻的非空闲的存储空间的存储容量，则将上述非空闲的存储空间中的数据进行转移，使该非空闲的存储空间转化为空闲存储空间，然后对转化后的位置相邻且连续的多个空闲数据单元进行整合，等等。
另外，本领域技术人员还可以根据实际情况将针对位置相邻且连续的多个空闲数据单元进行整理的整理方式设置为除上述列举的方式以外的其它实现方式，只要能够对位置相邻且连续的多个空闲数据单元进行有效整合，本公开对其整理方式不作限制。
步骤S340：根据整理结果更新与该数据分片集群的数据存储状态相对应的元数据信息。
具体地，元数据信息能够记录数据分片集群的数据存储状态，以便针对每个数据分片集群进行分配和管理。在本步骤中，根据步骤S330中的整理结果，更新配置服务器中存储的与该数据分片集群的数据存储状态相对应的元数据信息(即更新config节点中的元数据信息)，以便保持数据分片集群的数据存储状态信息与其对应的元数据信息中记录的对应信息的一致性，同时，经过更新的元数据信息还能够进一步用于对整理结果进行管理，确保后续针对存储空间进行管理时的准确性。
在本公开提供的基于分布式系统的存储空间整理的方法中，能够获取分布式系统包括的多个分布式的数据分片集群对应的存储状态的元数据信息， 并通过分析该元数据信息来确定每个数据分片集群中对应的空闲存储空间，并通过对该空闲存储空间的存储容量和/或位置分布进行分析来对数据分片集群中包含的空闲存储空间进行整理，将碎片化的、不能进行利用的空闲存储空间整合为一个较大的、可利用的存储空间，提升了存储空间的利用率及分配效率，同时也实现了针对存储空间的有效管理。另外，本实施例中的方案还可以对空闲存储空间进行更加细致的划分，使针对碎片化的空闲存储空间整理的准确性及针对碎片化的空闲存储空间利用率更高。
实施例三
图4示出了根据本公开实施例三提供的基于分布式系统的存储空间整理装置的结构框图。如图4所示，该装置包括：分析模块41、确定模块42、整理模块43以及更新模块44。
分析模块41适于分别针对每个数据分片集群，获取并分析与该数据分片集群的数据存储状态相对应的信息。
具体地，分布式系统至少包括多个分布式的数据分片集群。在分布式系统中，为了提升存储容量，可以将待存储的数据切分为多个分片，将各个分片分别存储在不同的数据分片集群上。因此，数据分片集群的具体数量可以根据存储的数据量进行设定或调整，本公开对此不作限定。
分析模块41所获取的信息可以包括元数据信息。元数据主要用于记录已存储的数据的属性信息，元数据的具体内容可以包括多种。例如，元数据用于记录各条存储数据的具体内容、存储位置、数据大小等。利用元数据能够实现历史数据查询、资源查找、文件记录等多种功能。其中，通过元数据记录的上述信息统称为元数据信息。
可选地，在本实施例的分布式系统中，还可以设置用于存储与各个数据分片集群的数据存储状态相对应的元数据信息的配置服务器，因此，在获取与每个数据分片集群的数据存储状态相对应的元数据信息时，通过上述配置服务器来获取。针对每个数据分片集群，分析模块41首先获取与该数据分片集群的数据存储状态相对应的元数据信息，然后对该元数据信息进行分析，从元数据信息中查找包含空闲标记的存储空间，并将查找到的存储空间 确定为空闲存储空间。其中，上述空闲标记可以包括：因数据均衡操作、数据更新操作和/或数据删除操作而产生的无效标记和/或删除标记。具体地，在分布式系统中，当一条数据被删除后，该数据所占用的存储空间会被对应设置删除标记，设置有删除标记的存储空间中的数据可以被涂改，以表明删除数据后的存储空间能够进行复用(即该存储空间可以被重新写入新的数据)。同样地，数据在进行数据均衡操作、数据更新操作等相关操作后也会产生上述类似的可以被复用的存储空间，同时该存储空间也会被对应设置删除标记和/或无效标记，以表明该存储空间可以被进行复用。因此，在确定空闲存储空间时，只要确定与该数据分片集群的数据存储状态相对应的元数据信息中包含有空闲标记，与该元数据对应的存储空间则可以确定为空闲存储空间。
确定模块42适于根据分析结果确定该数据分片集群中包含的空闲存储空间的存储容量和/或位置分布。
具体地，确定模块42根据分析模块41确定的空闲存储空间，从与该空闲存储空间对应的元数据信息中获取该空闲存储空间的存储容量和/或位置分布信息。
其中，每个数据分片集群还可以被划分为多个用于存储数据的数据单元。则确定数据分片集群中包含的空闲存储空间的存储容量具体可以为：确定该数据分片集群中的各个空闲数据单元的存储容量；同样地，确定数据分片集群中包含的空闲存储空间的位置分布具体可以为：确定各个空闲数据单元的位置分布。其中，上述划分数据单元的划分方式以及划分的数据单元的容量大小可以由本领域技术人员根据实际情况进行设置，本公开对此不作限制。
可选地，具体实施中，为了便于实现数据的存储和管理，每个数据分片集群包括多个用于存储数据的数据单元，相应地，每个数据分片集群中包含的空闲存储空间的存储容量和/或位置分布具体为：该数据分片集群中的各个空闲数据单元的存储容量，和/或各个空闲数据单元的位置分布。其中，数据单元的划分粒度可由本领域技术人员灵活设定。例如，针对上述每个数据单元，其存储空间还可以被进行进一步的细分。具体地，上述数据单元还可以 进一步包括以下中的至少一种：数据块、数据分区、以及数据子分区。其中，一个数据块包括至少一个数据分区，一个数据分区包括至少一个数据子分区。则确定模块42在确定空闲存储空间的存储容量和/或位置分布时，可以进一步确定该数据分片集群中的各个空闲数据单元中的数据块、数据分区、或者数据子分区的存储容量，和/或各个空闲数据单元中的数据块、数据分区、或者数据子分区的位置分布。其中，每个数据块的容量大小一般为64MB，具体实施中，每个数据块的大小还可以根据存储文件的大小进行调整，例如某个文件太大了，一个数据块的容量不够存储一个文件，则可以将该数据块的容量由默认的64MB调整为128MB、256MB或更大容量，以实现对对应文件的存储。在这里，上述数据块的容量大小可以由本领域技术人员根据实际情况进行调整，本公开对此不作限制。同样地，数据分区、或者数据子分区的设置以及调整方式与数据块类似，此处不再赘述。本领域技术人员可根据实际需求确定数据单元的具体粒度，例如，若只需针对大块的空闲空间进行整理，可以将各个数据块作为最小的数据单元；又如，若需要针对细碎的空闲空间进行整理，可以将各个数据子分区作为最小的数据单元。
具体地，确定模块42在确定各个空闲数据单元的位置分布时，其具体过程可以为：根据各个空闲数据单元的位置分布，确定位置相邻且连续的多个空闲数据单元。具体地，在一种可选的方案中，若多个空闲数据单元中，每两个相邻的空闲数据单元之间没有非空闲数据单元，则确定模块42将上述多个空闲数据单元确定为位置相邻且连续的多个空闲数据单元。
在另一种可选的方案中，若多个空闲数据单元中，至少两个相邻的空闲数据单元之间具有非空闲数据单元，且上述非空闲数据单元中的已存储数据量小于预设阈值，则确定模块42针对上述非空闲数据单元中已存储的全部数据执行数据迁移操作，并将上述多个空闲数据单元以及执行数据迁移操作后的非空闲数据单元确定为位置相邻且连续的多个空闲数据单元。其中，上述预设阈值的具体数值可以由本领域技术人员根据实际情况进行设置，本公开对此不作限制。上述经过数据迁移操作的存储单元被设置有空闲标记，因此该存储单元可以看作是空闲存储单元。
在这里，要说明的是，位置相邻且连续的多个空闲数据单元的确定方式 包括但不限于上述两种实现方式，在具体实施中，本领域技术人员还可以根据实际情况将确定模块42的确定方式设置为其它实现方式，只要能够确定位置相邻且连续的多个空闲数据单元，本公开对确定模块42的确定方式不作限制。
本领域技术人员能够理解的是，上述的空闲存储单元既可以是数据块、也可以是数据分区或者数据子分区，具体取决于本公开的整理粒度。
整理模块43适于根据空闲存储空间的存储容量和/或位置分布对该数据分片集群中包含的空闲存储空间进行整理。
具体地，整理模块43对空闲存储空间进行整理的方式可以有多种。其中，在一种可选的方案中，整理模块43可以将位置相邻且连续的多个空闲数据单元合并为一个待分配数据单元，并为该待分配数据单元设置对应的待分配标记。本方案直接将位置相邻且连续的多个空闲数据单元合并为一个更大的存储空间(待分配数据单元)，实现了多个空闲数据单元直接且快速地合并。在该种方式中，只是将位置相邻且连续的多个空闲数据单元合并为一个待分配数据单元，具体地，将位置相邻且连续的多个空闲数据单元合并为一个整体性空间，并为该整体性空间赋予待分配标记。该待分配标记用于表明：该整体性空间为空闲的、尚未分配的空间。即：尚未将该整体性空间划分为若干个具体大小的数据单元。例如，假设位置相邻且连续的多个空闲数据单元合并后得到的整体性空间的大小为900MB，该900MB空间对应于待分配标记，因而在后续过程中，本领域技术人员可以将这900MB空间划分为若干个数据单元，并根据待存储的数据大小分别设定各个数据单元的大小。由此可见，待分配的数据单元在后续过程中的灵活性较大，可根据存储需要灵活划分为至少一个相等或不等的数据单元。
在另一种可选的方案中，整理模块43将位置相邻且连续的多个空闲数据单元回收后重新分配为一个或多个待使用数据单元。其中，设定各个待使用数据单元的存储容量，并为各个待使用数据单元设置对应的待使用标记。在本方案中，整理模块43首先将位置相邻且连续的多个空闲数据单元进行回收，然后根据存储数据的大小对回收的多个空闲数据单元进行整合，获取与存储数据大小对应的存储空间，使整合后的空闲数据单元能够被直接使 用；或者，也可以根据数据块、数据分区或者数据子分区的容量大小对回收的多个空闲数据单元进行整合，将多个空闲数据单元整合成多个数据块、数据分区或者数据子分区等等，并为其设置对应的待使用标记，以表明该数据单元能够被直接使用。例如，假设位置相邻且连续的多个空闲数据单元回收后得到的整体性空间的大小为600MB，进一步地，将该600MB存储空间进一步划分为三个设定大小的数据单元，例如，每个数据单元分别为100MB、200MB和300MB，并分别为每个数据单元设置待使用标记。由此可见，在本方案中，针对回收的存储空间设定了数据单元的数量及每个数据单元的容量，因此在后续对数据进行分配存储时，可以直接根据各个待使用的存储空间的存储容量对数据进行存储。在这里，要说明的是，上述两种方案可以单独使用，也可以结合使用。
可选地，整理模块43在针对位置相邻且连续的多个空闲数据单元进行整理时，其过程还可以为：根据位置相邻且连续的多个空闲数据单元的存储容量总和，对位置相邻且连续的多个空闲数据单元进行整理。具体实施中，例如，可以设置一个存储容量阈值，根据位置相邻且连续的多个空闲数据单元的存储容量总和与该存储容量阈值之间的关系确定具体的整理方案。例如，当上述多个空闲数据单元的存储容量总和大于上述存储容量阈值时，对上述位置相邻且连续的多个空闲数据单元进行合并。或者，还可以将位置相邻且连续的多个空闲数据单元的存储容量总和与相邻的非空闲的存储空间的存储容量进行比较，若上述存储容量总和大于相邻的非空闲的存储空间的存储容量，则对位置相邻且连续的多个空闲数据单元进行整合；或将非空闲的存储空间中的数据进行转移，使该非空闲的存储空间转化为空闲存储空间，然后对转化后的位置相邻且连续的多个空闲数据单元进行整合，等等。
另外，本领域技术人员还可以根据实际情况将整理模块43的整理方式设置为除上述列举的方式以外的其它实现方式，只要能够对位置相邻且连续的多个空闲数据单元进行有效整合，本公开对整理模块43的具体整理方式不作限制。
更新模块44适于根据整理结果更新与该数据分片集群的数据存储状态相对应的信息。
具体地，信息可以包括元数据信息，元数据信息能够记录数据分片集群的数据存储状态，以便针对每个数据分片集群进行分配和管理。在本步骤中，更新模块44将整理模块43的整理结果更新到与相应的数据分片集群的数据存储状态相对应的元数据信息中，以便保持数据分片集群的数据存储状态信息与其对应的元数据信息中记录的对应信息的一致性，同时，经过更新的元数据信息还能够进一步用于对整理结果进行管理，确保后续针对存储空间进行管理时的准确性。
可选地，当分布式系统中设置有配置服务器，并从该配置服务器中获取与各个数据分片集群的数据存储状态相对应的元数据信息时，更新模块44对应的更新过程为：根据整理结果更新上述配置服务器中存储的与该数据分片集群的数据存储状态相对应的元数据信息。
上述各个模块的具体结构和工作原理可参照方法实施例中相应步骤的描述，此处不再赘述。
在本公开提供的基于分布式系统的存储空间整理的装置中，分布式系统至少包括多个分布式的数据分片集群，在对存储空间进行整理时，首先通过分析模块41分别获取并分析每个数据分片集群的数据存储状态想对应的信息，然后通过确定模块42根据分析结果确定该数据分片集群中包含的空闲存储空间的存储容量和/或位置分布，并通过整理模块43根据确定结果对数据分片集群中包含的空闲存储空间进行整理，最后通过更新模块44根据整理结果更新与该数据分片集群存储状态相对应的信息。由此可见，本公开中的方案能够解决在分布式系统中，由于无法对碎片化的存储空间进行有效整合和利用，导致大量可利用的存储空间被浪费，同时给存储空间的分配和管理都造成不便的问题。本公开能够对碎片化的存储空间进行有效整合和分配，提升了存储空间的利用率及分配效率，同时也实现了针对存储空间的有效管理。另外，本实施例中的方案还可以对空闲存储空间进行更加细致的划分，使针对碎片化的空闲存储空间整理的准确性及针对碎片化的空闲存储空间利用率更高。
实施例四
图5示出了根据本公开实施例四提供的分布式系统500的结构框图。如图5所示，该系统包括：基于分布式系统的存储空间整理装置400以及多个分布式的数据分片集群510。其中，基于分布式系统的存储空间整理装置400包括：分析模块41、确定模块42、整理模块43以及更新模块44。
实施例五
本申请实施例五提供了一种非易失性计算机可读存储介质，非易失性计算机可读存储介质存储有至少一可执行指令，该计算机可执行指令可执行上述任意方法实施例中的基于分布式系统的存储空间整理方法。
实施例六
图6示出了根据本公开实施例六的一种电子设备的结构示意图，本公开具体实施例并不对电子设备的具体实现做限定。
如图6所示，该电子设备可以包括：处理器(processor)602、通信接口(Communications Interface)604、存储器(memory)606、以及通信总线608。
其中：
处理器602、通信接口604、以及存储器606通过通信总线608完成相互间的通信。
通信接口604，用于与其它设备比如客户端或其它服务器等的网元通信。
处理器602，用于执行程序610，具体可以执行上述基于分布式系统的存储空间整理方法实施例中的相关步骤。
具体地，程序610可以包括程序代码，该程序代码包括计算机操作指令。
处理器602可能是中央处理器CPU，或者是特定集成电路ASIC(Application Specific Integrated Circuit)，或者是被配置成实施本公开实施例的一个或多个集成电路。电子设备包括的一个或多个处理器，可以是同一类型的处理器，如一个或多个CPU；也可以是不同类型的处理器，如一个或多个CPU以及一个或多个ASIC。
存储器606，用于存放程序610。存储器606可能包含高速RAM存储器，也可能还包括非易失性存储器(non-volatile memory)，例如至少一个磁盘存储器。
程序610具体可以用于使得处理器602执行以下操作：分别针对每个数据分片集群，获取并分析与该数据分片集群的数据存储状态相对应的元数据信息；根据分析结果确定该数据分片集群中包含的空闲存储空间的存储容量和/或位置分布；根据所述空闲存储空间的存储容量和/或位置分布对该数据分片集群中包含的空闲存储空间进行整理；根据整理结果更新与该数据分片集群的数据存储状态相对应的元数据信息。
在一种可选的实施方式中，每个数据分片集群包括多个用于存储数据的数据单元，则每个数据分片集群中包含的空闲存储空间的存储容量和/或位置分布具体为：该数据分片集群中的各个空闲数据单元的存储容量，和/或各个空闲数据单元的位置分布；
程序610进一步使得处理器602执行以下操作：根据各个空闲数据单元的位置分布，确定位置相邻且连续的多个空闲数据单元，针对所述位置相邻且连续的多个空闲数据单元进行整理。
在一种可选的实施方式中，程序610进一步使得处理器602执行以下操作：若多个空闲数据单元中，每两个相邻的空闲数据单元之间没有非空闲数据单元，则将所述多个空闲数据单元确定为位置相邻且连续的多个空闲数据单元；或者，若多个空闲数据单元中，至少两个相邻的空闲数据单元之间具有非空闲数据单元，且所述非空闲数据单元中的已存储数据量小于预设阈值，则针对所述非空闲数据单元中已存储的全部数据执行数据迁移操作，并将所述多个空闲数据单元以及执行数据迁移操作后的非空闲数据单元确定为位置相邻且连续的多个空闲数据单元。
在一种可选的实施方式中，程序610进一步使得处理器602执行以下操作：将所述位置相邻且连续的多个空闲数据单元合并为一个待分配数据单元；其中，为该待分配数据单元设置对应的待分配标记；或者，将所述位置相邻且连续的多个空闲数据单元回收后重新分配为一个或多个待使用数据单元；其中，设定各个待使用数据单元的存储容量，并为各个待使用数据单 元设置对应的待使用标记。
在一种可选的实施方式中，程序610进一步使得处理器602执行以下操作：根据所述位置相邻且连续的多个空闲数据单元的存储容量总和，对所述位置相邻且连续的多个空闲数据单元进行整理。
在一种可选的实施方式中，程序610进一步使得处理器602执行以下操作：从所述元数据信息中查找包含空闲标记的存储空间，并将查找到的存储空间确定为空闲存储空间；其中，所述空闲标记包括：因数据均衡操作、数据更新操作和/或数据删除操作而产生的无效标记和/或删除标记。
在一种可选的实施方式中，所述数据单元包括以下中的至少一种：数据块、数据分区、以及数据子分区；其中，一个数据块包括至少一个数据分区，一个数据分区包括至少一个数据子分区。
在一种可选的实施方式中，所述分布式系统进一步包括：用于存储与各个数据分片集群的数据存储状态相对应的元数据信息的配置服务器；
则程序610进一步使得处理器602执行以下操作：
从所述配置服务器中获取与各个数据分片集群的数据存储状态相对应的元数据信息；
根据整理结果更新所述配置服务器中存储的与该数据分片集群的数据存储状态相对应的元数据信息。
在此提供的算法和显示不与任何特定计算机、虚拟系统或者其它设备固有相关。各种通用系统也可以与基于在此的示教一起使用。根据上面的描述，构造这类系统所要求的结构是显而易见的。此外，本公开也不针对任何特定编程语言。应当明白，可以利用各种编程语言实现在此描述的本公开的内容，并且上面对特定语言所做的描述是为了披露本公开的最佳实施方式。
在此处所提供的说明书中，说明了大量具体细节。然而，能够理解，本公开的实施例可以在没有这些具体细节的情况下实践。在一些实例中，并未详细示出公知的方法、结构和技术，以便不模糊对本说明书的理解。
类似地，应当理解，为了精简本公开并帮助理解各个公开方面中的一个或多个，在上面对本公开的示例性实施例的描述中，本公开的各个特征有时 被一起分组到单个实施例、图、或者对其的描述中。然而，并不应将该公开的方法解释成反映如下意图：即所要求保护的本公开要求比在每个权利要求中所明确记载的特征更多的特征。更确切地说，如下面的权利要求书所反映的那样，公开方面在于少于前面公开的单个实施例的所有特征。因此，遵循具体实施方式的权利要求书由此明确地并入该具体实施方式，其中每个权利要求本身都作为本公开的单独实施例。
本领域那些技术人员可以理解，可以对实施例中的设备中的模块进行自适应性地改变并且把它们设置在与该实施例不同的一个或多个设备中。可以把实施例中的模块或单元或组件组合成一个模块或单元或组件，以及此外可以把它们分成多个子模块或子单元或子组件。除了这样的特征和/或过程或者单元中的至少一些是相互排斥之外，可以采用任何组合对本说明书(包括伴随的权利要求、摘要和附图)中公开的所有特征以及如此公开的任何方法或者设备的所有过程或单元进行组合。除非另外明确陈述，本说明书(包括伴随的权利要求、摘要和附图)中公开的每个特征可以由提供相同、等同或相似目的的替代特征来代替。
此外，本领域的技术人员能够理解，尽管在此所述的一些实施例包括其它实施例中所包括的某些特征而不是其它特征，但是不同实施例的特征的组合意味着处于本公开的范围之内并且形成不同的实施例。例如，在下面的权利要求书中，所要求保护的实施例的任意之一都可以以任意的组合方式来使用。
本公开的各个部件实施例可以以硬件实现，或者以在一个或者多个处理器上运行的软件模块实现，或者以它们的组合实现。本领域的技术人员应当理解，可以在实践中使用微处理器或者数字信号处理器(DSP)来实现根据本公开实施例的基于分布式系统的存储空间整理设备中的一些或者全部部件的一些或者全部功能。本发明还可以实现为用于执行这里所描述的方法的一部分或者全部的设备或者装置程序(例如，计算机程序和计算机程序产品)。这样的实现本发明的程序可以存储在计算机可读介质上，或者可以具有一个或者多个信号的形式。这样的信号可以从因特网网站上下载得到，或者在载体信号上提供，或者以任何其他形式提供。
应该注意的是上述实施例对本发明进行说明而不是对本发明进行限制，并且本领域技术人员在不脱离所附权利要求的范围的情况下可设计出替换实施例。在权利要求中，不应将位于括号之间的任何参考符号构造成对权利要求的限制。单词“包含”不排除存在未列在权利要求中的元件或步骤。位于元件之前的单词“一”或“一个”不排除存在多个这样的元件。本发明可以借助于包括有若干不同元件的硬件以及借助于适当编程的计算机来实现。在列举了若干装置的单元权利要求中，这些装置中的若干个可以是通过同一个硬件项来具体体现。单词第一、第二、以及第三等的使用不表示任何顺序。可将这些单词解释为名称。

Claims (22)

  1. 一种基于分布式系统的存储空间整理方法，其中，所述分布式系统至少包括多个分布式的数据分片集群，所述方法包括：分别针对每个数据分片集群，获取并分析与该数据分片集群的数据存储状态相对应的信息；根据分析结果确定该数据分片集群中包含的空闲存储空间的存储容量和/或位置分布；根据所述空闲存储空间的存储容量和/或位置分布对该数据分片集群中包含的空闲存储空间进行整理；以及根据整理结果更新与该数据分片集群的数据存储状态相对应的信息。
  2. 根据权利要求1所述的方法，其中，所述信息包括元数据信息。
  3. 根据权利要求2所述的方法，其中，每个数据分片集群包括多个用于存储数据的数据单元，则每个数据分片集群中包含的空闲存储空间的存储容量和/或位置分布具体为：该数据分片集群中的各个空闲数据单元的存储容量，和/或各个空闲数据单元的位置分布；
    且所述根据所述空闲存储空间的存储容量和/或位置分布对该数据分片集群中包含的空闲存储空间进行整理的步骤具体包括：
    根据各个空闲数据单元的位置分布，确定位置相邻且连续的多个空闲数据单元，针对所述位置相邻且连续的多个空闲数据单元进行整理。
  4. 根据权利要求3所述的方法，其中，所述根据各个空闲数据单元的位置分布，确定位置相邻且连续的多个空闲数据单元的步骤具体包括：
    若多个空闲数据单元中，每两个相邻的空闲数据单元之间没有非空闲数据单元，则将所述多个空闲数据单元确定为位置相邻且连续的多个空闲数据单元；或者，
    若多个空闲数据单元中，至少两个相邻的空闲数据单元之间具有非空闲数据单元，且所述非空闲数据单元中的已存储数据量小于预设阈值，则针对所述非空闲数据单元中已存储的全部数据执行数据迁移操作，并将所述多个空闲数据单元以及执行数据迁移操作后的非空闲数据单元确定为位置相邻且连续的多个空闲数据单元。
  5. 根据权利要求3或4所述的方法，其中，所述针对所述位置相邻且连 续的多个空闲数据单元进行整理的步骤具体包括：
    将所述位置相邻且连续的多个空闲数据单元合并为一个待分配数据单元；其中，为该待分配数据单元设置对应的待分配标记；或者，
    将所述位置相邻且连续的多个空闲数据单元回收后重新分配为一个或多个待使用数据单元；其中，设定各个待使用数据单元的存储容量，并为各个待使用数据单元设置对应的待使用标记。
  6. 根据权利要求3-5任一所述的方法，其中，所述针对所述位置相邻且连续的多个空闲数据单元进行整理的步骤具体包括：
    根据所述位置相邻且连续的多个空闲数据单元的存储容量总和，对所述位置相邻且连续的多个空闲数据单元进行整理。
  7. 根据权利要求2-6任一所述的方法，其中，所述分析与该数据分片集群的数据存储状态相对应的信息的步骤具体包括：
    从所述元数据信息中查找包含空闲标记的存储空间，并将查找到的存储空间确定为空闲存储空间；
    其中，所述空闲标记包括：因数据均衡操作、数据更新操作和/或数据删除操作而产生的无效标记和/或删除标记。
  8. 根据权利要求3-7任一所述的方法，其中，所述数据单元包括以下中的至少一种：数据块、数据分区、以及数据子分区；
    其中，一个数据块包括至少一个数据分区，一个数据分区包括至少一个数据子分区。
  9. 根据权利要求2-8任一所述的方法，其中，所述分布式系统进一步包括：用于存储与各个数据分片集群的数据存储状态相对应的元数据信息的配置服务器；
    则所述分别针对每个数据分片集群，获取并分析与该数据分片集群的数据存储状态相对应的信息的步骤具体包括：从所述配置服务器中获取与各个数据分片集群的数据存储状态相对应的元数据信息；
    且所述根据整理结果更新与该数据分片集群的数据存储状态相对应的信息的步骤具体包括：根据整理结果更新所述配置服务器中存储的与该数据 分片集群的数据存储状态相对应的元数据信息。
  10. 一种基于分布式系统的存储空间整理装置，其中，所述分布式系统至少包括多个分布式的数据分片集群，所述装置包括：
    分析模块，适于分别针对每个数据分片集群，获取并分析与该数据分片集群的数据存储状态相对应的信息；
    确定模块，适于根据分析结果确定该数据分片集群中包含的空闲存储空间的存储容量和/或位置分布；
    整理模块，适于根据所述空闲存储空间的存储容量和/或位置分布对该数据分片集群中包含的空闲存储空间进行整理；以及
    更新模块，适于根据整理结果更新与该数据分片集群的数据存储状态相对应的信息。
  11. 根据权利要求10所述的装置，其中，所述信息包括元数据信息。
  12. 根据权利要求11所述的装置，其中，每个数据分片集群包括多个用于存储数据的数据单元，则每个数据分片集群中包含的空闲存储空间的存储容量和/或位置分布具体为：该数据分片集群中的各个空闲数据单元的存储容量，和/或各个空闲数据单元的位置分布；
    且所述整理模块具体用于：根据各个空闲数据单元的位置分布，确定位置相邻且连续的多个空闲数据单元，针对所述位置相邻且连续的多个空闲数据单元进行整理。
  13. 根据权利要求12所述的装置，其中，所述确定模块具体用于：
    若多个空闲数据单元中，每两个相邻的空闲数据单元之间没有非空闲数据单元，则将所述多个空闲数据单元确定为位置相邻且连续的多个空闲数据单元；或者，
    若多个空闲数据单元中，至少两个相邻的空闲数据单元之间具有非空闲数据单元，且所述非空闲数据单元中的已存储数据量小于预设阈值，则针对所述非空闲数据单元中已存储的全部数据执行数据迁移操作，并将所述多个空闲数据单元以及执行数据迁移操作后的非空闲数据单元确定为位置相邻且连续的多个空闲数据单元。
  14. 根据权利要求12或13所述的装置，其中，所述整理模块具体用于：
    将所述位置相邻且连续的多个空闲数据单元合并为一个待分配数据单元；其中，为该待分配数据单元设置对应的待分配标记；或者，
    将所述位置相邻且连续的多个空闲数据单元回收后重新分配为一个或多个待使用数据单元；其中，设定各个待使用数据单元的存储容量，并为各个待使用数据单元设置对应的待使用标记。
  15. 根据权利要求12-14任一所述的装置，其中，所述整理模块具体用于：
    根据所述位置相邻且连续的多个空闲数据单元的存储容量总和，对所述位置相邻且连续的多个空闲数据单元进行整理。
  16. 根据权利要求11-15任一所述的装置，其中，所述分析模块具体用于：
    从所述元数据信息中查找包含空闲标记的存储空间，并将查找到的存储空间确定为空闲存储空间；
    其中，所述空闲标记包括：因数据均衡操作、数据更新操作和/或数据删除操作而产生的无效标记和/或删除标记。
  17. 根据权利要求12-16任一所述的装置，其中，所述数据单元包括以下中的至少一种：数据块、数据分区、以及数据子分区；
    其中，一个数据块包括至少一个数据分区，一个数据分区包括至少一个数据子分区。
  18. 根据权利要求11-17任一所述的装置，其中，所述分布式系统进一步包括：用于存储与各个数据分片集群的数据存储状态相对应的元数据信息的配置服务器；
    则所述分析模块具体用于：从所述配置服务器中获取与各个数据分片集群的数据存储状态相对应的元数据信息；
    且所述更新模块具体用于：根据整理结果更新所述配置服务器中存储的与该数据分片集群的数据存储状态相对应的元数据信息。
  19. 一种分布式系统，包括权利要求10-18任一所述的存储空间整理装置，以及所述多个分布式的数据分片集群。
  20. 一种电子设备，包括：处理器、存储器、通信接口和通信总线，所述处理器、所述存储器和所述通信接口通过所述通信总线完成相互间的通信；
    所述存储器用于存放至少一可执行指令，所述可执行指令使所述处理器执行如权利要求1-9中任一项所述的存储空间整理方法对应的操作。
  21. 一种计算机程序，包括计算机可读代码，当所述计算机可读代码在计算设备上运行时，导致所述计算设备执行根据权利要求1-9中的任一项所述的存储空间整理方法对应的操作。
  22. 一种非易失性计算机可读存储介质，所述非易失性计算机可读存储介质中存储有至少一可执行指令，所述可执行指令使处理器执行如权利要求1-9中任一项所述的存储空间整理方法对应的操作。
PCT/CN2018/079241 2017-06-30 2018-03-16 基于分布式系统的存储空间整理方法、装置及系统 WO2019001020A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710555164.4A CN107368260A (zh) 2017-06-30 2017-06-30 基于分布式系统的存储空间整理方法、装置及系统
CN201710555164.4 2017-06-30

Publications (1)

Publication Number Publication Date
WO2019001020A1 true WO2019001020A1 (zh) 2019-01-03

Family

ID=60306015

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/079241 WO2019001020A1 (zh) 2017-06-30 2018-03-16 基于分布式系统的存储空间整理方法、装置及系统

Country Status (2)

Country Link
CN (1) CN107368260A (zh)
WO (1) WO2019001020A1 (zh)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107368260A (zh) * 2017-06-30 2017-11-21 北京奇虎科技有限公司 基于分布式系统的存储空间整理方法、装置及系统
CN110019125B (zh) * 2017-11-27 2021-12-14 北京京东尚科信息技术有限公司 数据库管理的方法和装置
CN108762678B (zh) * 2018-05-30 2021-11-09 郑州云海信息技术有限公司 一种存储空间回收方法、系统、装置及可读存储介质
CN109039801B (zh) * 2018-06-29 2021-09-28 北京奇虎科技有限公司 分布式集群的套餐超用检测方法及装置、计算设备
CN109445701B (zh) * 2018-10-26 2021-02-23 北京计算机技术及应用研究所 一种双控磁盘阵列的自动精简配置同步方法
US11029872B2 (en) * 2018-12-18 2021-06-08 Western Digital Technologies, Inc. Non-volatile storage system with data shaping memory partitions
CN109831540B (zh) * 2019-04-12 2022-02-11 成都四方伟业软件股份有限公司 分布式存储方法、装置、电子设备及存储介质
CN111913909B (zh) 2019-05-08 2024-02-23 厦门网宿有限公司 一种分布式存储系统中的重新分片方法及系统
CN111580840A (zh) * 2020-03-27 2020-08-25 惠州市德赛西威汽车电子股份有限公司 一种基于分布式内存管理实现ecu更新的方法
CN111831752B (zh) * 2020-07-20 2024-06-04 北京百度网讯科技有限公司 分布式数据库的空间整理方法、装置、设备以及存储介质
CN113051221B (zh) * 2021-03-31 2023-06-30 网易（杭州）网络有限公司 数据存储方法、装置、介质、设备及分布式文件系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103366016A (zh) * 2013-08-01 2013-10-23 南京大学 基于hdfs的电子文件集中存储及优化方法
CN103678562A (zh) * 2011-12-27 2014-03-26 北京奇虎科技有限公司 容量获取方法和文件数据分配方法
CN104731921A (zh) * 2015-03-26 2015-06-24 江苏物联网研究发展中心 Hadoop分布式文件系统针对日志型小文件的存储和处理方法
CN107368260A (zh) * 2017-06-30 2017-11-21 北京奇虎科技有限公司 基于分布式系统的存储空间整理方法、装置及系统

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120159098A1 (en) * 2010-12-17 2012-06-21 Microsoft Corporation Garbage collection and hotspots relief for a data deduplication chunk store
CN103116552B (zh) * 2013-03-18 2017-03-15 华为技术有限公司 用于在分布式存储系统中分配存储空间的方法和装置
CN105589812B (zh) * 2015-12-16 2018-10-30 成都华为技术有限公司 磁盘碎片整理方法、装置及主机
CN106469120A (zh) * 2016-08-30 2017-03-01 华为技术有限公司 碎片整理方法、装置及设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103678562A (zh) * 2011-12-27 2014-03-26 北京奇虎科技有限公司 容量获取方法和文件数据分配方法
CN103366016A (zh) * 2013-08-01 2013-10-23 南京大学 基于hdfs的电子文件集中存储及优化方法
CN104731921A (zh) * 2015-03-26 2015-06-24 江苏物联网研究发展中心 Hadoop分布式文件系统针对日志型小文件的存储和处理方法
CN107368260A (zh) * 2017-06-30 2017-11-21 北京奇虎科技有限公司 基于分布式系统的存储空间整理方法、装置及系统

Also Published As

Publication number Publication date
CN107368260A (zh) 2017-11-21

Similar Documents

Publication Publication Date Title
WO2019001020A1 (zh) 基于分布式系统的存储空间整理方法、装置及系统
US11082206B2 (en) Layout-independent cryptographic stamp of a distributed dataset
TWI778157B (zh) 固態驅動器、分散式資料儲存系統和利用鍵值儲存的方法
US10268716B2 (en) Enhanced hadoop framework for big-data applications
WO2017107414A1 (zh) 文件操作方法和装置
US10073648B2 (en) Repartitioning data in a distributed computing system
WO2017107015A1 (zh) 存储空间的分配方法及存储设备
US10976946B2 (en) Method and computer system for managing blocks
US10515055B2 (en) Mapping logical identifiers using multiple identifier spaces
US11977548B2 (en) Allocating partitions for executing operations of a query
US10585613B2 (en) Small storage volume management
CN109407985B (zh) 一种数据管理的方法以及相关装置
WO2022120522A1 (zh) 内存空间的分配方法、装置及存储介质
CN111949210A (zh) 分布式存储系统中元数据存储方法、系统及存储介质
US8386741B2 (en) Method and apparatus for optimizing data allocation
US9696919B1 (en) Source/copy reference tracking with block pointer sets
US20160012075A1 (en) Computer system and data management method
CN115129621A (zh) 一种内存管理方法、设备、介质及内存管理模块
KR100907477B1 (ko) 플래시 메모리에 저장된 데이터의 인덱스 정보 관리 장치및 방법
US10482012B1 (en) Storage system and method of operating thereof
US11226769B2 (en) Large-scale storage system and data placement method in large-scale storage system
CN117331487A (zh) 一种数据重删方法及相关系统
CN116932196A (zh) 融合系统的数据处理方法、装置、设备和系统
CN113835613B (zh) 一种文件读取方法、装置、电子设备和存储介质
CN115203133A (zh) 数据处理方法、装置、归约服务器及映射服务器

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18825408

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18825408

Country of ref document: EP

Kind code of ref document: A1