CN111984549B - Hierarchical storage method and system for splitting data according to objects - Google Patents
Hierarchical storage method and system for splitting data according to objects Download PDFInfo
- Publication number
- CN111984549B CN111984549B CN202010726612.4A CN202010726612A CN111984549B CN 111984549 B CN111984549 B CN 111984549B CN 202010726612 A CN202010726612 A CN 202010726612A CN 111984549 B CN111984549 B CN 111984549B
- Authority
- CN
- China
- Prior art keywords
- data
- pool
- storage system
- hot
- setting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 25
- 238000012546 transfer Methods 0.000 claims description 54
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 23
- 238000013500 data storage Methods 0.000 claims description 16
- 238000012423 maintenance Methods 0.000 claims description 14
- 238000012986 modification Methods 0.000 claims description 11
- 230000004048 modification Effects 0.000 claims description 11
- 238000001514 detection method Methods 0.000 claims description 4
- 235000019633 pungent taste Nutrition 0.000 claims description 4
- 238000006467 substitution reaction Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0806—Multiuser, multiprocessor or multiprocessing cache systems
- G06F12/0811—Multiuser, multiprocessor or multiprocessing cache systems with multilevel cache hierarchies
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0844—Multiple simultaneous or quasi-simultaneous cache accessing
- G06F12/0846—Cache with multiple tag or data arrays being simultaneously accessible
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0893—Caches characterised by their organisation or structure
- G06F12/0897—Caches characterised by their organisation or structure with two or more cache hierarchy levels
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
Abstract
The invention provides a hierarchical storage method and a hierarchical storage system for data split according to objects, wherein the method comprises the following steps: s1, setting a storage system to split uploaded data into a head object and a plurality of tail objects; the header object includes metadata and information data; s2, setting a storage system to store the head objects of the uploaded data into a hot pool, and completely storing the tail objects into a cold pool; s3, setting the access heat of statistical data of the storage system, transferring the information data in the head object with the access heat lower than the threshold data from the hot pool to the cold pool, and transferring the information data with the access heat higher than the threshold data from the cold pool to the hot pool. The data of the invention is split according to the objects, so that a large number of tail objects are directly stored in the cold pool, and the tail objects are prevented from being transferred from the hot pool to the cold pool; metadata in the head object is used as a data reading basis, information data in the head object is used as transferable data, a margin is reserved for a hot pool, the performance of the storage system is provided, and the functions of the distributed object storage system are enriched.
Description
Technical Field
The invention belongs to the technical field of data storage, and particularly relates to a hierarchical storage method and system for splitting data according to objects.
Background
The hierarchical storage function is to utilize the ssd storage medium in the storage system as the cache of the hdd disk, so as to improve the read-write performance of the storage system as a whole. In the storage process, the object is firstly stored in the ssd disk with high read-write speed, and then the data is transferred to the hdd disk after being cooled. There are significant disadvantages to this approach in practical use.
Because the space of the ssd disk is relatively small, not much data can be stored, after all data is written into the ssd, the ssd disk is quickly full, and then the data in the ssd needs to be transferred to the hdd. When the speed of the external write storage system is faster than the speed of ssd transferring data to hdd, it causes the ssd disk space to be full all the time and the performance of the storage system writing data is severely affected.
Therefore, it is very necessary to provide a hierarchical storage method and system for splitting data according to objects, which address the above-mentioned drawbacks in the prior art.
Disclosure of Invention
Aiming at the defects that the space of an ssd disk is always full and the data writing performance of a storage system is seriously affected due to the fact that the traditional mode that object data is stored into the ssd disk first and then stored into an hhd disk in the prior art, the invention provides a hierarchical storage method and a hierarchical storage system for splitting the data according to objects, and aims to solve the technical problems.
In a first aspect, the present invention provides a hierarchical storage method for data split according to an object, including the following steps:
s1, setting a storage system to split uploaded data into a head object and a plurality of tail objects; the header object includes metadata and information data;
s2, setting a storage system to store the head objects of the uploaded data into a hot pool, and completely storing the tail objects into a cold pool;
s3, setting the access heat of statistical data of the storage system, transferring the information data in the head object with the access heat lower than the threshold data from the hot pool to the cold pool, and transferring the information data with the access heat higher than the threshold data from the cold pool to the hot pool.
Further, in step S1, the head object occupation space is a fixed value, and each tail object occupation space is equal to or less than a set threshold.
Further, the step S2 specifically includes the following steps:
s21, arranging a storage system to detect the water level of the hot pool;
when the hot pool water level is lower than the threshold value, the step S22 is carried out;
when the hot pool water level is higher than the threshold value, the step S23 is carried out;
s22, setting a storage system to store all head objects of the uploaded data into a hot pool and store all tail objects into a cold pool;
s23, a storage system is arranged to store the metadata in the head objects of the uploaded data into a hot pool, and store the information data and all tail objects into a cold pool. And determining the storage position of the information data in the head object according to the water level condition of the hot pool.
Further, in step S2, the hot pool employs a redundancy policy, and the cold pool employs an erasure correction policy.
Further, the step S3 specifically includes the following steps:
s31, setting a heat meter and a heat maintenance thread in a storage system, and setting a heat access thread to maintain the heat meter according to data access and modification conditions;
and S32, setting a transfer thread in the storage system, wherein the transfer thread is set to transfer the information data in the head object with the access heat degree lower than the threshold data from the hot pool to the cold pool, and to transfer the information data with the access heat degree higher than the threshold data from the cold pool to the hot pool. The hot value of the data is increased when the data is accessed and modified, and the hot value of the data is decreased with time when the data is not accessed for a long time.
Further, the step S31 specifically includes the following steps:
s311, setting a storage pool where the hotlist recording data is located and a data access hotness value;
s312, when the data access and modification times are increased within the time threshold, setting a heat meter maintenance thread to increase the access heat of the data in the heat meter;
s313, when the data access time exceeds a time threshold, setting a heat meter maintenance thread to reduce the access heat of the data in the heat meter. And the heat meter maintenance thread maintains the heat meter according to the data access condition.
Further, the step S32 specifically includes the following steps:
s321, a storage system is set to add data with the access heat degree lower than a threshold value into a hot-to-cold queue to be transferred;
s322, setting a storage system to add data with the access heat degree higher than a threshold value in the cold pool into a cold-to-hot queue to be transferred;
s323, a transfer thread is set to transfer information data in a head object of data in a hot-to-cold transfer queue to a cold pool as a No. 0 tail object, metadata information of the data is reserved in the hot pool head object, and the metadata is modified to record position information of the head object information data;
s324, a transfer thread is set to transfer the tail object No. 0 of the data in the cold-hot transfer tape transfer queue to the head object in the hot pool, and the position information of the head object information data is recorded by the head object metadata. The transfer thread transfers the data to the appropriate storage location according to the hotlist.
Further, the method also comprises the following steps:
and S4, setting a storage system to acquire the head object from the hot pool when downloading data, and acquiring the tail object from the cold pool according to metadata in the head object. Data downloading is respectively carried out in parallel from the hot pool and the cold pool, and the data reading speed is improved.
In a second aspect, the present invention provides a hierarchical storage system for splitting data according to objects, including:
the data splitting module is used for setting a storage system to split the uploaded data into a head object and a plurality of tail objects; the header object includes metadata and information data;
the data storage module is used for setting a storage system to store the head objects of the uploaded data into a hot pool and store all the tail objects into a cold pool;
and the data transfer module is used for setting the statistical data access heat of the storage system, transferring the information data in the head object with the access heat lower than the threshold data from the hot pool to the cold pool, and transferring the information data with the access heat higher than the threshold data from the cold pool to the hot pool.
Further, the data storage module includes:
the hot pool water level detection unit is used for setting a storage system to detect the water level of the hot pool;
the data storage first unit is used for setting a storage system to store all head objects of the uploaded data into the hot pool and store all tail objects into the cold pool when the water level of the hot pool is lower than a threshold value;
and the data storage second unit is used for setting a storage system to store metadata in head objects of uploaded data into the hot pool and store information data and all tail objects into the cold pool when the water level of the hot pool is higher than a threshold value.
Further, the data transfer module includes:
the system comprises a heat counting unit, a heat maintenance unit and a heat management unit, wherein the heat counting unit is used for setting a heat table and a heat maintenance thread in a storage system, and setting a heat access thread to maintain the heat table according to data access and modification conditions;
and the data transfer unit is used for setting a transfer thread in the storage system, setting the transfer thread to transfer the information data in the head object with the access heat degree lower than the threshold data from the hot pool to the cold pool, and transferring the information data with the access heat degree higher than the threshold data from the cold pool to the hot pool.
The beneficial effect of the invention is that,
according to the hierarchical storage method and the system for splitting the data according to the objects, the data is split according to the objects, so that a large number of tail objects are directly stored in the cold pool, the tail objects are prevented from being transferred from the hot pool to the cold pool, and the data writing performance of the storage system is greatly improved; metadata in the head object is used as a data reading basis, information data in the head object is used as transferable data, a margin is reserved for a hot pool, the performance of the storage system is provided, and the functions of the distributed object storage system are enriched.
In addition, the invention has reliable design principle, simple structure and very wide application prospect.
Therefore, compared with the prior art, the invention has prominent substantive features and remarkable progress, and the beneficial effects of the implementation are also obvious.
Drawings
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present invention, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
FIG. 1 is a first schematic flow diagram of the method of the present invention;
FIG. 2 is a second flow chart of the method of the present invention;
FIG. 3 is a schematic diagram of the system of the present invention;
in the figure, 1-data splitting module; 2-a data storage module; 2.1-a hot pool water level detection unit; 2.2-a data storage first unit; 2.3-a data storage second unit; 3-a data transfer module; 3.1-heat statistic unit; 3.2-data transfer Unit.
Detailed Description
In order to make those skilled in the art better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1:
as shown in fig. 1, the present invention provides a hierarchical storage method for splitting data according to objects, which includes the following steps:
s1, setting a storage system to split uploaded data into a head object and a plurality of tail objects; the header object includes metadata and information data;
s2, setting a storage system to store the head objects of the uploaded data into a hot pool, and completely storing the tail objects into a cold pool;
s3, setting the access heat of statistical data of the storage system, transferring the information data in the head object with the access heat lower than the threshold data from the hot pool to the cold pool, and transferring the information data with the access heat higher than the threshold data from the cold pool to the hot pool.
In certain embodiments, the method further comprises the steps of:
and S4, setting a storage system to acquire the head object from the hot pool when downloading data, and acquiring the tail object from the cold pool according to metadata in the head object.
Example 2:
as shown in fig. 2, the present invention provides a hierarchical storage method for splitting data according to objects, which includes the following steps:
s1, setting a storage system to split uploaded data into a head object and a plurality of tail objects; the header object includes metadata and information data; the head object occupation space is a fixed value, and the tail object occupation space is less than or equal to a set threshold;
s2, setting a storage system to store the head objects of the uploaded data into a hot pool, and completely storing the tail objects into a cold pool; the hot pool adopts a redundancy strategy, and the cold pool adopts an erasure strategy; the method comprises the following specific steps:
s21, arranging a storage system to detect the water level of the hot pool;
when the hot pool water level is lower than the threshold value, the step S22 is carried out;
when the hot pool water level is higher than the threshold value, the step S23 is carried out;
s22, setting a storage system to store all head objects of the uploaded data into a hot pool and store all tail objects into a cold pool;
s23, a storage system is arranged to store metadata in the head objects of the uploaded data into a hot pool, and store information data and all tail objects into a cold pool;
s3, setting a statistical data access heat of a storage system, transferring information data in a head object with the access heat lower than threshold data from a hot pool to a cold pool, and transferring the information data with the access heat higher than the threshold data from the cold pool to the hot pool; the method comprises the following specific steps:
s31, setting a heat meter and a heat maintenance thread in a storage system, and setting a heat access thread to maintain the heat meter according to data access and modification conditions;
and S32, setting a transfer thread in the storage system, wherein the transfer thread is set to transfer the information data in the head object with the access heat degree lower than the threshold data from the hot pool to the cold pool, and to transfer the information data with the access heat degree higher than the threshold data from the cold pool to the hot pool.
In some embodiments, the head object footprint is set to 512k, and each tail object is set to 4M or less; because the head objects are small, the hot pools ssd with the same size can buffer more objects; the tail object is used as a 4 m-sized strip, and the performance difference between the tail object stored in the hot pool ssd and the cold pool Hdd is small, so that the storage of the tail object in the hot pool ssd and the cold pool Hdd is more suitable, and the space of the hot pool ssd is saved; when reading and uploading data, the reading and writing head object from the hot pool ssd and the reading and writing tail object from the hot pool Hdd can achieve the concurrence effect, thereby greatly improving the storage performance.
Compared with the traditional storage scheme, the scheme solves the problem that when the uploading speed is too high, the ssd disk easily reaches the threshold value to influence the performance of the storage system
In some embodiments, step S31 includes the following steps:
s311, setting a storage pool where the hotlist recording data is located and a data access hotness value;
s312, when the data access and modification times are increased within the time threshold, setting a heat meter maintenance thread to increase the access heat of the data in the heat meter;
s313, when the data access time exceeds a time threshold, setting a heat meter maintenance thread to reduce the access heat of the data in the heat meter.
The step S32 includes the following steps:
s321, a storage system is set to add data with the access heat degree lower than a threshold value into a hot-to-cold queue to be transferred;
s322, setting a storage system to add data with the access heat degree higher than a threshold value in the cold pool into a cold-to-hot queue to be transferred;
s323, a transfer thread is set to transfer information data in a head object of data in a queue to be transferred from hot to cold to a cold pool as a No. 0 tail object, metadata information of the data is reserved in the head object of the hot pool, and position information of the head object information data is recorded by modifying the metadata;
s324, a transfer thread is set to transfer the tail object No. 0 of the data in the cold-hot transfer tape transfer queue to the head object in the hot pool, and the position information of the head object information data is recorded by the head object metadata.
Example 3:
as shown in fig. 3, the present invention provides a hierarchical storage system with data split according to objects, including:
the data splitting module 1 is used for setting a storage system to split the uploaded data into a head object and a plurality of tail objects; the header object includes metadata and information data;
the data storage module 2 is used for setting a storage system to store the head objects of the uploaded data into a hot pool and store all the tail objects into a cold pool; the data storage module 2 includes:
the hot pool water level detection unit 2.1 is used for setting a storage system to detect the water level of the hot pool;
a first data storage unit 2.2, configured to set the storage system to store all head objects of the uploaded data in the hot pool and all tail objects in the cold pool when the water level of the hot pool is lower than a threshold;
a second data storage unit 2.3, configured to, when the water level of the hot pool is higher than the threshold, set the storage system to store metadata in the head objects of the uploaded data into the hot pool, and store the information data and all tail objects into the cold pool;
the data transfer module 3 is used for setting the statistical data access heat of the storage system, transferring the information data in the head object with the access heat lower than the threshold data from the hot pool to the cold pool, and transferring the information data with the access heat higher than the threshold data from the cold pool to the hot pool; the data transfer module 3 includes:
the heat counting unit 3.1 is used for setting a heat table and a heat maintenance thread in the storage system, and setting a heat access thread to maintain the heat table according to the data access and modification conditions;
and the data transfer unit 3.2 is used for setting a transfer thread in the storage system, wherein the transfer thread is set to transfer the information data in the head object with the access heat degree lower than the threshold data from the hot pool to the cold pool, and transfer the information data with the access heat degree higher than the threshold data from the cold pool to the hot pool.
Although the present invention has been described in detail by referring to the drawings in connection with the preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made on the embodiments of the present invention by those skilled in the art without departing from the spirit and scope of the present invention, and these modifications or substitutions are within the scope of the present invention/any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A hierarchical storage method for splitting data according to objects is characterized by comprising the following steps:
s1, setting a storage system to split uploaded data into a head object and a plurality of tail objects; the header object includes metadata and information data;
s2, setting a storage system to store the head objects of the uploaded data into a hot pool, and completely storing the tail objects into a cold pool;
s3, setting the access heat of statistical data of the storage system, transferring the information data in the head object with the access heat lower than the threshold data from the hot pool to the cold pool, and transferring the information data with the access heat higher than the threshold data from the cold pool to the hot pool.
2. The hierarchical storage method for data splitting according to objects as claimed in claim 1, wherein step S2 specifically comprises the following steps:
s21, arranging a storage system to detect the water level of the hot pool;
when the hot pool water level is lower than the threshold value, the step S22 is carried out;
when the hot pool water level is higher than the threshold value, the step S23 is carried out;
s22, setting a storage system to store all head objects of the uploaded data into a hot pool and store all tail objects into a cold pool;
s23, a storage system is arranged to store the metadata in the head objects of the uploaded data into a hot pool, and store the information data and all tail objects into a cold pool.
3. The method for hierarchical storage of data according to object splitting according to claim 1 or 2, wherein step S2 employs a redundancy policy for the hot pool and an erasure policy for the cold pool.
4. The hierarchical storage method for data splitting according to objects as claimed in claim 1, wherein step S3 specifically comprises the following steps:
s31, setting a heat meter and a heat maintenance thread in a storage system, and setting a heat access thread to maintain the heat meter according to data access and modification conditions;
and S32, setting a transfer thread in the storage system, wherein the transfer thread is set to transfer the information data in the head object with the access heat degree lower than the threshold data from the hot pool to the cold pool, and to transfer the information data with the access heat degree higher than the threshold data from the cold pool to the hot pool.
5. The hierarchical storage method for data splitting according to objects as claimed in claim 4, wherein step S31 specifically comprises the following steps:
s311, setting a storage pool where the hotlist recording data is located and a data access hotness value;
s312, when the data access and modification times are increased within the time threshold, setting a heat meter maintenance thread to increase the access heat of the data in the heat meter;
s313, when the data access time exceeds a time threshold, setting a hotlist maintenance thread to reduce the access hotness of the data in the hotlist.
6. The method for hierarchical storage of data according to object splitting according to claim 4 or 5, wherein step S32 specifically comprises the following steps:
s321, setting a storage system to add data with the access heat degree lower than a threshold value in the hot pool into a hot-to-cold queue to be transferred;
s322, setting a storage system to add data with the access heat degree higher than a threshold value in the cold pool into a cold-to-hot queue to be transferred;
s323, a transfer thread is set to transfer information data in a head object of data in a queue to be transferred from hot to cold to a cold pool as a No. 0 tail object, metadata information of the data is reserved in the head object of the hot pool, and position information of the head object information data is recorded by modifying the metadata;
s324, a transfer thread is set to transfer the tail object No. 0 of the data in the cold-to-hot transfer tape transfer queue to the head object in the hot pool, and the position information of the head object information data is recorded by the head object metadata.
7. The method for hierarchical storage of data split by object according to claim 1, further comprising the steps of:
and S4, setting a storage system to acquire the head object from the hot pool when downloading data, and acquiring the tail object from the cold pool according to metadata in the head object.
8. A hierarchical storage system for splitting data according to objects, comprising:
the data splitting module (1) is used for setting a storage system to split the uploaded data into a head object and a plurality of tail objects; the header object includes metadata and information data;
the data storage module (2) is used for setting the storage system to store the head objects of the uploaded data into the hot pool and store all the tail objects into the cold pool;
and the data transfer module (3) is used for setting the access heat of statistical data of the storage system, transferring the information data in the head object with the access heat lower than the threshold data from the hot pool to the cold pool, and transferring the information data with the access heat higher than the threshold data from the cold pool to the hot pool.
9. The hierarchical data-by-object splitting storage system according to claim 8, wherein the data storage module (2) comprises:
the hot pool water level detection unit (2.1) is used for setting a storage system to detect the water level of the hot pool;
a first data storage unit (2.2) for setting the storage system to store all head objects of the uploaded data into the hot pool and all tail objects into the cold pool when the water level of the hot pool is lower than a threshold value;
and a data storage second unit (2.3) for setting the storage system to store the metadata in the head objects of the uploaded data into the hot pool and the information data and all tail objects into the cold pool when the water level of the hot pool is higher than the threshold value.
10. The hierarchical storage system with data split by object according to claim 8, wherein the data transfer module (3) comprises:
the heat counting unit (3.1) is used for setting a heat table and a heat maintenance thread in the storage system, and setting a heat access thread to maintain the heat table according to the data access and modification conditions;
and the data transfer unit (3.2) is used for setting a transfer thread in the storage system, the transfer thread is set to transfer the information data in the head object with the access heat degree lower than the threshold data from the hot pool to the cold pool, and the information data with the access heat degree higher than the threshold data is transferred from the cold pool to the hot pool.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010726612.4A CN111984549B (en) | 2020-07-25 | 2020-07-25 | Hierarchical storage method and system for splitting data according to objects |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010726612.4A CN111984549B (en) | 2020-07-25 | 2020-07-25 | Hierarchical storage method and system for splitting data according to objects |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111984549A CN111984549A (en) | 2020-11-24 |
| CN111984549B true CN111984549B (en) | 2022-07-19 |
Family
ID=73438131
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010726612.4A Active CN111984549B (en) | 2020-07-25 | 2020-07-25 | Hierarchical storage method and system for splitting data according to objects |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111984549B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104360961A (en) * | 2014-12-10 | 2015-02-18 | 浪潮(北京)电子信息产业有限公司 | Object storage-based self-adaptive graded processing method and object storage-based self-adaptive graded processing system |
| WO2016165441A1 (en) * | 2015-09-06 | 2016-10-20 | 中兴通讯股份有限公司 | Migration policy adjustment method, capacity-change suggestion method and device |
-
2020
- 2020-07-25 CN CN202010726612.4A patent/CN111984549B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104360961A (en) * | 2014-12-10 | 2015-02-18 | 浪潮(北京)电子信息产业有限公司 | Object storage-based self-adaptive graded processing method and object storage-based self-adaptive graded processing system |
| WO2016165441A1 (en) * | 2015-09-06 | 2016-10-20 | 中兴通讯股份有限公司 | Migration policy adjustment method, capacity-change suggestion method and device |
Non-Patent Citations (1)
| Title |
|---|
| 对象存储系统的合作缓存方案;罗益辉等;《华中科技大学学报(自然科学版)》;20081115(第11期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111984549A (en) | 2020-11-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Sun et al. | DPPDL: a dynamic partial-parallel data layout for green video surveillance storage | |
| CN106095342B (en) | A kind of watt recording disc array construction method and the system of dynamically changeable long strip | |
| CN103631536B (en) | A kind of method utilizing the invalid data of SSD to optimize RAID5/6 write performance | |
| CN108829341B (en) | Data management method based on hybrid storage system | |
| CN106293511A (en) | A kind of dynamic local parallel data layout towards continuous data storage | |
| WO2010066098A1 (en) | Method and device for constructing high speed solid state storage disk with larger capacity dram involved in management of flash media | |
| CN102999428A (en) | Four-stage addressing method for tile recording disk | |
| CN103226965B (en) | Based on the audio/video data access method of time bitmap | |
| CN103702057A (en) | Block storage algorithm applicable to multiple paths of concurrent-written stream media data | |
| CN107221351B (en) | Optimization processing method of error correcting code in solid-state disk system and application thereof | |
| CN110888600B (en) | Buffer area management method for NAND flash memory | |
| CN105278656A (en) | Energy saving method of cold storage servers | |
| CN101625586A (en) | Method, equipment and computer for managing energy conservation of storage device | |
| CN108959589A (en) | Accelerate the method for solid-state memory journal file saving/restoring based on STT-MRAM | |
| CN105005510A (en) | Error correction protection architecture and method applied to resistive random access memory cache of solid state disk | |
| CN106909323B (en) | Page caching method suitable for DRAM/PRAM mixed main memory architecture and mixed main memory architecture system | |
| WO2019022837A1 (en) | Minimizing performance degradation due to refresh operations in memory sub-systems | |
| CN111984549B (en) | Hierarchical storage method and system for splitting data according to objects | |
| CN106469119A (en) | A kind of data write buffer method based on NVDIMM and its device | |
| CN102520885A (en) | Data management system for hybrid hard disk | |
| CN116250041A (en) | Refresh management list for DRAM | |
| CN105138277A (en) | Cache management method for solid-state disc array | |
| KR101436442B1 (en) | Dynamic random access memory device having selective refresh function according to read and write access and the selective refresh method the same | |
| CN108647157A (en) | A kind of mapping management process and solid state disk based on phase transition storage | |
| KR101031338B1 (en) | Memory management method for reducing disk energy consumption in video playback |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |