CN111367469A - Layered storage data migration method and system - Google Patents
Layered storage data migration method and system Download PDFInfo
- Publication number
- CN111367469A CN111367469A CN202010094756.2A CN202010094756A CN111367469A CN 111367469 A CN111367469 A CN 111367469A CN 202010094756 A CN202010094756 A CN 202010094756A CN 111367469 A CN111367469 A CN 111367469A
- Authority
- CN
- China
- Prior art keywords
- file
- migration
- storage
- storage layer
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000005012 migration Effects 0.000 title claims abstract description 149
- 238000013508 migration Methods 0.000 title claims abstract description 149
- 238000000034 method Methods 0.000 title claims abstract description 31
- 238000012544 monitoring process Methods 0.000 claims description 6
- 230000000694 effects Effects 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 4
- 230000004044 response Effects 0.000 claims description 3
- 238000011161 development Methods 0.000 description 3
- 238000006467 substitution reaction Methods 0.000 description 3
- 230000015654 memory Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0685—Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method and a system for migrating layered storage data, wherein the method comprises the following steps: acquiring metadata information of a file; calculating the value of the file according to the metadata information; generating a file migration schedule according to the file value; the migration tasks in the file migration schedule are arranged according to the file value in the priority order; and taking out the migration task from the file migration schedule, and migrating the file on different storage layers. The file value judgment method with the access mode matched with the storage performance can effectively improve the accuracy and matching degree of file migration and reduce the influence on the system performance during data migration.
Description
Technical Field
The invention relates to the technical field of storage data migration management, in particular to a method and a system for migrating layered storage data.
Background
The development of computer and communication technologies has led to the explosive development of data, and the rapid growth of data has made higher demands on the efficient use of storage systems. However, according to the statistics of the relevant organization, a large amount of data is accessed very infrequently after being created, some data is in a dormant state for a long time, but the data is still stored on an expensive storage device, and enterprises generally want to put more important data in a high-performance storage device to obtain higher performance and utilization rate. More and more organizations are becoming aware of the importance of tiered storage.
Hierarchical storage is a technique for migration management of data between different storage tiers. Hierarchical storage management has emerged to save more cost while providing the desired performance. Of all data, the data with higher access frequency is stored in the high-performance storage layer, most other data is stored in the low-performance storage layer with large capacity and low price, and the user does not need to know where the data is stored, and the system can automatically retrieve the data. The latest development of hierarchical storage is a combination of mechanical hard disks and flash memories in view of the physical structure of the storage medium. Hierarchical storage in a storage system places hot data, which is more frequently accessed, in a high-rate storage tier, such as an SSD storage tier, while cold data, which is less frequently accessed, is placed in a low-rate storage tier, such as an HDD storage tier. Most of file-based hierarchical storage system data migration is mainly determined according to the creation time and the access frequency of data, data exceeding a certain age or files with low access frequency are placed in low-speed storage equipment, but the association between a data access mode and different layers of storage media and actual service scenes is ignored, and meanwhile, the timeliness of services and the fluctuation possibly brought to a storage system are not considered, so that the efficiency of the storage equipment is not effectively improved.
Disclosure of Invention
Aiming at the problem that the efficiency of the storage equipment is not effectively improved because the data migration of most of file-based hierarchical storage systems is mainly determined according to the creation time and the access frequency of data, the data exceeding a certain age or the files with lower access frequency are placed in low-speed storage equipment, but the association between a data access mode and different layers of storage media and actual service scenes is ignored, and the timeliness of the service and the fluctuation possibly brought to the storage systems are not considered, the invention provides a method and a system for migrating the hierarchical storage data based on the data value.
The technical scheme of the invention is as follows:
on one hand, the technical scheme of the invention provides a method for migrating layered storage data, which comprises the following steps:
acquiring metadata information of a file;
calculating the value of the file according to the metadata information;
generating a file migration schedule according to the file value; the migration tasks in the file migration schedule are arranged according to the file value in the priority order;
and taking out the migration task from the file migration schedule, and migrating the file on different storage layers.
Further, in the metadata information of the obtained file, the metadata information includes a read-write frequency, a sequence/random access ratio, a read-write size of the file, and a read-write sequence/random coefficient of the file;
file read-write order coefficient
Random coefficient for file read and write
δ ∈ (0, 1), and assuming that the average response time of X period is t:
δ is a constant between 0 and 1, representing the magnitude of the effect of historical access frequency on file worth determination.
Further, the step of calculating the value of the file according to the metadata information comprises:
the step of calculating the value of the file according to the metadata information comprises the following steps:
acquiring the time required by all files in the system to finish all read-write operations in a high-rate storage layer and a low-rate storage layer;
calculating a storage layer matching degree coefficient according to the acquired time and the file read-write sequence/random coefficient;
calculating a file value F according to the storage layer matching degree coefficient, the file reading and writing sequence/random coefficient and the size of the current file; the file value F is used for reflecting the value obtained by storing the file in the high-speed storage layer;
Further, the step of calculating the storage layer matching degree coefficient according to the acquired time and the file read-write sequence/random coefficient includes:
calculating time difference between the acquired time required by sequential access in the low-rate storage layer and the acquired time required by sequential access in the high-rate storage layer, and multiplying the calculated time difference by a file read-write sequence coefficient to obtain a sequence matching coefficient;
calculating the time difference between the acquired time required by random access in the low-rate storage layer and the acquired time required by random access in the high-rate storage layer, and multiplying the calculated time difference by a file read-write random coefficient to obtain a random matching coefficient;
and adding the sequential matching coefficient and the random matching coefficient to obtain a storage layer matching degree coefficient. The matching degree coefficient represents the degree to which the file is suitable for storage in one storage tier but not in another storage tier.
Further, the step of generating the file migration schedule according to the file value comprises:
sorting the files according to the value of the files and outputting a file list; the sequence of the file list reflects the relative size of the file values obtained by different files at the current layer;
traversing each file in the file list to output a file migration schedule; when traversing the file, if the current file is not in the high-speed storage layer and the size of the file is smaller than the available space of the high-speed storage layer, adding a file migration plan in the migration plan table.
Further, the step of taking out the migration task from the file migration schedule and migrating the file on different storage tiers includes:
taking out a migration task from the file migration schedule, detecting the performance information of the current file system, and migrating the files on different storage layers according to the performance information; the performance parameters comprise the load rate of the file system, the residual storage space and the historical access mode.
Further, in the step of taking out the migration task from the file migration schedule, detecting performance information of the current file system, and migrating the file on different storage tiers according to the performance information, the specific steps include:
detecting the load rate of the current file system;
when the load rate is detected to be smaller than a set load threshold value, detecting the file storage capacity of the target storage layer, and when the file storage capacity of the target storage layer is lower than a set first threshold value, taking out the migration task from the file migration schedule table to execute migration; stopping migration after the current storage space is full or the files are completely migrated;
and when the file storage capacity of the target storage layer is higher than a set second threshold value, taking out the migration task from the file migration schedule to execute migration when the size of the file to be migrated is judged to be smaller than the current available space of the target storage layer, otherwise, not executing the migration process, wherein the first threshold value is smaller than the second threshold value.
Further, if the storage layer where the file to be migrated is currently located is a low-rate storage layer and the size of the file is larger than a set large file threshold value, querying a historical access mode of the file, and if the historical access mode of the file is a sequential access mode, not executing migration on the file; and if the historical access mode of the file is a random access mode, migrating the file to a high-speed storage layer.
Further, if the storage layer where the file to be migrated is currently located is a low-rate storage layer and the file size is smaller than the set small file threshold, the file is not migrated.
On the other hand, the technical scheme of the invention also provides a layered storage data migration system which comprises a metadata management module, a file value judgment module, a migration control module, a file system monitoring module and a migration module;
the metadata management module is used for acquiring and managing metadata information of the file;
the file value judging module is used for judging the value of the file and outputting the judgment result to the migration control module;
the migration control module receives the file value calculation result of the file value judgment module, generates a file migration schedule and outputs control information to the migration module;
the file system monitoring module is used for collecting the performance information of the storage system and outputting the information to the migration control module;
and the migration module is used for receiving the control information of the migration control module, taking out the migration task from the inside of the migration plan table and migrating the file on different storage layers.
The system also comprises an access redirection module which is used for providing a virtual layer and enabling an upper layer interface to access different storage layers by using a uniform application program interface.
The factors such as data size, access time, access frequency, access mode and influence on system performance are comprehensively considered, and when the data needing to be migrated is more, compared with the traditional migration method, the data migration method has higher accuracy and smaller influence on system performance.
According to the technical scheme, the invention has the following advantages: the file value judgment method with the access mode matched with the storage performance can effectively improve the accuracy and matching degree of file migration and reduce the influence on the system performance during data migration.
In addition, the invention has reliable design principle, simple structure and very wide application prospect.
Therefore, compared with the prior art, the invention has prominent substantive features and remarkable progress, and the beneficial effects of the implementation are also obvious.
Drawings
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present invention, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
Fig. 1 is a schematic flow chart of a method for providing migration of layered storage data according to an embodiment of the present invention.
Fig. 2 is a schematic block diagram of a hierarchical storage data migration system according to a fourth embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention, and it is obvious that the described embodiment is only a part of the embodiment of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
As shown in fig. 1, the technical solution of the present invention provides a method for migrating layered storage data, including the following steps:
s1-1: acquiring metadata information of a file; here, the metadata information includes the attribute, the file position, and the history access information of the file, where the history access information includes the read-write frequency, the sequence/random access ratio, the read-write size, and the read-write sequence/random coefficient of the file;
s1-2: calculating the value of the file according to the metadata information; calculating the value of the file according to the time required by the file to complete all read-write operations in the current storage layer and the adjacent storage layer and the read-write sequence/random coefficient of the file;
s1-3: generating a file migration schedule according to the file value;
s1-4: in this embodiment, a person skilled in the art may set when to perform migration and how to perform migration. The migration time interval is determined according to the load change speed of the file system, the residual storage space, the file access hit rate and other factors, so that a reasonable file migration time interval is set, the file value judgment module is promoted to better find hot files, and the influence of the migration activity of the system on the normal service of the system is ensured within a controllable range.
Example two
The technical scheme of the invention provides a layered storage data migration method, which comprises the following steps:
s1-1: acquiring metadata information of a file; here, the metadata information includes the attribute, the file position, and the history access information of the file, where the history access information includes the read-write frequency, the sequence/random access ratio, the read-write size, and the read-write sequence/random coefficient of the file;
s1-2: calculating the value of the file according to the metadata information; calculating the value of the file according to the time required by the file to complete all read-write operations in the current storage layer and the adjacent storage layer and the read-write sequence/random coefficient of the file;
s1-3: generating a file migration schedule according to the file value;
s1-4: taking out a migration task from the file migration schedule, and migrating the file on different storage layers; the performance parameters comprise the load rate of the file system, the residual storage space and the historical access mode. The method comprises the following specific steps:
detecting the load rate of the current file system;
when the load rate is detected to be smaller than a set load threshold value, detecting the file storage capacity of the target storage layer, and when the file storage capacity of the target storage layer is lower than a set first threshold value, taking out the migration task from the file migration schedule table to execute migration; stopping migration after the current storage space is full or the files are completely migrated;
and when the file storage capacity of the target storage layer is higher than a set second threshold value, taking out the migration task from the file migration schedule to execute migration when the size of the file to be migrated is judged to be smaller than the current available space of the target storage layer, otherwise, not executing the migration process, wherein the first threshold value is smaller than the second threshold value.
If the storage layer where the file to be migrated is currently located is a low-rate storage layer and the size of the file is larger than a set large file threshold value, inquiring a historical access mode of the file, and if the historical access mode of the file is a sequential access mode, not executing migration on the file; and if the historical access mode of the file is a random access mode, migrating the file to a high-speed storage layer. And if the storage layer where the file to be migrated is currently located is a low-speed storage layer and the size of the file is smaller than the set small file threshold value, the file is not migrated.
EXAMPLE III
The technical scheme of the invention provides a layered storage data migration method, which comprises the following steps:
s2-1: acquiring metadata information of a file; in this step, in the metadata information of the obtained file, the metadata information includes a read-write frequency, a sequence/random access ratio, a read-write size, and a read-write sequence/random coefficient of the file;
file read-write order coefficient
Random coefficient for file read and write
δ ∈ (0, 1), and assuming that the average response time of X period is t:
δ is a constant between 0 and 1, representing the magnitude of the effect of historical access frequency on file worth determination.
S2-2: calculating the value of the file according to the metadata information; it should be noted that the present step includes:
s2-21: acquiring the time required by all files in the system to finish all read-write operations in a high-rate storage layer and a low-rate storage layer; in the embodiment, the high-rate storage layer is an SSD storage layer, and the low-rate storage layer is an HDD storage layer;
s2-22: calculating a storage layer matching degree coefficient rho according to the acquired time and the file read-write sequence/random coefficient;
in an embodiment of the present invention, ρ represents the extent to which a file is suitable for storage in the storage layer of an SSD but not in the storage layer of an HDD. Therefore, the time required for the file to complete the operation corresponding to the read-write coefficient of the last cycle on the storage layer of the SSD and the storage layer of the HDD needs to be calculated respectively. And subtracting the calculated required time to obtain the degree coefficient rho. The calculation formula of rho is that rho is Fs(i)×[Ts(k+1)-Ts(k)]+Fr(i)×[Tr(k+1)-Tr(k)]The symbols are defined as follows: t iss(k +1) represents a time required for memory sequential access in a storage layer of the HDD, Ts(k) Indicating the time required for sequential access of storage in an SSDr(k +1) represents a time required for storing random access in the storage layer of the HDD, Tr(k) Indicating the time required to store the random access at the SSD storage layer.
S2-23: calculating a file value F according to the storage layer matching degree coefficient, the file reading and writing sequence/random coefficient and the size of the current file;
S2-4: generating a file migration schedule according to the file value; sorting the files according to the size of the F and outputting a file list; traversing each file in the file list to output a file migration schedule; when traversing the file, if the current file is not in the SSD storage layer and the size of the file is smaller than the available space stored in the SSD layer, adding a file migration plan in the migration plan table, and if the file is in the SSD storage layer before, no operation is needed. And the migration tasks in the file migration schedule are arranged according to the file value in the priority order.
S2-5: and taking out the migration task from the file migration schedule, and migrating the file on different storage layers. Taking out a migration task from the file migration schedule, wherein in the embodiment, a target migration layer corresponding to the migration task is an SSD storage layer, detecting performance information of a current file system of the SSD storage layer, and migrating files on different storage layers according to the performance information; the performance parameters comprise the load rate of the file system, the residual storage space and the historical access mode.
The method comprises the following specific steps:
s2-51: detecting the load rate of the current file system of the SSD storage layer;
s2-52: when the load rate is detected to be smaller than a set load threshold, detecting the file storage capacity of the SSD storage layer, and when the file storage capacity of the SSD storage layer is lower than a set first threshold, taking out the migration task from the file migration schedule to execute migration; stopping migration after the current storage space is full or the files are completely migrated;
s2-53: and when the file storage capacity of the SSD storage layer is higher than a set second threshold value, when the size of the file to be migrated is judged to be smaller than the current available space of the SSD storage layer, taking out the migration task from the file migration schedule table to execute migration, otherwise, not executing the migration process, wherein the first threshold value is smaller than the second threshold value.
S2-54: if the storage layer where the file to be migrated is currently located is an HDD storage layer and the size of the file is larger than a set large file threshold value, inquiring a historical access mode of the file, and if the historical access mode of the file is a sequential access mode, not executing migration on the file; and if the historical access mode of the file is a random access mode, migrating the file to an SSD storage layer.
S2-55: and if the storage layer where the file to be migrated is currently located is an HDD storage layer and the file size is smaller than the set small file threshold, the file is not migrated. It should be noted that the purpose of migration is to migrate the hot spot data stored in the low-rate storage layer to the high-rate storage layer, and migrate the non-hot spot data in the high-rate storage layer from the high-rate storage layer at a suitable migration time point, thereby further and reasonably utilizing resources.
Example four
As shown in fig. 2, the technical solution of the present invention further provides a layered storage data migration system, which includes a metadata management module, a file value determination module, a migration control module, a file system monitoring module, and a migration module;
the metadata management module is used for acquiring and managing metadata information of the file; the attribute, the file position and the historical access information of the reserved file comprise information such as the reading and writing frequency, the sequence, the random access proportion, the reading and writing size and the like of the file, and the information is the main input for carrying out value judgment on the file in the file value judgment stage. The file value judging module is used for judging the value of the file and outputting the judgment result to the migration control module; the judged result reflects the activity degree of the file; the migration control module receives the file value calculation result of the file value judgment module, generates a file migration schedule and outputs control information to the migration module; the file system monitoring module is used for collecting the performance information of the storage system of the target storage layer and outputting the information to the migration control module; the method comprises the following steps of file system access delay, access hit rate, sequential read-write proportion, storage system residual space and the like, and is an important basis for generating a migration plan and controlling migration speed of a migration module; and the migration module is used for receiving the control information of the migration control module, taking out the migration task from the inside of the migration plan table and migrating the file on different storage layers. In order to ensure that the migrated data access is not limited by a system, an access redirection module is also arranged and used for providing a virtual layer, and an upper layer interface can access different storage layers by using a uniform application program interface.
Although the present invention has been described in detail by referring to the drawings in connection with the preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made on the embodiments of the present invention by those skilled in the art without departing from the spirit and scope of the present invention, and these modifications or substitutions are within the scope of the present invention/any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (10)
1. A method for migrating layered storage data is characterized by comprising the following steps:
acquiring metadata information of a file;
calculating the value of the file according to the metadata information;
generating a file migration schedule according to the file value; the migration tasks in the file migration schedule are arranged according to the file value in the priority order;
and taking out the migration task from the file migration schedule, and migrating the file on different storage layers.
2. The migration method of layered storage data according to claim 1, wherein in the metadata information of the obtained file, the metadata information includes a read-write frequency, a sequential/random access ratio, a read-write size, a read-write sequence/random coefficient of the file;
file read-write order coefficient
Random coefficient for file read and write
δ ∈ (0, 1), setting the average response time of X period as t;
Fs X(i) indicating the frequency of sequential access of the file with the number i in the X period;
Fr X(i) representing the frequency of random access of the file with the number i in an X period;
δ is a constant between 0 and 1, representing the magnitude of the effect of historical access frequency on file worth determination.
3. The method of claim 1, wherein the step of calculating the file value based on the metadata information comprises:
acquiring the time required by all files in the system to finish all read-write operations in a high-rate storage layer and a low-rate storage layer;
calculating a storage layer matching degree coefficient according to the acquired time and the file read-write sequence/random coefficient;
calculating a file value F according to the storage layer matching degree coefficient, the file reading and writing sequence/random coefficient and the size of the current file; the file value F is used for reflecting the value obtained by storing the file in the high-speed storage layer;
4. The method for migrating layered storage data according to claim 3, wherein the step of calculating the storage layer matching degree coefficient according to the acquired time and the file read-write sequence/random coefficient comprises:
calculating time difference between the acquired time required by sequential access in the low-rate storage layer and the acquired time required by sequential access in the high-rate storage layer, and multiplying the calculated time difference by a file read-write sequence coefficient to obtain a sequence matching coefficient;
calculating the time difference between the acquired time required by random access in the low-rate storage layer and the acquired time required by random access in the high-rate storage layer, and multiplying the calculated time difference by a file read-write random coefficient to obtain a random matching coefficient;
and adding the sequential matching coefficient and the random matching coefficient to obtain a storage layer matching degree coefficient.
5. The method of claim 1, wherein the step of generating the file migration schedule according to the file value comprises:
sorting the files according to the value of the files and outputting a file list;
traversing each file in the file list to output a file migration schedule; when traversing the file, if the current file is not in the high-speed storage layer and the size of the file is smaller than the available space of the high-speed storage layer, adding a file migration plan in the migration plan table.
6. The method for migrating layered storage data according to claim 1, wherein the step of taking out the migration task from the file migration schedule and migrating the file on different storage tiers comprises:
taking out a migration task from the file migration schedule, detecting the performance information of the current file system, and migrating the files on different storage layers according to the performance information; the performance parameters comprise the load rate of the file system, the residual storage space and the historical access mode.
7. The method for migrating layered storage data according to claim 6, wherein in the step of taking out the migration task from the file migration schedule, detecting the performance information of the current file system, and migrating the file on different storage tiers according to the performance information, the specific steps include:
detecting the load rate of the current file system;
when the load rate is detected to be smaller than a set load threshold value, detecting the file storage capacity of the target storage layer, and when the file storage capacity of the target storage layer is lower than a set first threshold value, taking out the migration task from the file migration schedule table to execute migration; stopping migration after the current storage space is full or the files are completely migrated;
and when the file storage capacity of the target storage layer is higher than a set second threshold value, taking out the migration task from the file migration schedule to execute migration when the size of the file to be migrated is judged to be smaller than the current available space of the target storage layer, otherwise, not executing the migration process, wherein the first threshold value is smaller than the second threshold value.
8. The method for migrating layered storage data according to claim 7, wherein if the storage tier where the file to be migrated is currently located is a low-rate storage tier and the size of the file is greater than a set large file threshold, the historical access mode of the file is queried, and if the historical access mode of the file is a sequential access mode, the file is not migrated; if the historical access mode of the file is a random access mode, transferring the file to a high-speed storage layer;
and if the storage layer where the file to be migrated is currently located is a low-speed storage layer and the size of the file is smaller than the set small file threshold value, the file is not migrated.
9. A layered storage data migration system is characterized by comprising a metadata management module, a file value judgment module, a migration control module, a file system monitoring module and a migration module;
the metadata management module is used for acquiring and managing metadata information of the file;
the file value judging module is used for judging the value of the file and outputting the judgment result to the migration control module;
the migration control module receives the file value calculation result of the file value judgment module, generates a file migration schedule and outputs control information to the migration module;
the file system monitoring module is used for collecting the performance information of the storage system and outputting the information to the migration control module;
and the migration module is used for receiving the control information of the migration control module, taking out the migration task from the inside of the migration plan table and migrating the file on different storage layers.
10. The system for migrating layered storage data according to claim 9, further comprising an access redirection module, said access redirection module being configured to provide a virtual layer, such that the upper layer interface accesses different storage layers using a uniform application program interface.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010094756.2A CN111367469B (en) | 2020-02-16 | 2020-02-16 | Method and system for migrating layered storage data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010094756.2A CN111367469B (en) | 2020-02-16 | 2020-02-16 | Method and system for migrating layered storage data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111367469A true CN111367469A (en) | 2020-07-03 |
| CN111367469B CN111367469B (en) | 2022-07-08 |
Family
ID=71206123
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010094756.2A Active CN111367469B (en) | 2020-02-16 | 2020-02-16 | Method and system for migrating layered storage data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111367469B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112269759A (en) * | 2020-10-23 | 2021-01-26 | 北京浪潮数据技术有限公司 | Migration method and related device for shared file storage |
| CN112506875A (en) * | 2020-12-15 | 2021-03-16 | 科大讯飞股份有限公司 | File storage method, related device and file storage system |
| CN113515238A (en) * | 2021-07-27 | 2021-10-19 | 华云数据控股集团有限公司 | Data scheduling method and system based on hierarchical storage and electronic equipment |
| CN113741819A (en) * | 2021-09-15 | 2021-12-03 | 第四范式（北京）技术有限公司 | Method and device for hierarchical storage of data |
| CN114816749A (en) * | 2022-04-22 | 2022-07-29 | 江苏华存电子科技有限公司 | Intelligent management method and system for memory |
| CN116974468A (en) * | 2023-07-18 | 2023-10-31 | 广东华原网络工程有限公司 | Equipment data storage management method and system based on big data |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103150263A (en) * | 2012-12-13 | 2013-06-12 | 深圳先进技术研究院 | Hierarchical storage method |
| CN103548004A (en) * | 2011-12-28 | 2014-01-29 | 华为技术有限公司 | Method and device for realizing multilevel storage in file system |
| CN105653591A (en) * | 2015-12-22 | 2016-06-08 | 浙江中控研究院有限公司 | Hierarchical storage and migration method of industrial real-time data |
-
2020
- 2020-02-16 CN CN202010094756.2A patent/CN111367469B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103548004A (en) * | 2011-12-28 | 2014-01-29 | 华为技术有限公司 | Method and device for realizing multilevel storage in file system |
| CN103150263A (en) * | 2012-12-13 | 2013-06-12 | 深圳先进技术研究院 | Hierarchical storage method |
| CN105653591A (en) * | 2015-12-22 | 2016-06-08 | 浙江中控研究院有限公司 | Hierarchical storage and migration method of industrial real-time data |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112269759A (en) * | 2020-10-23 | 2021-01-26 | 北京浪潮数据技术有限公司 | Migration method and related device for shared file storage |
| CN112269759B (en) * | 2020-10-23 | 2023-12-22 | 北京浪潮数据技术有限公司 | Migration method and related device for shared file storage |
| CN112506875A (en) * | 2020-12-15 | 2021-03-16 | 科大讯飞股份有限公司 | File storage method, related device and file storage system |
| CN112506875B (en) * | 2020-12-15 | 2023-01-17 | 科大讯飞股份有限公司 | File storage method, related device and file storage system |
| CN113515238A (en) * | 2021-07-27 | 2021-10-19 | 华云数据控股集团有限公司 | Data scheduling method and system based on hierarchical storage and electronic equipment |
| CN113515238B (en) * | 2021-07-27 | 2024-02-06 | 华云数据控股集团有限公司 | Data scheduling method and system based on hierarchical storage and electronic equipment |
| CN113741819A (en) * | 2021-09-15 | 2021-12-03 | 第四范式（北京）技术有限公司 | Method and device for hierarchical storage of data |
| CN114816749A (en) * | 2022-04-22 | 2022-07-29 | 江苏华存电子科技有限公司 | Intelligent management method and system for memory |
| CN116974468A (en) * | 2023-07-18 | 2023-10-31 | 广东华原网络工程有限公司 | Equipment data storage management method and system based on big data |
| CN116974468B (en) * | 2023-07-18 | 2024-02-20 | 广东华原网络工程有限公司 | Equipment data storage management method and system based on big data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111367469B (en) | 2022-07-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111367469B (en) | Method and system for migrating layered storage data | |
| CN105653591B (en) | A kind of industrial real-time data classification storage and moving method | |
| US9229653B2 (en) | Write spike performance enhancement in hybrid storage systems | |
| CN107943867B (en) | High-performance hierarchical storage system supporting heterogeneous storage | |
| US8909887B1 (en) | Selective defragmentation based on IO hot spots | |
| EP3869316B1 (en) | Hybrid storage | |
| CN104115134B (en) | For managing the method and system to be conducted interviews to complex data storage device | |
| CN103399713B (en) | The data buffering method of balance dynamic data attemper performance and solid-state disk service life | |
| CN104025054B (en) | Dynamic memory layering in virtual environment | |
| CN104516471B (en) | Method and device for managing power supply of storage system | |
| US8782324B1 (en) | Techniques for managing placement of extents based on a history of active extents | |
| US8478939B2 (en) | Relative heat index based hot data determination for block based storage tiering | |
| CN103440208B (en) | A kind of method that data store and device | |
| CN102136005B (en) | Data searching method and device | |
| WO2018019119A1 (en) | Method and device for dynamic partial-parallel data layout for continuous data storage | |
| CN104699424B (en) | A kind of isomery EMS memory management process based on page temperature | |
| CN102117248A (en) | Caching system and method for caching data in caching system | |
| CN105242871A (en) | Data writing method and apparatus | |
| CN103914516A (en) | Method and system for layer-management of storage system | |
| WO2018113317A1 (en) | Data migration method, apparatus, and system | |
| CN104102454A (en) | Method for automatically realizing hierarchical storage and system for managing hierarchical storage | |
| CN103916459A (en) | Big data filing and storing system | |
| CN107340975A (en) | A kind of method and device of file storage | |
| CN106371762A (en) | Optimization method and system of storage data | |
| CN115858510A (en) | Method for evaluating data temperature and performing dynamic storage management and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |