CN114217740A - Storage management method, equipment, system and computer readable storage medium - Google Patents
Storage management method, equipment, system and computer readable storage medium Download PDFInfo
- Publication number
- CN114217740A CN114217740A CN202111447378.2A CN202111447378A CN114217740A CN 114217740 A CN114217740 A CN 114217740A CN 202111447378 A CN202111447378 A CN 202111447378A CN 114217740 A CN114217740 A CN 114217740A
- Authority
- CN
- China
- Prior art keywords
- storage area
- storage
- migration
- data files
- unavailable
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 56
- 238000013508 migration Methods 0.000 claims abstract description 130
- 230000005012 migration Effects 0.000 claims abstract description 130
- 230000014759 maintenance of location Effects 0.000 claims abstract description 64
- 239000011800 void material Substances 0.000 claims abstract description 51
- 238000000034 method Methods 0.000 claims description 34
- 238000004590 computer program Methods 0.000 claims description 13
- 238000010586 diagram Methods 0.000 description 14
- 238000012217 deletion Methods 0.000 description 6
- 230000037430 deletion Effects 0.000 description 6
- 230000007613 environmental effect Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a storage management method, equipment, a system and a computer readable storage medium, wherein an SMR disk comprises a plurality of storage areas, and the plurality of storage areas comprise a hole storage area and an empty storage area, and the storage management method comprises the following steps: acquiring a void storage area list in an SMR disk, wherein the void storage area list comprises a void storage area, at least one data file is stored in the void storage area, and the void storage area is a storage area in which metadata of the data file is deleted; selecting a current migration strategy from the prediction migration strategy set; and migrating the data files in the hollow storage area to the hollow storage area based on the current migration strategy and the retention period information of the data files. Through the mode, the storage space can be released, and the space utilization rate is greatly improved.
Description
Technical Field
The present application relates to the field of data storage technologies, and in particular, to a storage management method, device, system, and computer-readable storage medium.
Background
The SMR (corrugated stack) disk only supports sequential writing from head to tail, and has the characteristic of whole block deletion, when a plurality of different data files are stored in a storage area in the SMR disk, if a certain data file is expired, the data file cannot be deleted independently in the storage area, corresponding metadata can only be deleted in a cloud server, so that a capacity hole is caused, the physical capacity of the expired data file cannot be released in time, the storage space of the SMR disk is continuously occupied, and the space utilization rate of the SMR disk is greatly reduced.
Disclosure of Invention
The application provides a storage management method, equipment, a system and a computer readable storage medium, which can release storage space and greatly improve space utilization rate.
In order to solve the technical problem, the technical scheme adopted by the application is as follows: provided is a storage management method based on an SMR disk, which comprises the following steps: acquiring a void storage area list in an SMR disk, wherein the void storage area list comprises a void storage area, at least one data file is stored in the void storage area, and the void storage area is a storage area in which metadata of the data file is deleted; selecting a current migration strategy from the prediction migration strategy set; and migrating the data files in the hollow storage area to the hollow storage area based on the current migration strategy and the retention period information of the data files.
In order to solve the above technical problem, another technical solution adopted by the present application is: there is provided a storage management device, which includes a memory and a processor connected to each other, wherein the memory is used for storing a computer program, and the computer program is used for implementing the storage management method based on the SMR disk in the above technical solution when being executed by the processor.
In order to solve the above technical problem, another technical solution adopted by the present application is: the storage management device is used for managing the SMR disk, wherein the storage management device is the storage management device in the technical scheme, and a storage area in the SMR disk is used for storing data files.
In order to solve the above technical problem, the present application adopts another technical solution: a computer-readable storage medium is provided for storing a computer program, and the computer program is used for implementing the method for SMR disk-based storage management in the above technical solution when executed by a processor.
Through the scheme, the beneficial effects of the application are that: the method comprises the steps of firstly obtaining a hollow storage area list in an SMR disk, then selecting a current migration strategy from a pre-established prediction migration strategy set, then migrating data files in a hollow storage area to the hollow storage area according to the current migration strategy and retention period information of the data files, and selecting the most appropriate migration strategy according to a current application environment by setting the prediction migration strategy set so as to release a storage space of the hollow storage area according to the current migration strategy, rapidly providing storage service for the outside and greatly increasing the utilization rate of the storage space; moreover, due to the fact that multiple migration strategies are provided, application requirements of different scenes can be met, and applicability is expanded.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts. Wherein:
FIG. 1 is a schematic flow chart diagram illustrating an embodiment of a method for SMR disk-based storage management provided herein;
FIG. 2 is a schematic diagram of the EC mode of "4 + 1" provided herein;
FIG. 3 is a schematic diagram of a storage area 1 provided herein;
FIG. 4 is a schematic diagram of two storage area groups provided herein storing data files;
FIG. 5 is a schematic flow chart of writing a data file provided herein;
FIG. 6 is a schematic diagram of a storage area provided herein storing a plurality of different data files;
FIG. 7 is a schematic flow chart diagram illustrating another embodiment of a method for SMR disk-based storage management provided herein;
FIG. 8 is a schematic flow chart diagram illustrating one embodiment of migrating data files provided herein;
FIG. 9 is a flowchart illustrating a first migration policy provided herein;
FIG. 10 is a flowchart illustrating a second migration policy provided herein;
FIG. 11 is a schematic structural diagram of an embodiment of a storage management device provided in the present application;
FIG. 12 is a schematic diagram of an embodiment of a storage management system provided herein;
FIG. 13 is a schematic structural diagram of an embodiment of a computer-readable storage medium provided in the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be noted that the following examples are only illustrative of the present application, and do not limit the scope of the present application. Likewise, the following examples are only some examples and not all examples of the present application, and all other examples obtained by a person of ordinary skill in the art without any inventive work are within the scope of the present application.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
It should be noted that the terms "first", "second" and "third" in the present application are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of the feature. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless explicitly specifically limited otherwise. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
Referring to fig. 1, fig. 1 is a schematic flowchart illustrating an embodiment of a method for SMR disk-based storage management, where the method includes:
step 11: and acquiring a hole storage area list in the SMR disk.
The SMR disk comprises a plurality of storage areas (zones), wherein the storage areas comprise a hole storage area and a hollow storage area, each storage area is used for storing at least one data file, a hole storage area list comprises the hole storage areas, the hole storage areas are the storage areas with deleted metadata of the data files, and the hole storage areas store at least one data file; because the SMR disk only supports the whole deletion of the storage area, namely only all data files in the whole storage area can be deleted, and certain data files in the storage area cannot be deleted independently, when a certain data file in the storage area is overdue, the synchronous deletion with metadata of the overdue data file cannot be realized, so that the phenomenon of capacity holes is generated, and a hole storage area is generated.
Specifically, the storage areas in the SMR disk are areas that are formed by continuous tracks and can be written in continuously, and data files acquired from a client may be stored in each storage area, where the number of the storage areas is related to the actual memory size of the SMR disk, and the memory size of one storage area may be generally 256MB, for example: the memory size of the SMR disk is 18TB, so that the SMR disk comprises 68664 storage areas; further, the client includes, but is not limited to, a device such as a camera, and when the client stores the data file into the storage area, the client may also store object storage information (bucket) of the client in addition to detailed information of the data file (including a name, a size, and user information to which the data file belongs), and each client corresponds to one object storage information, so that the data file input by the client can be correspondingly found through the object storage information.
In a specific embodiment, an Erasure Code (EC) mode may be used to store the data file in the SMR disk, where the EC mode is: adding m parts of coded data to n parts of original data, so that any n parts of data in the n + m parts of data are restored into the original data, and the data storage is safer and more reliable; as shown in fig. 2, taking an EC mode of "4 + 1" (i.e., n is 4, m is 1) as an example for explanation, and fig. 2 is a schematic diagram of a storage manner of a data file in the EC mode of "4 + 1", if a data file with a memory size of 256MB is input at this time, the data file with the memory size of 256MB may be equally divided into four data files with memory sizes of 64MB respectively, and then each data file is stored in a corresponding storage area 1-4, and corresponding check data is generated based on the data file through a corresponding EC algorithm and stored in a corresponding check storage area (i.e., storage area 5); then, when inputting the next data file, the next data file can be stored into the storage area group (i.e. storage areas 1-5) shown in fig. 2 according to the same EC mode, and the next storage area group is not switched to store until the remaining storage space in the storage area group is smaller than the minimum memory size (generally 4MB) of the file, for example: the storage areas 6-10, such as the final storage area 1 shown in FIG. 3, can store a plurality of data files. It is understood that n may be 4, 5, 6, etc., and m may be 1, 2, 3, etc., which are not limited herein and may be set according to practical applications.
Further, when the data file to be stored is large (for example, more than 1GB) and the storage space of 1-4 of four storage areas is insufficient, the data file can be stored by being divided into a plurality of storage area groups, the data file can be equally divided into two, three or more than three parts, and then the corresponding number of storage area groups are obtained, so that each part of the data file is stored into the corresponding storage area group according to the corresponding EC mode; as shown in FIG. 4, the data file can be stored in two storage zone groups (i.e., objects 1-2), each storage zone group comprising five storage zones, object 1 comprising zone 1-zone 4 and ECzone, and object2 comprising zone 1 '-zone 4' and ECzone ', wherein ECzone and ECzone' are used to indicate the storage zones for storing encoded data.
In another specific embodiment, the server may comprise a storage node for storing data files, data files may be imported into the storage node via a Software Development Kit (SDK) in the client, so as to utilize the storage nodes to store data files, four data files as shown in FIG. 2 can be stored in the corresponding storage areas 1-4 through the storage nodes 1-4 respectively, when storing data files in the storage areas, the storage nodes can be used for adding index information to the corresponding storage areas, the index information comprises the position of the current write pointer of the storage area and the residual storage space of the storage area, when writing the next data file into the storage area, it can be determined whether the size of the remaining storage space satisfies the storage condition based on the index information, the position of the current write pointer is then shifted back when the condition is satisfied, thereby enabling sequential writing of the SMR disk.
Further, the server may further include a management node, where the management node is configured to manage the storage nodes, that is, manage the data files in the storage area, as shown in fig. 5, the client may obtain the required storage nodes 1 to N from the management node, and then store the data files to be stored in the storage nodes 1 to N; the migration and deletion of the data files are uniformly controlled by the management node, so that the reliability of management can be improved, and the consistency of data can be ensured; for example: when all data files in the storage area are deleted, the management node can be used for controlling the write pointer to be reset to the initialization position so as to recycle the storage area.
Step 12: and selecting the current migration strategy from the prediction migration strategy set.
The predicted migration strategy set comprises at least two migration strategies which are preset migration schemes; the current migration strategy can be selected from the prediction migration strategy set to migrate the storage space in the hollow storage area, so that the storage space of the hollow storage area is released.
It will be appreciated that the current migration policy may be selected based on current environmental conditions or user application requirements, etc., such as: when the current environmental pressure is larger/a user urgently needs the memory space, a migration strategy for quickly releasing the memory space can be adopted, so that the aim of releasing the memory space in time is fulfilled.
Step 13: and migrating the data files in the hollow storage area to the hollow storage area based on the current migration strategy and the retention period information of the data files.
When the data files are stored, the corresponding retention period information of each data file is stored at the same time, the retention period information can be recorded in the corresponding object storage information, the retention period information can comprise the storage time and the retention period time of the data files, the retention period time is the effective storage time of the data files, and is generally measured in hours or days. It will be appreciated that the retention period time may be custom generated according to the current application scenario or set by the user according to the current application requirements, for example: the retention period time of the data files stored in the daytime can be set to be 30 days, and the retention period time of the data files stored at night can be set to be 15 days; or, in a security storage scene, the retention period time of the video record can be 90 days, the retention period time of the picture can be 180 days, and after the retention period time expires, the data file is invalid, and the data file is deleted in time to release the storage space, so that the writing of a new data file is facilitated.
Specifically, the same storage area may contain a plurality of different data files, and the retention periods of the plurality of data files may all be different, as shown in fig. 6, for example: data files A to D are stored in a hollow storage area, the retention period time corresponding to each of the data files A to D is respectively 10 days, 5 days, 15 days and 30 days, the current state is that the data file B is expired, and because the data file B cannot be deleted in the storage area alone, if the storage space in the hollow storage area is required to be released and the utilization rate of the storage space is increased, the data files in the hollow storage area can be migrated to the hollow storage area based on the current migration strategy and the retention period information of the data files, so that the storage space in the current hollow storage area is released. Further, the data files may be migrated to one, two, or more than two empty storage areas, that is, all the data files may be migrated to the same empty storage area based on the current migration policy, or the data files may be migrated to a plurality of different empty storage areas, respectively. Because the empty storage area is selected as a new storage area during data migration, the invalid migration action of 2 storage areas with different retention periods immediately after migration can be avoided.
According to the scheme provided by the embodiment, the predicted migration strategy set is set, and the most suitable migration strategy is selected according to the current application environment, so that the storage space of the hollow storage area can be released conveniently according to the current migration strategy, the storage service can be provided for the outside quickly, and the utilization rate of the storage space is greatly increased; moreover, due to the fact that multiple migration strategies are provided, application requirements of different scenes can be met, and applicability is improved.
Referring to fig. 7, fig. 7 is a schematic flowchart illustrating another embodiment of a method for SMR disk-based storage management according to the present application, where the method includes:
step 71: all the storage areas are divided into usable storage areas, unusable storage areas and hole storage areas based on the residual storage space of the storage areas.
The SMR disk comprises a hole storage area, an available storage area and an unavailable storage area, wherein the available storage area comprises the hole storage area, all the storage areas can be classified by judging the size of the residual storage space of each storage area, the available storage area is stored in an available storage space list, the unavailable storage area is stored in an unavailable storage space list, and the hole storage area is stored in the hole storage space list, so that the storage areas can be classified and managed conveniently.
In a specific embodiment, whether the remaining storage space of the storage area is smaller than a preset storage space is judged, if the remaining storage space of the storage area is smaller than the preset storage space, the storage area is determined to be an unavailable storage area, and the unavailable storage area is stored in an unavailable storage area list; and if the residual storage space of the storage area is larger than or equal to the preset storage space, determining that the storage area is an available storage area, and storing the available storage area into an available storage area list.
The preset storage space can be the minimum writing unit of the data file, namely the minimum memory size of the writable data file, the minimum writing unit can be set according to the actual situation and can be generally set to be 4MB, namely whether the residual storage space of the storage area is less than 4MB or not is judged, and if the residual storage space of the storage area is more than or equal to 4MB, the storage area is determined to be an available storage area; it can be understood that, when the data file is stored in the EC mode, the remaining storage spaces of the available storage areas, which are in the standard of "n + m" storage areas, are all greater than or equal to the preset storage space, so that the data file can be stored in five storage areas in a distributed and average manner.
In other specific embodiments, the available storage area list may be traversed to mark an available storage area, in which the remaining storage space in the available storage area list is smaller than a preset storage space, as an unavailable storage area, and add the unavailable storage area list to the unavailable storage area list, so as to update the available storage area list and the unavailable storage area list in time; it is understood that the operation of traversing the available storage area list can be performed periodically, and the specific time interval can be set according to actual situations.
In addition to classifying the storage areas, the data files in each storage area may also be managed for a retention period, as shown in the following steps 72-74:
step 72: an unavailable memory area is obtained from the list of unavailable memory areas.
Step 73: and judging whether the number of the data files in the unavailable storage area is larger than a preset value or not.
And acquiring an unavailable storage area from the unavailable storage area list, and judging whether the number of the data files in the unavailable storage area is greater than a preset value, wherein the preset value can be 1, namely judging whether the number of the data files in the unavailable storage area is greater than 1.
Step 74: and if the number of the data files in the unavailable storage area is larger than a preset value, marking the unavailable storage area as an available storage area/a hollow storage area based on the maximum retention period time.
When two or more data files are contained in the unavailable storage area, acquiring the maximum retention time of all the data files, wherein the maximum retention time is the maximum value of the retention time of all the data files in the unavailable storage area, for example: storing the data files 1-4, wherein the retention time corresponding to each data file is 10 days, 5 days, 15 days and 30 days respectively, the maximum retention time is 30 days, and then marking the unavailable storage area as an available storage area/a hole storage area based on the maximum retention time.
In a specific embodiment, it is determined whether the maximum retention time is smaller than a time difference, where the time difference is a difference between the current system time and a storage time of the data file corresponding to the maximum retention time, and if the maximum retention time is smaller than the time difference, which indicates that all the data files in the storage area are expired, all the data files in the unavailable storage area are deleted, and the unavailable storage area is marked as an available storage area and stored in the available storage area list.
If the maximum retention time is greater than or equal to the time difference, it is indicated that all the data files in the storage area are not overdue, and all the data files cannot be deleted directly, the retention time of each data file is compared with the time difference, the metadata corresponding to the data files with the retention time greater than the time difference is deleted, that is, the metadata corresponding to the overdue data files are deleted, so that a hole storage area is obtained, the unavailable storage area is marked as a hole storage area, and the hole storage area is stored in a hole storage area list.
And if the number of the data files in the unavailable storage area is equal to 1, returning to the step of acquiring an unavailable storage area from the unavailable storage area list until the traversal of the unavailable storage area list is finished. Specifically, when the number of the data files in the unavailable storage area is 1, the unavailable storage area can be directly recovered when the data files are overdue, the data files are deleted, the situation of capacity void can not occur, then the next unavailable storage area can be directly returned to the unavailable storage area list to obtain the next unavailable storage area until the list of the unavailable storage areas is completely traversed, retention period judgment is performed on all the unavailable storage areas in the unavailable storage area list, the screened void storage areas are stored in the void storage area list, and the available storage areas generated after the storage space is released are stored in the available storage area list.
It can be understood that when the number of data files in the unavailable storage area is equal to 1, whether the retention period time of the data files is less than the time difference value is judged; and if the retention period time of the data file is less than the time difference value, deleting the data file, initializing and resetting a write pointer in the unavailable storage area, marking the unavailable storage area as an empty storage area, and storing the empty storage area into an available storage area list.
Through the above steps, all storage areas in the SMR disk can be classified and managed, a large number of void storage areas may occupy storage space, the storage space occupied by an expired data file cannot be released, and a new data file is not written with storage space, at this time, a void storage area list can be obtained from the SMR disk, and then a current migration policy is selected from the prediction migration policy set, so that the data file in the void storage area is migrated to the void storage area according to the current migration policy and the retention period information of the data file, thereby releasing the storage space of the void storage area, and the specific steps are shown in fig. 8:
step 81: and acquiring the number of the hole storage areas.
Step 82: and judging whether the ratio of the number of the void storage areas to the number of all the storage areas is larger than a preset ratio or not.
Calculating the ratio of the number of the void storage areas to the number of all the storage areas, and calculating the number ratio of the current void storage areas in all the storage areas, so as to judge whether the number of the void storage areas is excessive or not and whether the data files in the void storage areas need to be migrated or not by comparing the number ratio of the current void storage areas in all the storage areas with a preset ratio; specifically, the preset ratio may be set according to actual application requirements, the preset ratio is illustrated as 0.01 in this embodiment, and in other embodiments, the preset ratio may also be 0.02 or 0.03, and the like, which is not limited herein.
The method can be used for periodically searching the hollow storage area list to obtain the number of the hollow storage areas so as to judge whether the occupation ratio of the current hollow storage areas exceeds a preset ratio, the time interval can be set according to the actual condition, and the searching can be performed once every one day or two days without limitation.
Step 83: and if the ratio of the number of the void storage areas to the number of all the storage areas is larger than a preset ratio, selecting the current migration strategy from the prediction migration strategy set.
When the number ratio of all the storage areas in the void storage areas is more than 0.01, the number of the current void storage areas is large, the data files in the void storage areas need to be migrated, and the current migration strategy can be selected from the predicted migration strategy set to release the storage space occupied by the void storage areas; it can be understood that, if the ratio of the number of the hole storage areas to the number of all the storage areas is equal to or smaller than the preset ratio, the step of obtaining the number of the hole storage areas is returned, and the traversal of the hole storage areas is continuously performed until the ratio of the number of the hole storage areas to the number of all the storage areas is larger than the preset ratio, so that the step of migrating the data file is performed.
Step 84: and acquiring current strategy selection information, and judging whether the current strategy selection information meets a preset condition.
The predicted migration policy set may include a first migration policy and a second migration policy, and the selection of the first migration policy/the second migration policy may be determined by determining whether the current policy selection information satisfies a preset condition.
Specifically, it may be determined whether the current policy selection information includes information corresponding to the first migration policy, and if the current policy selection information includes information corresponding to the first migration policy, it is determined that the current policy selection information satisfies a preset condition; for example: the current policy selection information may include identification information corresponding to a currently required migration policy, the identification information corresponding to the first migration policy may be "1", and the identification corresponding to the second migration policy may be "2", and then it may be determined whether the current policy selection information includes the identification information "1", so as to determine whether the current policy selection information satisfies a preset condition.
The current strategy selection information can also comprise a bandwidth value, wherein the bandwidth value can be an overall bandwidth value of the server operation, is used for reflecting the current operation environment pressure, judging whether the bandwidth value in the current strategy selection information is smaller than a preset threshold value or not, and if the bandwidth value in the current strategy selection information is smaller than the preset threshold value, determining that the current strategy selection information meets a preset condition; specifically, when the bandwidth value in the current policy selection information is smaller than a preset threshold, which indicates that the current operating environment pressure is lower, a migration policy that occupies more resources but has a lower secondary migration frequency may be used to perform migration of the data file, that is, a first migration policy described below; it is understood that the preset threshold may be set according to the actual network condition and the application environment, and the current policy selection information may be set by a user in a customized manner or automatically generated by the server according to the current application environment.
Step 85: and if the current strategy selection information meets the preset condition, determining that the current migration strategy is a first migration strategy.
The first migration policy may be to migrate data files with the same valid deadline to the same empty storage area, and when migrating, the empty storage area is used to ensure that the valid deadline of the data files contained in the storage area after migration is the same, and the retention period information corresponding to each data file includes the storage time of the data file and the retention period time of the data file, where the valid deadline is the sum of the storage time and the retention period time of the data file, for example: the storage time of the data file is 2021/01/1110: 25:32, the retention period time is 1 day, and the corresponding effective deadline time is 2021/01/1210: 25: 32.
As shown in fig. 9, taking an example that the empty storage area includes four data files File 1 to 4, and the retention periods of the four data files are respectively 30 days, 15 days, 5 days, and 40 days as an example for description, it can be understood that for clear expression, the retention period is used in this example to represent valid deadlines, that is, the storage periods of the data files are the same, and the storage periods of the data files are different in actual situations, where File 3 is an expired data File, the metadata corresponding to File 3 is deleted, the data files with the same valid deadlines (the retention periods in this example) are migrated to the same empty storage area according to a first migration policy, then three different empty storage areas 1 to 3 are respectively obtained, then File 1 with a retention period of 30 days is stored in empty storage area 1, File 2 with a retention period of 15 days is stored in empty storage area 2, files 4 with the retention period of 40 days are stored in the empty storage area 3, and similarly, data files with the same retention period in other empty storage areas are also stored in the empty storage areas 1-3 until the storage space in the empty storage areas is smaller than the preset storage space, namely, new data files cannot be stored.
It will be appreciated that the migrated data file is also stored in the empty storage area in the same EC pattern as when it was stored, for example: the data files before migration are stored in 5 storage areas in a distributed mode, and the data files after migration are still stored in other 5 empty storage areas in a distributed mode.
Step 86: and if the current strategy selection information does not meet the preset condition, determining that the current migration strategy is a second migration strategy.
The second migration strategy is to migrate all the data files in the void storage area to the same void storage area, and the arrangement sequence of all the data files in the void storage area before migration is the same as the arrangement sequence of all the data files in the void storage area after migration; the arrangement sequence of the data files is related to the storage time, the smaller the storage time is, the more the arrangement sequence is, taking the example that the storage regions store the data files from top to bottom, the data file stored firstly is located at the first position of the top of the storage region, namely the arrangement sequence is 1, the next data file stored is stored in the next position of the storage region in sequence, namely the arrangement sequence is 2, and so on, the data files are also transferred to the corresponding position in the empty storage region according to the arrangement sequence during storage during transfer.
As shown in fig. 10, the same void storage area as that in fig. 9 is taken as an example for explanation, at this time, if a second migration policy is adopted to migrate a data File, a void storage area can be obtained, and then the data File is migrated into the void storage area, the original arrangement order of the data files File 1, File 2, and File4 is File 1, File 2, and File4, and File 1 is located at the uppermost position of the storage area, then after migrating to the void storage area at this time, File 1 is still located at the uppermost position of the storage area, the arrangement order of the data files File 1, File 2, and File4 is File 1, File 2, and File4, the storage space occupied by data File 3 originally located in the middle of the storage area is released, and the tail end position of the migrated void storage area can also be stored with a new data File.
It can be understood that, after the data file is migrated by using the first migration policy/the second migration policy, the write pointer of the void storage area may be initialized and reset, and the storage space of the original void storage area is released, so as to obtain a new void storage area, and store the new void storage area in the available storage space list.
In a specific embodiment, when the data file is migrated by adopting the first migration strategy/the second migration strategy, the data file can be migrated by selecting a parallel migration or serial migration mode according to the current environmental pressure; specifically, when the current environmental pressure is low, a parallel migration method can be adopted to migrate the data files so as to simultaneously migrate all the data files in the void storage area to the void storage area, so that the migration speed is increased, and the migration duration is greatly reduced; when the current environmental pressure is higher, the data files can be migrated by adopting a serial migration method, and the data files in the void storage areas are sequentially migrated to the void storage areas, so that fewer system resources are occupied, and the stability of the storage system is ensured.
The embodiment divides all storage areas into the usable storage areas, the unusable storage areas and the hollow storage areas, regularly manages the retention period time of the data files in all the storage areas, realizes fine management of all the storage areas, and greatly improves the overall space utilization rate of the storage system. 2 different migration strategies are provided, the data files in the hollow storage area can be migrated to the hollow storage area by utilizing the migration strategies, and the storage space occupied by the expired data files can be released in time, so that the utilization rate of the storage space is greatly improved; meanwhile, different migration strategies can be selected based on the current application environment, the migration strategies suitable for the current environment resources can be selected, the data files with the same effective deadline time can be migrated to the same empty storage area by adopting the first migration strategy, and the effective deadline time of the data files in the storage area after migration can be unified by one-time migration, so that the whole storage area can be recycled when the data files in the storage area after migration are overdue, batch deletion is realized, the number of empty storage areas is greatly reduced, and the frequency of secondary migration is reduced; and all data files in the void storage area can be migrated to the same void storage area by adopting a second migration strategy, so that the rapid release of the storage space can be realized, the capacity can be rapidly provided, and the stability of the storage system can be ensured. In addition, in an application scene with low cluster system pressure, a parallel migration mode is selected, so that the migration speed can be increased, and the migration time can be greatly reduced; when the cluster system is under a high-pressure application scene, a serial migration mode is selected, so that fewer system resources can be occupied, and the stability of the storage system is ensured. In addition, the proportion and the migration frequency for providing data migration can be dynamically adjusted, the timeliness of the data migration and the occupation of environmental resources can be met, and the fastest data migration and the fastest space release can be achieved under the condition that the writing of upper-layer service specifications is not influenced.
Referring to fig. 11, fig. 11 is a schematic structural diagram of an embodiment of a storage management device provided in the present application, where the storage management device 110 includes a memory 111 and a processor 112 that are connected to each other, the memory 111 is used for storing a computer program, and the computer program is used for implementing the storage management method based on an SMR disk in the foregoing embodiment when being executed by the processor 112.
Referring to fig. 12, fig. 12 is a schematic structural diagram of an embodiment of the storage management system provided in the present application, where the storage management system 120 includes a storage management device 121 and an SMR disk 122 that are connected to each other, the storage management device 121 is used to manage the SMR disk 122, the storage management device 121 is a storage management device in the foregoing embodiment, and a storage area in the SMR disk 122 is used to store a data file.
The scheme of the embodiment has the advantages that:
1) be applicable to the storage field, during data write in the storage area of SMR disk with the different retention period, satisfied that monoblock storage area order is write in, monoblock storage area is retrieved, and the monoblock storage area can't be retrieved and carries out data recovery after the data migration again, satisfies the requirement of various storage schemes to capacity practicality, greatly promotes storage management system's whole space utilization.
2) 2 different migration strategy selections are provided, the secondary migration frequency can be reduced after the migration is selected, and the service can be rapidly provided to the outside after the migration is selected, so that the robustness of the system is ensured.
3) The frequency of data migration and the triggering proportion both support dynamic configuration, all scenes of upper-layer services are completely and dynamically compatible through 2 configurations, and the availability of the cluster is improved.
4) The method and the device meet concurrent migration and serial migration, support rapid migration of multiple files, provide services rapidly, and ensure that space detection services provide functions rapidly to the outside.
5) The front and back migration and deletion of the data are controlled by the management node uniformly, and the consistency of the data can be ensured with the highest reliability.
Referring to fig. 13, fig. 13 is a schematic structural diagram of an embodiment of a computer-readable storage medium 130 provided in the present application, where the computer-readable storage medium 130 is used to store a computer program 131, and when the computer program 131 is executed by a processor, the computer program is used to implement the method for storage management based on an SMR disk in the foregoing embodiment.
The computer readable storage medium 130 may be a server, a usb disk, a removable hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program codes.
In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of modules or units is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed.
Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The above description is only an example of the present application and is not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings, or which are directly or indirectly applied to other related technical fields, are intended to be included within the scope of the present application.
Claims (15)
1. A storage management method based on an SMR disk, wherein the SMR disk comprises a plurality of storage areas, and the storage areas comprise a hole storage area and an empty storage area, and the method comprises the following steps:
acquiring a hole storage area list in the SMR disk, wherein the hole storage area list comprises a hole storage area, at least one data file is stored in the hole storage area, and the hole storage area is a storage area in which metadata of the data file is deleted;
selecting a current migration strategy from the prediction migration strategy set;
and migrating the data files in the empty storage area to the empty storage area based on the current migration strategy and the retention period information of the data files.
2. The method of claim 1, wherein the set of predicted migration policies comprises a first migration policy and a second migration policy, and wherein the step of selecting the current migration policy from the set of predicted migration policies comprises:
acquiring current strategy selection information, and judging whether the current strategy selection information meets a preset condition or not;
if so, determining that the current migration policy is the first migration policy;
if not, determining that the current migration policy is the second migration policy.
3. The method of SMR disk-based storage management according to claim 2, comprising:
judging whether the current strategy selection information comprises information corresponding to the first migration strategy; if so, determining that the current strategy selection information meets the preset condition; or
And judging whether the bandwidth value in the current strategy selection information is smaller than a preset threshold value, if so, determining that the current strategy selection information meets the preset condition.
4. The SMR disk-based storage management method of claim 2,
the retention period information comprises the storage time of the data files and the retention period time of the data files, and the first migration strategy is to migrate the data files with the same effective deadline time to the same empty storage area;
the second migration strategy is to migrate all the data files in the empty storage area to the same empty storage area; and the arrangement sequence of all the data files in the void storage area before the migration is the same as the arrangement sequence of all the data files in the void storage area after the migration.
5. The method of SMR disk-based storage management according to claim 1, wherein the step of selecting the current migration policy from the set of predicted migration policies is preceded by the steps of:
acquiring the number of the void storage areas;
judging whether the ratio of the number of the cavity storage areas to the number of all the storage areas is larger than a preset ratio or not;
and if so, selecting the current migration strategy from the prediction migration strategy set.
6. The SMR disk-based storage management method of claim 1, wherein the SMR disk further comprises an available storage area and an unavailable storage area, the available storage area comprising the empty storage area, the method further comprising:
dividing all the storage areas into the available storage areas, the unavailable storage areas and the hole storage areas based on the residual storage space of the storage areas;
storing the available storage area into an available storage space list;
storing the unavailable storage area into an unavailable storage space list;
and storing the hole storage area into the hole storage space list.
7. The method of claim 6, wherein the step of dividing the storage areas into the usable storage area, the unusable storage area, and the hole storage area based on the remaining storage space of the storage areas comprises:
judging whether the residual storage space of the storage area is smaller than a preset storage space or not;
if so, determining the storage area as the unavailable storage area, and storing the unavailable storage area into the unavailable storage area list;
if not, determining the storage area as the available storage area, and storing the available storage area into the available storage area list.
8. The method of SMR disk-based storage management according to claim 7, the method further comprising:
acquiring an unavailable storage area from the unavailable storage area list, and judging whether the number of data files in the unavailable storage area is greater than a preset value or not;
if so, marking the unavailable storage area as the available storage area/the hole storage area based on the maximum retention time, wherein the maximum retention time is the maximum value of the retention time of all data files in the unavailable storage area;
if not, returning to the step of acquiring an unavailable storage area from the unavailable storage area list until the traversing of the unavailable storage area list is completed.
9. The SMR disk-based storage management method of claim 8, wherein the step of marking the unavailable storage as the available storage/the hole storage based on the maximum retention time comprises:
judging whether the maximum retention period time is smaller than a time difference value, wherein the time difference value is the difference value between the current system time and the storage time of the data file corresponding to the maximum retention period time;
if the maximum retention period time is less than the time difference value, deleting all data files in the unavailable storage area, marking the unavailable storage area as an available storage area, and storing the available storage area list;
if the maximum retention period time is larger than or equal to the time difference, comparing the retention period time of each data file with the time difference, deleting the metadata corresponding to the data files with the retention period time larger than the time difference, marking the unavailable storage areas as hole storage areas, and storing the hole storage areas into the hole storage area list.
10. The method of SMR disk-based storage management of claim 9, the method further comprising:
when the number of the data files in the unavailable storage area is equal to the preset value, judging whether the retention period time of the data files is smaller than the time difference value;
and if so, deleting the data file, marking the unavailable storage area as an empty storage area, and storing the empty storage area into the available storage area list.
11. The method of SMR disk-based storage management according to claim 7, the method further comprising:
and traversing the available storage area list to mark the available storage areas with the residual storage space smaller than the preset storage space in the available storage area list as unavailable storage areas, and adding the unavailable storage areas into the unavailable storage area list.
12. The SMR disk-based storage management method of claim 3, further comprising:
migrating the data files by adopting a parallel migration method so as to simultaneously migrate all the data files in the empty storage area to the empty storage area; or
And migrating the data files by adopting a serial migration method, and sequentially migrating the data files in the hollow storage area to the hollow storage area.
13. A storage management device comprising a memory and a processor connected to each other, wherein the memory is configured to store a computer program, and the computer program is configured to implement the SMR disk-based storage management method of any one of claims 1-12 when executed by the processor.
14. A storage management system, comprising a storage management device and an SMR disk connected to each other, the storage management device being configured to manage the SMR disk, wherein the storage management device is the storage management device according to claim 13, and a storage area in the SMR disk is configured to store a data file.
15. A computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, is configured to implement the method for SMR disk based storage management of any one of claims 1-12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111447378.2A CN114217740A (en) | 2021-11-30 | 2021-11-30 | Storage management method, equipment, system and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111447378.2A CN114217740A (en) | 2021-11-30 | 2021-11-30 | Storage management method, equipment, system and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114217740A true CN114217740A (en) | 2022-03-22 |
Family
ID=80699304
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111447378.2A Pending CN114217740A (en) | 2021-11-30 | 2021-11-30 | Storage management method, equipment, system and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114217740A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114816278A (en) * | 2022-06-30 | 2022-07-29 | 苏州浪潮智能科技有限公司 | Data migration method, system, equipment and storage medium of storage server |
| CN117289877A (en) * | 2023-11-23 | 2023-12-26 | 云筑信息科技(成都)有限公司 | Universal file storage dynamic proxy method based on service grid |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9395937B1 (en) * | 2013-12-27 | 2016-07-19 | Emc Corporation | Managing storage space in storage systems |
| CN107590283A (en) * | 2017-09-29 | 2018-01-16 | 浙江大华技术股份有限公司 | A kind of file recovery method, device, server and computer-readable recording medium |
| CN112269535A (en) * | 2020-10-16 | 2021-01-26 | 苏州浪潮智能科技有限公司 | Space resource allocation method and device of storage system and readable storage medium |
| CN112328549A (en) * | 2020-10-29 | 2021-02-05 | 无锡先进技术研究院 | Small file storage method, electronic device and storage medium |
| CN112463058A (en) * | 2020-11-27 | 2021-03-09 | 杭州海康威视系统技术有限公司 | Fragmented data sorting method and device and storage node |
| CN113485874A (en) * | 2021-09-07 | 2021-10-08 | 联想凌拓科技有限公司 | Data processing method and distributed storage system |
-
2021
- 2021-11-30 CN CN202111447378.2A patent/CN114217740A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9395937B1 (en) * | 2013-12-27 | 2016-07-19 | Emc Corporation | Managing storage space in storage systems |
| CN107590283A (en) * | 2017-09-29 | 2018-01-16 | 浙江大华技术股份有限公司 | A kind of file recovery method, device, server and computer-readable recording medium |
| CN112269535A (en) * | 2020-10-16 | 2021-01-26 | 苏州浪潮智能科技有限公司 | Space resource allocation method and device of storage system and readable storage medium |
| CN112328549A (en) * | 2020-10-29 | 2021-02-05 | 无锡先进技术研究院 | Small file storage method, electronic device and storage medium |
| CN112463058A (en) * | 2020-11-27 | 2021-03-09 | 杭州海康威视系统技术有限公司 | Fragmented data sorting method and device and storage node |
| CN113485874A (en) * | 2021-09-07 | 2021-10-08 | 联想凌拓科技有限公司 | Data processing method and distributed storage system |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114816278A (en) * | 2022-06-30 | 2022-07-29 | 苏州浪潮智能科技有限公司 | Data migration method, system, equipment and storage medium of storage server |
| CN114816278B (en) * | 2022-06-30 | 2022-11-11 | 苏州浪潮智能科技有限公司 | Data migration method, system, equipment and storage medium of storage server |
| CN117289877A (en) * | 2023-11-23 | 2023-12-26 | 云筑信息科技(成都)有限公司 | Universal file storage dynamic proxy method based on service grid |
| CN117289877B (en) * | 2023-11-23 | 2024-03-19 | 云筑信息科技(成都)有限公司 | Universal file storage dynamic proxy method based on service grid |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103106152B (en) | Based on the data dispatching method of level storage medium | |
| CN108628753B (en) | Memory space management method and device | |
| US9218135B2 (en) | Hierarchical allocation for file system storage device | |
| US8909887B1 (en) | Selective defragmentation based on IO hot spots | |
| CN114217740A (en) | Storage management method, equipment, system and computer readable storage medium | |
| JPWO2008114441A1 (en) | Storage management program, storage management method, and storage management device | |
| CN113722275B (en) | Object storage space management method, device, server and storage medium | |
| CN103095805A (en) | Cloud storage system of data intelligent and decentralized management | |
| CN113568582B (en) | Data management method, device and storage equipment | |
| US20170123975A1 (en) | Centralized distributed systems and methods for managing operations | |
| WO2016138839A1 (en) | Bitmap-based storage space management system and method thereof | |
| CN110858162A (en) | Memory management method and device and server | |
| CN110245129B (en) | Distributed global data deduplication method and device | |
| CN113495889A (en) | Distributed object storage method and device, electronic equipment and storage medium | |
| CN109558456A (en) | A kind of file migration method, apparatus, equipment and readable storage medium storing program for executing | |
| CN111638853A (en) | Data storage method and device, storage cluster, gateway equipment and main equipment | |
| CN114594914A (en) | Control method and system for distributed storage system | |
| CN116400855A (en) | Data processing method and data storage system | |
| CN116467267A (en) | Garbage recycling method, device, storage medium and system | |
| CN115617264A (en) | Distributed storage method and device | |
| CN111399761A (en) | Storage resource allocation method, device and equipment and storage medium | |
| CN109753224B (en) | Storage structure and storage structure configuration method | |
| CN111435286A (en) | Data storage method, device and system | |
| CN106354793B (en) | Method and device for monitoring hot spot object | |
| CN115904263B (en) | Data migration method, system, equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |