CN111880735A - Data migration method, device, equipment and storage medium in storage system - Google Patents
Data migration method, device, equipment and storage medium in storage system Download PDFInfo
- Publication number
- CN111880735A CN111880735A CN202010725623.0A CN202010725623A CN111880735A CN 111880735 A CN111880735 A CN 111880735A CN 202010725623 A CN202010725623 A CN 202010725623A CN 111880735 A CN111880735 A CN 111880735A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- data unit
- unit
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data migration method in a storage system, which comprises the following steps: when an access request for target data on a target memory is monitored, determining a target data unit to which the target data belongs, wherein the target memory is one or more memories which are not used for storing thermal data in a storage system; placing the target data unit at the head of the data linked list; updating the access times of the target data unit in the record table; when a set data migration triggering condition is reached, determining a hot data unit based on the data link table and the record table; data in the thermal data unit is migrated to a memory for storing the thermal data. By applying the technical scheme provided by the application, the hot data unit can be accurately determined by integrating the information in the data linked list and the record table, so that the hot data can be accurately migrated, and the access performance of the whole storage system is improved. The application also discloses a data migration device, equipment and a storage medium in the storage system, and the device, the equipment and the storage medium have corresponding technical effects.
Description
Technical Field
The present application relates to the field of computer application technologies, and in particular, to a method, an apparatus, a device, and a storage medium for data migration in a storage system.
Background
With the rapid development of computer technology, the application of storage systems in various industries is gradually increased. Storage systems rely on storage such as the bottom disk for data storage. In the cloud storage era, the access volume of data reading and writing and the like is increasingly large, so that the storage pressure of a bottom-layer memory is increasingly large.
The performance of the memory is high or low, but because the price of the high-performance memory is high, the high-performance memory is not completely used in the memory system, but the high-performance memory and the low-performance memory are used together. Therefore, hot data can be placed on the high-performance memory, and cold data can be placed on the low-performance memory, so that the access capability of the whole storage system is improved.
Therefore, how to accurately identify the hot data for accurate migration of the hot data is a technical problem that needs to be solved by those skilled in the art.
Disclosure of Invention
The application aims to provide a data migration method, a data migration device, data migration equipment and a storage medium in a storage system, so that hot data can be accurately identified and accurately migrated.
In order to solve the technical problem, the application provides the following technical scheme:
a method for data migration in a storage system comprises the following steps:
when an access request for target data on a target memory is monitored, determining a target data unit to which the target data belongs, wherein the target memory is one or more memories which are not used for storing thermal data in a storage system;
placing the target data unit at the head of a data linked list;
updating the access times of the target data unit in the record table;
when a set data migration triggering condition is reached, determining a hot data unit based on the data linked list and the record table;
migrating data in the thermal data units to a memory for storing thermal data.
In one embodiment of the present application, the target memory is partitioned from the beginning into a plurality of fixed-size memory blocks, each memory block being a unit of data.
In a specific embodiment of the present application, the placing the target data unit at the head of the data link list includes:
traversing a data linked list, and determining whether the target data unit exists in the data linked list;
if so, moving the target data unit to the head of the data linked list;
if not, the target data unit is placed directly at the head of the data link list.
In a specific embodiment of the present application, the determining a hot data unit based on the data chain table and the record table includes:
determining the first M data units in the data chain table and the first N data units with the maximum access times in the record table, wherein M, N is a positive integer;
the determined data unit is determined to be a thermal data unit.
In a specific embodiment of the present application, after determining the first M data units in the data chain table and the first N data units with the largest number of accesses in the record table, and before determining the determined data unit as a hot data unit, the method further includes:
and carrying out duplicate removal processing on the first M data units and the first N data units.
In a specific embodiment of the present application, after determining the first M data units in the data chain table and the first N data units with the largest number of accesses in the record table, and before determining the determined data unit as a hot data unit, the method further includes:
repeatedly judging the first M data units and the first N data units;
if P data units are repeated, P non-repeated data units are determined in the data chain table and/or the record table, and P is a positive integer;
correspondingly, the determining the determined data unit as a thermal data unit includes:
and determining the determined non-repeated data units in the first M data units and the first N data units and the re-determined P non-repeated data units as thermal data units.
In a specific embodiment of the present application, after the placing the target data unit at the head of the data link list, the method further includes:
and if the number of the data units in the data linked list is greater than the total number Q of the nodes, removing the data units at the tail part of the data linked list, wherein Q is a positive integer.
A data migration apparatus in a storage system, comprising:
the device comprises a target data unit determining module, a target data unit determining module and a target data unit determining module, wherein the target data unit determining module is used for determining a target data unit to which target data belongs when an access request for the target data on a target memory is monitored, and the target memory is one or more memories which are not used for storing thermal data in a storage system;
the data linked list updating module is used for placing the target data unit at the head of the data linked list;
the record table updating module is used for updating the access times of the target data unit in the record table;
the hot data unit determining module is used for determining a hot data unit based on the data linked list and the record table when a set data migration triggering condition is reached;
and the hot data migration module is used for migrating the data in the hot data unit to a memory for storing hot data.
A data migration apparatus in a storage system, comprising:
a memory for storing a computer program;
a processor, configured to implement the steps of the data migration method in the storage system according to any one of the above embodiments when the computer program is executed.
A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a data migration method in a storage system according to any one of the above.
By applying the technical scheme provided by the embodiment of the application, when an access request for the target data on the target memory is monitored, the target data unit to which the target data belongs is determined, the target data unit is arranged at the head of the data chain table, the access times of the target data unit in the record table are updated, when the set data migration triggering condition is reached, the hot data unit is determined based on the data chain table and the record table, and the data in the hot data unit is migrated to the memory for storing the hot data. By integrating the information in the data linked list and the record list, the hot data unit can be accurately determined, so that the hot data can be accurately migrated, and the access performance of the whole storage system is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart illustrating an implementation of a data migration method in a storage system according to an embodiment of the present application;
FIG. 2 is a schematic structural diagram of a data migration apparatus in a storage system according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a data migration apparatus in a storage system according to an embodiment of the present application.
Detailed Description
In order that those skilled in the art will better understand the disclosure, the following detailed description will be given with reference to the accompanying drawings. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1, a flowchart of an implementation of a data migration method in a storage system according to an embodiment of the present application is shown, where the method may include the following steps:
s110: when an access request for target data on the target memory is monitored, a target data unit to which the target data belongs is determined.
The target memory is one or more memories in the storage system that are not used to store hot data.
The storage system may rely on storage such as multiple underlying disks for data storage. Different memories have different performance levels, and high performance memories can be used to store thermal data. The target memory may be one or more memories in the storage system that are not used to store thermal data.
In the embodiment of the present application, the target memory is divided from the beginning into a plurality of fixed-size memory blocks, each of which is a data unit.
During normal operation of the storage system, access to data in the storage system may be monitored. When an access request for target data on the target memory is monitored, a target data unit to which the target data belongs can be determined. The target data is any data on the target memory.
For example, the target memory is divided into A, B, C, D four memory blocks, each memory block being a data unit. The data unit A stores data A1, A2 and A3, and the data unit B stores data B1 and B2. When an access request for the data a1 is monitored, the data unit to which the data a1 belongs may be determined to be a.
S120: the target data element is placed at the head of the data link list.
In the embodiment of the application, a data linked list can be preset. The data linked list may specifically be an LRU (least recently used) linked list. The length of the data link table can be adjusted according to actual conditions or set as a fixed value. Each node of the data link list corresponds to a data unit.
When an access request for the target data on the target memory is monitored, the target data unit to which the target data belongs is determined, and then the target data unit can be placed at the head of the data linked list. Specifically, the identifier of the target data unit, the storage data related information, and the like may be recorded in the head node of the data chain table.
In one embodiment of the present application, the step may comprise the steps of:
traversing the data linked list, and determining whether a target data unit exists in the data linked list;
if so, moving the target data unit to the head of the data link list;
if not, the target data unit is placed directly at the head of the data link list.
For convenience of description, the above steps are combined for illustration.
After the target data unit to which the target data belongs is determined, the data link table may be traversed first to determine whether the target data unit exists in the data link table.
If the target data unit exists in the data link list, the target data unit is added into the data link list, which indicates that the access request for the data belonging to the target data unit exists before. In this case, the target data element may be moved to the head of the data link list. Of course, if the target data element is at the head of the data link list, the location of the target data element in the data link list does not change after the move operation.
If the target data unit is not present in the data link list, this indicates that there has been no previous or longer period of time to access the data belonging to the target data unit, that the target data unit has not been added to the data link list, or has been removed from the data link list. In this case, the target data element may be placed directly at the head of the data link list.
The target data unit is arranged at the head of the data linked list, and other data units in the data linked list can move towards the tail direction in sequence.
In a specific embodiment of the present application, after the target data unit is placed at the head of the data chain table, if the number of data units in the data chain table is greater than the total number Q of nodes, the data unit at the tail of the data chain table is removed, where Q is a positive integer.
The data elements at the tail of the data link list are accessed earlier and less likely to be accessed again than the data elements at the front of the data link list, so that after the target data element is placed at the head of the data link list, if the number of data elements in the data link list is greater than the total number of nodes, the data elements at the tail of the data link list may be removed. Therefore, the total number of the nodes in the data link list is ensured to be unchanged, and the related information of Q data units is recorded at most. Q can be set and adjusted according to actual conditions, such as 100.
S130: and updating the access times of the target data unit in the record table.
In the embodiment of the present application, the number of accesses of each data unit may be recorded by a record table. When an access request for the target data in the target memory is monitored, after the target data unit to which the target data belongs is determined, the access frequency of the target data unit in the record table may be updated, for example, the access frequency of the target data unit recorded in the record table is increased by 1.
S140: and when the set data migration triggering condition is reached, determining the hot data unit based on the data link table and the record table.
In this embodiment of the present application, a data migration triggering condition may be preset, and when the data migration triggering condition is reached, the hot data unit is determined based on the data link table and the record table. For example, when the set migration period is reached, it may be considered that the data migration trigger condition is reached, or when the migration instruction is received, it may be considered that the data migration trigger condition is reached.
It will be appreciated that when there is an access request for data, the data element to which the data belongs is placed at the head of the data linked list. The data units with access time closer to the current time are closer to the head of the data link list. Correspondingly, the data units with access time farther from the current time are closer to the tail of the data link list. And the data which is accessed recently is more likely to be accessed again and is more suitable as hot data.
The access times of each data unit are recorded in the record table. For any one data unit, if the number of times of access of the data unit is greater, the probability of being accessed again is greater, and the data unit is more suitable to be used as hot data.
Based on the data linked list and the record table, the data unit that has been accessed recently and the data unit with a large number of accesses can be determined, and the data units can be determined as hot data units.
S150: data in the thermal data unit is migrated to a memory for storing the thermal data.
After the hot data unit is determined, the data in the hot data unit may be migrated from the target memory to the memory for storing the hot data. The memory for storing thermal data may be a high performance memory. The hot data is stored in the high-performance memory, so that the access speed can be improved, and the access performance of the storage system is improved.
By applying the method provided by the embodiment of the application, when an access request for the target data on the target memory is monitored, the target data unit to which the target data belongs is determined, the target data unit is arranged at the head of the data chain table, the access times of the target data unit in the record table are updated, when the set data migration triggering condition is reached, the hot data unit is determined based on the data chain table and the record table, and the data in the hot data unit is migrated to the memory for storing the hot data. By integrating the information in the data linked list and the record list, the hot data unit can be accurately determined, so that the hot data can be accurately migrated, and the access performance of the whole storage system is improved.
In one embodiment of the present application, step S140 may include the following steps:
the method comprises the following steps: determining the first M data units in the data chain table and the first N data units with the maximum access times in the record table, wherein M, N is a positive integer;
step two: the determined data unit is determined to be a thermal data unit.
For convenience of description, the above two steps are combined for illustration.
When the set data migration triggering condition is reached, the first M data units in the data chain table and the first N data units with the largest access times in the record table can be respectively determined.
The data units near the head in the data link list indicate that the data which has been accessed recently is more likely to be accessed again, so the first M data units can be determined from the data link list. M is less than the total number of nodes of the data link list.
The data unit with the largest number of accesses in the record table is more likely to be accessed again, so the first N data units can be determined from the record table. N is less than the total number of data units.
M and N can be set and adjusted according to actual conditions. If M is 5, N is 5.
After the first M data units in the data link table and the first N data units with the largest number of access times in the record table are determined, the determined data units may be determined as hot data units, and it may be considered that the probability that data in the data units are accessed again is high.
In a specific embodiment of the present application, after determining the first M data units in the data chain table and the first N data units with the largest access times in the record table, and before determining the determined data units as hot data units, deduplication processing may be performed on the first M data units and the first N data units.
That is, if there is duplication between the first M data units in the data linked list and the first N data units with the largest access times in the record table, the duplicated data units may be removed, and then the other data units from which the duplicated data units are removed are determined as hot data units. Therefore, the condition of disordered data migration can be effectively avoided.
In another specific embodiment of the present application, after determining the first M data units in the data chain table and the first N data units with the largest number of accesses in the record table, and before determining the determined data units as hot data units, repeat determination may be further performed on the first M data units and the first N data units, and if there is P data unit repeats, then P non-repeated data units are determined in the data chain table and/or the record table, where P is a positive integer;
accordingly, the non-duplicate data unit of the first M data units and the first N data units and the non-duplicate data unit of the second P data units can be determined as the thermal data unit.
In the embodiment of the present application, after determining the first M data units in the data chain table and the first N data units with the largest access times in the record table, repeated determination may be performed on the first M data units and the first N data units to determine whether a conflict exists therein. If there are P data unit duplicates, P non-duplicate data units may be determined again in the data link table and/or the record table, and then the determined first M data units, the non-duplicate data units in the first N data units, and the re-determined P non-duplicate data units are determined as thermal data units.
Specifically, the P nonrepeating data units are determined again in the data link table and/or the record table, and the selection may be started from the M +1 th data unit of the data link table, and/or the selection may be started from the N +1 th data unit with the largest number of accesses to the record table. When selecting from the data linked list and the record list at the same time, if the selected data units are repeated, the selection can be continued until P non-repeated data units are determined.
And finally determining the number of the obtained thermal data units to be M + N.
Therefore, the data units with enough quantity can be migrated to the memory for storing the hot data every time the hot data migration is carried out, and the access performance of the storage system is improved.
Corresponding to the above method embodiments, the present application further provides a data migration apparatus in a storage system, and the data migration apparatus in the storage system described below and the data migration method in the storage system described above may be referred to correspondingly.
Referring to fig. 2, the apparatus may include the following modules:
the target data unit determining module 210 is configured to determine a target data unit to which target data belongs when an access request for the target data on a target memory is monitored, where the target memory is one or more memories in the storage system that are not used for storing thermal data;
a data linked list updating module 220, configured to place the target data unit at the head of the data linked list;
a record table updating module 230, configured to update the access times of the target data unit in the record table;
a hot data unit determining module 240, configured to determine a hot data unit based on the data link table and the record table when a set data migration triggering condition is reached;
and a hot data migration module 250 for migrating the data in the hot data unit to a memory for storing the hot data.
By applying the device provided by the embodiment of the application, when an access request for the target data on the target memory is monitored, the target data unit to which the target data belongs is determined, the target data unit is arranged at the head of the data chain table, the access times of the target data unit in the record table are updated, when a set data migration triggering condition is reached, the hot data unit is determined based on the data chain table and the record table, and the data in the hot data unit is migrated to the memory for storing the hot data. By integrating the information in the data linked list and the record list, the hot data unit can be accurately determined, so that the hot data can be accurately migrated, and the access performance of the whole storage system is improved.
In one embodiment of the present application, the target memory is partitioned from the beginning into a plurality of fixed-size memory blocks, each memory block being a unit of data.
In a specific embodiment of the present application, the data linked list updating module 220 is configured to:
traversing the data linked list, and determining whether a target data unit exists in the data linked list;
if so, moving the target data unit to the head of the data link list;
if not, the target data unit is placed directly at the head of the data link list.
In one embodiment of the present application, the thermal data unit determining module 240 is configured to:
determining the first M data units in the data chain table and the first N data units with the maximum access times in the record table, wherein M, N is a positive integer;
the determined data unit is determined to be a thermal data unit.
In a specific embodiment of the present application, the apparatus further includes a deduplication module, configured to:
after determining the first M data units in the data chain table and the first N data units with the maximum access times in the record table, and before determining the determined data units as the hot data units, performing deduplication processing on the first M data units and the first N data units.
In an embodiment of the present application, the apparatus further includes a data unit re-determination module, configured to:
after determining the first M data units in the data chain table and the first N data units with the maximum access times in the record table and before determining the determined data units as the hot data units, repeatedly judging the first M data units and the first N data units;
if P data units are repeated, P non-repeated data units are determined in the data chain table and/or the record table, wherein P is a positive integer;
accordingly, the thermal data unit determination module 240 is configured to:
and determining the determined data units which are not repeated in the first M data units and the first N data units and the determined P data units which are not repeated as thermal data units.
In an embodiment of the present application, the apparatus further includes a data unit removing module, configured to:
after the target data unit is placed at the head of the data link list, if the number of the data units in the data link list is greater than the total number Q of the nodes, removing the data units at the tail of the data link list, wherein Q is a positive integer.
Corresponding to the above method embodiment, an embodiment of the present application further provides a data migration apparatus in a storage system, including:
a memory for storing a computer program;
and the processor is used for realizing the steps of the data migration method in the storage system when executing the computer program.
As shown in fig. 3, which is a schematic diagram of a structure of a data migration apparatus in a storage system, the data migration apparatus in the storage system may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through a communication bus 13.
In the embodiment of the present application, the processor 10 may be a Central Processing Unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, etc.
The processor 10 may call a program stored in the memory 11, and in particular, the processor 10 may perform operations in an embodiment of a data migration method in a storage system.
The memory 11 is used for storing one or more programs, the program may include program codes, the program codes include computer operation instructions, in this embodiment, the memory 11 stores at least the program for implementing the following functions:
when an access request for target data on a target memory is monitored, determining a target data unit to which the target data belongs, wherein the target memory is one or more memories which are not used for storing thermal data in a storage system;
placing the target data unit at the head of the data linked list;
updating the access times of the target data unit in the record table;
when a set data migration triggering condition is reached, determining a hot data unit based on the data link table and the record table;
data in the thermal data unit is migrated to a memory for storing the thermal data.
In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function (such as a data access function and an information recording function), and the like; the storage data area may store data created during use, such as access data, recording data, and the like.
Further, the memory 11 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device or other volatile solid state storage device.
The communication interface 13 may be an interface of a communication module for connecting with other devices or systems.
Of course, it should be noted that the structure shown in fig. 3 does not constitute a limitation on the data migration apparatus in the storage system in the embodiment of the present application, and in practical applications, the data migration apparatus in the storage system may include more or less components than those shown in fig. 3, or may combine some components.
Corresponding to the above method embodiments, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data migration method in the storage system are implemented.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The principle and the implementation of the present application are explained in the present application by using specific examples, and the above description of the embodiments is only used to help understanding the technical solution and the core idea of the present application. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
Claims (10)
1. A method for data migration in a storage system, comprising:
when an access request for target data on a target memory is monitored, determining a target data unit to which the target data belongs, wherein the target memory is one or more memories which are not used for storing thermal data in a storage system;
placing the target data unit at the head of a data linked list;
updating the access times of the target data unit in the record table;
when a set data migration triggering condition is reached, determining a hot data unit based on the data linked list and the record table;
migrating data in the thermal data units to a memory for storing thermal data.
2. The method of claim 1, wherein the target memory is partitioned from scratch into a plurality of fixed-size memory blocks, each memory block being a unit of data.
3. The method of claim 1, wherein placing the target data element at a head of a data linked list comprises:
traversing a data linked list, and determining whether the target data unit exists in the data linked list;
if so, moving the target data unit to the head of the data linked list;
if not, the target data unit is placed directly at the head of the data link list.
4. The method of claim 1, wherein determining the hot data unit based on the data link table and the record table comprises:
determining the first M data units in the data chain table and the first N data units with the maximum access times in the record table, wherein M, N is a positive integer;
the determined data unit is determined to be a thermal data unit.
5. The method of claim 4, wherein after determining the first M data elements in the data chain table and the first N data elements in the record table having the highest number of accesses and before determining the determined data element as a hot data element, further comprising:
and carrying out duplicate removal processing on the first M data units and the first N data units.
6. The method of claim 4, wherein after determining the first M data elements in the data chain table and the first N data elements in the record table having the highest number of accesses and before determining the determined data element as a hot data element, further comprising:
repeatedly judging the first M data units and the first N data units;
if P data units are repeated, P non-repeated data units are determined in the data chain table and/or the record table, and P is a positive integer;
correspondingly, the determining the determined data unit as a thermal data unit includes:
and determining the determined non-repeated data units in the first M data units and the first N data units and the re-determined P non-repeated data units as thermal data units.
7. The method of any of claims 1 to 6, further comprising, after the placing the target data element at a head of a data linked list:
and if the number of the data units in the data linked list is greater than the total number Q of the nodes, removing the data units at the tail part of the data linked list, wherein Q is a positive integer.
8. An apparatus for data migration in a storage system, comprising:
the device comprises a target data unit determining module, a target data unit determining module and a target data unit determining module, wherein the target data unit determining module is used for determining a target data unit to which target data belongs when an access request for the target data on a target memory is monitored, and the target memory is one or more memories which are not used for storing thermal data in a storage system;
the data linked list updating module is used for placing the target data unit at the head of the data linked list;
the record table updating module is used for updating the access times of the target data unit in the record table;
the hot data unit determining module is used for determining a hot data unit based on the data linked list and the record table when a set data migration triggering condition is reached;
and the hot data migration module is used for migrating the data in the hot data unit to a memory for storing hot data.
9. A data migration apparatus in a storage system, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data migration method in the storage system according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of a data migration method in a storage system according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010725623.0A CN111880735B (en) | 2020-07-24 | 2020-07-24 | Data migration method, device, equipment and storage medium in storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010725623.0A CN111880735B (en) | 2020-07-24 | 2020-07-24 | Data migration method, device, equipment and storage medium in storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111880735A true CN111880735A (en) | 2020-11-03 |
| CN111880735B CN111880735B (en) | 2023-07-14 |
Family
ID=73201543
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010725623.0A Active CN111880735B (en) | 2020-07-24 | 2020-07-24 | Data migration method, device, equipment and storage medium in storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111880735B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114138183A (en) * | 2021-10-29 | 2022-03-04 | 郑州云海信息技术有限公司 | Data layering method, device, equipment and storage medium of storage system |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103514106A (en) * | 2012-06-20 | 2014-01-15 | 北京神州泰岳软件股份有限公司 | Method for caching data |
| US20150095671A1 (en) * | 2013-09-27 | 2015-04-02 | International Business Machines Corporation | Method and apparatus for managing power of a storage system |
| WO2015081757A1 (en) * | 2013-12-05 | 2015-06-11 | 华为技术有限公司 | Cold and hot data identification threshold calculation method, apparatus and system |
| CN105205014A (en) * | 2015-09-28 | 2015-12-30 | 北京百度网讯科技有限公司 | Data storage method and device |
| US20160085585A1 (en) * | 2013-05-31 | 2016-03-24 | Huawei Technologies Co., Ltd. | Memory System, Method for Processing Memory Access Request and Computer System |
| CN105447059A (en) * | 2014-09-29 | 2016-03-30 | 华为技术有限公司 | Data processing method and device |
| CN107180118A (en) * | 2017-07-04 | 2017-09-19 | 郑州云海信息技术有限公司 | A kind of file system cache data managing method and device |
| CN107688443A (en) * | 2017-09-18 | 2018-02-13 | 郑州云海信息技术有限公司 | A kind of method of data storage, system and relevant apparatus |
| CN108519862A (en) * | 2018-03-30 | 2018-09-11 | 百度在线网络技术（北京）有限公司 | Storage method, device, system and the storage medium of block catenary system |
| CN108829341A (en) * | 2018-05-03 | 2018-11-16 | 上海科技大学 | A kind of data managing method based on mixing storage system |
| CN109213699A (en) * | 2018-09-21 | 2019-01-15 | 郑州云海信息技术有限公司 | A kind of metadata management method, system, equipment and computer readable storage medium |
| US20200057729A1 (en) * | 2017-04-27 | 2020-02-20 | Huawei Technologies Co., Ltd. | Memory access method and computer system |
| CN111159066A (en) * | 2020-01-07 | 2020-05-15 | 杭州电子科技大学 | Dynamically-adjusted cache data management and elimination method |
-
2020
- 2020-07-24 CN CN202010725623.0A patent/CN111880735B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103514106A (en) * | 2012-06-20 | 2014-01-15 | 北京神州泰岳软件股份有限公司 | Method for caching data |
| US20160085585A1 (en) * | 2013-05-31 | 2016-03-24 | Huawei Technologies Co., Ltd. | Memory System, Method for Processing Memory Access Request and Computer System |
| US20150095671A1 (en) * | 2013-09-27 | 2015-04-02 | International Business Machines Corporation | Method and apparatus for managing power of a storage system |
| WO2015081757A1 (en) * | 2013-12-05 | 2015-06-11 | 华为技术有限公司 | Cold and hot data identification threshold calculation method, apparatus and system |
| CN105447059A (en) * | 2014-09-29 | 2016-03-30 | 华为技术有限公司 | Data processing method and device |
| CN105205014A (en) * | 2015-09-28 | 2015-12-30 | 北京百度网讯科技有限公司 | Data storage method and device |
| US20200057729A1 (en) * | 2017-04-27 | 2020-02-20 | Huawei Technologies Co., Ltd. | Memory access method and computer system |
| CN107180118A (en) * | 2017-07-04 | 2017-09-19 | 郑州云海信息技术有限公司 | A kind of file system cache data managing method and device |
| CN107688443A (en) * | 2017-09-18 | 2018-02-13 | 郑州云海信息技术有限公司 | A kind of method of data storage, system and relevant apparatus |
| CN108519862A (en) * | 2018-03-30 | 2018-09-11 | 百度在线网络技术（北京）有限公司 | Storage method, device, system and the storage medium of block catenary system |
| CN108829341A (en) * | 2018-05-03 | 2018-11-16 | 上海科技大学 | A kind of data managing method based on mixing storage system |
| CN109213699A (en) * | 2018-09-21 | 2019-01-15 | 郑州云海信息技术有限公司 | A kind of metadata management method, system, equipment and computer readable storage medium |
| CN111159066A (en) * | 2020-01-07 | 2020-05-15 | 杭州电子科技大学 | Dynamically-adjusted cache data management and elimination method |
Non-Patent Citations (6)
| Title |
|---|
| LING ZHAN等: "Design and implementation of SSD aware Heterogeneous cache Algorithm: A Two-level caching algorithm for RAID storage systems", 2016 IEEE INTERNATIONAL CONFERENCE ON CLOUD COMPUTING AND BIG DATA ANALYSIS (ICCCBDA) * |
| 吕帅;刘光明;徐凯;刘欣;: "海量信息分级存储数据迁移策略研究", 计算机工程与科学, no. 1 * |
| 柴云鹏;杨楠;: "冷数据集中的流媒体存储系统节能方法", 计算机科学, no. 10 * |
| 谢长生等: "基于分条单元热度的RAID数据分布优化", 《计算机科学》 * |
| 谢长生等: "基于分条单元热度的RAID数据分布优化", 《计算机科学》, no. 03, 25 March 2006 (2006-03-25), pages 279 - 282 * |
| 钟诚;岳丽华;韦鹏;: "一种多路Flash存储结构的热数据判断方法", 计算机工程, no. 12 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114138183A (en) * | 2021-10-29 | 2022-03-04 | 郑州云海信息技术有限公司 | Data layering method, device, equipment and storage medium of storage system |
| CN114138183B (en) * | 2021-10-29 | 2023-09-12 | 郑州云海信息技术有限公司 | Data layering method, device and equipment of storage system and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111880735B (en) | 2023-07-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11853549B2 (en) | Index storage in shingled magnetic recording (SMR) storage system with non-shingled region | |
| CN108647151B (en) | Full flash system metadata disk-dropping method, device, equipment and storage medium | |
| CN111737265B (en) | Block data access method, block data storage method and device | |
| CN107329704B (en) | Cache mirroring method and controller | |
| CN110147203B (en) | File management method and device, electronic equipment and storage medium | |
| CN107402870B (en) | Method and device for processing log segment in metadata server | |
| CN110618789B (en) | Method and device for deleting repeated data | |
| CN111868705A (en) | Recording device, reading device, recording method, recording program, reading method, reading program, and magnetic tape | |
| CN112182010B (en) | Dirty page refreshing method and device, storage medium and electronic equipment | |
| US9063667B2 (en) | Dynamic memory relocation | |
| CN107798063B (en) | Snapshot processing method and snapshot processing device | |
| CN110504002B (en) | Hard disk data consistency test method and device | |
| CN107766067B (en) | Method for improving compressed mirror image burning speed, storage medium and computer | |
| CN114721594A (en) | Distributed storage method, device, equipment and machine readable storage medium | |
| CN111880735A (en) | Data migration method, device, equipment and storage medium in storage system | |
| CN109144403B (en) | Method and equipment for switching cloud disk modes | |
| CN110658994B (en) | Data processing method and device based on HDD (hard disk drive) and SSD (solid State disk) hybrid disk array | |
| CN111880745A (en) | Data processing method, device and equipment based on solid-state disk array and storage medium | |
| CN104408126A (en) | Persistent writing method, device and system of database | |
| CN106708445A (en) | Link selection method and device | |
| CN115373609A (en) | Task processing method and related equipment | |
| CN115328851A (en) | Data protection method, device, equipment and medium | |
| CN110795034B (en) | Data migration method, device and equipment of storage system and readable storage medium | |
| CN105573862B (en) | Method and equipment for recovering file system | |
| CN112015672A (en) | Data processing method, device, equipment and storage medium in storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |