CN115079958A - Multi-node load balancing cold and hot data migration device, method, terminal and medium - Google Patents
Multi-node load balancing cold and hot data migration device, method, terminal and medium Download PDFInfo
- Publication number
- CN115079958A CN115079958A CN202210864889.2A CN202210864889A CN115079958A CN 115079958 A CN115079958 A CN 115079958A CN 202210864889 A CN202210864889 A CN 202210864889A CN 115079958 A CN115079958 A CN 115079958A
- Authority
- CN
- China
- Prior art keywords
- data
- node
- migration
- migrated
- data block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005012 migration Effects 0.000 title claims abstract description 130
- 238000013508 migration Methods 0.000 title claims abstract description 121
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000013480 data collection Methods 0.000 claims abstract description 12
- 230000007246 mechanism Effects 0.000 claims description 7
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 230000004044 response Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of data migration, and particularly discloses a multi-node load balancing cold and hot data migration device, a method, a terminal and a medium, wherein a data collection module collects IO (input/output) request access information of each data block in each node; the data value calculation module calculates the data value of the data block according to the IO request access information; the cold and hot data identification module divides a data value interval, and determines a node to be migrated and a node to be migrated corresponding to each data value interval according to the distribution condition of data blocks on each node in the data value interval; and the migration module migrates the data block on the node to be migrated to the node to be migrated. The method determines the data value of the data block, divides the data value interval, appropriately migrates the storage file according to the distribution interval of the data value, migrates some data to the storage nodes with smaller load pressure by searching the storage nodes with smaller load pressure, and realizes the load balance of the system, thereby improving the stability of the system.
Description
Technical Field
The invention relates to the field of data migration, in particular to a multi-node load balancing cold and hot data migration device, method, terminal and medium.
Background
In the big data era, the storage and transmission rate of data plays an important role in the normal and fast operation of a storage server, and for a multi-node storage system, the resource usage of the storage node of the data can seriously affect the performance of the storage server. The nodes storing the data are often heavily loaded by the burst access and concurrent access of the users, so that it is difficult for the heavily loaded nodes to bear the burst access pressure, thereby causing serious access delay and network bandwidth problems to the whole system.
Disclosure of Invention
In order to solve the above problems, the present invention provides a multi-node load balancing cold and hot data migration apparatus, method, terminal, and medium, which determine data value for a data block, partition a data value interval, perform appropriate migration on a storage file according to a distribution interval of the data value, and migrate some data to storage nodes with smaller load pressure by searching for the storage nodes with smaller load pressure, thereby implementing load balancing of a system, and thus improving stability of the system.
In a first aspect, an aspect of the present invention provides a multi-node load balancing cold and hot data migration apparatus, including,
a data collection module: collecting IO request access information of each data block in each node;
a data value calculation module: calculating the data value of the data block according to the IO request access information;
cold and hot data identification module: dividing a data value interval, and determining a node to be migrated and a node to be migrated corresponding to each data value interval according to the distribution condition of the data blocks on each node in the data value interval;
a migration module: and migrating the data block on the node to be migrated to the node to be migrated.
Further, the IO access request information of the data block collected by the data collection module includes a total access frequency, a random read-write access frequency, a sequential read-write access frequency, a last access time of the file, a last access type, and a last access destination address;
the data value calculation module calculates the data value of the data block according to the IO request access information, and specifically comprises the following steps:
and distributing weights for each access information factor, and performing weighted summation to obtain the data value of the data block.
Further, the cold and hot data identification module is specifically configured to:
dividing N data value intervals;
calculating the data block percentage of each data value interval aiming at each node;
for each data value interval, comparing the node with the highest data block percentage with the node with the lowest data block percentage, if the difference between the two data block percentages exceeds a threshold percentage, marking the node with the highest data block percentage as a node to be migrated, and marking the node with the lowest data block percentage as a node to be migrated, otherwise, not migrating; comparing the nodes with the second highest percentage of the data blocks with the nodes with the second lowest percentage of the data blocks, if the percentage difference of the two data blocks exceeds the threshold percentage, marking the nodes with the second highest percentage of the data blocks as nodes to be migrated, and marking the nodes with the second lowest percentage of the data blocks as nodes to be migrated, otherwise, not migrating; and so on.
Further, the device also comprises a control device,
a metadata module: recording the file saving physical position of each data block, wherein the file saving physical position comprises a storage node, a storage device type and a specific storage position in a storage device;
a migration plan generation module: setting a data migration rule which comprises a migration mode, a migration period, migration time and a wear balance mechanism;
the migration module is specifically configured to:
migrating the data blocks based on the data migration rule set by the generation migration calculation module according to the nodes to be migrated and the nodes to be migrated which are determined by the cold and hot data identification module; and after the migration is completed, the metadata module is notified to update the metadata information.
In a second aspect, a technical solution of the present invention provides a multi-node load balancing cold and hot data migration method, including the following steps:
s1, collecting IO request access information of each data block in each node;
s2, calculating the data value of the data block according to the IO request access information;
s3, dividing data value intervals, and determining a node to be migrated and a node to be migrated corresponding to each data value interval according to the distribution condition of the data blocks on each node in the data value intervals;
and S4, migrating the data block on the node to be migrated to the node to be migrated.
Further, the IO access request information of the data block collected in step S1 includes a total access frequency, a random read-write access frequency, a sequential read-write access frequency, a last access time of the file, a last access type, and a last access destination address;
step S2 specifically includes:
and distributing weights for each access information factor, and performing weighted summation to obtain the data value of the data block.
Further, step S3 specifically includes:
dividing N data value intervals;
calculating the data block percentage of each data value interval aiming at each node;
for each data value interval, comparing the node with the highest data block percentage with the node with the lowest data block percentage, if the difference between the two data block percentages exceeds a threshold percentage, marking the node with the highest data block percentage as a node to be migrated, and marking the node with the lowest data block percentage as a node to be migrated, otherwise, not migrating; comparing the nodes with the second highest data block percentage with the nodes with the second lowest data block percentage, if the percentage difference of the two data blocks exceeds the threshold percentage, marking the nodes with the second highest data block percentage as nodes to be migrated, and marking the nodes with the second lowest data block percentage as nodes to be migrated, otherwise, not migrating; and so on.
Further, the method may further comprise,
s0, recording the file saving physical position of each data block, including storage node, storage device type and concrete storage position in the storage device; setting a data migration rule which comprises a migration mode, a migration period, migration time and a wear balance mechanism;
step S4 specifically includes:
migrating the data block based on the set data migration rule according to the determined nodes to be migrated and the determined nodes to be migrated; and after the migration is completed, the metadata module is notified to update the metadata information.
In a third aspect, a technical solution of the present invention provides a terminal, including:
the memory is used for storing a multi-node load balancing cold and hot data migration program;
and the processor is used for implementing the steps of the multi-node load balancing cold and hot data migration method when the multi-node load balancing cold and hot data migration program is executed.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where a multi-node load balancing cold and hot data migration program is stored on the computer-readable storage medium, and when executed by a processor, the multi-node load balancing cold and hot data migration program implements the steps of the multi-node load balancing cold and hot data migration method according to any one of the above-mentioned embodiments.
Compared with the prior art, the multi-node load balancing cold and hot data migration device, method, terminal and medium provided by the invention have the following beneficial effects: determining a data value of a data block, dividing a data value interval, properly migrating a storage file according to a distribution interval of the data value, and migrating some data to the storage nodes with lower load pressure by searching the storage nodes with lower load pressure, so that the access pressure among the nodes can be balanced to a greater extent, the delay problem caused by uneven load of node resources is reduced, and the visit time of a user is saved; meanwhile, due to the balance of resource usage among the nodes, the problems of delay and slow response caused by sudden access or concurrent access of a single node in a short time are solved, the hot data are separated to each working node, the bandwidth among the nodes is balanced, the IOPS and the bandwidth of the storage system are improved, and finally the user experience and the IOPS effect are improved.
Drawings
In order to clearly illustrate the embodiments or technical solutions of the present application, the drawings used in the embodiments or technical solutions of the present application will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic block diagram of a multi-node load balancing cold and hot data migration apparatus according to an embodiment of the present invention.
Fig. 2 is a schematic block diagram of a multi-node load balancing cold and hot data migration apparatus according to a second embodiment of the present invention.
Fig. 3 is a schematic flow chart of a multi-node load balancing cold and hot data migration method according to a third embodiment of the present invention.
Fig. 4 is a flowchart illustrating a multi-node load balancing cold and hot data migration method according to a fourth embodiment of the present invention.
Fig. 5 is a schematic structural diagram of a terminal according to a fifth embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the disclosure, the following detailed description will be given with reference to the accompanying drawings. It should be apparent that the described embodiments are only a few embodiments of the present application, and not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.
Example one
Fig. 1 is a schematic block diagram of a multi-node load balancing cold and hot data migration apparatus according to an embodiment of the present invention, including a data collection module, a data value calculation module, a cold and hot data identification module, and a migration module.
A data collection module: and collecting IO request access information of each data block in each node.
It should be noted that before a user issues an IO and downloads the IO into a disk, the data in the disk is divided or aggregated into data blocks with the same granularity, and the data collection module collects IO request access information of the data blocks to provide information for subsequently calculating the data value of the data blocks.
A data value calculation module: and calculating the data value of the data block according to the IO request access information.
The data value module calculates the data value of the data block according to IO request access information of the data block, namely, the cold and heat value of the data block is evaluated.
Cold and hot data identification module: and dividing a data value interval, and determining a node to be migrated and a node to be migrated corresponding to each data value interval according to the distribution condition of the data blocks on each node in the data value interval.
Dividing a data value interval according to actual requirements, for example, dividing the data value interval into 10 divisions, then distributing each data block in a corresponding data value interval, obtaining the distribution condition of the data block in the data value interval for each node, then determining a node to be migrated and a node to be migrated for each data value interval, and enabling the data in each data value interval to be distributed evenly as much as possible in each node.
A migration module: and migrating the data block on the node to be migrated to the node to be migrated.
And the migration module is used for executing the migration of the data block and executing the migration according to the determined nodes to be migrated and the nodes to be migrated.
According to the multi-node load balancing cold and hot data migration device provided by the embodiment of the invention, the data value of a data block is determined, a data value interval is divided, a storage file is properly migrated according to the distribution interval of the data value, some data are migrated to the storage nodes with lower load pressure by searching the storage nodes with lower load pressure, the access pressure among the nodes can be balanced to a greater degree, the delay problem caused by uneven load of node resources is reduced, and the visit time of a user is saved; meanwhile, due to the balance of resource usage among the nodes, the problems of delay and slow response caused by sudden access or concurrent access of a single node in a short time are solved, the hot data are separated to each working node, the bandwidth among the nodes is balanced, the IOPS and the bandwidth of the storage system are improved, and finally the user experience and the IOPS effect are improved.
On the basis of the above embodiment, as a preferred implementation manner, the data block IO access request information collected by the data collection module includes total access frequency, random read-write access frequency, sequential read-write access frequency, last access time of a file, last access type, and last end address of last access
It should be noted that the user may also set other access information as needed, and specifically select which accesses do not affect the implementation of the present embodiment.
Correspondingly, the data value calculation module calculates the data value of the data block according to the IO request access information, specifically: and distributing weights for each access information factor, and performing weighted summation to obtain the data value of the data block.
It should be noted that, weights may be assigned to the new factors according to user experience, and finally, the data value of the file is calculated by performing weighted summation.
On the basis of the above embodiment, as a preferred implementation manner, the hot and cold data identification module determines the to-be-migrated node and the to-be-migrated node of each data value interval through the following steps.
Dividing N data value intervals;
step two, calculating the data block percentage of each data value interval aiming at each node;
comparing the node with the highest data block percentage with the node with the lowest data block percentage for each data value interval, if the difference of the two data block percentages exceeds a threshold percentage, marking the node with the highest data block percentage as a node to be migrated, and marking the node with the lowest data block percentage as a node to be migrated, otherwise, not migrating; comparing the nodes with the second highest percentage of the data blocks with the nodes with the second lowest percentage of the data blocks, if the percentage difference of the two data blocks exceeds the threshold percentage, marking the nodes with the second highest percentage of the data blocks as nodes to be migrated, and marking the nodes with the second lowest percentage of the data blocks as nodes to be migrated, otherwise, not migrating; and so on.
It can be understood that N data value intervals are divided, each data value interval needs to be judged whether migration is needed, and if migration is needed, the node to be migrated and the node to be migrated are determined. For example, 10 data value intervals are divided, 10 nodes are total, for a tenth data value interval (an interval with a data value of 9-10), the data block distribution percentage of the 1 st node in the data value interval is the highest and is 10%, the data block distribution percentage of the 4 th node in the data value interval is the lowest and is 1%, if the difference between the two is more than 5%, the data of the 1 st node in the data value interval needs to be migrated to the 4 th node, the 1 st node in the data value interval is marked as a node to be migrated, and the 4 th node is marked as a node to be migrated.
Example two
Fig. 2 is a schematic block diagram of a multi-node load balancing cold and hot data migration apparatus according to a second embodiment of the present invention, including a metadata module, a migration plan generation module, a data collection module, a data value calculation module, a cold and hot data identification module, and a migration module.
A metadata module: the file-saving physical location of each data block is recorded, including storage node, storage device type, and specific storage location in the storage device.
The metadata module records the physical location of a data block where a file is stored, determines on which node the file is stored, in which storage device (HDD/SSD), and the specific storage location in the storage device. When a new I/O request comes, the data information of the storage block on the disk is updated or searched by using the I/O request and the file access information in the mapping table.
A migration plan generation module: setting data migration rules, including a migration mode, a migration period, migration time and a wear leveling mechanism.
Setting a migration requirement of a user, automatically or manually migrating, setting a migration period and migration time of a system if the migration is automatic, generally setting 0 point of each day, setting related migration requirements, such as requirements for realizing load migration balance on nodes, and simultaneously adding a wear balance mechanism, selecting a data block with the least recent migration frequency in the same heat interval as much as possible when selecting the migration block, sending the final metadata information of the migration data block to a data migration module, and notifying the metadata module to update the physical address information of the block after the migration is completed.
A data collection module: and collecting IO request access information of each data block in each node.
It should be noted that before a user issues an IO and downloads the IO into a disk, the data in the disk is divided or aggregated into data blocks with the same granularity, and the data collection module collects IO request access information of the data blocks to provide information for subsequently calculating the data value of the data blocks.
A data value calculation module: and calculating the data value of the data block according to the IO request access information.
The data value module calculates the data value of the data block according to the IO request access information of the data block, namely, the cold and heat value of the data block is evaluated.
Cold and hot data identification module: and dividing a data value interval, and determining a node to be migrated and a node to be migrated corresponding to each data value interval according to the distribution condition of the data blocks on each node in the data value interval.
Dividing a data value interval according to actual requirements, for example, dividing the data value interval into 10 divisions, then distributing each data block in a corresponding data value interval, obtaining the distribution condition of the data block in the data value interval for each node, then determining a node to be migrated and a node to be migrated for each data value interval, and enabling the data in each data value interval to be distributed evenly as much as possible in each node.
A migration module: and migrating the data block on the node to be migrated to the node to be migrated.
And the migration module is used for executing the migration of the data block and executing the migration according to the determined nodes to be migrated and the nodes to be migrated. Meanwhile, the migration module migrates the data block based on the data migration rule set by the generation migration calculation module; and after the migration is completed, the metadata module is notified to update the metadata information.
The multi-node load balancing cold and hot data migration device provided by the embodiment of the invention determines the data value of the data block, divides the data value interval, appropriately migrates the storage file according to the distribution interval of the data value, migrates some data to the storage nodes with smaller load pressure by searching the storage nodes with smaller load pressure, can balance the access pressure among the nodes to a greater extent, reduces the delay problem caused by uneven load of node resources, and saves the visit time of a user; meanwhile, due to the balance of resource usage among the nodes, the problems of delay and slow response caused by sudden access or concurrent access of a single node in a short time are solved, the hot data are separated to each working node, the bandwidth among the nodes is balanced, the IOPS and the bandwidth of the storage system are improved, and finally the user experience and the IOPS effect are improved.
EXAMPLE III
In the first embodiment, a detailed description is given to an embodiment of a multi-node load balancing cold and hot data migration apparatus, and based on the multi-node load balancing cold and hot data migration apparatus described in the first embodiment, an embodiment of the present invention further provides a multi-node load balancing cold and hot data migration method corresponding to the apparatus.
Fig. 3 is a flowchart illustrating a multi-node load balancing cold and hot data migration method according to a third embodiment of the present invention, including the following steps.
And S1, collecting the IO request access information of each data block in each node.
The collected data block IO access request information comprises total access frequency, random read-write access frequency, sequential read-write access frequency, last access time of a file, last access type and last access destination address.
And S2, calculating the data value of the data block according to the IO request access information.
And distributing weights for each access information factor, and performing weighted summation to obtain the data value of the data block.
S3, dividing the data value interval, and determining the nodes to be migrated and the nodes to be migrated corresponding to each data value interval according to the distribution of the data blocks on each node in the data value interval, which specifically comprises:
s31, dividing N data value intervals;
s32, calculating the data block percentage of each data value interval aiming at each node;
s33, for each data value interval, comparing the node with the highest data block percentage with the node with the lowest data block percentage, if the difference between the two data block percentages exceeds a threshold percentage, marking the node with the highest data block percentage as a node to be migrated, and marking the node with the lowest data block percentage as a node to be migrated, otherwise, not migrating; comparing the nodes with the second highest data block percentage with the nodes with the second lowest data block percentage, if the percentage difference of the two data blocks exceeds the threshold percentage, marking the nodes with the second highest data block percentage as nodes to be migrated, and marking the nodes with the second lowest data block percentage as nodes to be migrated, otherwise, not migrating; and so on.
And S4, migrating the data block on the node to be migrated to the node to be migrated.
The multi-node load balancing cold and hot data migration method of the present embodiment is implemented based on the aforementioned multi-node load balancing cold and hot data migration apparatus, and therefore, a detailed implementation of the method can be seen in the foregoing section of the embodiment of the multi-node load balancing cold and hot data migration apparatus, and therefore, reference may be made to the description of the corresponding embodiments of the respective sections for the detailed implementation, and a description thereof will not be provided herein.
In addition, since the multi-node load balancing cold and hot data migration method of the present embodiment is implemented based on the aforementioned multi-node load balancing cold and hot data migration apparatus, the functions thereof correspond to the functions of the above-mentioned apparatus, and are not described herein again.
Example four
In the foregoing embodiment, a detailed description is given to an embodiment of a multi-node load balancing cold and hot data migration apparatus, and based on the multi-node load balancing cold and hot data migration apparatus described in the foregoing embodiment, an embodiment of the present invention further provides a multi-node load balancing cold and hot data migration method corresponding to the apparatus.
Fig. 4 is a schematic flowchart of a multi-node load balancing cold and hot data migration method according to a fourth embodiment of the present invention, including the following steps.
S0, recording the file saving physical position of each data block, including storage node, storage device type and concrete storage position in the storage device; setting data migration rules, including a migration mode, a migration period, migration time and a wear leveling mechanism.
And S1, collecting the IO request access information of each data block in each node.
The collected data block IO access request information comprises total access frequency, random read-write access frequency, sequential read-write access frequency, last access time of a file, last access type and last access destination address.
And S2, calculating the data value of the data block according to the IO request access information.
And distributing weights for each access information factor, and performing weighted summation to obtain the data value of the data block.
S3, dividing the data value interval, and determining the to-be-migrated node and the to-be-migrated node corresponding to each data value interval according to the distribution of the data block in the data value interval on each node, which specifically includes:
s31, dividing N data value intervals;
s32, calculating the data block percentage of each data value interval aiming at each node;
s33, for each data value interval, comparing the node with the highest data block percentage with the node with the lowest data block percentage, if the difference between the two data block percentages exceeds a threshold percentage, marking the node with the highest data block percentage as a node to be migrated, and marking the node with the lowest data block percentage as a node to be migrated, otherwise, not migrating; comparing the nodes with the second highest data block percentage with the nodes with the second lowest data block percentage, if the percentage difference of the two data blocks exceeds the threshold percentage, marking the nodes with the second highest data block percentage as nodes to be migrated, and marking the nodes with the second lowest data block percentage as nodes to be migrated, otherwise, not migrating; and so on.
S4, migrating the data block on the node to be migrated to the node to be migrated, which specifically includes:
migrating the data block based on the set data migration rule according to the determined nodes to be migrated and the determined nodes to be migrated; and after the migration is completed, the metadata module is notified to update the metadata information.
The multi-node load balancing cold and hot data migration method of the present embodiment is implemented based on the aforementioned multi-node load balancing cold and hot data migration apparatus, and therefore, a detailed implementation of the method can be seen in the foregoing section of the embodiment of the multi-node load balancing cold and hot data migration apparatus, and therefore, reference may be made to the description of the corresponding embodiments of the respective sections for the detailed implementation, and a description thereof will not be provided herein.
In addition, since the multi-node load balancing cold and hot data migration method of the present embodiment is implemented based on the multi-node load balancing cold and hot data migration apparatus, the role thereof corresponds to the role of the apparatus described above, and details thereof are not repeated here.
EXAMPLE five
Fig. 5 is a schematic structural diagram of a terminal device 500 according to an embodiment of the present invention, including: a processor 510, a memory 520, and a communication unit 530. The processor 510 is configured to implement a multi-node load balancing cold and hot data migration program stored in the memory 520, and implement the following steps:
s1, collecting IO request access information of each data block in each node;
s2, calculating the data value of the data block according to the IO request access information;
s3, dividing data value intervals, and determining a node to be migrated and a node to be migrated corresponding to each data value interval according to the distribution condition of the data blocks on each node in the data value intervals;
and S4, migrating the data block on the node to be migrated to the node to be migrated.
The method determines the data value of the data block, divides the data value interval, appropriately migrates the storage file according to the distribution interval of the data value, migrates some data to the storage nodes with smaller load pressure by searching the storage nodes with smaller load pressure, can balance the access pressure among the nodes to a greater extent, reduces the delay problem caused by uneven load of node resources, and saves the visit time of a user; meanwhile, due to the balance of resource usage among the nodes, the problems of delay and slow response caused by sudden access or concurrent access of a single node in a short time are solved, the hot data are separated to each working node, the bandwidth among the nodes is balanced, the IOPS and the bandwidth of the storage system are improved, and finally the user experience and the IOPS effect are improved.
The terminal apparatus 500 includes a processor 510, a memory 520, and a communication unit 530. The components communicate via one or more buses, and those skilled in the art will appreciate that the architecture of the server shown in the figures is not intended to be limiting, and that it may be a bus architecture, a star architecture, a combination of more or fewer components than shown, or a different arrangement of components.
The memory 520 may be used for storing instructions executed by the processor 510, and the memory 520 may be implemented by any type of volatile or non-volatile storage terminal or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The executable instructions in memory 520, when executed by processor 510, enable terminal 500 to perform some or all of the steps in the method embodiments described below.
The processor 510 is a control center of the storage terminal, connects various parts of the entire electronic terminal using various interfaces and lines, and performs various functions of the electronic terminal and/or processes data by operating or executing software programs and/or modules stored in the memory 520 and calling data stored in the memory. The processor may be composed of an Integrated Circuit (IC), for example, a single packaged IC, or a plurality of packaged ICs connected with the same or different functions. For example, processor 510 may include only a Central Processing Unit (CPU). In the embodiment of the present invention, the CPU may be a single operation core, or may include multiple operation cores.
A communication unit 530, configured to establish a communication channel so that the storage terminal can communicate with other terminals. And receiving user data sent by other terminals or sending the user data to other terminals.
EXAMPLE six
The present invention also provides a computer storage medium, wherein the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a Random Access Memory (RAM).
The computer storage medium stores a multi-node load balancing cold and hot data migration program, which when executed by a processor implements the steps of:
s1, collecting IO request access information of each data block in each node;
s2, calculating the data value of the data block according to the IO request access information;
s3, dividing data value intervals, and determining a node to be migrated and a node to be migrated corresponding to each data value interval according to the distribution condition of the data blocks on each node in the data value intervals;
and S4, migrating the data block on the node to be migrated to the node to be migrated.
The method determines the data value of the data block, divides the data value interval, appropriately migrates the storage file according to the distribution interval of the data value, migrates some data to the storage nodes with smaller load pressure by searching the storage nodes with smaller load pressure, can balance the access pressure among the nodes to a greater extent, reduces the delay problem caused by uneven load of node resources, and saves the visit time of a user; meanwhile, due to the balance of resource usage among the nodes, the problems of delay and slow response caused by sudden access or concurrent access of a single node in a short time are solved, the hot data are separated to each working node, the bandwidth among the nodes is balanced, the IOPS and the bandwidth of the storage system are improved, and finally the user experience and the IOPS effect are improved.
Those skilled in the art will readily appreciate that the techniques of the embodiments of the present invention may be implemented as software plus a required general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied in the form of a software product, where the computer software product is stored in a storage medium, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like, and the storage medium can store program codes, and includes instructions for enabling a computer terminal (which may be a personal computer, a server, or a second terminal, a network terminal, and the like) to perform all or part of the steps of the method in the embodiments of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The above disclosure is only for the preferred embodiments of the present invention, but the present invention is not limited thereto, and any non-inventive changes that can be made by those skilled in the art and several modifications and amendments made without departing from the principle of the present invention shall fall within the protection scope of the present invention.
Claims (10)
1. A multi-node load balancing cold and hot data migration device is characterized by comprising,
a data collection module: collecting IO request access information of each data block in each node;
a data value calculation module: calculating the data value of the data block according to the IO request access information;
cold and hot data identification module: dividing a data value interval, and determining a node to be migrated and a node to be migrated corresponding to each data value interval according to the distribution condition of the data blocks on each node in the data value interval;
a migration module: and migrating the data block on the node to be migrated to the node to be migrated.
2. The multi-node load balancing cold and hot data migration device according to claim 1, wherein the IO access request information of the data block collected by the data collection module includes a total access frequency, a random read and write access frequency, a sequential read and write access frequency, a last access time of the file, a last access type, and a last destination address of the access;
the data value calculation module calculates the data value of the data block according to the IO request access information, and specifically comprises the following steps:
and distributing weights for each access information factor, and performing weighted summation to obtain the data value of the data block.
3. The multi-node load balancing cold and hot data migration apparatus according to claim 2, wherein the cold and hot data identification module is specifically configured to:
dividing N data value intervals;
calculating the data block percentage of each data value interval aiming at each node;
for each data value interval, comparing the node with the highest data block percentage with the node with the lowest data block percentage, if the difference between the two data block percentages exceeds a threshold percentage, marking the node with the highest data block percentage as a node to be migrated, and marking the node with the lowest data block percentage as a node to be migrated, otherwise, not migrating; comparing the nodes with the second highest percentage of the data blocks with the nodes with the second lowest percentage of the data blocks, if the percentage difference of the two data blocks exceeds the threshold percentage, marking the nodes with the second highest percentage of the data blocks as nodes to be migrated, and marking the nodes with the second lowest percentage of the data blocks as nodes to be migrated, otherwise, not migrating; and so on.
4. The multi-node load-balancing cold and hot data migration apparatus according to claim 3, further comprising,
a metadata module: recording the file storage physical position of each data block, including a storage node, a storage device type and a specific storage position in the storage device;
a migration plan generation module: setting a data migration rule which comprises a migration mode, a migration period, migration time and a wear balance mechanism;
the migration module is specifically configured to:
migrating the data blocks based on the data migration rule set by the generation migration calculation module according to the nodes to be migrated and the nodes to be migrated which are determined by the cold and hot data identification module; and after the migration is completed, the metadata module is notified to update the metadata information.
5. A multi-node load balancing cold and hot data migration method is characterized by comprising the following steps:
s1, collecting IO request access information of each data block in each node;
s2, calculating the data value of the data block according to the IO request access information;
s3, dividing data value intervals, and determining a node to be migrated and a node to be migrated corresponding to each data value interval according to the distribution condition of the data blocks on each node in the data value intervals;
and S4, migrating the data block on the node to be migrated to the node to be migrated.
6. The multi-node load balancing cold and hot data migration method according to claim 5, wherein the data block IO access request information collected in step S1 includes a total access frequency, a random read and write access frequency, a sequential read and write access frequency, a last access time of a file, a last access type, and a last access destination address;
step S2 specifically includes:
and distributing weights for each access information factor, and performing weighted summation to obtain the data value of the data block.
7. The multi-node load balancing cold and hot data migration method according to claim 6, wherein step S3 specifically includes:
dividing N data value intervals;
calculating the data block percentage of each data value interval aiming at each node;
for each data value interval, comparing the node with the highest data block percentage with the node with the lowest data block percentage, if the difference between the two data block percentages exceeds a threshold percentage, marking the node with the highest data block percentage as a node to be migrated, and marking the node with the lowest data block percentage as a node to be migrated, otherwise, not migrating; comparing the nodes with the second highest data block percentage with the nodes with the second lowest data block percentage, if the percentage difference of the two data blocks exceeds the threshold percentage, marking the nodes with the second highest data block percentage as nodes to be migrated, and marking the nodes with the second lowest data block percentage as nodes to be migrated, otherwise, not migrating; and so on.
8. The multi-node load balancing cold and hot data migration method according to claim 7, further comprising,
s0, recording the file saving physical position of each data block, including storage node, storage device type and concrete storage position in the storage device; setting a data migration rule which comprises a migration mode, a migration period, migration time and a wear balance mechanism;
step S4 specifically includes:
migrating the data block based on the set data migration rule according to the determined nodes to be migrated and the determined nodes to be migrated; and after the migration is completed, the metadata module is notified to update the metadata information.
9. A terminal, comprising:
the memory is used for storing a multi-node load balancing cold and hot data migration program;
a processor, configured to implement the steps of the multi-node load balancing cold and hot data migration method according to any one of claims 5 to 8 when executing the multi-node load balancing cold and hot data migration program.
10. A computer-readable storage medium, having stored thereon a multi-node load-balancing cold and hot data migration program, which when executed by a processor, performs the steps of the multi-node load-balancing cold and hot data migration method according to any one of claims 5 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210864889.2A CN115079958A (en) | 2022-07-22 | 2022-07-22 | Multi-node load balancing cold and hot data migration device, method, terminal and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210864889.2A CN115079958A (en) | 2022-07-22 | 2022-07-22 | Multi-node load balancing cold and hot data migration device, method, terminal and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115079958A true CN115079958A (en) | 2022-09-20 |
Family
ID=83241993
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210864889.2A Pending CN115079958A (en) | 2022-07-22 | 2022-07-22 | Multi-node load balancing cold and hot data migration device, method, terminal and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115079958A (en) |
-
2022
- 2022-07-22 CN CN202210864889.2A patent/CN115079958A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9606735B2 (en) | Storage management apparatus, and performance adjusting method | |
| JP2019511054A (en) | Distributed cluster training method and apparatus | |
| KR20170110708A (en) | Opportunistic Resource Migration for Resource Deployment Optimization | |
| CN109408590B (en) | Method, device and equipment for expanding distributed database and storage medium | |
| CN111290699B (en) | Data migration method, device and system | |
| CN111966449B (en) | Virtual machine backup management method, system, terminal and storage medium | |
| JP5672386B2 (en) | Computer and resource search method | |
| CN113867644B (en) | Disk array optimization method, disk array optimization device, computer equipment and storage medium | |
| CN110297743B (en) | Load testing method and device and storage medium | |
| CN115469813A (en) | Data processing method and device, storage medium and electronic device | |
| CN111338579B (en) | Read-write cache optimization method, system, terminal and storage medium based on storage pool | |
| JP5515889B2 (en) | Virtual machine system, automatic migration method and automatic migration program | |
| CN108833592A (en) | Cloud host schedules device optimization method, device, equipment and storage medium | |
| CN109725835A (en) | For managing the method, equipment and computer program product of disk array | |
| CN109788006B (en) | Data equalization method and device and computer equipment | |
| CN116483740B (en) | Memory data migration method and device, storage medium and electronic device | |
| CN115344358A (en) | Resource scheduling method, device and management node | |
| CN111007988B (en) | RAID internal wear balancing method, system, terminal and storage medium | |
| CN115079958A (en) | Multi-node load balancing cold and hot data migration device, method, terminal and medium | |
| IL301738A (en) | Predictive block storage size provisioning for cloud storage volumes | |
| CN115061813A (en) | Cluster resource management method, device, equipment and medium | |
| CN106909472B (en) | Input/output load adjusting method and device of distributed file system | |
| CN112748850B (en) | Method, apparatus and computer program product for storage management | |
| CN115686342A (en) | Data storage method and device in storage system | |
| CN111124260B (en) | Method, electronic device and computer program product for managing redundant array of independent disks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |