CN117991997A - Method and device for balancing disk storage load - Google Patents
Method and device for balancing disk storage load Download PDFInfo
- Publication number
- CN117991997A CN117991997A CN202410406454.2A CN202410406454A CN117991997A CN 117991997 A CN117991997 A CN 117991997A CN 202410406454 A CN202410406454 A CN 202410406454A CN 117991997 A CN117991997 A CN 117991997A
- Authority
- CN
- China
- Prior art keywords
- heat
- disk
- disks
- data
- expected
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000005192 partition Methods 0.000 claims abstract description 91
- 230000000737 periodic effect Effects 0.000 claims abstract description 40
- 238000004458 analytical method Methods 0.000 claims abstract description 26
- 230000005012 migration Effects 0.000 claims abstract description 13
- 238000013508 migration Methods 0.000 claims abstract description 13
- 238000007726 management method Methods 0.000 claims description 52
- 238000012544 monitoring process Methods 0.000 claims description 38
- 238000004364 calculation method Methods 0.000 claims description 21
- 238000010276 construction Methods 0.000 claims description 14
- 238000000638 solvent extraction Methods 0.000 claims description 14
- 238000012545 processing Methods 0.000 claims description 9
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention is applicable to the technical field of disk management, and provides a method and a device for balancing disk storage load. The invention pre-partitions a plurality of physical disks into a temporary storage disk and a plurality of expected heat disks; storing the initial data to a temporary storage disk; performing time period heat analysis on the initial data, and calculating the time period data heat; matching a target hot disk from a plurality of expected hot disks, and migrating initial data to the target hot disk; calculating the heat of the magnetic disk in a plurality of periods and the corresponding gradient proportion of the heat; partition adjustment is performed on a plurality of expected hot magnetic disks. The method and the device can calculate the time period data heat of initial data, match a target heat disk to perform data migration of the initial data, monitor the heat periodically on a plurality of expected heat disks, calculate the heat of the plurality of periodic disks and the corresponding heat gradient proportion, and perform partition adjustment on the plurality of expected heat disks, so that load balancing is more intelligent, and more reasonable load balancing can be realized.
Description
Technical Field
The invention belongs to the technical field of disk management, and particularly relates to a method and a device for balancing disk storage load.
Background
Load balancing is an important concept in computer networks and storage systems, and by reasonably distributing resources, the overall performance of the system is ensured to be optimal. The load balancing of disk storage is mainly realized by dispersing data to a plurality of disks so as to fully utilize the performance of each disk, realize load balancing and high availability and avoid that a single disk becomes a performance bottleneck.
In the prior art, load balancing of disk storage is achieved only by simply polling or randomly selecting and distributing I/O requests, but factors such as real-time performance of the disk, data distribution, disk capacity and the like are not fully considered, so that the load balancing is not intelligent enough, and more reasonable load balancing cannot be achieved.
Disclosure of Invention
The embodiment of the invention aims to provide a method and a device for balancing disk storage load, which aim to solve the technical problems in the prior art mentioned in the background art.
The embodiment of the invention is realized as follows:
A method for disk storage load balancing, the method comprising the steps of:
Determining a plurality of physical disks, pre-partitioning the physical disks into a temporary storage disk and a plurality of expected heat disks, wherein the expected heat disks correspond to a plurality of expected heat intervals, and the expected heat intervals are arranged in a gradient manner;
Initial data are acquired, the initial data are stored in the temporary storage disk, and data link is carried out on the temporary storage disk and a plurality of expected heat disks, so that an integrated management list is constructed;
performing time period heat analysis on the initial data, and calculating time period data heat;
matching a target heat disk from a plurality of expected heat disks according to the time period data heat and a plurality of expected heat intervals, and migrating the initial data to the target heat disk;
performing periodic heat monitoring on a plurality of expected heat disks, calculating heat of the plurality of periodic disks, and calculating heat gradient proportions corresponding to the heat of the plurality of periodic disks;
And carrying out partition adjustment on a plurality of expected heat magnetic disks according to a plurality of heat gradient proportions.
As a further limitation of the technical solution of the embodiment of the present invention, the determining a plurality of physical disks, pre-partitioning the plurality of physical disks into a temporary storage disk and a plurality of expected heat disks, where the plurality of expected heat disks correspond to a plurality of expected heat intervals, and the gradient arrangement of the plurality of expected heat intervals specifically includes the following steps:
Determining a plurality of physical disks;
receiving basic partition requirements;
Carrying out partition analysis on the basic partition requirements, determining the number of basic partitions, and acquiring a plurality of gradient arrangement expected heat intervals;
according to the number of the basic partitions, uniformly pre-partitioning the physical disks to generate temporary storage disks and a plurality of expected heat disks;
And correspondingly matching the expected heat intervals with the expected heat disks.
As a further limitation of the technical solution of the embodiment of the present invention, the obtaining initial data, storing the initial data in the temporary storage disk, and performing data linking on the temporary storage disk and the plurality of expected heat disks, and constructing an integrated management list specifically includes the following steps:
acquiring initial data and triggering generation of an initial storage instruction;
Storing the initial data into the temporary storage disk according to the initial storage instruction;
Creating an integrated management interface;
performing data monitoring on the temporary storage disk and the plurality of expected heat disks to acquire a plurality of disk data addresses and corresponding data subject information;
and managing and arranging the plurality of data subject information in the integrated management interface, and carrying out data link with the plurality of disk data addresses to construct an integrated management list.
As a further limitation of the technical solution of the embodiment of the present invention, the time period heat analysis is performed on the initial data, and in calculating the time period data heat, a calculation formula of the time period data heat is as follows:
；
；
Wherein, For period data heat,/>For time period access times,/>Weighting coefficient for time period visit times,/>For the most recent access time difference,/>For the weighting factor of the most recent access time difference,/>For data importance level,/>Is the weighting coefficient of the importance level of the data,/>For data size,/>Is a weighting coefficient for the data size.
As a further limitation of the technical solution of the embodiment of the present invention, the matching a target hot disk from a plurality of expected hot disks according to the period data hot and a plurality of expected hot intervals, and the migrating the initial data to the target hot disk specifically includes the following steps:
from a plurality of expected heat intervals, matching time period data heat corresponding to the heat falling into a heat interval;
Marking the expected heat disk corresponding to the heat interval as a target heat disk;
Monitoring the read-write state of the target heat magnetic disk and the temporary storage magnetic disk, and selecting a read-write idle period;
Migrating the initial data to the target hot disk in the read-write idle period;
And deleting the initial data in the temporary storage disk.
As a further limitation of the technical solution of the embodiment of the present invention, the periodic heat monitoring is performed on the plurality of expected heat disks, the heat of the plurality of periodic disks is calculated, and in the heat gradient proportion corresponding to the heat of the plurality of periodic disks, the calculation formula of the heat of the plurality of periodic disks is as follows:
；
Wherein, Represents the/>The expected heat disk,/>For/>Periodic disk heats of disk of desired heat,/>Is the loss coefficient of the magnetic disk,/>For/>Disk capacity of a disk of desired heat,/>For/>The number of I/O requests for a disk of expected heat,For/>Queue length of each expected hot disk,/>Representing a time period,/>For the average length of the access time interval,/>For/>Error rate of each expected hot disk;
the calculation formulas of the heat gradient ratios are as follows:
；
Wherein, Representing the number of hot disks expected,/>For/>The ratio of the thermal gradients of the desired thermal disks.
As a further limitation of the technical solution of the embodiment of the present invention, the partition adjustment for the plurality of expected thermal disks according to the thermal gradient ratios specifically includes the following steps:
The temporary storage disk and a plurality of the expected heat disks are put into an umount state;
checking a plurality of empty spaces of a plurality of the expected heat disks;
And in the umount state, expanding or shrinking and adjusting the free space of a plurality of expected thermal magnetic disks according to a plurality of thermal gradient proportions.
The device comprises a physical disk pre-partition module, a management list construction module, a data heat calculation module, an initial data migration module, a disk heat calculation module and a disk partition adjustment module, wherein:
The physical disk pre-partitioning module is used for determining a plurality of physical disks, pre-partitioning the physical disks into a temporary storage disk and a plurality of expected heat disks, wherein the expected heat disks correspond to a plurality of expected heat intervals, and the expected heat intervals are arranged in a gradient manner;
The management list construction module is used for acquiring initial data, storing the initial data into the temporary storage disk, and carrying out data link on the temporary storage disk and a plurality of expected heat disks to construct an integrated management list;
the data heat calculation module is used for carrying out time period heat analysis on the initial data and calculating the time period data heat;
The initial data migration module is used for matching a target heat disk from a plurality of expected heat disks according to the time period data heat and a plurality of expected heat intervals, and migrating the initial data to the target heat disk;
The magnetic disk heat calculation module is used for carrying out periodic heat monitoring on a plurality of magnetic disks with expected heat, calculating the heat of a plurality of periodic magnetic disks and calculating heat gradient proportions corresponding to the heat of the plurality of periodic magnetic disks;
and the disk partition adjusting module is used for carrying out partition adjustment on a plurality of expected heat disks according to a plurality of heat gradient proportions.
As a further limitation of the technical solution of the embodiment of the present invention, the physical disk pre-partition module specifically includes:
A physical disk determining unit configured to determine a plurality of physical disks;
the demand receiving unit is used for receiving the demand of the basic partition;
the partition analysis unit is used for carrying out partition analysis on the basic partition requirements, determining the number of basic partitions and acquiring a plurality of gradient arranged expected heat intervals;
The pre-partition processing unit is used for carrying out uniform pre-partition processing on a plurality of physical disks according to the number of the basic partitions to generate a temporary storage disk and a plurality of expected heat disks;
And the heat interval matching unit is used for correspondingly matching the expected heat intervals with the expected heat disks.
As a further limitation of the technical solution of the embodiment of the present invention, the management list construction module specifically includes:
The instruction generation unit is used for acquiring initial data and triggering generation of an initial storage instruction;
the temporary storage unit is used for storing the initial data into the temporary storage disk according to the initial storage instruction;
The interface creation unit is used for creating an integrated management interface;
The data monitoring unit is used for carrying out data monitoring on the temporary storage disk and the plurality of expected heat disks and acquiring a plurality of disk data addresses and corresponding data subject information;
And the list construction unit is used for carrying out management arrangement on the plurality of data subject information in the integrated management interface, and carrying out data link with the plurality of disk data addresses to construct an integrated management list.
Compared with the prior art, the invention has the beneficial effects that:
The embodiment of the invention pre-partitions a plurality of physical disks into a temporary storage disk and a plurality of expected heat disks; storing the initial data to a temporary storage disk; performing time period heat analysis on the initial data, and calculating the time period data heat; matching a target hot disk from a plurality of expected hot disks, and migrating initial data to the target hot disk; calculating the heat of the magnetic disk in a plurality of periods and the corresponding gradient proportion of the heat; partition adjustment is performed on a plurality of expected hot magnetic disks. The method and the device can calculate the time period data heat of initial data, match a target heat disk to perform data migration of the initial data, monitor the heat periodically on a plurality of expected heat disks, calculate the heat of the plurality of periodic disks and the corresponding heat gradient proportion, and perform partition adjustment on the plurality of expected heat disks, so that load balancing is more intelligent, and more reasonable load balancing can be realized.
Drawings
FIG. 1 is a flowchart of a method for balancing disk storage load according to an embodiment of the present invention;
FIG. 2 is a flowchart of a physical disk pre-partition process in a method according to an embodiment of the present invention;
FIG. 3 shows a flowchart of constructing an integrated management list in a method provided by an embodiment of the present invention;
FIG. 4 shows a flowchart of initial data matching migration in a method provided by an embodiment of the present invention;
FIG. 5 is a flow chart illustrating the expected hot disk partition adjustment in a method provided by an embodiment of the present invention;
FIG. 6 shows an application architecture diagram of an apparatus for disk storage load balancing provided by an embodiment of the present invention;
FIG. 7 is a block diagram illustrating a physical disk pre-partition module in an apparatus according to an embodiment of the present invention;
Fig. 8 is a block diagram illustrating a structure of a management list construction module in an apparatus according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
It can be understood that in the prior art, load balancing of disk storage is achieved only by simply polling or randomly selecting and distributing I/O requests, but factors such as real-time performance, data distribution, disk capacity and the like of the disk are not fully considered, so that load balancing is not intelligent enough, and more reasonable load balancing cannot be achieved.
In order to solve the above problems, according to the method and device for balancing the storage load of the disk disclosed by the embodiments of the present invention, by determining a plurality of physical disks, the plurality of physical disks are pre-partitioned into a temporary storage disk and a plurality of expected heat disks, and the plurality of expected heat disks correspond to a plurality of expected heat intervals, and the plurality of expected heat intervals are arranged in a gradient manner; initial data are acquired, the initial data are stored in a temporary storage disk, and data link is carried out on the temporary storage disk and a plurality of expected heat disks, so that an integrated management list is constructed; performing time period heat analysis on the initial data, and calculating the time period data heat; according to the time period data heat and a plurality of expected heat intervals, matching a target heat disk from a plurality of expected heat disks, and migrating initial data to the target heat disk; periodically monitoring the heat of a plurality of expected heat disks, calculating the heat of the plurality of periodic disks, and calculating the heat gradient proportion corresponding to the heat of the plurality of periodic disks; and carrying out partition adjustment on a plurality of expected heat magnetic disks according to a plurality of heat gradient proportions. The method and the device can calculate the time period data heat of initial data, match a target heat disk to perform data migration of the initial data, monitor the heat periodically on a plurality of expected heat disks, calculate the heat of the plurality of periodic disks and the corresponding heat gradient proportion, and perform partition adjustment on the plurality of expected heat disks, so that load balancing is more intelligent, and more reasonable load balancing can be realized.
FIG. 1 is a flowchart illustrating a method for balancing disk storage load according to an embodiment of the present invention.
In particular, in a preferred embodiment provided by the present invention, a method for balancing disk storage load, the method specifically includes the following steps:
Step S101, determining a plurality of physical disks, pre-partitioning the physical disks into a temporary storage disk and a plurality of expected heat disks, wherein the expected heat disks correspond to a plurality of expected heat intervals, and the expected heat intervals are arranged in a gradient mode.
In the embodiment of the invention, the plurality of physical disks are determined, the basic partition requirement is received, partition analysis is carried out on the basic partition requirement, the required content of the basic partition requirement is identified, the number of basic partitions is determined, a plurality of gradient arranged expected heat intervals are obtained, the disk spaces of the plurality of physical disks are identified, the plurality of physical disks are uniformly pre-partitioned according to the number of basic partitions, a temporary storage disk and a plurality of expected heat disks are generated, and the plurality of expected heat intervals and the plurality of expected heat disks are correspondingly matched, so that each expected heat disk is in a corresponding expected heat interval.
It will be appreciated that the gradient arrangement is arranged from large to small in terms of the interval values of a plurality of expected heat intervals.
Specifically, fig. 2 shows a flowchart of physical disk pre-partition processing in the method provided by the embodiment of the present invention.
In the preferred embodiment provided by the invention, the determining a plurality of physical disks pre-partitions the plurality of physical disks into a temporary storage disk and a plurality of expected heat disks, wherein the plurality of expected heat disks correspond to a plurality of expected heat intervals, and the gradient arrangement of the plurality of expected heat intervals specifically comprises the following steps:
step S1011, determining a plurality of physical disks.
Step S1012, receiving the basic partition requirement.
Step S1013, carrying out partition analysis on the basic partition requirements, determining the number of basic partitions, and obtaining a plurality of gradient arranged expected heat intervals.
And step S1014, performing uniform pre-partition processing on a plurality of physical disks according to the number of the basic partitions, and generating a temporary storage disk and a plurality of expected heat disks.
Step S1015, performing corresponding matching between the plurality of expected heat intervals and the plurality of expected heat disks.
Further, the method for balancing the disk storage load further comprises the following steps:
step S102, initial data are acquired, the initial data are stored in the temporary storage disk, and data link is carried out on the temporary storage disk and a plurality of expected heat disks, so that an integrated management list is constructed.
In the embodiment of the invention, in the use process of a plurality of physical disks, an initial storage instruction is triggered and generated by acquiring initial data, then the initial data is stored in a temporary storage disk according to the initial storage instruction, an integrated management interface is created, a plurality of storage data are determined by identifying and monitoring the data in the temporary storage disk and a plurality of expected heat disks, disk data addresses and data subject information corresponding to the plurality of storage data are acquired, and then the plurality of data subject information are arranged and displayed in the integrated management interface, and data links are carried out between the data subject information of the plurality of storage data and the disk data addresses to construct an integrated management list, so that a user does not need to inquire and use the data according to different physical disks or expected heat disks, but directly accesses, views and uses the data in the plurality of physical disks through the data links of the integrated management list.
It is understood that the initial data is newly generated by a computer device installed with a plurality of physical disks, and may be newly generated internet of things data, social media data, application data, transaction data, multimedia data, etc.
Specifically, fig. 3 shows a flowchart of building an integrated management list in the method provided by the embodiment of the present invention.
In the preferred embodiment of the present invention, the obtaining initial data, storing the initial data in the temporary storage disk, and performing data linking on the temporary storage disk and the plurality of expected heat disks, and constructing an integrated management list specifically includes the following steps:
Step S1021, obtaining initial data and triggering generation of an initial storage instruction.
Step S1022, storing the initial data to the temporary storage disk according to the initial storage instruction.
Step S1023, creating an integrated management interface.
And step S1024, data monitoring is carried out on the temporary storage disk and the plurality of expected heat disks, and a plurality of disk data addresses and corresponding data subject information are obtained.
Step S1025, managing and arranging the plurality of data subject information in the integrated management interface, and carrying out data link with the plurality of disk data addresses to construct an integrated management list.
Further, the method for balancing the disk storage load further comprises the following steps:
step S103, carrying out time period heat analysis on the initial data, and calculating the time period data heat.
In the embodiment of the invention, the time from the initial data storage to the temporary storage disk is taken as the starting time, the use condition of the initial data in the temporary storage disk is monitored according to a preset data monitoring period, the initial monitoring data is obtained, the period data heat of the initial data is calculated by carrying out period heat analysis on the initial monitoring data, and the calculation formula of the period data heat is specifically as follows:
；
；
Wherein, For period data heat,/>For time period access times,/>Weighting coefficient for time period visit times,/>For the most recent access time difference,/>For the weighting factor of the most recent access time difference,/>For data importance level,/>Is the weighting coefficient of the importance level of the data,/>For data size,/>Is a weighting coefficient for the data size.
It can be understood that the initial monitoring data includes the time period access times, the latest access time difference, the data importance level, the data size and the like of the initial data.
And step S104, matching a target heat disk from a plurality of expected heat disks according to the time period data heat and a plurality of expected heat intervals, and migrating the initial data to the target heat disk.
In the embodiment of the invention, the time period data heat is compared with a plurality of expected heat intervals, the time period data heat is matched with an expected heat disk corresponding to the time period data heat in the expected heat intervals, the expected heat disk corresponding to the time period data heat is marked as a target heat disk, the read-write idle time is selected by monitoring the read-write state of the target heat disk and the temporary storage disk, an initial migration task is created in the read-write idle time, the initial data is migrated to the target heat disk, and the initial data in the temporary storage disk is deleted, so that after the initial data is monitored for a period of time, the initial data is migrated to a proper logic disk, and the load balance of more intelligent and accurate data distribution is realized.
It is understood that the read-write idle period is a period in which the computer device on which a plurality of physical disks are mounted is in an operating state, but the target hot disk and the temporary storage disk do not have a read-write task.
Specifically, fig. 4 shows a flowchart of initial data matching migration in the method provided by the embodiment of the present invention.
In a preferred embodiment of the present invention, the matching a target thermal disk from a plurality of the expected thermal disks according to the period data thermal degree and a plurality of the expected thermal intervals, and migrating the initial data to the target thermal disk specifically includes the following steps:
step S1041, wherein the data heat of the matching period falls into a heat interval corresponding to the expected heat intervals.
Step S1042, marking the expected heat disk corresponding to the heat interval as the target heat disk.
Step S1043, performing read-write status monitoring on the target hot disk and the temporary storage disk, and selecting a read-write idle period.
Step S1044, migrating the initial data to the target hot disk in the read-write idle period.
Step S1045, deleting the initial data in the temporary storage disk.
Further, the method for balancing the disk storage load further comprises the following steps:
Step 105, performing periodic heat monitoring on the plurality of expected heat disks, calculating heat of the plurality of periodic disks, and calculating heat gradient ratios corresponding to the heat of the plurality of periodic disks.
In the embodiment of the invention, according to a preset disk monitoring period, periodically acquiring disk monitoring data, calculating the periodic disk heat corresponding to the plurality of expected heat disks by performing disk heat analysis on the disk monitoring data, and then calculating the heat gradient ratio corresponding to the plurality of expected heat disks according to the plurality of periodic disk heat, wherein the calculation formula of the plurality of periodic disk heat is as follows:
；
Wherein, Represents the/>The expected heat disk,/>For/>Periodic disk heats of disk of desired heat,/>Is the loss coefficient of the magnetic disk,/>For/>Disk capacity of a disk of desired heat,/>For/>The number of I/O requests for a disk of expected heat,For/>Queue length of each expected hot disk,/>Representing a time period,/>For the average length of the access time interval,/>For/>Error rate of each expected hot disk;
the calculation formulas of the heat gradient ratios are as follows:
；
Wherein, Representing the number of hot disks expected,/>For/>The ratio of the thermal gradients of the desired thermal disks.
It will be appreciated that the disk monitoring data includes the disk capacity, I/O request count, queue length, access time interval, error rate, etc. of the plurality of hot disks.
And S106, carrying out partition adjustment on a plurality of expected heat magnetic disks according to a plurality of heat gradient proportions.
In the embodiment of the invention, the temporary storage disk and the plurality of expected heat disks are controlled to enter the umount state, and the plurality of empty spaces of the plurality of expected heat disks are checked, and in the umount state, the expansion or the reduction of the empty spaces of the plurality of expected heat disks are adjusted according to the plurality of heat gradient proportions, so that the disk capacity of the plurality of adjusted expected heat disks meets the corresponding heat gradient proportions, and the load balancing of the disk partition is realized more intelligently and reasonably.
Specifically, fig. 5 shows a flowchart of the partition adjustment of the expected hot disk in the method provided by the embodiment of the invention.
In the preferred embodiment provided by the invention, the partition adjustment of the plurality of expected thermal disks according to the thermal gradient ratios specifically comprises the following steps:
step S1061, putting the temporary storage disk and the plurality of expected heat disks into an umount state.
Step S1062, checking a plurality of empty spaces of a plurality of the expected thermal disks.
Step S1063, in the umount state, performing expansion or contraction adjustment on the free space of the plurality of expected thermal disks according to the thermal gradient ratios.
Further, fig. 6 shows an application architecture diagram of the apparatus for balancing disk storage load according to the embodiment of the present invention.
In another preferred embodiment of the present invention, an apparatus for balancing disk storage load includes:
The physical disk pre-partitioning module 101 is configured to determine a plurality of physical disks, pre-partition the plurality of physical disks into a temporary storage disk and a plurality of expected heat disks, where the plurality of expected heat disks correspond to a plurality of expected heat intervals, and the plurality of expected heat intervals are arranged in a gradient manner.
In the embodiment of the present invention, the physical disk pre-partitioning module 101 determines the number of basic partitions by determining a plurality of physical disks, receiving the basic partition requirement, performing partition analysis on the basic partition requirement, identifying the requirement content of the basic partition requirement, determining the number of basic partitions, and obtaining a plurality of gradient arranged expected heat intervals, further identifying the disk spaces of the plurality of physical disks, performing uniform pre-partitioning processing on the plurality of physical disks according to the number of basic partitions, generating a temporary storage disk and a plurality of expected heat disks, and performing corresponding matching on the plurality of expected heat intervals and the plurality of expected heat disks, so that each expected heat disk is a corresponding expected heat interval.
Specifically, fig. 7 shows a block diagram of a physical disk pre-partition module 101 in an apparatus according to an embodiment of the present invention.
In a preferred embodiment of the present invention, the physical disk pre-partition module 101 specifically includes:
the physical disk determining unit 1011 is configured to determine a plurality of physical disks.
The requirement receiving unit 1012 is configured to receive the base partition requirement.
The partition analysis unit 1013 is configured to perform partition analysis on the basic partition requirement, determine the number of basic partitions, and obtain a plurality of gradient arranged expected heat intervals.
The pre-partition processing unit 1014 is configured to perform uniform pre-partition processing on the plurality of physical disks according to the number of basic partitions, and generate a temporary storage disk and a plurality of expected hot disks.
And a heat interval matching unit 1015, configured to match the expected heat intervals with the expected heat disks.
Further, the device for balancing the disk storage load further comprises:
The management list construction module 102 is configured to obtain initial data, store the initial data in the temporary storage disk, and perform data linking on the temporary storage disk and the plurality of expected heat disks to construct an integrated management list.
In the embodiment of the invention, in the use process of a plurality of physical disks, the management list construction module 102 triggers to generate an initial storage instruction by acquiring initial data, further stores the initial data into a temporary storage disk according to the initial storage instruction, creates an integrated management interface, identifies and monitors the data in the temporary storage disk and a plurality of expected heat disks, determines a plurality of storage data, acquires disk data addresses and data subject information corresponding to the plurality of storage data, further arranges and displays the plurality of data subject information in the integrated management interface, and links the data subject information of the plurality of storage data with the disk data addresses to construct an integrated management list, so that a user does not need to inquire and use the data according to different physical disks or expected heat disks, but directly accesses, views and uses the data in the plurality of physical disks through the data link of the integrated management list.
Specifically, fig. 8 shows a block diagram of the management list construction module 102 in the device according to the embodiment of the present invention.
In a preferred embodiment of the present invention, the management list construction module 102 specifically includes:
The instruction generating unit 1021 is configured to acquire initial data and trigger generation of an initial storage instruction.
A temporary storage unit 1022, configured to store the initial data in the temporary storage disk according to the initial storage instruction.
An interface creation unit 1023 for creating an integrated management interface.
And a data monitoring unit 1024, configured to perform data monitoring on the temporary storage disk and the plurality of expected heat disks, and obtain a plurality of disk data addresses and corresponding data subject information.
The list construction unit 1025 is configured to manage and arrange the plurality of data subject information in the integrated management interface, and perform data linking with the plurality of disk data addresses, so as to construct an integrated management list.
Further, the device for balancing the disk storage load further comprises:
And the data heat calculation module 103 is used for carrying out time period heat analysis on the initial data and calculating the time period data heat.
In the embodiment of the present invention, the data heat calculating module 103 monitors the use condition of the initial data in the temporary storage disk according to a preset data monitoring period by taking the time from the initial data to the temporary storage disk as the starting time, obtains the initial monitoring data, and calculates the time-period data heat of the initial data by performing time-period heat analysis on the initial monitoring data, where the specific calculation formula of the time-period data heat is:
；
；
Wherein, For period data heat,/>For time period access times,/>Weighting coefficient for time period visit times,/>For the most recent access time difference,/>For the weighting factor of the most recent access time difference,/>For data importance level,/>Is the weighting coefficient of the importance level of the data,/>For data size,/>Is a weighting coefficient for the data size.
An initial data migration module 104, configured to match a target hot disk from a plurality of the expected hot disks according to the period data hot and the plurality of the expected hot intervals, and migrate the initial data to the target hot disk.
In the embodiment of the present invention, the initial data migration module 104 compares the time-period data heat with a plurality of expected heat intervals, matches the expected heat disk corresponding to the time-period data heat with the expected heat interval corresponding to the heat interval from the expected heat intervals, marks the expected heat disk as the target heat disk, monitors the read-write state of the target heat disk and the temporary storage disk, selects a read-write idle period, creates an initial migration task in the read-write idle period, migrates the initial data to the target heat disk, and deletes the initial data in the temporary storage disk, thereby migrating the initial data to a suitable logic disk after monitoring the initial data for a period of time, and realizing load balancing of more intelligent and accurate data allocation.
The disk heat calculating module 105 is configured to monitor the heat of the plurality of expected heat disks periodically, calculate the heat of the plurality of periodic disks, and calculate a heat gradient ratio corresponding to the heat of the plurality of periodic disks.
In the embodiment of the present invention, the disk heat calculating module 105 performs periodic heat monitoring on a plurality of expected heat disks according to a preset disk monitoring period, periodically obtains disk monitoring data, performs disk heat analysis on the disk monitoring data, calculates periodic disk heat corresponding to the plurality of expected heat disks, and calculates heat gradient ratios corresponding to the plurality of expected heat disks according to the plurality of periodic disk heat, where a calculation formula of the plurality of periodic disk heat is specifically:
；
Wherein, Represents the/>The expected heat disk,/>For/>Periodic disk heats of disk of desired heat,/>Is the loss coefficient of the magnetic disk,/>For/>Disk capacity of a disk of desired heat,/>For/>The number of I/O requests for a disk of expected heat,For/>Queue length of each expected hot disk,/>Representing a time period,/>For the average length of the access time interval,/>For/>Error rate of each expected hot disk;
the calculation formulas of the heat gradient ratios are as follows:
；
Wherein, Representing the number of hot disks expected,/>For/>The ratio of the thermal gradients of the desired thermal disks.
The disk partition adjustment module 106 is configured to perform partition adjustment on a plurality of expected thermal disks according to a plurality of thermal gradient proportions.
In the embodiment of the present invention, the disk partition adjustment module 106 controls the temporary storage disk and the plurality of expected heat disks to enter the umount state, and checks a plurality of empty spaces of the plurality of expected heat disks, and in the umount state, the expansion or the shrinkage adjustment of the empty spaces is performed on the plurality of expected heat disks according to a plurality of heat gradient ratios, so that the disk capacities of the plurality of adjusted expected heat disks satisfy the corresponding heat gradient ratios, and more intelligent and reasonable load balancing of the disk partition is realized.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.
Claims (10)
1. A method for balancing disk storage load, the method comprising the steps of:
Determining a plurality of physical disks, pre-partitioning the physical disks into a temporary storage disk and a plurality of expected heat disks, wherein the expected heat disks correspond to a plurality of expected heat intervals, and the expected heat intervals are arranged in a gradient manner;
Initial data are acquired, the initial data are stored in the temporary storage disk, and data link is carried out on the temporary storage disk and a plurality of expected heat disks, so that an integrated management list is constructed;
performing time period heat analysis on the initial data, and calculating time period data heat;
matching a target heat disk from a plurality of expected heat disks according to the time period data heat and a plurality of expected heat intervals, and migrating the initial data to the target heat disk;
performing periodic heat monitoring on a plurality of expected heat disks, calculating heat of the plurality of periodic disks, and calculating heat gradient proportions corresponding to the heat of the plurality of periodic disks;
And carrying out partition adjustment on a plurality of expected heat magnetic disks according to a plurality of heat gradient proportions.
2. The method of claim 1, wherein the determining a plurality of physical disks, pre-partitioning the plurality of physical disks into a temporary storage disk and a plurality of expected heat disks, wherein the plurality of expected heat disks correspond to a plurality of expected heat intervals, and wherein the gradient arrangement of the plurality of expected heat intervals specifically comprises the steps of:
Determining a plurality of physical disks;
receiving basic partition requirements;
Carrying out partition analysis on the basic partition requirements, determining the number of basic partitions, and acquiring a plurality of gradient arrangement expected heat intervals;
according to the number of the basic partitions, uniformly pre-partitioning the physical disks to generate temporary storage disks and a plurality of expected heat disks;
And correspondingly matching the expected heat intervals with the expected heat disks.
3. The method for balancing the storage load of the magnetic disk according to claim 1, wherein the steps of obtaining initial data, storing the initial data in the temporary storage magnetic disk, and performing data linking on the temporary storage magnetic disk and the plurality of expected heat magnetic disks, and constructing an integrated management list specifically include the following steps:
acquiring initial data and triggering generation of an initial storage instruction;
Storing the initial data into the temporary storage disk according to the initial storage instruction;
Creating an integrated management interface;
performing data monitoring on the temporary storage disk and the plurality of expected heat disks to acquire a plurality of disk data addresses and corresponding data subject information;
and managing and arranging the plurality of data subject information in the integrated management interface, and carrying out data link with the plurality of disk data addresses to construct an integrated management list.
4. The method for balancing the disk storage load according to claim 1, wherein the time period heat analysis is performed on the initial data, and in calculating the time period data heat, a calculation formula of the time period data heat is as follows:
；
；
Wherein, For period data heat,/>For time period access times,/>Weighting coefficient for time period visit times,/>For the most recent access time difference,/>For the weighting factor of the most recent access time difference,/>For data importance level,/>Is the weighting coefficient of the importance level of the data,/>For data size,/>Is a weighting coefficient for the data size.
5. The method of claim 1, wherein said matching a target hot disk from among a plurality of said expected hot disks and said migrating said initial data to said target hot disk according to said period of data hot and a plurality of said expected hot intervals comprises the steps of:
from a plurality of expected heat intervals, matching time period data heat corresponding to the heat falling into a heat interval;
Marking the expected heat disk corresponding to the heat interval as a target heat disk;
Monitoring the read-write state of the target heat magnetic disk and the temporary storage magnetic disk, and selecting a read-write idle period;
Migrating the initial data to the target hot disk in the read-write idle period;
And deleting the initial data in the temporary storage disk.
6. The method for balancing the storage load of the magnetic disk according to claim 1, wherein the periodic heat monitoring is performed on the plurality of expected heat magnetic disks, a plurality of periodic magnetic disk heats are calculated, and a heat gradient ratio corresponding to the plurality of periodic magnetic disk heats is calculated according to a calculation formula:
；
Wherein, Represents the/>The expected heat disk,/>For/>Periodic disk heats of disk of desired heat,/>Is the loss coefficient of the magnetic disk,/>For/>Disk capacity of a disk of desired heat,/>For/>I/O request number of expected hot disk,/>For/>Queue length of each expected hot disk,/>Representing a time period,/>For the average length of the access time interval,/>For/>Error rate of each expected hot disk;
the calculation formulas of the heat gradient ratios are as follows:
；
Wherein, Representing the number of hot disks expected,/>For/>The ratio of the thermal gradients of the desired thermal disks.
7. The method for balancing the storage load of the magnetic disk according to claim 1, wherein the partition adjustment of the plurality of expected hot magnetic disks according to the hot gradient ratios specifically comprises the following steps:
The temporary storage disk and a plurality of the expected heat disks are put into an umount state;
checking a plurality of empty spaces of a plurality of the expected heat disks;
And in the umount state, expanding or shrinking and adjusting the free space of a plurality of expected thermal magnetic disks according to a plurality of thermal gradient proportions.
8. The device for balancing the disk storage load is characterized by comprising a physical disk pre-partition module, a management list construction module, a data heat calculation module, an initial data migration module, a disk heat calculation module and a disk partition adjustment module, wherein:
The physical disk pre-partitioning module is used for determining a plurality of physical disks, pre-partitioning the physical disks into a temporary storage disk and a plurality of expected heat disks, wherein the expected heat disks correspond to a plurality of expected heat intervals, and the expected heat intervals are arranged in a gradient manner;
The management list construction module is used for acquiring initial data, storing the initial data into the temporary storage disk, and carrying out data link on the temporary storage disk and a plurality of expected heat disks to construct an integrated management list;
the data heat calculation module is used for carrying out time period heat analysis on the initial data and calculating the time period data heat;
The initial data migration module is used for matching a target heat disk from a plurality of expected heat disks according to the time period data heat and a plurality of expected heat intervals, and migrating the initial data to the target heat disk;
The magnetic disk heat calculation module is used for carrying out periodic heat monitoring on a plurality of magnetic disks with expected heat, calculating the heat of a plurality of periodic magnetic disks and calculating heat gradient proportions corresponding to the heat of the plurality of periodic magnetic disks;
and the disk partition adjusting module is used for carrying out partition adjustment on a plurality of expected heat disks according to a plurality of heat gradient proportions.
9. The apparatus of claim 8, wherein the physical disk pre-partition module specifically comprises:
A physical disk determining unit configured to determine a plurality of physical disks;
the demand receiving unit is used for receiving the demand of the basic partition;
the partition analysis unit is used for carrying out partition analysis on the basic partition requirements, determining the number of basic partitions and acquiring a plurality of gradient arranged expected heat intervals;
The pre-partition processing unit is used for carrying out uniform pre-partition processing on a plurality of physical disks according to the number of the basic partitions to generate a temporary storage disk and a plurality of expected heat disks;
And the heat interval matching unit is used for correspondingly matching the expected heat intervals with the expected heat disks.
10. The apparatus of claim 8, wherein the management list construction module specifically comprises:
The instruction generation unit is used for acquiring initial data and triggering generation of an initial storage instruction;
the temporary storage unit is used for storing the initial data into the temporary storage disk according to the initial storage instruction;
The interface creation unit is used for creating an integrated management interface;
The data monitoring unit is used for carrying out data monitoring on the temporary storage disk and the plurality of expected heat disks and acquiring a plurality of disk data addresses and corresponding data subject information;
And the list construction unit is used for carrying out management arrangement on the plurality of data subject information in the integrated management interface, and carrying out data link with the plurality of disk data addresses to construct an integrated management list.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410406454.2A CN117991997B (en) | 2024-04-07 | 2024-04-07 | Method and device for balancing disk storage load |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410406454.2A CN117991997B (en) | 2024-04-07 | 2024-04-07 | Method and device for balancing disk storage load |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117991997A true CN117991997A (en) | 2024-05-07 |
| CN117991997B CN117991997B (en) | 2024-06-11 |
Family
ID=90901493
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410406454.2A Active CN117991997B (en) | 2024-04-07 | 2024-04-07 | Method and device for balancing disk storage load |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117991997B (en) |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102841931A (en) * | 2012-08-03 | 2012-12-26 | 中兴通讯股份有限公司 | Storage method and storage device of distributive-type file system |
| US20140013032A1 (en) * | 2012-07-03 | 2014-01-09 | Research & Business Foundation Sungkyunkwan University | Method and apparatus for controlling writing data in storage unit based on nand flash memory |
| CN105138290A (en) * | 2015-08-20 | 2015-12-09 | 浪潮(北京)电子信息产业有限公司 | High-performance storage pool organization method and device |
| CN105808620A (en) * | 2014-12-31 | 2016-07-27 | 华为技术有限公司 | Method and device for managing disk file |
| CN106339181A (en) * | 2016-08-19 | 2017-01-18 | 华为技术有限公司 | Method and system for processing data in storage system |
| CN106933511A (en) * | 2017-02-27 | 2017-07-07 | 武汉大学 | Consider the GML data storage method for organizing and system of load balancing and disk efficiency |
| US20170285971A1 (en) * | 2016-04-05 | 2017-10-05 | International Business Machines Corporation | Hotness based data storage for facilitating garbage collection |
| CN107608631A (en) * | 2017-09-12 | 2018-01-19 | 郑州云海信息技术有限公司 | A kind of data file storage method, device, equipment and storage medium |
| CN107870735A (en) * | 2016-09-28 | 2018-04-03 | 杭州华为数字技术有限公司 | Data managing method, device and equipment |
| CN108733308A (en) * | 2017-04-17 | 2018-11-02 | 伊姆西Ip控股有限责任公司 | Method and apparatus for managing Pan Chi |
| CN108932150A (en) * | 2017-05-24 | 2018-12-04 | 中兴通讯股份有限公司 | Caching method, device and medium based on SSD and disk mixing storage |
| CN110362277A (en) * | 2019-07-19 | 2019-10-22 | 重庆大学 | Data classification storage method based on mixing storage system |
| CN112100293A (en) * | 2020-09-23 | 2020-12-18 | 腾讯科技（深圳）有限公司 | Data processing method, data access method, data processing device, data access device and computer equipment |
| CN114090637A (en) * | 2021-11-29 | 2022-02-25 | 平安科技（深圳）有限公司 | Data access method, device, equipment and storage medium |
-
2024
- 2024-04-07 CN CN202410406454.2A patent/CN117991997B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140013032A1 (en) * | 2012-07-03 | 2014-01-09 | Research & Business Foundation Sungkyunkwan University | Method and apparatus for controlling writing data in storage unit based on nand flash memory |
| CN102841931A (en) * | 2012-08-03 | 2012-12-26 | 中兴通讯股份有限公司 | Storage method and storage device of distributive-type file system |
| CN105808620A (en) * | 2014-12-31 | 2016-07-27 | 华为技术有限公司 | Method and device for managing disk file |
| CN105138290A (en) * | 2015-08-20 | 2015-12-09 | 浪潮(北京)电子信息产业有限公司 | High-performance storage pool organization method and device |
| US20170285971A1 (en) * | 2016-04-05 | 2017-10-05 | International Business Machines Corporation | Hotness based data storage for facilitating garbage collection |
| CN106339181A (en) * | 2016-08-19 | 2017-01-18 | 华为技术有限公司 | Method and system for processing data in storage system |
| CN107870735A (en) * | 2016-09-28 | 2018-04-03 | 杭州华为数字技术有限公司 | Data managing method, device and equipment |
| CN106933511A (en) * | 2017-02-27 | 2017-07-07 | 武汉大学 | Consider the GML data storage method for organizing and system of load balancing and disk efficiency |
| CN108733308A (en) * | 2017-04-17 | 2018-11-02 | 伊姆西Ip控股有限责任公司 | Method and apparatus for managing Pan Chi |
| CN108932150A (en) * | 2017-05-24 | 2018-12-04 | 中兴通讯股份有限公司 | Caching method, device and medium based on SSD and disk mixing storage |
| CN107608631A (en) * | 2017-09-12 | 2018-01-19 | 郑州云海信息技术有限公司 | A kind of data file storage method, device, equipment and storage medium |
| CN110362277A (en) * | 2019-07-19 | 2019-10-22 | 重庆大学 | Data classification storage method based on mixing storage system |
| CN112100293A (en) * | 2020-09-23 | 2020-12-18 | 腾讯科技（深圳）有限公司 | Data processing method, data access method, data processing device, data access device and computer equipment |
| CN114090637A (en) * | 2021-11-29 | 2022-02-25 | 平安科技（深圳）有限公司 | Data access method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117991997B (en) | 2024-06-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Raicu et al. | The quest for scalable support of data-intensive workloads in distributed systems | |
| Lee et al. | PFRF: An adaptive data replication algorithm based on star-topology data grids | |
| CN103176849B (en) | A kind of dispositions method of the cluster virtual machine based on resource classification | |
| Sethi et al. | RecShard: statistical feature-based memory optimization for industry-scale neural recommendation | |
| CN103795781B (en) | A kind of distributed caching method based on file prediction | |
| CN106156283B (en) | Heterogeneous Hadoop storage method based on data temperature and node performance | |
| JP2006011860A (en) | System configuration management program and system configuration management device | |
| CN108900626B (en) | Data storage method, device and system in cloud environment | |
| CN109075988A (en) | Task schedule and resource delivery system and method | |
| Rahman et al. | Replica placement strategies in data grid | |
| Huang et al. | Achieving load balance for parallel data access on distributed file systems | |
| CN111737168A (en) | Cache system, cache processing method, device, equipment and medium | |
| CN107409142A (en) | It is synchronous to store affined shared content item | |
| CN109002503A (en) | A kind of metadata read method, device, equipment and readable storage medium storing program for executing | |
| US9501485B2 (en) | Methods for facilitating batch analytics on archived data and devices thereof | |
| Sato et al. | File clustering based replication algorithm in a grid environment | |
| CN110457307A (en) | Metadata management system, user's cluster creation method, device, equipment and medium | |
| CN117991997A (en) | Method and device for balancing disk storage load | |
| CN111858656A (en) | Static data query method and device based on distributed architecture | |
| Yang et al. | Latency-sensitive data allocation and workload consolidation for cloud storage | |
| Su et al. | Predicting response latency percentiles for cloud object storage systems | |
| Luo et al. | Data placement algorithm for improving I/O load balance without using popularity information | |
| AT&T | ||
| CN110941396A (en) | Copy placement method based on airflow organization and oriented to cloud data center | |
| CN110874344B (en) | Data migration method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |