CN114253456A - Cache load balancing method and device - Google Patents
Cache load balancing method and device Download PDFInfo
- Publication number
- CN114253456A CN114253456A CN202010995223.1A CN202010995223A CN114253456A CN 114253456 A CN114253456 A CN 114253456A CN 202010995223 A CN202010995223 A CN 202010995223A CN 114253456 A CN114253456 A CN 114253456A
- Authority
- CN
- China
- Prior art keywords
- cache
- disk group
- load parameter
- cache disk
- file block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
According to the cache load balancing method and device provided by the embodiment of the invention, a first load parameter of a first cache disk group and a second load parameter of a second cache disk group are obtained; when the first load parameter is larger than the second load parameter, determining a first number of file block groups to be migrated according to the difference between the first load parameter and the second load parameter; and migrating at least part of the file block groups stored in the first cache disk group to the second cache disk group according to the first number. The invention can realize data migration among different cache devices, so that the loads born by the different cache devices in the actual operation process are more balanced, and further the overall performance of the cache system can be fully exerted.
Description
Technical Field
The invention relates to the technical field of data storage, in particular to a cache load balancing method and device.
Background
The current cache system is characterized in that each cache device (such as a solid-state disk) is served by a fixed IP and a port and monopolizes a thread. Each caching device may be considered to have a separate service instance. If the client needs to obtain data from the cache device, the client needs to read the data by taking the large file block as a unit, so that the read-write access of the whole large file block is positioned on the service instance. In addition, the access of different clients to the large file block must be serial, and the access of the same client to the large file block is also serial.
In addition, the current cache system has no method for data migration between different cache devices, so that a situation that some cache devices store a large amount of hot spot data and some cache devices store a small amount of hot spot data occurs. This further makes the cache device storing a large amount of hot data hit by the data read request of the client frequently, and since the access to the data is serial, the load of the cache device is large, even exceeds the maximum load of the cache device, so that the performance of the cache device is seriously degraded, and the overall performance of the cache system in which the cache device is located is further reduced. The cache device storing a small amount of hot spot data is in an idle state most of the time, that is, the probability that the cache device is hit by a data reading request of a client is extremely low, and resources of the cache device are seriously wasted. Therefore, it can be seen that, in the existing cache system, the loads borne by different cache devices are unbalanced in the actual operation process, which causes the performance of the cache device with a high load to be seriously reduced, and the resources of the cache device with a low load to be seriously wasted, so that the overall performance of the cache system is not fully exerted, and the performance of the cache system is also reduced.
Disclosure of Invention
Embodiments of the present invention provide a method and an apparatus for balancing cache loads, so as to implement data migration between different cache devices, so that loads borne by different cache devices in an actual operation process are relatively balanced, and further, the overall performance of a cache system can be fully exerted. The specific technical scheme is as follows:
in a first aspect, a method for cache load balancing includes:
obtaining a first load parameter of a first cache disk group and a second load parameter of a second cache disk group;
when the first load parameter is larger than the second load parameter, determining a first number of file block groups to be migrated according to the difference between the first load parameter and the second load parameter;
and migrating at least part of the file block groups stored in the first cache disk group to the second cache disk group according to the first number.
With reference to the first aspect, in some optional embodiments, the first load parameter and the second load parameter are both storage capacity parameters, or the first load parameter and the second load parameter are both bandwidth parameters.
With reference to the first aspect, in some optional embodiments, the cache device includes a plurality of cache disk groups, where the plurality of cache disk groups includes the first cache disk group and a second cache disk group, the first cache disk group is a cache disk group with a largest load parameter in the plurality of cache disk groups, and the second disk cache group is a cache disk group with a smallest load parameter in the plurality of cache disk groups.
Optionally, in some optional embodiments, when the first load parameter is greater than the second load parameter, determining the first number of file block groups to be migrated according to a difference between the first load parameter and the second load parameter includes:
and when the first load parameter is greater than the second load parameter and the difference between the first load parameter and the second load parameter is greater than a preset difference, determining a first number of file block groups to be migrated according to the difference between the first load parameter and the second load parameter.
With reference to the first aspect, in some optional embodiments, the determining a first number of file block groups to be migrated according to a difference between the first load parameter and the second load parameter includes:
and determining a first number of file block groups to be migrated according to the difference between the first load parameter and the second load parameter and according to a first total load capacity of the first cache disk group and a second total load capacity of the second cache disk group.
With reference to the previous embodiment, in some optional embodiments, the determining, according to the difference between the first load parameter and the second load parameter, and according to the first total load capacity of the first cache disk group and the second total load capacity of the second cache disk group, the first number of file block groups to be migrated includes:
according to the formula:
calculating a first number x of file block groups to be migrated, wherein n is1Is the first total load capacity, u1Is the used amount of the first total load capacity, n2Is the second total load capacity, u2Is the used amount of the second total load capacity.
Optionally, in some optional embodiments, the determining, according to the difference between the first load parameter and the second load parameter, and according to the first total load capacity of the first cache disk group and the second total load capacity of the second cache disk group, the first number of file block groups to be migrated includes:
according to the formula:
calculating a first number x of file block groups to be migrated, wherein n is1Is the first total load capacity, u1Is the used amount of the first total load capacity, n2Is the second total load capacity, u2Is the used amount of the second total load capacity, and p is a preset ratio.
With reference to the first aspect, in some optional embodiments, the migrating at least a portion of file chunk groups saved in the first cache disk group to the second cache disk group according to the first number includes:
and judging whether the first number is larger than a preset number, if so, migrating the preset number of file block groups stored in the first cache disk group to the second cache disk group, otherwise, migrating the first number of file block groups stored in the first cache disk group to the second cache disk group.
In some optional embodiments, in combination with the previous embodiment, the migrating the preset number of file block groups saved in the first cache disk group to the second cache disk group includes: searching the preset number of file block groups stored in the first cache disk group in a non-cache disk group; storing the searched file block groups of the preset number to the second cache disk group; deleting the preset number of file block groups stored in the first cache disk group;
and/or the presence of a gas in the gas,
the migrating the first number of file block groups saved in the first cache disk group to the second cache disk group includes: searching the first number of file block groups stored in the first cache disk group in a non-cache disk group; storing the searched file block groups of the first number to the second cache disk group; deleting the first number of file block groups saved in the first cache disk group.
In a second aspect, a cache load balancing apparatus includes: the system comprises a load parameter obtaining unit, a first quantity determining unit and a transferring unit;
the load parameter obtaining unit is configured to obtain a first load parameter of a first cache disk group and a second load parameter of a second cache disk group;
the first number determining unit is configured to determine a first number of file block groups to be migrated according to a difference between the first load parameter and the second load parameter when the first load parameter is greater than the second load parameter;
the migration unit is configured to perform migration of at least a part of the file block groups saved in the first cache disk group to the second cache disk group according to the first number.
According to the cache load balancing method and device provided by the embodiment of the invention, a first load parameter of a first cache disk group and a second load parameter of a second cache disk group are obtained; when the first load parameter is larger than the second load parameter, determining a first number of file block groups to be migrated according to the difference between the first load parameter and the second load parameter; and migrating at least part of the file block groups stored in the first cache disk group to the second cache disk group according to the first number. With reference to the first aspect, in some optional embodiments, the first load parameter and the second load parameter are both storage capacity parameters, or the first load parameter and the second load parameter are both bandwidth parameters. Therefore, the invention can realize data migration among different cache devices, so that the loads born by the different cache devices in the actual operation process are more balanced, and further the overall performance of the cache system can be fully exerted. Of course, it is not necessary for any product or method of practicing the invention to achieve all of the above-described advantages at the same time.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart illustrating a client requesting hot data from a cache device according to the present invention;
fig. 2 is another flowchart of a client requesting hot data from a cache device according to the present invention;
fig. 3 is a flowchart of a cache load balancing method according to the present invention;
fig. 4 is a schematic structural diagram of a cache load balancing apparatus according to the present invention;
fig. 5 is a schematic structural diagram of an apparatus provided in the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Since some data are frequently accessed data in an actual system, the frequently accessed data may be referred to as hot spot data. For hot spot data, the hot spot data is usually stored in a cache device in a cache system at present, and the cache device has a high data reading and writing speed, and can quickly provide data for other devices or systems so as to improve the speed of acquiring the data by the other devices or systems.
Specifically, the storage system with the cache system may include: the system comprises client equipment, a data server, a cache server, cache equipment, an index server and non-cache equipment, wherein the cache equipment can be memory, solid-state storage disks and other storage equipment with high data read-write speed. The non-cache device may be a mechanical storage disk or the like that is not a cache device. The data read-write speed of the non-cache device is generally lower than that of the cache device, for example, the read-write speed of the mechanical storage disk is generally lower than that of the memory and the solid-state storage disk. Wherein, the cache system can include: a cache server and a cache device.
The client device can be various electronic devices such as a computer, a mobile phone, a tablet computer, a wearable device and the like.
The data server may communicate with the client device via an SDK (software development kit).
The cache server may store therein index information of data stored in the cache device, and when the data is stored in units of data blocks, the index information stored in the cache server may be: (file _ id, block _ id) > (disk _ id, block _ id). Wherein, the file _ ID in (file _ ID, block _ ID) is the file ID, and the block _ ID in (file _ ID, block _ ID) is the data block ID. Disk _ ID in (disk _ ID, block _ ID) is disk ID of the cache device, and block _ ID in (disk _ ID, block _ ID) is disk block ID.
Wherein the storage space of one disk block is the same as the data volume of one complete data block.
Whether each data block of each file is stored in the cache device and which disk block of which disk of the cache device is stored can be determined through the index information.
Similar to the cache server, the index server may store index information of data stored in the non-cache device, and when the data is stored in units of data blocks, the index server may store the index information of the data stored in the index server as follows: (file _ id, block _ id) > (disk _ id, block _ id). Wherein, the file _ ID in (file _ ID, block _ ID) is the file ID, and the block _ ID in (file _ ID, block _ ID) is the data block ID. Disk _ ID in (disk _ ID, block _ ID) is disk ID of the non-cache device, and block _ ID in (disk _ ID, block _ ID) is disk block ID.
Whether each data block of each file is stored in the non-cache device and which disk block of which disk of the non-cache device is stored can be determined through the index information.
For ease of understanding, a data reading process of a storage system with a cache system is disclosed below:
when the data requested to be read by the client device is stored in the cache device, the data reading process is as shown in fig. 1, and includes:
s1, the client device sends a data access request to the data server;
s2, the data server sends the data access request to the cache server;
s3, the cache server determines whether the data requested by the client device is stored in the cache device, if yes, S4 is executed;
s4, the cache server sends the storage address of the data requested by the client device in the cache device to the data server;
s5, after receiving the storage address returned by the cache server, the data server obtains the data requested by the client device from the storage address of the cache device;
and S6, the data server returns the data requested by the client device to the client device.
When the data requested to be read by the client device is not stored in the cache device, the data reading process is as shown in fig. 2, and includes:
s1, the client device sends a data access request to the data server;
s2, the data server sends the data access request to the cache server;
s3, the cache server determines whether the data requested by the client device is stored in the cache device, if not, the step S7 is executed;
s7, the cache server sends a notice that the data requested by the client device is not stored in the cache device to the data server;
s8, after receiving the notification that the data requested by the client device is not stored in the cache device and returned by the cache server, the data server sends the data access request to the index server;
s9, the index server sends the storage address of the data requested by the client device in the non-cache device to the data server;
s10, after receiving the storage address returned by the index server, the data server obtains the data requested by the client device from the storage address of the non-cache device;
and S11, the data server returns the data requested by the client device to the client device.
The inventor researches and discovers that under the processing flow, when a client requests data from a cache system, the cache system returns the data requested by the client to the client as long as the requested data is hot spot data cached in the cache system. When a large amount of hot data which are continuously requested to be accessed by a client are concentrated on one disk, due to the fact that the input/output performance of the disk is limited and the network bandwidth of a network card is limited, the output of the hot data is limited, the data transmission performance of the disk is seriously affected, the capacity of a cache system for rapidly providing the hot data is further reduced, and the invention provides the following scheme for solving the problems.
As shown in fig. 3, the present invention provides a cache load balancing method, which may include:
s100, obtaining a first load parameter of a first cache disk group and a second load parameter of a second cache disk group;
it should be understood that the executing subject of the method of the present invention may be a device communicatively connected to both the first cache disk group and the second cache disk group, for example, may be a cache server, and the present invention is not limited thereto.
It should be understood that the cache apparatus mentioned herein may include a plurality of cache disk groups, and the plurality of cache disk groups may include the first cache disk group and the second cache disk group, where the first cache disk group may be a cache disk group with a largest load parameter among the plurality of cache disk groups, and the second cache disk group may be a cache disk group with a smallest load parameter among the plurality of cache disk groups.
Alternatively, the first cache disk group and the second cache disk group may be pre-divided cache disk groups. For example, a plurality of cache disks included in one cache device may be divided into a plurality of cache disk groups, and one cache disk group may include at least one cache disk. In this way, the data grouping can be performed on the cache data to be stored in the cache disk group. The current data processing method generally uses a complete data file as an object, and stores the complete data file determined as hot-point data in a cache device, or divides the complete data file into a plurality of small file blocks, and stores the file blocks determined as hot-point data in the cache device, and neither method groups data. The method provided by the invention can divide the small file blocks into a plurality of file block groups, wherein one file block group comprises a plurality of file blocks. File blocks divided into the same file block group may be stored in the same caching device, e.g. may be stored in the same set of caching disks in the same caching device.
The first cache disk group and the second cache disk group may be connected to the same cache server, or may be connected to different cache servers, which is not limited in the present invention.
It should be understood that the role of the cache disk group is to store the cache data and provide the client with a reading service of the cache data when the client needs to access the cache data. Therefore, for the cache disk group, it is generally desirable that the larger the storage capacity of one cache disk group is, the better the speed of providing the cache data to the client is. In other words, in the actual engineering, two indexes, namely the storage capacity and the bandwidth, are more concerned, the storage capacity may refer to the current available storage capacity of the cache disk group, and the bandwidth may refer to the speed at which the cache disk group provides the cache data to the client, which is not limited in the present invention.
Optionally, the first load parameter and the second load parameter may be both storage capacity parameters, or the first load parameter and the second load parameter may be both bandwidth parameters. Optionally, the storage capacity parameter may include: at least one of a storage capacity usage ratio, a storage capacity usage amount, a storage capacity non-usage ratio, and a storage capacity non-usage amount, and the bandwidth parameter may include: at least one of bandwidth usage duty, bandwidth usage amount, bandwidth unused duty, and bandwidth unused amount.
Optionally, the first load parameter and the second load parameter are both parameters for describing storage capacities of the first cache disk group and the second cache disk group, or parameters for describing data reading speeds of the first cache disk group and the second cache disk group.
Optionally, the first load parameter and the second load parameter may keep the parameter types consistent. For example, if the first load parameter is the storage capacity usage percentage, the second load parameter may be the storage capacity usage percentage; if the first load parameter is bandwidth usage, the second load parameter may also be bandwidth usage. If the first load parameter is a combination of a plurality of parameters, the second load parameter may also be a combination of the plurality of parameters, which is not limited in the present invention.
It should be understood that the first load parameter and the second load parameter should be kept consistent in parameter type, so that the first load parameter and the second load parameter are comparable. Further, when data migration is performed subsequently, the result obtained by the step of comparing the difference between the first load parameter and the second load parameter has a higher reference value, and the invention is not limited to this.
S200, when the first load parameter is larger than the second load parameter, determining a first number of file block groups to be migrated according to the difference between the first load parameter and the second load parameter;
it should be understood that if the first load parameter and the second load parameter are storage capacity parameters, by comparing the first load parameter with the second load parameter, it can be obtained which of the first cache disk group and the second cache disk group has a larger current storage capacity and which has a smaller current storage capacity, so that part of cache data of the cache disk group with the smaller current storage capacity is subsequently migrated to the cache disk group with the larger current storage capacity, thereby balancing the cache data in the first cache disk group and the second cache disk group, and also balancing the current storage capacities of the first cache disk group and the second cache disk group.
It should be understood that if the first load parameter and the second load parameter are both bandwidth parameters, by comparing the first load parameter with the second load parameter, it can be obtained which of the first cache disk group and the second cache disk group has a larger current available bandwidth and which of the first cache disk group and the second cache disk group has a smaller current available bandwidth, so that part of cache data of the cache disk group having the smaller current available bandwidth is subsequently migrated to the cache disk group having the larger current available bandwidth, thereby balancing the cache data in the first cache disk group and the second cache disk group, and also balancing the bandwidths of the first cache disk group and the second cache disk group.
Alternatively, the first load parameter and the second load parameter may both be a combination of a storage capacity parameter and a bandwidth parameter, that is, the storage capacity parameter and the bandwidth parameter are comprehensively referred to determine to which cache data is migrated. In this way, the migration results obtained are more reasonable. That is, from the two aspects of the current storage capacity and bandwidth, the cached data of the first cache disk group and the second cache disk group are more balanced, which is not limited by the present invention.
Optionally, the step S200 may include:
and when the first load parameter is greater than the second load parameter and the difference between the first load parameter and the second load parameter is greater than a preset difference, determining a first number of file block groups to be migrated according to the difference between the first load parameter and the second load parameter.
It should be understood that, the present invention may balance the data in the first cache disk group and the second cache disk group, but if the difference between the first load parameter and the second load parameter is not very large, the cache data in the first cache disk group and the second cache disk group may not be balanced. Therefore, a preset gap may be set in advance, if the gap between the first load parameter and the second load parameter is greater than the preset gap, the first number of the file block groups to be migrated may be determined, otherwise, the process may be ended to execute other processes, and the present invention does not limit other processes.
It should be understood that, when the first load parameter is greater than the second load parameter and the difference between the first load parameter and the second load parameter is greater than the preset difference, it indicates that a part of cache data in one of the first cache disk group and the second cache disk group may be migrated to the other cache disk group, but how much cache data is migrated specifically may enable data of the first cache disk group and the second cache disk group to be compared and balanced, and a further determination is required, that is, a first number of file block groups to be migrated needs to be determined.
Optionally, the first number may represent the number of file blocks included in the file block group to be migrated, which is not limited in the present invention.
Optionally, the difference between the first load parameter and the second load parameter may be an absolute value of a difference between numerical values of the two parameters, which is not limited in the present invention.
Optionally, the determining the first number of the file block groups to be migrated according to the difference between the first load parameter and the second load parameter in step S200 may include:
and determining a first number of file block groups to be migrated according to the difference between the first load parameter and the second load parameter and according to a first total load capacity of the first cache disk group and a second total load capacity of the second cache disk group.
It should be understood that, for any one cache disk group, after the cache disk group is divided, the maximum data amount that it can store, that is, the total load capacity thereof may be fixed. Therefore, the first total load capacity and the second total load capacity may be parameters already obtained after the corresponding cache disk groups are divided, and the cache server may obtain not only the first total load capacity and the second total load capacity but also the total load capacity of the cache disk groups of other cache devices to which the cache server is communicatively connected. Of course, when the first load parameter is obtained, the first total load capacity may be obtained together, or the first load parameter is obtained first and then the first total load capacity is obtained, or the first total load capacity is obtained first and then the first load parameter is obtained.
It should be appreciated that the first total load capacity may account to some extent for the storage capacity of the first set of cache disks. The second total load capacity may also account for the storage capacity of the second set of cache disks to some extent. And determining the first number of the file block groups to be migrated according to the storage capacities of the first cache disk group and the second cache disk group, so that the method is scientific and can reasonably balance the cache data of the first cache disk group and the second cache disk group.
Optionally, the determining, according to the difference between the first load parameter and the second load parameter, and according to the first total load capacity of the first cache disk group and the second total load capacity of the second cache disk group, the first number of file block groups to be migrated may include:
according to the formula:
calculating a first number x of file block groups to be migrated, wherein n is1Is the first total load capacity, u1Is the used amount of the first total load capacity, n2Is the second total load capacity, u2Is the used amount of the second total load capacity.
It should be understood that the calculation of the first number by the above formula is only an alternative embodiment, and any embodiment that is modified appropriately based on the above formula or any embodiment that obtains the first number by calculating the first total load capacity and the second total load capacity belongs to the protection scope of the present invention, and the present invention is not limited to this.
Optionally, the first number calculated by the above formula may be the total number of file blocks included in the file block group to be migrated, that is, all of the file blocks to be migrated. In the subsequent steps, all the file blocks to be migrated may be migrated all at one time, or may be migrated in multiple batches, which is not limited in the present invention.
The following will provide an embodiment of the set of file blocks to be migrated that are to be migrated in multiple batches: optionally, the determining, according to the difference between the first load parameter and the second load parameter, and according to the first total load capacity of the first cache disk group and the second total load capacity of the second cache disk group, the first number of file block groups to be migrated may include:
according to the formula:
calculating a first number x of file block groups to be migrated, wherein n is1Is the first total load capacity, u1Is the used amount of the first total load capacity, n2Is the second total load capacity, u2Is the used amount of the second total load capacity, and p is a preset ratio.
It should be understood that the first number in this embodiment may be the number of file blocks that can be migrated at a single time in the process of migrating the desired group of file blocks in multiple batches. The number of file blocks to be migrated at each time is calculated according to a preset ratio, and the preset ratio can be set according to actual needs, which is not limited by the present invention.
Optionally, the value range of the preset ratio may be from 0% to 100%, which is not limited in the present invention.
It should be understood that, by migrating the file block groups to be migrated in multiple batches, it is reasonable to avoid that the communication in other processes is blocked for a long time and the processes in other processes are seriously affected because the process of migrating the file block groups continuously occupies communication resources for a long time. For example, after performing a process of migrating a part of file blocks of a file block group, the device executing the method of the present invention may perform at least one other process (a process of not migrating a file block group) and then return to perform a process of migrating a part of file blocks of the file block group, and so on, which is repeatedly performed, but the present invention does not limit other processes.
It should be understood that there may be two schemes for migrating groups of file blocks to be migrated by dividing into multiple batches:
first, n is as defined above1、u1、n2And u2After the equal parameters are migratedAll file blocks of the file block group to be migrated are not updated and are kept unchanged. That is, the number of file blocks per migration may be the same across multiple batches.
Second, in multiple batches, after each migration is completed, n is1、u1、n2And u2All the parameters are updated, i.e. n is above1、u1、n2And u2The isoparametric values differ from the previous values each time the first quantity is calculated. Therefore, the first number calculated before each migration may also be different. In this way, "over head" of migration may be avoided, that is, as each migration is completed, the difference between the amount of the cache data stored in the first cache disk group and the amount of the cache data stored in the second cache disk group is gradually decreased, and at this time, the amount of the file blocks migrated at each time may be gradually decreased, that is, the first amount may exhibit a gradually decreasing trend.
S300, according to the first number, migrating at least part of file block groups stored in the first cache disk group to the second cache disk group.
It should be understood that the first number is generally smaller than the number of file blocks stored in the first cache disk group, so that whether all file block groups of the first number are migrated to the second cache disk group, or part of file blocks of the first number are migrated to the second cache disk group, it can be understood that at least part of file block groups stored in the first cache disk group are migrated to the second cache disk group.
Optionally, the present invention implements balancing cache data of the first cache disk group and the second cache disk group. Therefore, the number of the file blocks stored in the first cache disk group is not generally equal to or greater than the number of the file blocks stored in the first cache disk group, otherwise, after the file blocks stored in the first cache disk group are migrated to the second cache disk group, the cache data cached by the first cache disk group may be empty, and the effect of balancing the cache data of the first cache disk group and the cache data of the second cache disk group is not achieved.
Optionally, the step S300 may include:
and judging whether the first number is larger than a preset number, if so, migrating the preset number of file block groups stored in the first cache disk group to the second cache disk group, otherwise, migrating the first number of file block groups stored in the first cache disk group to the second cache disk group.
It should be appreciated that, as a result of migrating file block groups, the first number of file block groups saved in the first cache disk group is searched from non-cache disk groups in relation to subsequent steps; storing the searched file block groups of the first number to the second cache disk group; deleting the first number of file block groups saved in the first cache disk group. The speed of searching and obtaining the first number of file block groups from the non-cache disk group is limited by the data reading performance and the communication bandwidth of the non-cache disk group; the speed of saving the first number of file block groups to the second cache disk group is limited by the data writing performance and the communication bandwidth of the second cache disk group. An upper limit, referred to herein as a preset number, may be set for each migration.
It is to be understood that the predetermined amount may be a value determined by experiment, and the present invention is not limited thereto.
Optionally, when the first number obtained by the current calculation is greater than or equal to the preset number, it is determined that the number of file blocks included in the file block group migrated this time is the preset number, otherwise, it is determined that the number of file blocks included in the file block group migrated this time is the first number, which is not limited in the present invention.
Optionally, the migrating the preset number of file block groups stored in the first cache disk group to the second cache disk group may include: searching the preset number of file block groups stored in the first cache disk group in a non-cache disk group; storing the searched file block groups of the preset number to the second cache disk group; deleting the preset number of file block groups stored in the first cache disk group;
and/or the presence of a gas in the gas,
the migrating the first number of file block groups saved in the first cache disk group to the second cache disk group includes: searching the first number of file block groups stored in the first cache disk group in a non-cache disk group; storing the searched file block groups of the first number to the second cache disk group; deleting the first number of file block groups saved in the first cache disk group.
As shown in fig. 4, the present invention provides a cache load balancing apparatus, including: a load parameter obtaining unit 100, a first number determining unit 200, and a migration unit 300;
the load parameter obtaining unit 100 is configured to obtain a first load parameter of a first cache disk group and a second load parameter of a second cache disk group;
the first number determining unit 200 is configured to determine, when the first load parameter is greater than the second load parameter, a first number of file block groups to be migrated according to a difference between the first load parameter and the second load parameter;
the migration unit 300 is configured to perform migration of at least a part of the file block groups saved in the first cache disk group to the second cache disk group according to the first number.
Optionally, in some optional embodiments, the first quantity determining unit 200 includes: a first subunit;
the first subunit is configured to determine a first number of file block groups to be migrated according to a difference between the first load parameter and the second load parameter when the first load parameter is greater than the second load parameter and a difference between the first load parameter and the second load parameter is greater than a preset difference.
Optionally, in some optional embodiments, the first quantity determining unit 200 includes: a second subunit;
the second subunit is configured to execute a gap according to the first load parameter and the second load parameter, and determine a first number of file block groups to be migrated according to a first total load capacity of the first cache disk group and a second total load capacity of the second cache disk group.
Optionally, in some optional embodiments, the second subunit includes: a third subunit;
the third subunit configured to perform, according to a formula:
calculating a first number x of file block groups to be migrated, wherein n is1Is the first total load capacity, u1Is the used amount of the first total load capacity, n2Is the second total load capacity, u2Is the used amount of the second total load capacity, and p is a preset ratio.
Optionally, in some optional embodiments, the migration unit 300 includes: a fourth subunit, a fifth subunit, and a sixth subunit;
the fourth subunit is configured to perform judgment on whether the first number is greater than a preset number, if so, the fifth subunit is triggered, otherwise, the sixth subunit is triggered;
the fifth subunit is configured to perform migration of the preset number of file block groups saved in the first cache disk group to the second cache disk group;
the sixth subunit is configured to perform migration of the first number of file block groups saved in the first cache disk group to the second cache disk group.
Optionally, in some optional embodiments, the fifth sub-unit includes: the device comprises a first searching subunit, a first saving subunit and a first deleting subunit;
the first searching subunit is configured to perform searching for the preset number of file block groups saved in the first cache disk group in a non-cache disk group;
the first saving subunit is configured to perform saving of the found preset number of file block groups to the second cache disk group;
the first deletion subunit is configured to perform deletion of the preset number of file block groups saved in the first cache disk group;
and/or the presence of a gas in the gas,
the sixth sub-unit includes: the second searching subunit, the second saving subunit and the second deleting subunit;
the second searching subunit is configured to perform searching for the first number of file block groups saved in the first cache disk group in a non-cache disk group;
the second saving subunit is configured to perform saving the found first number of file block groups to the second cache disk group;
the second deletion subunit is configured to perform deletion of the first number of file group groups saved in the first cache disk group.
The cache load balancing device comprises a processor and a memory, wherein the load parameter obtaining unit 100, the first quantity determining unit 200, the migrating unit 300 and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.
The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. One or more than one kernel can be set, and data migration can be performed among different cache devices by adjusting kernel parameters, so that loads borne by the different cache devices in the actual operation process are relatively balanced, and further the overall performance of the cache system can be fully exerted.
An embodiment of the present invention provides a storage medium, on which a program is stored, where the program, when executed by a processor, implements the cache load balancing method.
The embodiment of the invention provides a processor, which is used for running a program, wherein the cache load balancing method is executed when the program runs.
As shown in fig. 5, an embodiment of the present invention provides an apparatus 70, where the apparatus 70 includes at least one processor 701, and at least one memory 702 and a bus 703 connected to the processor 701; the processor 701 and the memory 702 complete mutual communication through a bus 703; the processor 701 is configured to call program instructions in the memory 702 to perform the above-described cache load balancing method. The device herein may be a server, a PC, a PAD, a mobile phone, etc.
The present application also provides a computer program product adapted to execute a program initialized with the steps comprised by the above described cache load balancing method when executed on a data processing device.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a device includes one or more processors (CPUs), memory, and a bus. The device may also include input/output interfaces, network interfaces, and the like.
The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include transitory computer readable media (transmyedia) such as modulated data signals and carrier waves.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A method for cache load balancing, comprising:
obtaining a first load parameter of a first cache disk group and a second load parameter of a second cache disk group;
when the first load parameter is larger than the second load parameter, determining a first number of file block groups to be migrated according to the difference between the first load parameter and the second load parameter;
and migrating at least part of the file block groups stored in the first cache disk group to the second cache disk group according to the first number.
2. The method of claim 1, wherein the first load parameter and the second load parameter are both storage capacity parameters or wherein the first load parameter and the second load parameter are both bandwidth parameters.
3. The method according to claim 1, wherein a cache device comprises a plurality of cache disk groups, the plurality of cache disk groups comprises the first cache disk group and a second cache disk group, the first cache disk group is a cache disk group with a largest load parameter among the plurality of cache disk groups, and the second disk cache group is a cache disk group with a smallest load parameter among the plurality of cache disk groups.
4. The method according to claim 1 or 3, wherein the determining a first number of file block groups to be migrated according to a difference between the first load parameter and the second load parameter when the first load parameter is greater than the second load parameter comprises:
and when the first load parameter is greater than the second load parameter and the difference between the first load parameter and the second load parameter is greater than a preset difference, determining a first number of file block groups to be migrated according to the difference between the first load parameter and the second load parameter.
5. The method of claim 1, wherein determining the first number of file block groups to be migrated according to the difference between the first load parameter and the second load parameter comprises:
and determining a first number of file block groups to be migrated according to the difference between the first load parameter and the second load parameter and according to a first total load capacity of the first cache disk group and a second total load capacity of the second cache disk group.
6. The method of claim 5, wherein determining the first number of file block groups to be migrated according to the difference between the first load parameter and the second load parameter and according to the first total load capacity of the first cache disk group and the second total load capacity of the second cache disk group comprises:
according to the formula:
calculating a first number x of file block groups to be migrated, wherein n is1Is the first total load capacity, u1Is the used amount of the first total load capacity, n2Is the second total load capacity, u2Is the used amount of the second total load capacity.
7. The method of claim 5, wherein determining the first number of file block groups to be migrated according to the difference between the first load parameter and the second load parameter and according to the first total load capacity of the first cache disk group and the second total load capacity of the second cache disk group comprises:
according to the formula:
calculating a first number x of file block groups to be migrated, wherein n is1Is the first total load capacity, u1Is the used amount of the first total load capacity, n2Is the second total load capacity, u2Is the used amount of the second total load capacity, and p is a preset ratio.
8. The method of claim 1, wherein migrating at least some of the file chunk sets saved in the first cache disk set to the second cache disk set according to the first number comprises:
and judging whether the first number is larger than a preset number, if so, migrating the preset number of file block groups stored in the first cache disk group to the second cache disk group, otherwise, migrating the first number of file block groups stored in the first cache disk group to the second cache disk group.
9. The method of claim 8,
the migrating the preset number of file block groups stored in the first cache disk group to the second cache disk group includes: searching the preset number of file block groups stored in the first cache disk group in a non-cache disk group; storing the searched file block groups of the preset number to the second cache disk group; deleting the preset number of file block groups stored in the first cache disk group;
and/or the presence of a gas in the gas,
the migrating the first number of file block groups saved in the first cache disk group to the second cache disk group includes: searching the first number of file block groups stored in the first cache disk group in a non-cache disk group; storing the searched file block groups of the first number to the second cache disk group; deleting the first number of file block groups saved in the first cache disk group.
10. A cache load balancing apparatus, comprising: the system comprises a load parameter obtaining unit, a first quantity determining unit and a transferring unit;
the load parameter obtaining unit is configured to obtain a first load parameter of a first cache disk group and a second load parameter of a second cache disk group;
the first number determining unit is configured to determine a first number of file block groups to be migrated according to a difference between the first load parameter and the second load parameter when the first load parameter is greater than the second load parameter;
the migration unit is configured to perform migration of at least a part of the file block groups saved in the first cache disk group to the second cache disk group according to the first number.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010995223.1A CN114253456A (en) | 2020-09-21 | 2020-09-21 | Cache load balancing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010995223.1A CN114253456A (en) | 2020-09-21 | 2020-09-21 | Cache load balancing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114253456A true CN114253456A (en) | 2022-03-29 |
Family
ID=80788284
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010995223.1A Pending CN114253456A (en) | 2020-09-21 | 2020-09-21 | Cache load balancing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114253456A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115550080A (en) * | 2022-09-19 | 2022-12-30 | 苏州浪潮智能科技有限公司 | Network card, data transmission system, method, computer equipment and storage medium |
| CN117234435A (en) * | 2023-11-14 | 2023-12-15 | 苏州元脑智能科技有限公司 | File storage method and device |
-
2020
- 2020-09-21 CN CN202010995223.1A patent/CN114253456A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115550080A (en) * | 2022-09-19 | 2022-12-30 | 苏州浪潮智能科技有限公司 | Network card, data transmission system, method, computer equipment and storage medium |
| CN117234435A (en) * | 2023-11-14 | 2023-12-15 | 苏州元脑智能科技有限公司 | File storage method and device |
| CN117234435B (en) * | 2023-11-14 | 2024-02-20 | 苏州元脑智能科技有限公司 | File storage method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11734271B2 (en) | Data query method, apparatus and device | |
| US9454407B2 (en) | Service resource allocation | |
| US10356150B1 (en) | Automated repartitioning of streaming data | |
| CN106407207B (en) | Real-time newly-added data updating method and device | |
| US20180167461A1 (en) | Method and apparatus for load balancing | |
| CN109981702B (en) | File storage method and system | |
| CN103747047A (en) | CDN file storage method, file distribution control center and system thereof | |
| CN108900626B (en) | Data storage method, device and system in cloud environment | |
| CN109240946A (en) | The multi-level buffer method and terminal device of data | |
| CN106713028B (en) | Service degradation method and device and distributed task scheduling system | |
| CN114253456A (en) | Cache load balancing method and device | |
| CN106534308A (en) | Method and device for solving data block access hotspot problem in distributed storage system | |
| CN111737168A (en) | Cache system, cache processing method, device, equipment and medium | |
| CN110737717A (en) | database migration method and device | |
| CN115421924A (en) | Memory allocation method, device and equipment | |
| WO2020094064A1 (en) | Performance optimization method, device, apparatus, and computer readable storage medium | |
| CN107423425A (en) | A kind of data quick storage and querying method to K/V forms | |
| CN111259012B (en) | Data homogenizing method, device, computer equipment and storage medium | |
| CN111770150A (en) | Access flow control method and device and electronic equipment | |
| CN110851327A (en) | Hardware state information acquisition method and device, terminal and readable storage medium | |
| CN111506254A (en) | Distributed storage system and management method and device thereof | |
| CN115033551A (en) | Database migration method and device, electronic equipment and storage medium | |
| CN110719306B (en) | Network request limiting method, computer equipment and storage medium | |
| CN112114962A (en) | Memory allocation method and device | |
| CN108762679B (en) | Method for combining online DDP (distributed data processing) and offline DDP (distributed data processing) and related device thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |