CN114064570A - Metadata load balancing method and device and electronic equipment - Google Patents
Metadata load balancing method and device and electronic equipment Download PDFInfo
- Publication number
- CN114064570A CN114064570A CN202111401376.XA CN202111401376A CN114064570A CN 114064570 A CN114064570 A CN 114064570A CN 202111401376 A CN202111401376 A CN 202111401376A CN 114064570 A CN114064570 A CN 114064570A
- Authority
- CN
- China
- Prior art keywords
- shared directory
- metadata
- load parameter
- metadata information
- shared
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/119—Details of migration of file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a metadata load balancing method, a metadata load balancing device and electronic equipment, which are applied to a server cluster consisting of a plurality of metadata servers, wherein the method comprises the following steps: creating a shared directory in each metadata server to store metadata information through the shared directory; counting load parameters corresponding to each shared directory, and determining a difference value between the maximum load parameter and the minimum load parameter; and if the difference value reaches a preset threshold value, transferring part of metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter. According to the technical scheme provided by the invention, the load parameter difference between the maximum load metadata server and the minimum load metadata server is taken as the measure, the metadata information is migrated only when the difference meets the condition of a preset threshold value, the migration times in the metadata management process are reduced, and the influence of interrupting the user operation service is reduced.
Description
Technical Field
The invention relates to the field of distributed file storage systems, in particular to a metadata load balancing method and device and electronic equipment.
Background
With the development of technologies such as high-definition video, image processing, video monitoring and the like, the data capacity of users is becoming larger and larger, the requirements of the users on the read-write performance of stored data are synchronously increasing, and distributed file storage systems are produced. Metadata management, namely management of metadata information, is an important link of a distributed file storage system, manages all data distribution information of the distributed file storage system, is a core component of the distributed file storage system, has high management efficiency, and directly determines the performance of the distributed file storage system, so that efficient metadata management is very important, and a large number of shared directory files cause excessive load of a metadata server, thereby seriously affecting the metadata management efficiency.
In the prior art, all metadata information of a directory structure is stored in a metadata server, and as directory files under a directory are continuously increased, whether loads among a plurality of metadata servers are consistent or not is judged in a real-time load calculation mode, and if the loads are not consistent, the metadata information in the directory is migrated in real time, so that the loads among the plurality of metadata servers are consistent. However, when the metadata is migrated, the file or directory in the actual physical storage address cannot be written, which may cause interruption of user operations, and migrating the metadata information in real time may seriously affect the efficiency of the user in reading and writing the directory or file. Therefore, how to reduce the influence of user operation interruption in the metadata management process is an urgent problem to be solved.
Disclosure of Invention
In view of this, embodiments of the present invention provide a metadata load balancing method, apparatus, and electronic device, so as to reduce the influence of user operation interruption during metadata management.
According to a first aspect, the present invention provides a metadata load balancing method applied in a server cluster composed of a plurality of metadata servers, the method comprising: creating a shared directory in each of the metadata servers to store metadata information through the shared directory; counting load parameters corresponding to the shared directories, and determining a difference value between a maximum load parameter and a minimum load parameter; and if the difference reaches a preset threshold value, transferring part of metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter.
Optionally, the migrating part of the metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter includes: and migrating part of metadata information in the first shared directory to the second shared directory until the difference value of the number of files in the first shared directory and the second shared directory is smaller than a second preset threshold value.
Optionally, the load parameter further includes a number of times of calling a metadata information file in the shared directory, and the determining a difference between the maximum load parameter and the minimum load parameter includes: calculating the weighted result of the file number and the calling times of each metadata server; calculating a difference between a maximum weighting result and a minimum weighting result as a difference between the maximum load parameter and the minimum load parameter.
Optionally, the migrating a part of metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter includes: and migrating part of metadata information in the first shared directory to the second shared directory until the difference value of the weighting results between the first shared directory and the second shared directory is smaller than a third preset threshold value.
Optionally, the method further comprises: and adding the metadata information corresponding to the newly added subdirectories into the shared directories of the metadata servers in a circulating mode.
Optionally, before migrating part of metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter, the method further includes: periodically detecting the calling times of the metadata information in all the shared directories, and acquiring a time period when the calling times are lower than a fourth preset threshold value as a migration time period; if the current time point is within the migration time period, returning to the step of migrating part of the metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter; and if the current time point is not in the migration time period, suspending the migration task until the time reaches the migration time period, and returning to the step of migrating part of the metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter.
According to a second aspect, the present invention provides a metadata load balancing apparatus applied in a server cluster composed of a plurality of metadata servers, the apparatus comprising: an initialization module, configured to create a shared directory in each metadata server, so as to store metadata information through the shared directory; the load calculation module is used for counting load parameters corresponding to the shared directories and determining a difference value between a maximum load parameter and a minimum load parameter; and the load balancing module is used for migrating part of metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter if the difference reaches a preset threshold.
Optionally, before the equalizing module, the method further includes: the time detection module is used for periodically detecting the calling times of the metadata information in all the shared directories and acquiring a time period with the calling times lower than a fourth preset threshold value as a migration time period; a migration module, configured to return the step of migrating a part of metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter if the current time point is within the migration time period; and the suspension module is used for suspending the migration task if the current time point is not in the migration time period, and returning to the step of migrating part of the metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter until the time reaches the migration time period.
According to a third aspect, an embodiment of the present invention provides an electronic device, including: a memory and a processor, the memory and the processor being communicatively coupled to each other, the memory having stored therein computer instructions, and the processor performing the method of the first aspect, or any one of the optional embodiments of the first aspect, by executing the computer instructions.
According to a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions for causing a computer to thereby perform the method of the first aspect, or any one of the optional implementation manners of the first aspect.
The technical scheme provided by the application has the following advantages:
according to the technical scheme, the shared directory is firstly created in each metadata server and used for storing metadata information of actual subdirectories or files in an external data storage space, so that a plurality of metadata servers can manage the whole directory structure in a labor-sharing mode. The load parameters of the shared directories in each metadata server are calculated in real time, and only when the maximum load parameter and the minimum load parameter exceed preset thresholds, the metadata information in the corresponding first shared directory is migrated to the second shared directory, so that the load balance between the metadata server with the maximum load and the metadata server with the minimum load is completed, the load is prevented from being adjusted in real time, the metadata migration times are reduced on the premise of ensuring the load balance of each metadata server, and the influence of interrupting the use of users is reduced.
In addition, the file quantity and the calling times of the metadata information in the shared directory are used as load parameters, and the accuracy of load balancing calculation is further improved. And when a new subdirectory needs to be added, circularly storing the newly added subdirectory files in each metadata server, so that the addition of the subdirectories is as much as possible according to the principle of average distribution, and the subsequent metadata information migration times are reduced. On the other hand, the migration time of the metadata information is set in the time period when the user accesses the directory or the file for the least times, so that the interruption influence of the metadata migration on the user operation is further reduced.
Drawings
The features and advantages of the present invention will be more clearly understood by reference to the accompanying drawings, which are illustrative and not to be construed as limiting the invention in any way, and in which:
FIG. 1 is a diagram illustrating steps of a metadata load balancing method according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating a metadata load balancing method according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a metadata load balancing apparatus according to an embodiment of the present invention;
fig. 4 is another schematic structural diagram of a metadata load balancing apparatus according to an embodiment of the present invention;
fig. 5 shows a schematic structural diagram of an electronic device in an embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings of the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without inventive step based on the embodiments of the present invention, are within the scope of the present invention.
Referring to fig. 1 and fig. 2, in an embodiment, a metadata load balancing method is applied to a server cluster composed of a plurality of metadata servers, and specifically includes the following steps:
step S101: a shared directory is created in each metadata server to store metadata information through the shared directory.
Step S102: and counting the load parameters corresponding to the shared directories, and determining the difference between the maximum load parameter and the minimum load parameter.
Step S103: and if the difference value reaches a preset threshold value, transferring part of metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter.
Specifically, in a distributed file system, an entire directory and subdirectories therein and locations where files are actually stored are referred to as storage tiers, data for describing the directory and files is referred to as metadata, and metadata information is stored on a metadata server to manage the directory structure, referred to as a metadata tier. The computing layer can access the corresponding position in the storage layer through the metadata information of the metadata layer, so that the corresponding file is read and written. The storage tier, metadata tier, and compute tier may each include multiple devices. In the embodiment of the present invention, a shared directory is created in each metadata server to manage metadata information, for example: a certain directory structure comprises 10 pieces of metadata information, each metadata server comprises a sharing directory, and the 10 pieces of metadata information are respectively dispersed in the sharing directories, so that a plurality of metadata servers can jointly manage the directory structure. In order to solve the problem of serious user interruption caused by the real-time migration of metadata information in the prior art, a preset threshold value is preset, and when the load parameter difference value between the sharing directory with the maximum load and the sharing directory with the minimum load exceeds the threshold value under an extreme condition, namely the load of the metadata server with the maximum load is much higher than that of the metadata server with the minimum load, the migration of the metadata information is carried out, so that the load balance is ensured. Generally, based on the number of files managed by the metadata server, migration is executed only when the difference reaches million levels, so that based on the steps, load balancing can be achieved on the premise that the metadata server works normally, compared with real-time load calculation and migration, the number of times of metadata information migration is greatly reduced, and the influence of interrupting a user to operate the files is also greatly reduced.
Specifically, in an embodiment, the load parameter is the number of files of the metadata information in the shared directory of each metadata server, and the step S103 specifically includes the following steps:
the method comprises the following steps: and migrating part of metadata information in the first shared directory to a second shared directory until the difference value of the number of files in the first shared directory and the second shared directory is less than a second preset threshold value. Specifically, when the number of managed files in the metadata server is too large, operations such as running, starting, calling and the like of the server are slow, and therefore, when the difference between the number of shared directories with the largest number of files in the metadata server and the number of files in the shared directory with the smallest number of files is too large (generally in the order of millions), the number of files in the two metadata servers needs to be balanced until the difference between the number of files in the shared directories in the two metadata servers is smaller than a second preset threshold, and in this embodiment, the difference between the number of files in the two shared directories is regarded as load balancing within 1000. The number of files is used as the standard of the load parameters, and the description of the load parameters is more specific, so that the accuracy of load balancing is improved. For example: the first shared directory contains 10000 pieces of metadata information, the second shared directory contains 2000 pieces of metadata information, the metadata information in the first shared directory is migrated to the second shared directory in batches, 100 pieces of metadata information are migrated each time, after 36 times of migration, the first shared directory contains 6400 pieces of files, the second shared directory contains 5600 pieces of files, the difference value is 800 and is smaller than 1000, and therefore the condition of a second preset threshold value is met, and migration is completed.
Specifically, in an embodiment, the load parameter further includes a number of times of calling the metadata information file in the shared directory, and the step S102 specifically includes the following steps:
step two: and calculating the weighted result of the file number and the calling times of each metadata server.
Step three: the difference between the maximum weighted result and the minimum weighted result is calculated as the difference between the maximum load parameter and the minimum load parameter.
Specifically, in this embodiment, the load parameter includes, in addition to the number of files, the number of calls by which each file in the shared directory is called, when a user accesses an actual file of the storage layer through the computing layer, a metadata information file managed by the metadata server needs to be called to obtain information such as an actual position and an attribute of the file in the storage layer, and when the user accesses a certain type of files many times, the metadata server frequently calls the directory and the file of the metadata information, so that the CPU and memory use pressure of the metadata server increases. Therefore, the number of metadata information files in the shared directory and the number of times of calling the metadata information directory files are comprehensively considered, and a weighting algorithm (in this embodiment, average weighting is adopted) is adopted to take a result of combining the two types of parameters as a load parameter, so that a comprehensive index for measuring the load size of the metadata server is obtained, a difference value of the load parameter obtained later is more accurate, and the accuracy of load balancing of the metadata server is further improved. For example: the number of files in the first shared directory is 2000, the number of times of the shared directory internal metadata information file being called is 3000, in this embodiment, the result obtained by the load parameter according to the weighted average algorithm is 2000 × 0.5+3000 × 0.5 ═ 2500, the number of the second shared directory metadata information files is 600, the number of times of the shared directory internal file being called is 800, the result obtained by the load parameter according to the weighted average algorithm is 600 × 0.5+800 × 0.5 ═ 700, and then the difference of the weighted results is 1800.
Specifically, in an embodiment, based on the second step and the third step, the step S103 specifically includes the following steps:
step four: and migrating part of metadata information in the first shared directory to the second shared directory until the difference value of the weighting results between the first shared directory and the second shared directory is smaller than a third preset threshold value.
Specifically, part of the metadata information in the first shared directory is migrated to the second shared directory, and the migration is stopped until the difference of the weighting results is smaller than a third preset threshold (in this embodiment, the third preset threshold is 200). Thereby further ensuring the accuracy and reliability of the load balancing operation. For example, based on the example in the second step and the third step, the difference of the weighting result between the first shared directory and the second shared directory is 1800, 50 pieces of metadata information are migrated from the first shared directory to the second shared directory each time, after the first migration, the number of files in the first shared directory is 1950, and the number of files in the second shared directory is 650, assuming that the difference of the weighting result obtained through calculation is 1700 and is not less than the third preset threshold 200, the migration needs to be continued; after 20 times of migration, the number of files in the first shared directory is 1000, the number of files in the second shared directory is 1600, the difference of the weighting results of the two shared directories is recalculated to be 180, and the migration is completed when the condition that the difference is smaller than the third preset threshold 200 is met.
Specifically, in an embodiment, a metadata load balancing method further includes the following steps:
step five: and adding the metadata information corresponding to the newly added subdirectories into the shared directories of the metadata servers in a circulating mode.
Specifically, when a user creates a new sub-directory to an actual storage location of the directory structure, the metadata information of the sub-directory is sequentially and cyclically stored in the shared directory of each different metadata server, for example: the first newly created subdirectory is stored in the metadata server 1, the next newly created subdirectory is stored in the metadata server 2, and the operation of storing metadata information is circulated among the plurality of metadata servers. Therefore, the newly-built subdirectories can be averagely distributed to a plurality of metadata servers as far as possible, and if the difference of the number of files in each subdirectory is not large, the load of each shared directory can not have the condition of overlarge difference at least in a short time, so that the migration times of metadata information are further reduced, and the influence of interrupting user operation brought by the migration of the metadata information is reduced.
Specifically, in an embodiment, before the step S103, a metadata load balancing method further includes the following steps:
step six: and periodically detecting the calling times of the metadata information in all the shared directories, and acquiring a time period with the calling times lower than a fourth preset threshold value as a migration time period.
Step seven: and if the current time point is in the migration time period, returning to the step of migrating part of the metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter.
Step eight: and if the current time point is not in the migration time period, suspending the migration task, and returning to the step of migrating part of the metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter until the time reaches the migration time period.
Specifically, in this embodiment, when the user service is normally performed, the computing layer may periodically count the number of times that the metadata information in each metadata server is entirely called (for example, detection is performed once per hour), and when the number of times that the metadata information is called is greater, the number of times that the actual data file is read or written in the storage layer is greater (the service is more frequent), and the access flow rate of the server corresponding to the server storing the actual data file is greater. Therefore, in the periodic detection process, a time period in which the number of times that the user calls the metadata information is lower than a fourth preset threshold (for example, a time period in which the number of times that the metadata information is called is counted to be lower than 200 times) is recorded, in other words, a time period in which the service is least frequently counted is taken as a migration time period, for example, 3 to 4 am. And then, the step of migrating the metadata information is carried out within the migration time period, so that the influence of service interruption on the user can be further reduced to the maximum extent, and the use experience of the user is improved.
Through the steps, according to the technical scheme provided by the application, the shared directory is firstly created in each metadata server and used for storing the metadata information of the actual subdirectories or files in the external data storage space, so that a plurality of metadata servers can manage the whole directory structure in a labor-sharing manner. The load parameters of the shared directories in each metadata server are calculated in real time, and only when the maximum load parameter and the minimum load parameter exceed preset thresholds, the metadata information in the corresponding first shared directory is migrated to the second shared directory, so that the load balance between the metadata server with the maximum load and the metadata server with the minimum load is completed, the load is prevented from being adjusted in real time, the metadata migration times are reduced on the premise of ensuring the load balance of each metadata server, and the influence of interrupting the use of users is reduced.
In addition, the file quantity and the calling times of the metadata information in the shared directory are used as load parameters, and the accuracy of load balancing calculation is further improved. And when a new subdirectory needs to be added, circularly storing the newly added subdirectory files in each metadata server, so that the addition of the subdirectories is as much as possible according to the principle of average distribution, and the subsequent metadata information migration times are reduced. On the other hand, the migration time of the metadata information is set in the time period when the user accesses the directory or the file for the least times, so that the interruption influence of the metadata migration on the user operation is further reduced.
As shown in fig. 3, this embodiment further provides a metadata load balancing apparatus, which is applied to a server cluster composed of a plurality of metadata servers, and the apparatus includes:
an initialization module 101 is configured to create a shared directory in each metadata server, so as to store metadata information through the shared directory. For details, refer to the related description of step S101 in the above method embodiment, and no further description is provided here.
The load calculating module 102 is configured to count load parameters corresponding to each shared directory, and determine a difference between a maximum load parameter and a minimum load parameter. For details, refer to the related description of step S102 in the above method embodiment, and no further description is provided here.
And the load balancing module 103 is configured to migrate a part of metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter if the difference reaches a preset threshold. For details, refer to the related description of step S103 in the above method embodiment, and no further description is provided here.
Specifically, as shown in fig. 4, in an embodiment, before the equalizing module, the apparatus further includes:
the time detection module 104 is configured to periodically detect the number of calls of the metadata information in all the shared directories, and acquire a time period in which the number of calls is lower than a fourth preset threshold as a migration time period. For details, refer to the related description of step six in the above method embodiments, and no further description is provided herein.
The migration module 105, if the current time point is within the migration time period, is configured to return to the step of migrating a part of the metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter. For details, refer to the description related to step seven in the above method embodiments, and no further description is provided herein.
The suspending module 106 is configured to suspend the migration task if the current time point is not within the migration time period, and return to the step of migrating a part of the metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter until the time reaches the migration time period. For details, refer to the description related to step eight in the above method embodiments, and no further description is provided herein.
The metadata load balancing apparatus provided in the embodiment of the present invention is configured to execute the metadata load balancing method provided in the above embodiment, and the implementation manner and the principle thereof are the same, and details of the metadata load balancing apparatus refer to the related description of the above method embodiment and are not described again.
Through the cooperative cooperation of the above components, the technical scheme provided by the application is that a shared directory is firstly created in each metadata server and used for storing metadata information of actual subdirectories or files in an external data storage space, so that a plurality of metadata servers can manage the whole directory structure in a labor-sharing manner. The load parameters of the shared directories in each metadata server are calculated in real time, and only when the maximum load parameter and the minimum load parameter exceed preset thresholds, the metadata information in the corresponding first shared directory is migrated to the second shared directory, so that the load balance between the metadata server with the maximum load and the metadata server with the minimum load is completed, the load is prevented from being adjusted in real time, the metadata migration times are reduced on the premise of ensuring the load balance of each metadata server, and the influence of interrupting the use of users is reduced.
In addition, the file quantity and the calling times of the metadata information in the shared directory are used as load parameters, and the accuracy of load balancing calculation is further improved. And when a new subdirectory needs to be added, circularly storing the newly added subdirectory files in each metadata server, so that the addition of the subdirectories is as much as possible according to the principle of average distribution, and the subsequent metadata information migration times are reduced. On the other hand, the migration time of the metadata information is set in the time period when the user accesses the directory or the file for the least times, so that the interruption influence of the metadata migration on the user operation is further reduced.
Fig. 5 shows an electronic device according to an embodiment of the present invention, where the device includes a processor 901 and a memory 902, which may be connected via a bus or in another manner, and fig. 5 illustrates an example of a connection via a bus.
The memory 902, which is a non-transitory computer-readable storage medium, may be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions/modules corresponding to the methods in the above-described method embodiments. The processor 901 executes various functional applications and data processing of the processor by executing non-transitory software programs, instructions and modules stored in the memory 902, that is, implements the methods in the above-described method embodiments.
The memory 902 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created by the processor 901, and the like. Further, the memory 902 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 902 may optionally include memory located remotely from the processor 901, which may be connected to the processor 901 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
One or more modules are stored in the memory 902, which when executed by the processor 901 performs the methods in the above-described method embodiments.
The specific details of the electronic device may be understood by referring to the corresponding related descriptions and effects in the above method embodiments, and are not described herein again.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, and the implemented program can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD) or a Solid State Drive (SSD), etc.; the storage medium may also comprise a combination of memories of the kind described above.
Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.
Claims (10)
1. A metadata load balancing method is applied to a server cluster consisting of a plurality of metadata servers, and the method comprises the following steps:
creating a shared directory in each of the metadata servers to store metadata information through the shared directory;
counting load parameters corresponding to the shared directories, and determining a difference value between a maximum load parameter and a minimum load parameter;
and if the difference reaches a preset threshold value, transferring part of metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter.
2. The method of claim 1, wherein the load parameter is a file number of metadata information in a shared directory, and migrating a part of metadata information in a first shared directory corresponding to the maximum load parameter to a second shared directory corresponding to the minimum load parameter comprises:
and migrating part of metadata information in the first shared directory to the second shared directory until the difference value of the number of files in the first shared directory and the second shared directory is smaller than a second preset threshold value.
3. The method of claim 2, wherein the load parameters further include a number of calls to a metadata information file in the shared directory, and wherein determining the difference between the maximum load parameter and the minimum load parameter comprises:
calculating the weighted result of the file number and the calling times of each metadata server;
calculating a difference between a maximum weighting result and a minimum weighting result as a difference between the maximum load parameter and the minimum load parameter.
4. The method of claim 3, wherein migrating a portion of the metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter comprises:
and migrating part of metadata information in the first shared directory to the second shared directory until the difference value of the weighting results between the first shared directory and the second shared directory is smaller than a third preset threshold value.
5. The method of claim 1, further comprising: and adding the metadata information corresponding to the newly added subdirectories into the shared directories of the metadata servers in a circulating mode.
6. The method according to claim 1, wherein before migrating the part of the metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter, the method further comprises:
periodically detecting the calling times of the metadata information in all the shared directories, and acquiring a time period when the calling times are lower than a fourth preset threshold value as a migration time period;
if the current time point is within the migration time period, returning to the step of migrating part of the metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter;
and if the current time point is not in the migration time period, suspending the migration task until the time reaches the migration time period, and returning to the step of migrating part of the metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter.
7. A metadata load balancing apparatus applied to a server cluster composed of a plurality of metadata servers, the apparatus comprising:
an initialization module, configured to create a shared directory in each metadata server, so as to store metadata information through the shared directory;
the load calculation module is used for counting load parameters corresponding to the shared directories and determining a difference value between a maximum load parameter and a minimum load parameter;
and the load balancing module is used for migrating part of metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter if the difference reaches a preset threshold.
8. The apparatus of claim 7, further comprising, prior to the equalization module:
the time detection module is used for periodically detecting the calling times of the metadata information in all the shared directories and acquiring a time period with the calling times lower than a fourth preset threshold value as a migration time period;
a migration module, configured to return the step of migrating a part of metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter if the current time point is within the migration time period;
and the suspension module is used for suspending the migration task if the current time point is not in the migration time period, and returning to the step of migrating part of the metadata information in the first shared directory corresponding to the maximum load parameter to the second shared directory corresponding to the minimum load parameter until the time reaches the migration time period.
9. An electronic device, comprising:
a memory and a processor communicatively coupled to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the method of any of claims 1-6.
10. A computer-readable storage medium having stored thereon computer instructions for causing a computer to thereby perform the method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111401376.XA CN114064570A (en) | 2021-11-19 | 2021-11-19 | Metadata load balancing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111401376.XA CN114064570A (en) | 2021-11-19 | 2021-11-19 | Metadata load balancing method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114064570A true CN114064570A (en) | 2022-02-18 |
Family
ID=80276949
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111401376.XA Pending CN114064570A (en) | 2021-11-19 | 2021-11-19 | Metadata load balancing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114064570A (en) |
-
2021
- 2021-11-19 CN CN202111401376.XA patent/CN114064570A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9454407B2 (en) | Service resource allocation | |
| US10263876B2 (en) | Adaptive service timeouts | |
| US20180300066A1 (en) | Method and device for managing disk pool | |
| CN112100293A (en) | Data processing method, data access method, data processing device, data access device and computer equipment | |
| CN109981702B (en) | File storage method and system | |
| CN112286459A (en) | Data processing method, device, equipment and medium | |
| CN111857592A (en) | Data storage method and device based on object storage system and electronic equipment | |
| CN111966449B (en) | Virtual machine backup management method, system, terminal and storage medium | |
| US11914894B2 (en) | Using scheduling tags in host compute commands to manage host compute task execution by a storage device in a storage system | |
| CN112256433B (en) | Partition migration method and device based on Kafka cluster | |
| KR20220125198A (en) | Data additional writing method, apparatus, electronic device, storage medium and computer programs | |
| CN112988066A (en) | Data processing method and device | |
| CN111562884A (en) | Data storage method and device and electronic equipment | |
| CN109325001B (en) | Method, device and equipment for deleting small files based on metadata server | |
| US10587527B1 (en) | Systems and methods for apportioning bandwidth in storage systems | |
| CN117032566A (en) | Data self-classifying heterogeneous distributed storage method and system | |
| CN114064570A (en) | Metadata load balancing method and device and electronic equipment | |
| CN115269289A (en) | Slow disk detection method and device, electronic equipment and storage medium | |
| CN115509853A (en) | Cluster data anomaly detection method and electronic equipment | |
| CN115499513A (en) | Data request processing method and device, computer equipment and storage medium | |
| CN107018163B (en) | Resource allocation method and device | |
| CN111124275B (en) | Monitoring service optimization method and device of distributed block storage system | |
| CN112905119A (en) | Data writing control method, device and equipment of distributed storage system | |
| CN114968073A (en) | Data prefetching method, equipment and system | |
| CN112446490A (en) | Network training data set caching method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication |