CN115277736A - Automatic data balancing method and device for distributed block storage - Google Patents
Automatic data balancing method and device for distributed block storage Download PDFInfo
- Publication number
- CN115277736A CN115277736A CN202210875898.1A CN202210875898A CN115277736A CN 115277736 A CN115277736 A CN 115277736A CN 202210875898 A CN202210875898 A CN 202210875898A CN 115277736 A CN115277736 A CN 115277736A
- Authority
- CN
- China
- Prior art keywords
- object storage
- osd
- groups
- utilization rate
- storage device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000001514 detection method Methods 0.000 claims abstract description 10
- 238000004590 computer program Methods 0.000 claims description 17
- 238000012544 monitoring process Methods 0.000 claims description 15
- 230000007774 longterm Effects 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 18
- 238000007726 management method Methods 0.000 description 9
- 238000012545 processing Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000005012 migration Effects 0.000 description 2
- 238000013508 migration Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000013475 authorization Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1029—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers using data related to the state of servers by a load balancer
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides a method and a device for automatically balancing data stored in distributed blocks, wherein the method comprises the following steps: pre-allocating, in a storage pool, a number of initial Placement Groups (PGs) on Object Storage Devices (OSD) for the respective object storage devices; and detecting the utilization rate of the object storage equipment and the number of the placing groups on the object storage equipment according to a preset period, and dynamically adjusting the number of the placing groups on each object storage equipment according to a detection result. By dynamically adjusting the number of the placement groups on each object storage device, the fine management of the distributed block storage system is realized, the problem of high OSD long-term utilization rate is effectively solved under the condition of enough storage pool redundancy performance, and the influence on the rear-end internal flow and the front-end service performance caused by uneven OSD utilization rate of distributed storage is avoided.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a method and an apparatus for automatically equalizing data stored in distributed blocks.
Background
With the development of information technology, the traditional centralized data storage faces many new challenges. With the development of architecture transformation, distributed block storage is widely applied to the supply of storage resources of virtual machines as a standard infrastructure cloud storage backend. At present, distributed block storage at the back end of an open-source cloud platform OpenStack widely uses Ceph distributed storage, and generally, data storage modes are three copies, so that the safety of stored data is ensured by changing space into reliability. Due to data redundancy, the true available capacity can only reach up to one third of the cluster. Moreover, the Ceph distributed Storage cluster has a barrel effect, and because the number of the Placing Groups (PGs) in the cluster is unreasonable, the number of PGs carried on each Object-based Storage Device (OSD) is inconsistent, and when the use capacity of a certain OSD reaches a full load, the whole cluster can only be read and cannot be written.
At present, in order to solve the above-mentioned problems, it is a common practice in the industry to adjust the OSD usage rate by using a reweight adjustment weight, so that the OSD usage rate is in a balanced state in a short time, and once a service is increased or the number of virtual machines is expanded, the OSD usage rate is in an unbalanced state again, and the OSD weight needs to be manually adjusted again. The principle that the use rate is balanced by adjusting the OSD weights through reweight is that the weight of the OSD with high or low use rate is reduced or increased through a reweight command, so that PG on the OSD is migrated out or migrated into other OSD, and the OSD use rate is balanced.
It should be noted that the above background description is provided only for the sake of clarity and complete description of the technical solutions of the present application, and for the sake of understanding of those skilled in the art. These solutions are not considered to be known to the person skilled in the art merely because they are set forth in the background section of the present application.
Disclosure of Invention
The inventor finds that in the prior art, due to the fact that manual adjustment is needed, workload is large, misoperation is easily caused, and the step length of the weight adjustment is difficult to control; on one hand, the distributed block storage cluster which is only suitable for capacity expansion and capacity reduction is adjusted manually, and once the capacity expansion and capacity reduction of the cluster occurs, the OSD utilization rate of the cluster is unbalanced again. On the other hand, the cluster is only suitable for deploying some light-weight applications subsequently, and once the application with large log amount is deployed, the OSD utilization rate of the cluster is unbalanced again.
In order to solve at least one of the above problems, embodiments of the present application provide a method and an apparatus for automatically balancing data stored in a distributed block, where the number of placement groups on each object storage device is dynamically adjusted, so as to implement fine management of a distributed block storage system, and effectively cope with the problem of high OSD long-term usage rate when the storage pool redundancy performance is sufficient, and avoid that the rear-end internal traffic is affected due to uneven OSD usage rate in the distributed storage, thereby affecting the front-end service performance.
According to a first aspect of the embodiments of the present application, a method for automatically equalizing data in distributed block storage is provided, where the method includes:
pre-assigning, in a storage pool, a number of initial Placement Groups (PGs) on Object Storage Devices (OSD) for each object storage device;
and detecting the utilization rate of the object storage equipment and the number of the placing groups on the object storage equipment according to a preset period, and dynamically adjusting the number of the placing groups on each object storage equipment according to a detection result.
According to a second aspect of embodiments herein, the initial placement group number is determined based on a total number of placement groups and the number of object storage devices in the storage pool.
According to a third aspect of the embodiments of the present application, the dynamically adjusting the number of placement groups on each object storage device according to the detection result includes:
and dynamically adjusting the number of the placement groups on each object storage device according to the difference value between the utilization rate of the object storage device and the utilization rates of other object storage devices in the storage pool.
According to a fourth aspect of embodiments herein, wherein the method further comprises: and when the difference value between the utilization rate of the object storage device and the utilization rates of other object storage devices in the storage pool is greater than a preset threshold value in a preset time length, alarming aiming at the object storage device.
According to a fifth aspect of embodiments herein, wherein the method further comprises:
and monitoring input/output (IO) data and flow of each object storage device in the storage pool according to a preset period.
According to a sixth aspect of embodiments herein, wherein the method further comprises:
and according to the IO data and the flow of each object storage device, part of the IO data is landed on the object storage device with low utilization rate.
According to a seventh aspect of embodiments herein, wherein the method further comprises:
and when the IO data or the flow of the object storage equipment is larger than a preset threshold value, alarming aiming at the object storage equipment.
According to an eighth aspect of the embodiments of the present application, there is provided an apparatus for automatically equalizing data stored in distributed blocks, the apparatus including:
an allocation module to pre-allocate a number of initial Placement Groups (PGs) on Object Storage Devices (OSD) for the respective object storage devices in a storage pool;
and the adjusting module detects the utilization rate of the object storage equipment and the number of the placing groups on the object storage equipment according to a preset period, and dynamically adjusts the number of the placing groups on each object storage equipment according to a detection result.
According to a ninth aspect of the embodiments of the present application, there is provided a computer device, including a memory, a processor, and a computer program stored on the memory and capable of running on the processor, wherein the processor, when executing the computer program, implements the method for automatically balancing data of distributed block storage according to any one of the first aspect to the eighth aspect of the embodiments of the present application.
According to a tenth aspect of the embodiments of the present application, there is provided a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the method for data auto-balancing of distributed block storage according to any one of the first to eighth aspects of the embodiments of the present application.
One of the beneficial effects of the embodiment of the application lies in:
the number of the placement groups on each object storage device is dynamically adjusted, so that the fine management of the distributed block storage system is realized, the problem of high OSD long-term utilization rate is effectively solved under the condition that the redundancy performance of the storage pool is sufficient, and the influence on the rear-end internal flow caused by the uneven OSD utilization rate of the distributed storage is avoided, so that the front-end service performance is influenced.
Specific embodiments of the present application are disclosed in detail with reference to the following description and drawings, indicating the manner in which the principles of the application may be employed. It should be understood that the embodiments of the present application are not so limited in scope. The embodiments of the application include many variations, modifications and equivalents within the spirit and scope of the appended claims.
The feature information described and illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, in combination with or instead of the feature information in the other embodiments.
It should be emphasized that the term "comprises/comprising" when used herein, is taken to specify the presence of stated features, integers, steps or components but does not preclude the presence or addition of one or more other features, integers, steps or components.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
FIG. 1 is a schematic diagram of a data automatic balancing method for distributed block storage according to an embodiment of the present application;
FIG. 2 is a schematic diagram illustrating a method for pre-allocating an initial placement group according to an embodiment of the present disclosure;
FIG. 3 is a schematic diagram illustrating a method for dynamically adjusting the number of placement groups according to an embodiment of the present disclosure;
FIG. 4 is a schematic diagram illustrating an embodiment of a method for dynamically adjusting the number of placement groups according to the present disclosure;
fig. 5 is a block diagram of a data automatic balancing apparatus for distributed block storage according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
Example 1
The embodiment 1 of the application provides a data automatic balancing method for distributed block storage. Fig. 1 is a schematic diagram of a data automatic balancing method for distributed block storage according to an embodiment of the present application, and as shown in fig. 1, the method includes:
Therefore, the number of the placement groups on each object storage device is dynamically adjusted, fine management of the distributed block storage system is achieved, and the problem of high OSD long-term utilization rate is effectively solved under the condition that the storage pool redundancy performance is sufficient.
The steps of the above method are described in detail below.
In the storage pool, the number of initial Placement Groups (PGs) on the Object Storage Devices (OSDs) is pre-assigned, that is, an expected number of PGs are assigned to each OSD, step 100.
Fig. 2 is a schematic diagram illustrating a method for pre-allocating an initial placement group according to an embodiment of the present disclosure.
As shown in fig. 2, step 100 includes:
the number of PGs is planned in advance when the storage pools of the distributed block storage cluster are built and the future service growth condition is considered, and the proportion of each storage pool is well controlled.
For example, the number of PG contained in a cluster can be calculated according to the following formula (1):
Total PGs＝(Total_number_of_OSD×100)/max_replication_count (1)
wherein, total _ number _ of _ OSD represents the Total OSD number in the cluster; max _ replication _ count represents the number of data copies within the cluster.
The Total PGs calculated in the storage pool is adjusted to a power close to or equal to 2 to correspond to a unit of computer storage in 1024 units.
Wherein, the MON is responsible for configuration and state management of the whole storage cluster node, and each virtual machine accessing the OSD needs to access the MON to obtain the cluster map first, so as to know which OSD nodes to communicate with.
The number of PGs on each OSD may be fixed as the total number of PGs divided by the total number of OSDs, and it is necessary to round up the result when the result is not an integer, and then obtain the calculation result, for example, determine the final number of PGs on each OSD to be 5% of the result obtained by performing the adjustment calculation upward or 5% of the result obtained by performing the adjustment calculation downward.
Next, in step 200, the usage rate of the object storage device and the number of the placement groups on the object storage device are detected according to a preset period, and the number of the placement groups on each object storage device is dynamically adjusted according to the detection result.
Fig. 3 is a schematic diagram illustrating a method for dynamically adjusting the number of placement groups according to an embodiment of the disclosure.
As shown in fig. 3, step 200 includes:
for example, in MON management storage node deployment, the overload control framework periodically detects the condition of cluster OSD usage rate, and judges whether the cluster OSD usage rate is close to an overload state, so as to provide a basis for dynamic adjustment of OSD usage rate;
for example, the difference between the usage of each object storage device and the average of the usage of the other object storage devices in the storage pool is monitored.
Alternatively, a difference between the usage rates of the respective object storage devices and a preset threshold may be detected.
Map monitors the OSD usage status of each node, e.g., by monitoring OSD.
For example, when the pressure of the storage system changes, the number of PGs on each OSD is adaptively adjusted, so that the health state of the storage cluster is prevented from being influenced by too much single OSD service pressure.
Therefore, in MON management storage node deployment, the PG quantity born by OSD is self-adaptively adjusted by monitoring the OSD utilization rate of each node, so that the OSD utilization rate is in a relatively balanced state, and the cluster has continuous and stable service capability.
In some embodiments, when the difference between the usage rate of the object storage device and the usage rates of other object storage devices in the storage pool is greater than a preset threshold within a preset time period, an alarm is issued to the object storage device.
For example, an OSD alarm that the difference of the OSD utilization rates is larger than 10% for a long time is reported, and operation and maintenance personnel are prompted to manually trigger PG migration.
In some embodiments, the method may further comprise: and monitoring input/output (IO) data and flow of each object storage device in the storage pool according to a preset period.
For example, IO data and flow on each OSD are monitored through an IO monitoring period; and acquiring the performance redundancy condition of the whole storage system from the overload control framework, and triggering partial IO data and flow to be dynamically adjusted to the rest OSD.
In some embodiments, the method may further comprise: and according to the IO data and the flow of each object storage device, part of the IO data is landed on the object storage device with low utilization rate.
For example, the PG number and the usage rate of the OSD in the cluster are monitored in real time, and partial IO data are landed on the OSD with low usage rate through IO monitoring dynamic adjustment under the condition that the difference between the PG number and the OSD usage rate is large.
For example, the low usage rate means that the usage rate of the OSD is lower than that of other OSDs by 10% or more.
In some embodiments, when IO data or traffic of a target storage device is greater than a preset threshold, an alarm is issued for the target storage device.
Next, in this embodiment, a data automatic equalization method for distributed block storage according to an embodiment of the present invention is described as an example. Fig. 4 is a schematic diagram illustrating an embodiment of a method for dynamically adjusting the number of placement groups according to the present disclosure.
As shown in fig. 4, in the building stage, a storage pool of a distributed block storage cluster is built at the back end, an initial fixed number of PGs is allocated to each OSD, and a virtual machine is allocated to each OSD at the front end.
In the using process, the OSD utilization rate of each node by the OSD is monitored by an overload control framework deployed on the MON management storage node, the PG number borne by the OSD is adaptively adjusted, the OSD utilization rate is in a relatively balanced state, and the cluster has continuous and stable service capability. When the overload control frame periodically detects the utilization rate of the cluster OSD, judging that the OSD is close to an overload state when the utilization rate of the OSD is 80-85%, simultaneously carrying out alarm uploading, and after the centralized monitoring receives the reported information, adaptively adjusting the PG number on each OSD. Meanwhile, the IO data and the flow of each OSD are fed back to the centralized monitoring by the IO monitoring period monitoring front-end virtual machine, the overall performance redundancy condition of the storage system is obtained from the overload control framework through the centralized monitoring, and partial IO data and flow are triggered to be dynamically adjusted to the rest of OSD.
Therefore, according to the overall monitoring, under the condition that the PG number on the OSD and the difference of the OSD utilization rate are large, partial IO data are landed on the OSD with low utilization rate through IO monitoring dynamic adjustment. Meanwhile, in the non-consistency check period and the service low peak period, the PG number and the utilization rate among the OSD are in a relatively balanced state again by dynamically adjusting the PG number on part of the OSD. On the other hand, an OSD alarm that the difference of the OSD utilization rates is beyond 10 percent for a long time is reported, and operation and maintenance personnel are prompted to manually trigger PG migration.
According to the embodiment of the application, the number of the placement groups on each object storage device is dynamically adjusted, so that the fine management of the distributed block storage system is realized, the problem of high OSD long-term utilization rate is effectively solved under the condition that the storage pool redundancy performance is enough, and the influence on the rear-end internal flow caused by the uneven OSD utilization rate of the distributed storage is avoided, so that the front-end service performance is influenced.
Example 2
The embodiment of the present application further provides an apparatus for automatically equalizing data stored in distributed blocks, which corresponds to the method for automatically equalizing data stored in distributed blocks in embodiment 1, so that the implementation of the apparatus may refer to the implementation of the method for automatically equalizing data stored in distributed blocks in embodiment 1, and repeated details are not described herein.
Fig. 5 is a block diagram of a data auto-leveling apparatus for distributed block storage according to an embodiment of the present application, and as shown in fig. 5, the data auto-leveling apparatus 10 for distributed block storage includes an allocation module 11 and an adjustment module 12, wherein,
an allocation module 11, which pre-allocates the number of initially Placed Groups (PG) on an Object Storage Device (OSD) for each object storage device in a storage pool;
the adjusting module 12 detects the usage rate of the object storage devices and the number of the placement groups on the object storage devices according to a preset period, and dynamically adjusts the number of the placement groups on each object storage device according to the detection result.
The embodiment of the present application further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements a data auto-balancing method for distributed block storage when executing the computer program.
Embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a method for automatically equalizing data in distributed block storage.
It should be noted that, in the technical solution of the present application, the acquisition, storage, use, processing, etc. of data all conform to the relevant regulations of the national laws and regulations.
The user information in the embodiment of the application is obtained through legal compliance, and the user information is obtained, stored, used, processed and the like through authorization approval of a client.
Although the present application provides method steps as described in an embodiment or flowchart, additional or fewer steps may be included based on conventional or non-inventive efforts. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. When implemented in practice, the apparatus or client products may be executed sequentially or in parallel (e.g., in the context of parallel processors or multi-threaded processing) according to the methods shown in the embodiments or figures.
As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, apparatus (system) or computer program product. Accordingly, embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment. In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. The terms "upper", "lower", and the like, indicate orientations or positional relationships that are based on the orientations or positional relationships shown in the drawings, are only for convenience in describing the present application and simplifying the description, and do not indicate or imply that the referenced devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and thus should not be construed as limiting the present application. Unless expressly stated or limited otherwise, the terms "mounted," "connected," and "connected" are intended to be inclusive and mean, for example, that they may be fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present application can be understood by those of ordinary skill in the art as appropriate. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application is not limited to any single aspect, nor is it limited to any single embodiment, nor is it limited to any combination and/or permutation of these aspects and/or embodiments. Moreover, each aspect and/or embodiment of the present application may be utilized alone or in combination with one or more other aspects and/or embodiments thereof.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit and scope of the present disclosure, and the present disclosure should be construed as being covered by the claims and the specification.
Claims (10)
1. A method for automatically equalizing data stored in distributed blocks, the method comprising:
pre-allocating, in a storage pool, a number of initial Placement Groups (PGs) on Object Storage Devices (OSD) for the respective object storage devices;
and detecting the utilization rate of the object storage equipment and the number of the placement groups on the object storage equipment according to a preset period, and dynamically adjusting the number of the placement groups on each object storage equipment according to the detection result.
2. The method of claim 1,
the initial placement group number is determined based on a total number of placement groups and the number of object storage devices in the storage pool.
3. The method of claim 1, wherein dynamically adjusting the number of placement groups on each object storage device according to the detection result comprises:
and dynamically adjusting the number of the placement groups on each object storage device according to the difference value between the utilization rate of the object storage device and the utilization rates of other object storage devices in the storage pool.
4. The method of claim 3, further comprising:
and when the difference value between the utilization rate of the object storage device and the utilization rates of other object storage devices in the storage pool is greater than a preset threshold value in a preset time length, alarming aiming at the object storage device.
5. The method of claim 1, further comprising:
and monitoring input/output (IO) data and flow of each object storage device in the storage pool according to a preset period.
6. The method of claim 5, further comprising:
and according to the IO data and the flow of each object storage device, part of the IO data is landed on the object storage device with low utilization rate.
7. The method of claim 6, further comprising:
and when the IO data or the flow of the object storage equipment is larger than a preset threshold value, alarming aiming at the object storage equipment.
8. An apparatus for automatic data equalization for distributed block storage, the apparatus comprising:
an allocation module to pre-allocate a number of initial Placement Groups (PG) on Object Storage Devices (OSD) for the respective OSD devices in a storage pool;
and the adjusting module detects the utilization rate of the object storage equipment and the number of the placing groups on the object storage equipment according to a preset period, and dynamically adjusts the number of the placing groups on each object storage equipment according to a detection result.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method according to any of claims 1 to 7 are implemented when the program is executed by the processor.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210875898.1A CN115277736A (en) | 2022-07-25 | 2022-07-25 | Automatic data balancing method and device for distributed block storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210875898.1A CN115277736A (en) | 2022-07-25 | 2022-07-25 | Automatic data balancing method and device for distributed block storage |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115277736A true CN115277736A (en) | 2022-11-01 |
Family
ID=83768366
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210875898.1A Pending CN115277736A (en) | 2022-07-25 | 2022-07-25 | Automatic data balancing method and device for distributed block storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115277736A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117519992A (en) * | 2024-01-04 | 2024-02-06 | 中电云计算技术有限公司 | Lookup table type load balancing method and device based on CRUSH algorithm |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20060067119A (en) * | 2004-12-14 | 2006-06-19 | 한국전자통신연구원 | Crash recovery system and method for a distributed file server using object based storage |
| CN106991170A (en) * | 2017-04-01 | 2017-07-28 | 广东浪潮大数据研究有限公司 | A kind of method and apparatus of distributed document capacity equilibrium |
| CN107391039A (en) * | 2017-07-27 | 2017-11-24 | 郑州云海信息技术有限公司 | A kind of data object storage method and device |
| CN111290699A (en) * | 2018-12-07 | 2020-06-16 | 杭州海康威视系统技术有限公司 | Data migration method, device and system |
| CN111880747A (en) * | 2020-08-01 | 2020-11-03 | 广西大学 | Automatic balanced storage method of Ceph storage system based on hierarchical mapping |
| KR20210054732A (en) * | 2019-11-06 | 2021-05-14 | 주식회사 케이티 | Apparatus for object based storage and data storage method using the same |
-
2022
- 2022-07-25 CN CN202210875898.1A patent/CN115277736A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20060067119A (en) * | 2004-12-14 | 2006-06-19 | 한국전자통신연구원 | Crash recovery system and method for a distributed file server using object based storage |
| CN106991170A (en) * | 2017-04-01 | 2017-07-28 | 广东浪潮大数据研究有限公司 | A kind of method and apparatus of distributed document capacity equilibrium |
| CN107391039A (en) * | 2017-07-27 | 2017-11-24 | 郑州云海信息技术有限公司 | A kind of data object storage method and device |
| CN111290699A (en) * | 2018-12-07 | 2020-06-16 | 杭州海康威视系统技术有限公司 | Data migration method, device and system |
| KR20210054732A (en) * | 2019-11-06 | 2021-05-14 | 주식회사 케이티 | Apparatus for object based storage and data storage method using the same |
| CN111880747A (en) * | 2020-08-01 | 2020-11-03 | 广西大学 | Automatic balanced storage method of Ceph storage system based on hierarchical mapping |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117519992A (en) * | 2024-01-04 | 2024-02-06 | 中电云计算技术有限公司 | Lookup table type load balancing method and device based on CRUSH algorithm |
| CN117519992B (en) * | 2024-01-04 | 2024-04-16 | 中电云计算技术有限公司 | Lookup table type load balancing method and device based on CRUSH algorithm |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109586952B (en) | Server capacity expansion method and device | |
| US20050154576A1 (en) | Policy simulator for analyzing autonomic system management policy of a computer system | |
| CN107665144B (en) | Balanced scheduling center, method, system and device for distributed tasks | |
| CN112003763B (en) | Network link monitoring method, monitoring device, monitoring equipment and storage medium | |
| CN103502944B (en) | Virutal machine memory method of adjustment and equipment | |
| CN103916396A (en) | Method for automatic expansion of application examples of cloud platform based on load self-adaption | |
| CN106452818A (en) | Resource scheduling method and resource scheduling system | |
| CN101595456A (en) | The method and system that is used for transaction resource control | |
| CN115277736A (en) | Automatic data balancing method and device for distributed block storage | |
| CN110677459A (en) | Resource adjusting method and device, computer equipment and computer storage medium | |
| CN113688001B (en) | Dynamic balancing method and device for power consumption of server hard disk, terminal and storage medium | |
| CN106100901B (en) | Flow velocity control method and device | |
| WO2020093502A1 (en) | Nominal bandwidth adjusting method and device | |
| CN112711479A (en) | Load balancing system, method and device of server cluster and storage medium | |
| CN103810109A (en) | Memory allocation method, device and system | |
| CN112422440A (en) | Flow control method applied to server and related device | |
| CN112559122A (en) | Virtualization instance management and control method and system based on electric power special security and protection equipment | |
| CN115334084A (en) | Cloud platform based on cloud computing and internet | |
| CN116467082A (en) | Big data-based resource allocation method and system | |
| JPH11346377A (en) | Method for processing overload of message in communication system | |
| CN107203256B (en) | Energy-saving distribution method and device under network function virtualization scene | |
| CN111352726B (en) | Stream data processing method and device based on containerized micro-service | |
| CN108234266B (en) | Data transmission method and device for bus type communication network and storage medium | |
| CN112416888B (en) | Dynamic load balancing method and system for distributed file system | |
| CN106060123B (en) | The collecting method and distributed data system of distributed data system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |