WO2024099444A1

WO2024099444A1 - Storage cluster upgrade control method and apparatus, device, and storage medium

Info

Publication number: WO2024099444A1
Application number: PCT/CN2023/131087
Authority: WO
Inventors: 韩宾
Original assignee: 苏州元脑智能科技有限公司
Priority date: 2022-11-11
Filing date: 2023-11-10
Publication date: 2024-05-16
Also published as: CN115658116B; CN115658116A

Abstract

The present application relates to the technical field of data storage, and discloses a storage cluster upgrade control method and apparatus, a device, and a non-volatile readable storage medium. The method comprises: determining all weight classes in a storage cluster and weights corresponding to the weight classes, wherein the weight classes comprise all storage services deployed in the storage cluster and a bottom fault domain of the storage cluster, and the weight represents the number of nodes allowed to be concurrently upgraded by the corresponding weight class; determining a target weight class corresponding to each storage node in the storage cluster and a target weight corresponding to the target weight class to generate a weight matrix of each storage node; and controlling an online concurrent upgrade process of each storage node in the storage cluster on the basis of the weight matrix of each storage node. Therefore, according to the present application, the efficient concurrent online upgrade of a large-scale cluster can be achieved while ensuring that a storage cluster service system is not affected.

Description

A storage cluster upgrade control method, device, equipment and storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to a Chinese patent application filed with the China Patent Office on November 11, 2022, with application number 202211412298.8, and entitled “A Storage Cluster Upgrade Control Method, Device, Equipment and Storage Medium”, all contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the technical field of data storage, and in particular to a storage cluster upgrade control method, device, equipment and non-volatile readable storage medium.

Background technique

As storage systems become larger and larger, after a storage cluster is launched, it is necessary to regularly upgrade the storage system to fix vulnerabilities, enhance performance, or experience new version features. To ensure that the upgrade process does not affect the normal business of the storage cluster, only online upgrades can be selected. Currently, most storage systems use serial online upgrades, that is, after one node is upgraded, the next node is upgraded, or concurrent upgrades are implemented in some scenarios. However, the existing online upgrade solutions have many restrictions and are not universal.

Therefore, the above technical problems need to be solved by those skilled in the art urgently.

Summary of the invention

In view of this, the purpose of this application is to provide a storage cluster upgrade control method, device, equipment and non-volatile readable storage medium, which can realize efficient concurrent online upgrade of large-scale clusters while ensuring that the storage cluster business system is not affected. The specific scheme is as follows:

A first aspect of the present application provides a storage cluster upgrade control method, comprising:

Determine all weight classes in the storage cluster and the weights corresponding to the weight classes; wherein the weight classes include all storage services deployed in the storage cluster and the underlying fault domain of the storage cluster; the weights represent the number of nodes that the corresponding weight class allows to be upgraded concurrently;

Determine the target weight class corresponding to each storage node in the storage cluster and the target weight corresponding to the target weight class to generate a weight matrix for each storage node;

Based on the weight matrix of each storage node, the online concurrent upgrade process of each storage node in the storage cluster is controlled.

Optionally, determining a target weight class corresponding to each storage node in the storage cluster and a target weight value corresponding to the target weight class includes:

Each storage service deployed on each storage node in the storage cluster and the underlying fault domain to which the storage node belongs are determined as the target weight class corresponding to the storage node, and the target weight value corresponding to the target weight class is determined based on all weight classes and the weight values corresponding to the weight classes.

Optionally, after generating the weight matrix of each storage node, the method further includes:

When other weight classes are added to the weight matrix, the weights corresponding to the other weight classes are added to the weight matrix.

Optionally, the online concurrent upgrade process of each storage node in the storage cluster is controlled based on the weight matrix of each storage node, including:

The storage nodes corresponding to the weight matrix of the target weight class without duplication are divided into concurrent upgrade node groups, and each storage node in the concurrent upgrade node group is controlled to perform online concurrent upgrade; wherein the concurrent upgrade node group contains at least one storage node;

The remaining storage nodes in the storage cluster that are not divided into the concurrent upgrade node group are divided into the remaining storage node group, and the target weight corresponding to the target weight class of the storage node divided into the concurrent upgrade node group in the weight matrix of the remaining storage node group is reduced by 1 to obtain a more weight matrix for each storage node;

Based on the updated weight matrix of each storage node, the online concurrent upgrade process of each storage node in the storage cluster is controlled.

Optionally, the storage nodes corresponding to the weight matrices of the target weight classes that do not have duplication are divided into concurrent upgrade node groups, including:

The weight matrices of all storage nodes are processed using the bubbling principle to determine the storage nodes corresponding to the weight matrices of the target weight classes that do not have duplicates, and divide them into concurrent upgrade node groups.

Optionally, the weight matrices of all storage nodes are processed using the bubbling principle to determine the storage nodes corresponding to the weight matrices of the target weight class that do not have duplication, including:

The weight matrix including nodes with completely different weight types is selected from the weight matrix table, wherein the weight matrix table includes the weight matrices of all nodes in the storage cluster, and the completely different weight types of the nodes are used to represent the isolation between the nodes;

Storage nodes with completely different weight categories are screened out from the weight matrix including nodes with completely different weight categories.

Optionally, the online concurrent upgrade process of each storage node in the storage cluster is controlled based on the updated weight matrix of each storage node, including:

Eliminate the storage nodes corresponding to the weight matrices with target weights of 0 in the remaining storage node group to obtain the remaining storage node group after elimination; wherein each target weight in the weight matrix of the storage nodes in the remaining storage node group after elimination is not 0;

Filter out the storage node with the least number of weight classes from the remaining storage node groups after elimination, and divide the storage node into the concurrent upgrade node group;

Subtract 1 from the target weight corresponding to the target weight class of the storage node that is currently assigned to the concurrent upgrade node group in the weight matrix of the remaining storage node group after the elimination, to obtain an updated weight matrix of each storage node;

Repeat the node elimination step, filter storage nodes according to the number of weight categories, and reduce the weight by 1 step until the latest weight matrix of each storage node in the remaining storage node group after elimination has a target weight of 0, and obtain the latest remaining storage node group.

Optionally, the storage node with the least number of weight classes is selected from the remaining storage node group after elimination, including:

If there are multiple storage nodes with the least number of weighted classes in the remaining storage node group after elimination, one storage node is randomly selected from the multiple storage nodes as the storage node with the least number of weighted classes.

Optionally, after obtaining the latest remaining storage node group, it also includes:

Determine whether each storage node in the concurrent upgrade node group has been upgraded. If each storage node in the concurrent upgrade node group has been upgraded, restore the weight matrix of the upgraded storage node, and add 1 to the target weight corresponding to the target weight class of the restored storage node in the weight matrix of the latest remaining storage node group to obtain the final remaining storage node group;

Repeat the node elimination steps, storage node screening according to the number of weight categories, weight reduction step, node weight matrix recovery and weight increase step for the final remaining storage node group until all storage nodes are upgraded.

Optionally, after obtaining the updated weight matrix of each storage node, the method further includes:

The screened nodes are upgraded concurrently, and the weights of the corresponding weight classes in the remaining non-upgraded nodes are reduced by 1 according to the weight classes of the upgraded nodes.

Optionally, the weight corresponding to each weight class is greater than or equal to 1.

Optionally, the storage cluster upgrade control method further includes:

The real-time business pressure of the storage cluster is obtained, and the weight corresponding to the weight class is adjusted according to the real-time business pressure.

Optionally, the weights corresponding to the weight classes are adjusted according to real-time business pressure, including:

If the real-time business pressure exceeds the preset pressure value, the weight corresponding to the weight class is increased to reduce the number of concurrencies;

If the real-time business pressure is lower than the preset pressure value, the weight corresponding to the weight class is lowered to increase the number of concurrencies.

Optionally, the storage cluster upgrade control method further includes:

The upgrade process controller is used to trigger the storage cluster upgrade to be in a paused state and archive the current upgrade progress, or to trigger the storage cluster to continue the upgrade.

Optionally, using the upgrade process controller to trigger the storage cluster upgrade to be in a paused state and archive the current upgrade progress includes:

When receiving an instruction from a user to interrupt and exit the upgrade process before the upgrade is completed, the upgrade process controller responds to the instruction and triggers the cluster to suspend the upgrade process.

Optionally, triggering the storage cluster to continue the upgrade includes:

When the upgrade process is in a paused state, the storage cluster can be restored to continue the upgrade process through the continue function of the upgrade process controller.

Optionally, the storage cluster is a cluster under a distributed storage system and has an underlying fault domain structure based on a scalable pseudo-random data distribution algorithm structure.

A second aspect of the present application provides a storage cluster upgrade control device, including:

The weight class and weight value determination module is configured to determine all weight classes in the storage cluster and weight values corresponding to the weight classes; wherein the weight classes include all storage services deployed in the storage cluster and the underlying fault domain of the storage cluster; the weight values represent the number of nodes that the corresponding weight class allows to be upgraded concurrently;

A weight matrix generation module is configured to determine a target weight class corresponding to each storage node in the storage cluster and a target weight corresponding to the target weight class to generate a weight matrix for each storage node;

The upgrade module is configured to upgrade each storage node in the storage cluster based on the weight matrix of each storage node. Control the online concurrent upgrade process of the node.

A third aspect of the present application provides an electronic device, the electronic device comprising a processor and a memory; wherein the memory is configured to store a computer program, and the computer program is loaded and executed by the processor to implement the aforementioned storage cluster upgrade control method.

A fourth aspect of the present application provides a computer non-volatile readable storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the aforementioned storage cluster upgrade control method is implemented.

In this application, all weight classes and weights corresponding to the weight classes in the storage cluster are first determined; wherein the weight classes include all storage services deployed in the storage cluster and the underlying fault domains of the storage cluster; the weights represent the number of nodes that the corresponding weight classes allow for concurrent upgrades; then the target weight classes corresponding to each storage node in the storage cluster and the target weights corresponding to the target weight classes are determined to generate a weight matrix for each storage node; finally, based on the weight matrix of each storage node, the online concurrent upgrade process of each storage node in the storage cluster is controlled. It can be seen that this application sets a weight matrix for the storage nodes in the storage cluster, and controls the upgrade process of the storage nodes by controlling the weight classes and weights in the weight matrix, thereby achieving efficient concurrent online upgrades of large-scale clusters while ensuring that the storage cluster business system is not affected.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

FIG1 is a flow chart of a storage cluster upgrade control method provided by the present application;

FIG2 is a flow chart of an optional storage cluster upgrade control method provided by the present application;

FIG3 is a schematic diagram of an optional storage cluster upgrade control method provided by the present application;

FIG4 is a schematic diagram of the structure of a storage cluster upgrade control device provided by the present application;

FIG5 is a structural diagram of a storage cluster upgrade control electronic device provided in the present application.

Detailed ways

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

The online upgrade method of the existing storage system is serial, that is, after one node is upgraded, the next node is upgraded or concurrent upgrades are achieved in some scenarios. However, the existing online upgrade solutions have many restrictions and are not universal. In response to the above technical defects, the present application provides a storage cluster upgrade control solution, which sets a weight matrix for the storage nodes in the storage cluster, and controls the upgrade process of the storage nodes by controlling the weight classes and weights in the weight matrix, thereby achieving efficient concurrent online upgrades of large-scale clusters while ensuring that the storage cluster business system is not affected.

FIG1 is a flow chart of a storage cluster upgrade control method provided by an embodiment of the present application. The storage cluster upgrade control method includes:

S11: Determine all weight classes in the storage cluster and weights corresponding to the weight classes; wherein the weight classes include all storage services deployed in the storage cluster and the underlying fault domain of the storage cluster; and the weights represent the number of nodes that the corresponding weight classes are allowed to upgrade concurrently.

In this embodiment, all weight classes in the storage cluster and the weights corresponding to the weight classes are first determined. Among them, the weight classes include all storage services deployed in the storage cluster and the underlying fault domain of the storage cluster; the weights represent the number of nodes that the corresponding weight classes allow to be upgraded concurrently. The weight classes are the various services deployed on the nodes and the underlying fault domain structure to which the nodes belong. One service or one fault domain structure is a weight class, and a storage node can contain multiple weight classes. The weight is the number of nodes that each weight class allows to be upgraded concurrently. The weight class determines whether the node can be upgraded, and the weight determines how many nodes can be upgraded concurrently at the same time. In addition, the storage cluster of this embodiment is a cluster under a distributed storage system and is based on an underlying fault domain structure based on an extensible pseudo-random data distribution algorithm structure (crush root structure).

The online upgrade of this embodiment has two basic rules, which are formulated by the upgrade process controller. Basic rule 1: The weight of the ownership class under the node must be greater than 0; Basic rule 2: The number of nodes with the same weight class that are upgraded concurrently cannot exceed their basic weight. The process controller upgrade rules are based on the storage underlying fault domain structure (crush root structure), so the impact of the upper layer protocol on the upgrade rules can be effectively avoided, thereby realizing concurrent online upgrades in all scenarios.

S12: Determine the target weight class corresponding to each storage node in the storage cluster and the target weight value corresponding to the target weight class to generate a weight matrix for each storage node.

In this embodiment, the target weight class corresponding to each storage node in the storage cluster and the target weight value corresponding to the target weight class are determined to generate a weight matrix for each storage node. Optionally, each storage service deployed on each storage node in the storage cluster and the underlying fault domain to which the storage node belongs are determined as the target weight class corresponding to the storage node, and the target weight value corresponding to the target weight class is determined based on all weight classes and the weight values corresponding to the weight classes.

Finally, the node weight matrix is obtained. Whether a node can be upgraded depends on the node weight matrix. The node weight matrix consists of two dimensions: one is the weight class and the other is the weight. The process controller based on the node weight matrix has high scalability, that is, when other weight classes need to be added, only the weight of the corresponding weight class needs to be added to the weight matrix. And the process controller based on the node weight matrix has high concurrency, which can ensure that the storage cluster is in the best concurrent upgrade state in real time.

S13: Controlling the online concurrent upgrade process of each storage node in the storage cluster based on the weight matrix of each storage node.

In this embodiment, the online concurrent upgrade process of each storage node in the storage cluster is controlled based on the weight matrix of each storage node. It should be noted that the process controller based on the node weight matrix in this embodiment has high agility. When the cluster business or service is under high pressure, it can quickly affect and adjust the number of concurrently upgraded nodes to ensure the normal cluster business. Optionally, first obtain the real-time business pressure of the storage cluster, and adjust the weight corresponding to the weight class according to the real-time business pressure. If the real-time business pressure exceeds the preset pressure value, the size of the weight corresponding to the weight class is increased to reduce the number of concurrency; if the real-time business pressure is lower than the preset pressure value, the size of the weight corresponding to the weight class is lowered to increase the number of concurrency. Automatic adjustment of the weight class weights during the online upgrade process can be achieved. When the upgrade process controller When the controller detects that the storage cluster business pressure is high or the pressure of a certain service is high, it can actively adjust the weight of the node weight class. Adjusting the weight size can directly affect the number of concurrent upgrades of the nodes of this weight class, thereby ensuring that more nodes can handle cluster business.

This embodiment can also use the upgrade process controller to trigger the storage cluster upgrade to be in a suspended state and archive the current upgrade progress, or trigger the storage cluster to continue the upgrade. The upgrade process controller has the function of pausing and continuing the upgrade. Before the upgrade is completed, the user can interrupt and exit the upgrade process at any time and can actively trigger the cluster to suspend the upgrade process. After the upgrade is suspended, the cluster can be restored through the continue function to continue the upgrade.

It can be seen that the embodiment of the present application first determines all weight classes in the storage cluster and the weights corresponding to the weight classes; wherein the weight classes include all storage services deployed in the storage cluster and the underlying fault domains of the storage cluster; the weights represent the number of nodes that the corresponding weight class allows for concurrent upgrades; then the target weight class corresponding to each storage node in the storage cluster and the target weight class corresponding to the target weight class are determined to generate a weight matrix for each storage node; finally, based on the weight matrix of each storage node, the online concurrent upgrade process of each storage node in the storage cluster is controlled. The embodiment of the present application sets a weight matrix for the storage nodes in the storage cluster, and controls the upgrade process of the storage nodes by controlling the weight classes and weights in the weight matrix, thereby achieving efficient concurrent online upgrades of large-scale clusters while ensuring that the storage cluster business system is not affected.

FIG2 is a flow chart of an optional storage cluster upgrade control method provided in an embodiment of the present application. Referring to FIG2 , the storage cluster upgrade control method includes:

S21: Determine all weight classes and weights corresponding to the weight classes in the storage cluster, and determine the target weight class corresponding to each storage node in the storage cluster and the target weight corresponding to the target weight class to generate a weight matrix for each storage node.

In this embodiment, regarding the process of the above step S21, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described in detail here.

S22: Divide the storage nodes corresponding to the weight matrix of the target weight class without duplication into a concurrent upgrade node group, and control each storage node in the concurrent upgrade node group to perform online concurrent upgrade.

In this embodiment, the storage nodes corresponding to the weight matrices of the target weight classes that do not have duplicates are divided into concurrent upgrade node groups. The weight matrices of all storage nodes can be processed using the bubbling principle to determine the storage nodes corresponding to the weight matrices of the target weight classes that do not have duplicates, and divide them into concurrent upgrade node groups. That is, nodes with completely different weight classes are screened out. Completely different node weight classes indicate that the nodes are isolated from each other and can be upgraded concurrently. Then, each storage node in the concurrent upgrade node group is controlled to perform online concurrent upgrades. Among them, the concurrent upgrade node group contains at least one storage node.

S23: divide the remaining storage nodes in the storage cluster that have not been divided into the concurrent upgrade node group into the remaining storage node group, and reduce the target weight corresponding to the target weight class of the storage nodes divided into the concurrent upgrade node group this time in the weight matrix of the remaining storage node group by 1, to obtain a more updated weight matrix for each storage node.

In this embodiment, the remaining storage nodes in the storage cluster that are not divided into the concurrent upgrade node group are divided into the remaining storage node group, and then the target weight corresponding to the target weight class of the storage node divided into the concurrent upgrade node group in the weight matrix of the remaining storage node group is reduced by 1 to obtain the target weight of each storage node. The node weight matrix is updated. The selected nodes are upgraded concurrently, and the weights of the corresponding weight classes of the remaining non-upgraded nodes are adjusted according to the weight classes of the upgraded nodes, and the weights of the corresponding weight classes are reduced by 1.

S24: Eliminate the storage nodes corresponding to the weight matrices having target weights of 0 in the remaining storage node group to obtain the remaining storage node group after elimination; wherein each target weight in the weight matrix of the storage nodes in the remaining storage node group after elimination is not 0.

S25: Filter out the storage node with the least number of weight classes from the remaining storage node group after elimination, and divide the storage node into the concurrent upgrade node group.

In this embodiment, the storage nodes corresponding to the weight matrices with target weights of 0 in the remaining storage node group are eliminated to obtain the remaining storage node group after elimination. That is, the nodes whose weights are all not zero among the remaining nodes are screened, wherein the fact that all weights are not zero indicates that they can be upgraded. Each target weight in the weight matrix of the storage nodes in the remaining storage node group after elimination is not 0. The storage node with the least number of weight classes is screened out from the remaining storage node group after elimination, and the storage node is divided into the concurrent upgrade node group. In particular, if there are multiple storage nodes with the least number of weight classes in the remaining storage node group after elimination, one is randomly selected from the multiple storage nodes as the storage node with the least number of weight classes.

S26: Subtract 1 from the target weight corresponding to the target weight class of the storage node currently divided into the concurrent upgrade node group in the weight matrix of the remaining storage node group after elimination, to obtain an updated weight matrix of each storage node.

S27: Repeat the node elimination step, the storage node screening step according to the number of weight categories, and the weight reduction step until the latest weight matrix of each storage node in the remaining storage node group after elimination has a target weight of 0, thereby obtaining the latest remaining storage node group.

In this embodiment, the target weight corresponding to the target weight class of the storage node divided into the concurrent upgrade node group in the weight matrix of the remaining storage node group after elimination is reduced by 1, and the updated weight matrix of each storage node is obtained. Then, the node elimination step, the storage node selection according to the number of weight classes, and the weight reduction step are repeated until the latest weight matrix of each storage node in the remaining storage node group after elimination has a target weight of 0, and the latest remaining storage node group is obtained.

In this embodiment, the node elimination step, the storage node screening according to the number of weight classes and the weight reduction step are repeatedly executed until there is zero in the weights of the remaining nodes. Among them, a weight value of 0 indicates that the number of concurrent upgrade nodes of the weight class has reached the threshold, and the nodes of the weight class are not allowed to be upgraded before the upgrade of the nodes of the weight class is completed.

S28: Determine whether the upgrade of each storage node in the concurrent upgrade node group is completed. If the upgrade of each storage node in the concurrent upgrade node group is completed, restore the weight matrix of the upgraded storage node, and add 1 to the target weight corresponding to the target weight class of the restored storage node in the weight matrix of the latest remaining storage node group to obtain the final remaining storage node group.

S29: Repeat the node elimination step, the storage node screening step according to the number of weight categories, the weight reduction step, the node weight matrix recovery step and the weight increase step for the final remaining storage node group until all storage nodes are upgraded.

In this embodiment, it is determined whether each storage node in the concurrently upgraded node group has been upgraded. If the storage nodes in the concurrently upgraded node group have been upgraded, the weight matrix of the upgraded storage node is restored, and the weight matrix of the latest remaining storage node group that matches the restored storage node is restored. The target weight corresponding to the target weight class of the storage node is added by 1 to obtain the final remaining storage node group. That is, after the upgrade of the node waiting to be upgraded is completed, the weight of the corresponding weight class in the remaining non-upgraded nodes is refreshed according to the weight class of the upgraded node, and the weight of the corresponding weight class is added by 1. On this basis, the node elimination steps, storage node screening according to the number of weight classes, weight reduction step, node weight matrix recovery and weight addition step are repeated for the final remaining storage node group until all storage nodes are upgraded.

The upgrade control algorithm provided in the embodiment of the present application is described below with an example, and the operation flow of the process controller is shown in FIG3 .

For a storage cluster consisting of 10 storage nodes, three services f1, f2, and f3 are deployed, and the underlying fault domains are d1 and d2. The weights of f1, f2, and f3, that is, the concurrency thresholds, are set to 2, 2, and the weights of d1 and d2, that is, the concurrency thresholds, are set to 2, 2.

Step 1: First, generate a storage cluster node weight matrix table, which contains the weight matrices of all storage nodes. The weight matrix tables of each storage node are constructed as follows: n1: [f1, 2] [d1, 2]; n2: [f1, 2] [d1, 2]; n3: [f1, 2] [d1, 2]; n4: [f2, 2] [d2, 2]; n5: [f2, 2] [d2, 2]; n6: [f2, 2] [d2, 2]; n7: [f3, 2] [d1, 2]; n8: [f3, 2] [d2, 2]; n9: [f3, 2]; n10: [d2, 2].

Step 2: Use the bubbling principle to filter nodes, and first filter out nodes with completely different weight classes. Completely different node weight classes indicate that the nodes are isolated from each other and can be upgraded concurrently. Filter storage nodes n1: [f1, 2] [d1, 2], n4 [f2, 2] [d2, 2], and n9 [f3, 2] with completely different weight classes. These three storage nodes can be upgraded concurrently.

Step 3: Upgrade the selected nodes concurrently, and adjust the weights of the corresponding weight classes in the remaining non-upgraded nodes according to the weight classes of the upgraded nodes, and reduce the weights of the corresponding weight classes by 1. The weights of the remaining nodes are reduced by 1, specifically, the weights of the five weight classes f1, d1, f2, d2, and f3 are reduced by 1. The updated weight matrix is: n2: [f1, 1][d1, 1]; n3: [f1, 1][d1, 1]; n4: [f2, 1][d2, 1]; n5: [f2, 1][d2, 1]; n6: [f2, 1][d2, 1]; n7: [f3, 1][d1, 1]; n8: [f3, 1][d2, 1]; n10: [d2, 1].

Step 4: Filter the remaining nodes whose weights are all non-zero (if the weights are all non-zero, it means they can be upgraded). Nodes whose weights are all non-zero: n2: [f1, 1] [d1, 1]; n3: [f1, 1] [d1, 1]; n4: [f2, 1] [d2, 1]; n5: [f2, 1] [d2, 1]; n6: [f2, 1] [d2, 1]; n7: [f3, 1] [d1, 1]; n8: [f3, 1] [d2, 1]; n10: [d2, 1].

Step 5: Select the node with the smallest number of weight classes from the nodes generated in step 4 to upgrade and adjust the weights of the corresponding weight classes in the remaining nodes accordingly, and reduce the weight of the corresponding weight class by 1. If the number of weight classes is the same, one is randomly selected. The corresponding weights of the remaining nodes are reduced by 1, specifically, the weight of the weight class d2 is reduced by 1: n2: [f1, 1][d1, 1]; n3: [f1, 1][d1, 1]; n5: [f2, 1][d2, 0]; n6: [f2, 1][d2, 0]; n7: [f3, 1][d1, 1]; n8: [f3, 1][d2, 0].

Step 6: Loop through steps 4-5 until the remaining nodes have zero weights (a weight value of 0 indicates that the number of concurrently upgraded nodes of this weight class has reached the threshold, and upgrading nodes of this weight class is not allowed before the upgrade of nodes of this weight class is completed):

{Step 4: Nodes whose weights are not 0: n2: [f1, 1] [d1, 1]; n3: [f1, 1] [d1, 1]; n7: [f3, 1] [d1, 1];

Step 5: Select the node with the smallest number of weight classes. If the number of weight classes is the same, select randomly: n2: [f1, 1] [d1, 1]; the corresponding weights of the remaining nodes are reduced by 1: n3: [f1, 0] [d1, 0]; n5: [f2, 1] [d2, 0]; n6: [f2, 1] [d2, 0]; n7: [f3, 1] [d1, 1]; n8: [f3, 1] [d2, 0]. }

Step 7: After the node waiting for the upgrade is completed, the weights of the corresponding weight classes of the remaining nodes that have not been upgraded are refreshed according to the weight classes of the nodes that have completed the upgrade, and the weights of the corresponding weight classes are increased by 1. For example, after the upgrade is completed, n1: [f1, 2][d1, 2], the weights of the remaining nodes are increased by 1: n3: [f1, 1][d1, 1]; n5: [f2, 1][d2, 0]; n6: [f2, 1][d2, 0]; n7: [f3, 1][d1, 1]; n8: [f3, 1][d2, 0]

Step 8: Repeat steps 4-7 until all nodes have completed the upgrade:

{Step 4: Nodes whose weights are not 0: n3: [f1, 1] [d1, 1]; n7: [f3, 1] [d1, 1];

Step 5: Select the node with the smallest number of weight classes. If the number of weight classes is the same, select randomly: n3: [f1, 1] [d1, 1]; the corresponding weights of the remaining nodes are reduced by 1: n5: [f2, 1] [d2, 0]; n6: [f2, 1] [d2, 0]; n7: [f3, 1] [d1, 0]; n8: [f3, 1] [d2, 0]; wait for the node weights to recover. }

During the above online upgrade process, the weights of the weight classes are automatically adjusted. When the upgrade process controller detects that the storage cluster business pressure is high or the pressure of a certain service is high, it can actively adjust the weights of the node weight classes. Adjusting the weights can directly affect the number of concurrent upgrades of the nodes of this weight class, thereby ensuring that more nodes can handle cluster business.

As shown in FIG. 4 , the embodiment of the present application also discloses a storage cluster upgrade control device, including:

The weight class and weight value determination module 11 is configured to determine all weight classes in the storage cluster and weight values corresponding to the weight classes; wherein the weight classes include all storage services deployed in the storage cluster and the underlying fault domain of the storage cluster; the weight values represent the number of nodes that the corresponding weight class allows to be upgraded concurrently;

The weight matrix generation module 12 is configured to determine the target weight class corresponding to each storage node in the storage cluster and the target weight corresponding to the target weight class to generate a weight matrix for each storage node;

The upgrade module 13 is configured to control the online concurrent upgrade process of each storage node in the storage cluster based on the weight matrix of each storage node.

In this embodiment, the weight class and weight value determination module 11 determines all weight classes in the storage cluster and the weight values corresponding to the weight classes. Among them, the weight classes include all storage services deployed in the storage cluster and the underlying fault domain of the storage cluster; the weight value represents the number of nodes that the corresponding weight class allows to be upgraded concurrently. The weight class is the various services deployed on the node and the underlying fault domain structure to which the node belongs. A service or a fault domain structure is a weight class, and a storage node can contain multiple weight classes. The weight value is the number of nodes that each weight class allows to be upgraded concurrently. The weight class determines whether the node can be upgraded, and the weight value determines how many nodes can be upgraded concurrently at the same time. In addition, the storage cluster of this embodiment is a cluster under a distributed storage system and is based on an underlying fault domain structure based on an extensible pseudo-random data distribution algorithm structure (crush root structure).

The online upgrade of this embodiment has two basic rules, which are formulated by the upgrade process controller. Basic rule 1: The weight of the ownership class under the node must be greater than 0; Basic rule 2: The number of nodes with the same weight class that are upgraded concurrently cannot exceed their basic weight. The process controller upgrade rules are based on the storage underlying fault domain structure (crush root structure), so it can effectively avoid the impact of the upper layer protocol on the upgrade rules, and thus realize full-scenario concurrent online upgrades.

In this embodiment, the weight matrix generation module 12 determines the target weight class corresponding to each storage node in the storage cluster and the target weight corresponding to the target weight class to generate the weight matrix of each storage node. Optionally, each storage service deployed on each storage node in the storage cluster and the underlying fault domain to which the storage node belongs are determined as the target weight class corresponding to the storage node, and the target weight corresponding to the target weight class is determined based on all weight classes and the weights corresponding to the weight classes.

In this embodiment, the upgrade module 13 controls the online concurrent upgrade process of each storage node in the storage cluster based on the weight matrix of each storage node. It should be noted that the process controller based on the node weight matrix in this embodiment has high agility. When the cluster business or service is under high pressure, it can quickly affect and adjust the number of nodes for concurrent upgrades to ensure the normal operation of the cluster business. Optionally, the real-time business pressure of the storage cluster is first obtained, and the weight corresponding to the weight class is adjusted according to the real-time business pressure. If the real-time business pressure exceeds the preset pressure value, the size of the weight corresponding to the weight class is increased to reduce the number of concurrent operations; if the real-time business pressure is lower than the preset pressure value, the size of the weight corresponding to the weight class is lowered to increase the number of concurrent operations. The automatic adjustment of the weight class weight in the online upgrade process can be achieved. When the upgrade process controller detects that the storage cluster business pressure is large, or the pressure of a certain service is large, the weight of the node weight class can be actively adjusted. The adjustment of the weight size can directly affect the number of concurrent upgrades of the weight class node, thereby ensuring that more nodes can handle cluster business.

As an optional embodiment, the weight matrix generation module 12 is configured to determine each storage service deployed on each storage node in the storage cluster and the underlying fault domain to which the storage node belongs as the target weight class corresponding to the storage node, and determine the target weight corresponding to the target weight class based on all weight classes and the weights corresponding to the weight classes.

As an optional embodiment, the upgrade module 13 includes:

The first division unit is configured to divide the storage nodes corresponding to the weight matrix of the target weight class without duplication into a concurrent upgrade node group, and control each storage node in the concurrent upgrade node group to perform online concurrent upgrade; wherein the concurrent upgrade node group includes at least one storage node;

The second division unit is configured to divide the remaining storage nodes in the storage cluster that are not divided into the concurrent upgrade node group into the remaining storage node group, and reduce the target weight corresponding to the target weight class of the storage node divided into the concurrent upgrade node group in the weight matrix of the remaining storage node group by 1, so as to obtain a more weight matrix of each storage node;

The elimination unit is configured to eliminate the storage nodes corresponding to the weight matrices having target weights of 0 in the remaining storage node group, so as to obtain the remaining storage node group after the elimination; wherein each target weight in the weight matrix of the storage nodes in the remaining storage node group after the elimination is not 0;

A screening unit is configured to screen out a storage node with the least number of weight classes from the remaining storage node group after elimination, and divide the storage node into a concurrent upgrade node group;

A weight reduction unit is configured to reduce by 1 the target weight corresponding to the target weight class of the storage node currently divided into the concurrent upgrade node group in the weight matrix of the remaining storage node group after the elimination, so as to obtain an updated weight matrix of each storage node;

The first loop unit is configured to repeatedly execute the node elimination step, the storage node screening according to the number of weight categories, and the weight reduction step until the latest weight matrix of each storage node in the remaining storage node group after the elimination has a target weight of 0, thereby obtaining the latest remaining storage node group;

A judgment unit is configured to judge whether each storage node in the concurrently upgraded node group has been upgraded. If each storage node in the concurrently upgraded node group has been upgraded, the weight matrix of the upgraded storage node is restored, and the target weight corresponding to the target weight class of the restored storage node in the weight matrix of the latest remaining storage node group is increased by 1 to obtain a final remaining storage node group;

The second loop unit is configured to repeatedly execute the node elimination step, the storage node screening according to the number of weight categories, the weight reduction step, the node weight matrix recovery and the weight increase step for the final remaining storage node group until all storage nodes are upgraded.

As an optional embodiment, the first partitioning unit is further configured to process the weight matrices of all storage nodes using the bubbling principle to determine the storage nodes corresponding to the weight matrices of the target weight classes without duplication, and divide them into concurrent upgrade node groups.

As an optional embodiment, the elimination unit is further configured to randomly select one storage node from among the plurality of storage nodes as the storage node with the least number of weight classes if there are multiple storage nodes with the least number of weight classes in the remaining storage node group after the elimination.

As an optional embodiment, the storage cluster upgrade control device further includes:

An adjustment module is configured to obtain real-time business pressure of the storage cluster and adjust the weight corresponding to the weight class according to the real-time business pressure;

The control module is configured to use the upgrade process controller to trigger the storage cluster upgrade to be in a suspended state and archive the current upgrade progress, or to trigger the storage cluster to continue to perform the upgrade.

As an optional embodiment, the adjustment module includes:

The first adjustment unit is configured to increase the weight if the real-time business pressure exceeds the preset pressure value. The size of the weight corresponding to the class to reduce the number of concurrency;

The second adjustment unit is configured to reduce the size of the weight corresponding to the weight class to increase the number of concurrency if the real-time business pressure is lower than the preset pressure value.

The embodiment of the present application further provides an electronic device. Fig. 5 is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the diagram cannot be regarded as any limitation on the scope of application of the present application.

FIG5 is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input/output interface 25, and a communication bus 26. The memory 22 is configured to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the storage cluster upgrade control method disclosed in any of the aforementioned embodiments.

In this embodiment, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is configured to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the storage cluster upgrade control method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. The data 223 can include weight information and weight information collected by the electronic device 20.

The embodiment of the present application further discloses a non-volatile readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the storage cluster upgrade control method disclosed in any of the aforementioned embodiments are implemented.

In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that including one A process, method, article, or device that includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device that includes the element.

The storage cluster upgrade control method, device, equipment and non-volatile readable storage medium provided by the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

A storage cluster upgrade control method, characterized by comprising:

Determine all weight classes in the storage cluster and weights corresponding to the weight classes; wherein the weight classes include all storage services deployed in the storage cluster and the underlying fault domains of the storage cluster; the weights represent the number of nodes that the corresponding weight classes allow to be upgraded concurrently;

Determine a target weight class corresponding to each storage node in the storage cluster and a target weight corresponding to the target weight class to generate a weight matrix for each storage node;

The online concurrent upgrade process of each storage node in the storage cluster is controlled based on the weight matrix of each storage node.
The storage cluster upgrade control method according to claim 1, characterized in that the step of determining the target weight class corresponding to each storage node in the storage cluster and the target weight value corresponding to the target weight class comprises:

The storage services deployed on each storage node in the storage cluster and the underlying fault domain to which the storage node belongs are determined as the target weight class corresponding to the storage node, and the target weight value corresponding to the target weight class is determined based on all weight classes and the weight values corresponding to the weight classes.
The storage cluster upgrade control method according to claim 1, characterized in that after generating the weight matrix of each storage node, the method further comprises:

In the case where other weight classes are added to the weight matrix, weights corresponding to the other weight classes are added to the weight matrix.
The storage cluster upgrade control method according to claim 1, characterized in that the online concurrent upgrade process of each storage node in the storage cluster is controlled based on the weight matrix of each storage node, comprising:

Divide the storage nodes corresponding to the weight matrix of the target weight class that does not have duplication into a concurrent upgrade node group, and control each storage node in the concurrent upgrade node group to perform online concurrent upgrade; wherein the concurrent upgrade node group includes at least one storage node;

The remaining storage nodes in the storage cluster that are not divided into the concurrent upgrade node group are divided into the remaining storage node group, and the target weight corresponding to the target weight class of the storage node divided into the concurrent upgrade node group this time in the weight matrix of the remaining storage node group is reduced by 1 to obtain an updated weight matrix of each storage node;

The online concurrent upgrade process of each storage node in the storage cluster is controlled based on the updated weight matrix of each storage node.
The storage cluster upgrade control method according to claim 4, characterized in that the step of dividing the storage nodes corresponding to the weight matrix of the target weight class that does not have duplication into concurrent upgrade node groups comprises:

The weight matrices of all storage nodes are processed using the bubbling principle to determine storage nodes corresponding to the weight matrices of the target weight class that do not have duplications, and divide them into the concurrent upgrade node group.
The storage cluster upgrade control method according to claim 5 is characterized in that the weight matrix of all storage nodes is processed using the bubble principle to determine whether there is no duplication. The storage node corresponding to the weight matrix of the target weight class includes:

Filtering out the weight matrix including nodes with completely different weight types from the weight matrix table, wherein the weight matrix table includes the weight matrices of all nodes in the storage cluster, and the completely different weight types of the nodes are used to represent that the nodes are isolated from each other;

The storage nodes with completely different weight categories are screened out from the weight matrix including nodes with completely different weight categories.
The storage cluster upgrade control method according to claim 4, characterized in that the controlling the online concurrent upgrade process of each storage node in the storage cluster based on the updated weight matrix of each storage node comprises:

Eliminate the storage nodes corresponding to the weight matrix having the target weight value of 0 in the remaining storage node group to obtain the remaining storage node group after elimination; wherein each of the target weights in the weight matrix of the storage nodes in the remaining storage node group after elimination is not 0;

Filter out the storage node with the least number of weight classes from the remaining storage node group after elimination, and divide the storage node into the concurrent upgrade node group;

Subtract 1 from the target weight corresponding to the target weight class of the storage node currently divided into the concurrent upgrade node group in the weight matrix of the remaining storage node group after elimination, to obtain the updated weight matrix of each storage node;

Repeat the node elimination step, filter the storage nodes according to the number of weight categories, and reduce the weight by 1 step until the target weight of 0 exists in the latest weight matrix of each storage node in the remaining storage node group after elimination, thereby obtaining the latest remaining storage node group.
The storage cluster upgrade control method according to claim 7, characterized in that the step of selecting the storage node with the least number of weight classes from the remaining storage node group after elimination comprises:

If there are multiple storage nodes with the least number of weighted classes in the remaining storage node group after elimination, one storage node is randomly selected from the multiple storage nodes as the storage node with the least number of weighted classes.
The storage cluster upgrade control method according to claim 7, characterized in that after obtaining the latest remaining storage node group, it also includes:

Determine whether each storage node in the concurrently upgraded node group has been upgraded. If each storage node in the concurrently upgraded node group has been upgraded, restore the weight matrix of the upgraded storage node, and add 1 to the target weight corresponding to the target weight class of the restored storage node in the weight matrix in the latest remaining storage node group, to obtain the final remaining storage node group;

Repeat the node elimination step, the storage node screening step according to the number of weight categories, the weight reduction step, the node weight matrix recovery and the weight increase step for the final remaining storage node group until all storage nodes are upgraded.
The storage cluster upgrade control method according to claim 7, characterized in that after obtaining the updated weight matrix of each storage node, the method further comprises:

The screened nodes are upgraded concurrently, and the weights of the corresponding weight classes in the remaining non-upgraded nodes are reduced by 1 according to the weight classes of the upgraded nodes.
The storage cluster upgrade control method according to claim 1, characterized in that The weight corresponding to each weight class is greater than or equal to 1.
The storage cluster upgrade control method according to claim 1, characterized in that it also includes:

The real-time business pressure of the storage cluster is obtained, and the weight corresponding to the weight class is adjusted according to the real-time business pressure.
The storage cluster upgrade control method according to claim 12, characterized in that the step of adjusting the weight corresponding to the weight class according to the real-time business pressure comprises:

If the real-time business pressure exceeds the preset pressure value, the weight corresponding to the weight class is increased to reduce the number of concurrencies;

If the real-time business pressure is lower than the preset pressure value, the weight corresponding to the weight class is adjusted down to increase the number of concurrencies.
The storage cluster upgrade control method according to claim 1, characterized in that it also includes:

The upgrade process controller is used to trigger the storage cluster upgrade to be in a suspended state and archive the current upgrade progress, or to trigger the storage cluster to continue to perform the upgrade.
The storage cluster upgrade control method according to claim 14, characterized in that the step of using the upgrade process controller to trigger the storage cluster upgrade to be in a paused state and archiving the current upgrade progress comprises:

In the case where an instruction to interrupt and exit the upgrade process is received from the user before the upgrade is completed, the upgrade process controller responds to the instruction to trigger the cluster to suspend the upgrade process.
The storage cluster upgrade control method according to claim 14, wherein triggering the storage cluster to continue to perform the upgrade comprises:

When the upgrade process is in the paused state, the storage cluster is restored to continue executing the upgrade process through the resume function of the upgrade process controller.
The storage cluster upgrade control method according to any one of claims 1 to 16 is characterized in that the storage cluster is a cluster under a distributed storage system and has an underlying fault domain structure based on a scalable pseudo-random data distribution algorithm structure.
A storage cluster upgrade control device, characterized by comprising:

A weight class and weight value determination module is configured to determine all weight classes in the storage cluster and weight values corresponding to the weight classes; wherein the weight classes include all storage services deployed in the storage cluster and the underlying fault domains of the storage cluster; and the weight values represent the number of nodes that the corresponding weight class allows to be upgraded concurrently;

A weight matrix generation module is configured to determine a target weight class corresponding to each storage node in the storage cluster and a target weight corresponding to the target weight class to generate a weight matrix for each storage node;

The upgrade module is configured to control the online concurrent upgrade process of each storage node in the storage cluster based on the weight matrix of each storage node.
An electronic device, characterized in that the electronic device comprises a processor and a memory; wherein the memory is configured to store a computer program, and the computer program is loaded and executed by the processor to implement the storage cluster upgrade control method according to any one of claims 1 to 17.
A computer non-volatile readable storage medium, characterized in that it is configured to store computer A computer executable instruction, when the computer executable instruction is loaded and executed by a processor, implements the storage cluster upgrade control method according to any one of claims 1 to 17.