CN112417041B - Parameter configuration method and device, electronic equipment and storage medium - Google Patents

Parameter configuration method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN112417041B
CN112417041B CN202011301312.8A CN202011301312A CN112417041B CN 112417041 B CN112417041 B CN 112417041B CN 202011301312 A CN202011301312 A CN 202011301312A CN 112417041 B CN112417041 B CN 112417041B
Authority
CN
China
Prior art keywords
parameter
value
parameters
initial
storage system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011301312.8A
Other languages
Chinese (zh)
Other versions
CN112417041A (en
Inventor
舒继武
陆游游
吕文豪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tsinghua University
Original Assignee
Tsinghua University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tsinghua University filed Critical Tsinghua University
Priority to CN202011301312.8A priority Critical patent/CN112417041B/en
Publication of CN112417041A publication Critical patent/CN112417041A/en
Application granted granted Critical
Publication of CN112417041B publication Critical patent/CN112417041B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/217—Database tuning

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention provides a parameter configuration method, a parameter configuration device, electronic equipment and a storage medium. The method comprises the following steps: receiving a parameter configuration request of the distributed storage system, wherein the parameter configuration request comprises a parameter configuration target; acquiring parameters to be configured and a value space of the parameters to be configured from the parameters of the distributed storage system according to the constraint conditions of the parameter values; acquiring N candidate values of the parameter to be configured from a value space of the parameter to be configured according to the parameter configuration target; in the Mth iteration, the test cluster uses the performance value under the value of the parameter to be configured obtained by the M-1 th iteration under the current hardware configuration and the current load to obtain the Mth candidate value of the parameter to be configured from the value space of the parameter to be configured; and determining a target value of the parameter to be configured in the N candidate values of the parameter to be configured. The invention achieves the effect of improving the efficiency and the accuracy of parameter configuration of the distributed storage system.

Description

Parameter configuration method, device, electronic equipment and storage medium
Technical Field
The present invention relates to distributed storage system technologies, and in particular, to a parameter configuration method and apparatus, an electronic device, and a storage medium.
Background
With the explosive increase of the data volume of the internet, the network storage system adopting a centralized storage server to store all data cannot meet the requirement of large-scale storage application. Therefore, a Distributed Storage System (DSS) has been proposed to store data in a Distributed manner on a plurality of independent devices to improve the reliability, availability, and access efficiency of the System. The user can store data to the distributed storage system or acquire data through the terminal equipment.
A plurality of parameters which can be adjusted by a user exist in the distributed storage system, and the values of the parameters affect the performance of the distributed storage system, such as bandwidth (bandwidth), latency (latency), throughput (throughput), and the like. However, distributed storage systems that use default values of parameters to work with different hardware configurations and different workloads may not deliver the best performance of the distributed storage system. Therefore, a user needs to manually configure parameters of the distributed storage system based on the hardware configuration and the workload of the distributed storage system, so that the performance of the distributed storage system reaches the target performance.
However, the distributed storage system has many parameters, a complex constraint relationship of parameter values, and a huge parameter value space, and a user needs to spend a lot of time for parameter configuration, resulting in low efficiency and accuracy of parameter configuration of the distributed storage system.
Disclosure of Invention
The invention provides a parameter configuration method, a parameter configuration device, electronic equipment and a storage medium, and aims to achieve the technical effect of improving the efficiency and accuracy of parameter configuration of a distributed storage system.
In a first aspect, the present invention provides a parameter configuration method, including:
receiving a parameter configuration request of a distributed storage system, wherein the parameter configuration request comprises a parameter configuration target, and the parameter configuration target is used for indicating target performance to be achieved by the distributed storage system under the current hardware configuration and the current load;
acquiring a parameter to be configured and a value space of the parameter to be configured from the parameter of the distributed storage system according to a constraint condition of parameter value; wherein the constraint condition of the parameter value is related to the influence of the parameter on the operation of the distributed storage system;
acquiring N candidate values of the parameter to be configured from the value space of the parameter to be configured according to the parameter configuration target; in the Mth iteration, based on a performance value of the distributed storage system under a parameter to be configured obtained by using the Mth-1 th iteration, the Mth candidate value of the parameter to be configured is obtained from a value space of the parameter to be configured; wherein M is an integer greater than 1 and less than or equal to N;
and determining a target value of the parameter to be configured in the N candidate values of the parameter to be configured.
Optionally, after determining the target value of the parameter to be configured, the method further includes:
and using the target value of the parameter to be configured to configure the parameter of the distributed storage system.
Optionally, the obtaining, according to the constraint condition of the parameter value, the parameter to be configured from the parameter of the distributed storage system, and the value space of the parameter to be configured include:
acquiring X initial parameters to be configured and a value space of each initial parameter to be configured from the parameters of the distributed storage system according to the constraint condition of parameter value, wherein X is a positive integer;
acquiring coefficients of the initial parameters to be configured based on the initial parameters to be configured and the value space of the initial parameters to be configured; wherein the coefficient is positively correlated with the influence of the initial parameter to be configured on the performance of the distributed storage system;
according to the sequence of coefficients from high to low, taking Y initial parameters to be configured before the sequencing as the parameters to be configured, and taking the value space of the Y initial parameters to be configured before the sequencing as the value space of the parameters to be configured; wherein Y is an integer greater than 1 and less than or equal to X.
Optionally, the constraint condition of the parameter value includes at least one of the following: the first constraint condition is that whether the value of the constraint parameter is modifiable, the second constraint condition is that whether the value of the constraint parameter has a boundary, the third constraint condition is that whether the value of the constraint parameter is affected by the values of other parameters, the fourth constraint condition is that whether the sum of the value of the constraint parameter and the values of other parameters is equal to a preset value, and the fifth constraint condition is that whether the value of the constraint parameter is used for limiting the boundary of the values of other parameters.
Optionally, the obtaining, according to the constraint condition of the parameter value, X initial parameters to be configured from the parameters of the distributed storage system, and the value space of each initial parameter to be configured includes:
obtaining the X initial parameters to be configured from a preset configuration parameter file according to the first constraint condition, the third constraint condition, the fourth constraint condition and the fifth constraint condition;
setting the value space of a first type of parameters in the X initial parameters to be configured as continuous integers starting from a first value; the first type of parameters are parameters taking values as character strings;
setting the boundary of the value space of a second type parameter in the X initial parameters to be configured as a boundary based on the value dynamic expansion of the second type parameter; wherein the second type of parameter is a parameter lacking a value boundary;
and for a third type of parameters except the first type of parameters and the second type of parameters in the X initial parameters to be configured, acquiring a value space of the third type of parameters according to the second constraint condition and/or the fourth constraint condition.
Optionally, the obtaining the coefficient of each initial parameter to be configured based on each initial parameter to be configured and the value space of each initial parameter to be configured includes:
randomly acquiring Z initial values of each initial parameter to be configured from the value space of the initial parameter to be configured, wherein Z is a positive integer;
for the Z initial values of each initial parameter to be configured, obtaining a performance value of the test cluster when each initial value of each initial parameter to be configured is used under the current hardware configuration and the current load;
and acquiring coefficients of the initial parameters to be configured according to the Z initial values of the initial parameters to be configured, the initial parameters to be configured and performance values of the distributed storage system when each initial value of the initial parameters to be configured is used.
Optionally, before obtaining the coefficient of each initial parameter to be configured, the method further includes:
converting the first type of parameters in the initial parameters to be configured into dummy variables; the value of the dummy variable is a second value or a third value, and the first type parameter is a parameter with a value of a character string; and/or the presence of a gas in the gas,
and normalizing the Z initial values of the initial parameter to be configured and the performance value of the distributed storage system when each initial value of the initial parameter to be configured is used.
In a second aspect, the present invention provides a parameter configuration apparatus, including:
the system comprises a receiving module, a parameter configuration module and a load module, wherein the receiving module is used for receiving a parameter configuration request of the distributed storage system, the parameter configuration request comprises a parameter configuration target, and the parameter configuration target is used for indicating target performance of the distributed storage system to be achieved under current hardware configuration and current load;
the first acquisition module is used for acquiring a parameter to be configured and a value space of the parameter to be configured from the parameter of the distributed storage system according to a constraint condition of parameter value; wherein the constraint condition of the parameter value is related to the influence of the parameter on the operation of the distributed storage system;
a second obtaining module, configured to obtain, according to the parameter configuration target, N candidate values of the parameter to be configured from the value space of the parameter to be configured; in the Mth iteration, based on a performance value of a test cluster under the current hardware configuration and the current load and under the value of a parameter to be configured obtained by using the M-1 th iteration, the Mth candidate value of the parameter to be configured is obtained from the value space of the parameter to be configured; wherein M is an integer greater than 1 and less than or equal to N;
and the determining module is used for determining a target value of the parameter to be configured in the N candidate values of the parameter to be configured.
In a third aspect, the present invention provides an electronic device comprising: at least one processor, a memory;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the electronic device to perform the method of any of the first aspects.
In a fourth aspect, the present invention provides a computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, implement the method of any one of the first aspects.
According to the parameter configuration method, the parameter configuration device, the electronic equipment and the storage medium, the parameter to be configured and the value space of the parameter to be configured are obtained from the parameter of the distributed storage system according to the constraint condition of the parameter value, so that the electronic equipment can search the candidate value of the parameter to be configured in the value space of the parameter to be configured, and the efficiency and the accuracy of searching the candidate value of the parameter to be configured are improved. And then acquiring candidate values of the parameters to be configured from the value space of the parameters to be configured according to the target performance of the distributed storage system under the current hardware configuration and the current load. And determining a target value of the parameter to be configured from the candidate values of the parameter to be configured. The target value of the parameter to be configured of the distributed storage system is automatically acquired. According to the process, the parameter configuration of the distributed storage system is not needed manually, the time spent on the parameter configuration of the distributed storage system is reduced, and the parameter configuration efficiency of the distributed storage system is further improved. And the target value of the parameter to be configured is determined based on the target performance of the distributed storage system which needs to be achieved under the current hardware configuration and the current load, so that the technical effect of improving the accuracy of the parameter configuration of the distributed storage system is achieved.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the following briefly introduces the drawings needed to be used in the description of the embodiments or the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without inventive labor.
FIG. 1 is a schematic diagram of a distributed storage system;
FIG. 2 is a schematic diagram of a parameter configuration system according to the present invention;
FIG. 3 is a schematic flow chart of a parameter configuration method according to the present invention;
fig. 4 is a schematic flow chart of a method for acquiring a parameter to be configured and a value space thereof according to the present invention;
FIG. 5 is a schematic flow chart of another parameter configuration method according to the present invention;
FIG. 6 is a schematic structural diagram of a parameter configuration apparatus according to the present invention;
fig. 7 is a schematic structural diagram of an electronic device according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
FIG. 1 is a schematic diagram of a distributed storage system. As shown in fig. 1, the distributed storage system may include a plurality of devices, which are independent of each other, and data is dispersedly stored in the independent devices. The plurality of devices collectively provide storage services for the user. That is, the user can access the distributed storage system through the client to store or read data. In some embodiments, the plurality of devices may also be referred to as a cluster of a distributed storage system.
At present, distributed storage systems with wider application include, for example, ceph distributed storage systems, lustre distributed storage systems, gluster distributed storage systems, and the like. Fig. 1 is a schematic diagram of an architecture of a Ceph distributed storage system as an example. The core and base of the system are Reliable self-healing Distributed Object Store (RADOS) clusters. Wherein each server may include a plurality of Object-based Storage devices (OSDs).
A plurality of parameters which can be adjusted by a user exist in the distributed storage system, and the values of the parameters affect the performance of the distributed storage system, such as bandwidth (bandwidth), latency (latency), throughput (throughput), and the like.
Different users have different performance requirements on the distributed storage system. For example, users with large access data volume have high requirements on the data transmission bandwidth performance of the distributed storage system. The requirement on the real-time performance of data is high for users, and the requirement on the delay performance of the distributed storage system is high. Therefore, when the distributed storage system with different hardware configurations and different workloads is used, the optimal performance of the distributed storage system may not be achieved and the use requirements of users may not be met. At present, a user usually configures parameters of the distributed storage system manually, so that the distributed storage system can achieve target performance required by the user, and the use requirement of the user on the distributed storage system is met.
However, the distributed storage system has many parameters, complex constraint relationship between the parameters, and huge parameter value space, and even an experienced user needs to spend a lot of time for parameter configuration, so that the efficiency of parameter configuration of the distributed storage system is low, and it is difficult to obtain the optimal performance of the distributed storage system, that is, the accuracy of parameter configuration of the distributed storage system is low.
Considering that the distributed storage system has numerous parameters, the constraint relationship among the parameters of the distributed storage system is complex, the parameter value space is huge, and the performance of the distributed storage system under the same parameter configuration is related to the workload and the hardware configuration of the distributed storage system. Therefore, the invention provides a parameter configuration method, which can automatically acquire the value of the parameter to be configured, which can enable the distributed storage system to achieve the target performance, based on the factors. The method does not need to manually adjust the parameter value of the distributed storage system, and improves the parameter configuration efficiency and accuracy of the distributed storage system.
Fig. 2 is a schematic diagram of a parameter configuration system according to the present invention. As shown in fig. 2, the parameter configuration system includes an electronic device and a test cluster. The electronic equipment is in communication connection with the test cluster. The electronic device may be, for example, a terminal, a server, or the like.
The electronic device can automatically acquire the value of the parameter to be configured of the distributed storage system based on the parameter configuration request input by the user. In the process of obtaining the value, the electronic device can interact with the test cluster to obtain a performance value of the test cluster when the test cluster uses the configuration parameter to obtain the value under the condition of adopting the same hardware configuration and load as the distributed storage system.
For ease of description, a cluster of a distributed storage system is referred to as an online cluster to distinguish it from a test cluster.
In some embodiments, the electronic device may also be communicatively connected with the online cluster. Therefore, after the electronic device obtains the target value of the parameter to be configured of the distributed storage system, the electronic device can automatically perform parameter configuration on the online cluster by using the target value of the parameter to be configured.
The following takes a Ceph distributed storage system as an example, and details the technical solution of the present invention with reference to specific embodiments. It should be understood that the method of parameter configuration proposed by the present invention is also applicable to other kinds of distributed storage systems. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.
Fig. 3 is a schematic flow chart of a parameter configuration method according to the present invention. As shown in fig. 3, the method comprises the steps of:
s101, receiving a parameter configuration request of the distributed storage system.
For example, the electronic device may receive a parameter configuration request of the distributed storage system through an Application Program Interface (API) or a Graphical User Interface (GUI). The parameter configuration request comprises a parameter configuration target, and the parameter configuration target is used for indicating the target performance of the distributed storage system to be achieved under the current hardware configuration and the current load. The parameter configuration request further includes information about a current hardware configuration and a current load of the distributed storage system, for example, at least one of a size of a storage space of the distributed storage system, an Internet Protocol (IP) address of the distributed storage system, a type of a storage device used by the distributed storage system (e.g., a mechanical hard disk, a solid state hard disk, etc.), a size of an amount of data accessed by a user, and the like. The target performance may be, for example, a data transmission bandwidth or a data throughput rate of the distributed storage system.
S102, acquiring a parameter to be configured from the parameter of the distributed storage system according to the constraint condition of the parameter value, and acquiring the value space of the parameter to be configured.
For example, according to the influence of different parameter values on the operation of the distributed storage system, the constraint condition of the parameter value of the distributed storage system may include at least one of the following:
1. and the first constraint condition is whether the value of the constraint parameter can be modified. For a distributed storage system, some parameters in the distributed storage system cannot be modified when the current hardware configuration and current load of the distributed storage system are determined. The parameters satisfying the first constraint are, for example, a globally unique identifier, a network address, and the like in the Ceph distributed storage system. Modifying the values of these parameters may cause the distributed storage system to fail to operate properly.
2. And whether the constraint parameter value has a second constraint condition of a boundary or not is determined. Some parameters of a distributed storage system are of a range of values. If the values of the parameters exceed the value range, the distributed storage system may not operate normally. The parameter satisfying the second constraint condition is, for example, a Place Group (PG) parameter in the Ceph distributed Storage system, where the parameter is used to control the number of the place groups of each Object-based Storage Device (OSD) in the distributed Storage system to be between 30 and 250, and if the value of the PG parameter exceeds this range, the system may report an error and may not operate normally.
3. And a third constraint condition whether the value of the constraint parameter is influenced by the values of other parameters. Whether some parameter values of the distributed storage system are valid is related to other parameter values. For example, the fileStore module and blueStore module of the OSD. Both modules are local storage modules in the OSD. The OSD _ objectstore parameter determines whether the OSD uses the fileStore module or the blueStore module as the local storage module. For example, when the OSD _ object store value is blueStore, the OSD uses the blueStore module to perform local storage, and the fileStore module is not used. In this example, the change in the value of the parameter in the blueStore module may affect the operation and performance of the distributed storage system, and the change in the value of the parameter in the fileStore module may not affect the operation and performance of the distributed storage system.
Or, when the OSD _ object store value is fileStore, the OSD uses the fileStore module to perform local storage, and the blueStore module is not used. In this example, parameter value changes within the fileStore module may affect the operation and performance of the distributed storage system, while parameter value changes within the blueStore module may not affect the operation and performance of the distributed storage system.
4. And whether the sum of the constraint parameter value and other parameter values is equal to the fourth constraint condition of the preset value or not. For example, some configuration parameters collectively control a system resource of the distributed storage system, and therefore, the sum of values of the configuration parameters needs to be maintained at a fixed value. When the fixed value is exceeded, the system may be rendered inoperable or problematic. Illustratively, the blueStore _ cache _ kv _ ratio parameter in the Ceph distributed storage system, together with the blueStore _ cache _ meta _ ratio parameter, controls the use of the buffer space in the blueStore module, and the sum of the two values is equal to 1, and when the sum of the two values exceeds 1, the system may be in a problem or cannot operate.
5. And whether the constraint parameter value is used for limiting a fifth constraint condition of other parameter value boundaries or not. The value boundary referred to herein may be a maximum value or a minimum value. The parameter satisfying the fifth constraint condition may affect the operation of the distributed storage system by defining the value boundary of the other parameter. For example, the ms _ async _ max _ op _ threads parameter in the Ceph distributed storage system is used for limiting the maximum value of the ms _ async _ op _ threads parameter. If the value of the ms _ async _ max _ op _ threads parameter is larger than that of the ms _ async _ op _ threads parameter, it is indicated that the value change of the ms _ async _ max _ op _ threads parameter does not influence the value of the ms _ async _ op _ threads parameter, namely, does not influence the operation of the distributed storage system. If the value of the ms _ async _ max _ op _ threads parameter is smaller than that of the ms _ async _ op _ threads parameter, it is described that the value of the ms _ async _ op _ threads parameter is limited by the value of the ms _ async _ max _ op _ threads parameter and may not reach a larger value, and therefore the value of the ms _ async _ max _ op _ threads parameter may affect the operation of the distributed storage system.
It should be understood that the constraint on the parameter may be divided in other ways according to the influence of the parameter on the operation of the distributed storage system. The invention does not limit the classification of the constraint condition of the parameter value of the distributed storage system.
Although the above examples are examples of 5 value constraints, the value constraints are illustrated and explained. However, it should be understood that the value constraint condition corresponding to the distributed storage system may be determined according to a specific configuration of the distributed storage system, and is not limited thereto.
In this embodiment, the electronic device may delete and clean the parameters of the distributed storage system based on the constraint condition of the parameter value to remove the non-adjustable parameters and the parameters that do not affect the operation and performance of the distributed storage system, so as to reduce the number of the parameters to be configured, thereby improving the efficiency of obtaining the value of the parameters to be configured, and further improving the parameter configuration efficiency of the distributed storage system. In addition, the electronic device can also accurately acquire the value space of the parameter to be adjusted based on the constraint condition of the parameter value, and can ensure the accuracy of subsequently acquiring the N candidate values of the parameter to be configured.
It should be understood that the parameter of the distributed storage system may be input to the electronic device along with the parameter configuration request, or may be carried in the parameter configuration request, or may be a preset configuration parameter file (also referred to as a general configuration parameter file) stored in the electronic device, or a configuration parameter file of the distributed storage system itself, or may be carried in a parameter configuration file acquired from the distributed storage system by the electronic device accessing the distributed storage system.
It should be understood that, the sequence of the step S101 and the step S102 executed by the electronic device is not limited in this application. That is, the electronic device may first execute step S101 and then step S102, or may first execute step S102 and then step S101.
S103, acquiring N candidate values of the parameter to be configured from the value space of the parameter to be configured according to the parameter configuration target.
After the electronic device obtains the parameter to be configured and the value space of the parameter to be configured, the candidate value of the parameter to be configured can be obtained from the value space of the parameter to be configured by adopting an iterative loop mode according to the parameter configuration target. The ending condition of the iterative loop may be, for example, that the maximum iteration number is reached, or that a candidate value satisfying the parameter configuration target is obtained before the maximum iteration number is not reached.
N candidate values of the parameter to be configured are obtained in an iterative loop mode. I.e. N iterations are performed throughout the iteration loop period. In the mth iteration, the electronic device may obtain, from the value space of the parameter to be configured, an mth candidate value of the parameter to be configured based on the performance value of the test cluster under the current hardware configuration and the current load, using the value of the parameter to be configured obtained in the M-1 th iteration. In other words, in each iteration cycle, the reference basis for updating the value of the parameter to be configured is the performance value of the parameter to be configured obtained in the M-1 th iteration.
After the mth candidate value of the parameter to be configured is obtained, the electronic device may send the mth candidate value of the parameter to be configured to the test cluster, so as to control the test cluster to operate using the value of the parameter to be configured obtained by the mth iteration under the current hardware configuration and the current load, so as to obtain the performance value of the test cluster. The performance value is used for obtaining the M +1 th candidate value of the parameter to be configured in the next iteration cycle.
For example, after the electronic device configures the test cluster by using the mth candidate value of the parameter to be configured, the electronic device controls the test cluster to restart, so that the test cluster works under the mth candidate value. Then, the electronic device may use a test tool of the distributed storage system to perform a performance test on the test cluster under the current hardware configuration and the current load to obtain a performance value, and the performance value is used as a basis for searching a candidate value of a parameter to be configured in the M +1 th iteration process.
It is to be understood that M described above is an integer greater than 1 and less than or equal to N. For example, when the electronic device performs a first iteration (i.e., M = 1), an initial value may be assigned to each parameter to be configured (for example, an initial value may be assigned to the parameter to be configured in a value space of the parameter to be configured in a random assignment manner), where the initial value is the 1 st candidate value of the parameter to be configured.
For example, the method for finding the value of the parameter to be configured may be a bayesian optimization algorithm based on a gaussian process. Because the performance value is obtained by means of experimental test, the experimental test process often has fluctuation and error, and thus the performance value may have a high noise problem. The Bayesian optimization algorithm based on the Gaussian process can tolerate the noise of the performance value, and further can improve the accuracy of finding the value of the parameter to be configured.
In addition, this embodiment does not limit how the electronic device sets the test cluster to work under the current hardware configuration and the current load, for example, the electronic device may set the test cluster to work under the current hardware configuration and the current load before executing the iteration loop, or may set the test cluster to work under the current hardware configuration and the current load in the first iteration period, or may set the test cluster to work under the current hardware configuration and the current load in each iteration period.
And S104, determining a target value of the parameter to be configured in the N candidate values of the parameter to be configured.
For example, the electronic device may select an optimal candidate value from all candidate values of the parameter to be configured obtained in the N iteration processes as a target value of the parameter to be configured.
When the iteration loop in the above-mentioned S103 is ended, the candidate value that satisfies the parameter configuration target is obtained before the maximum iteration count is not reached. That is, when N is less than the maximum number of iterations, the optimal candidate value may be, for example, a candidate value that enables the distributed storage system to achieve the target performance, or a candidate value whose difference from the target performance is within a preset threshold range.
The end of the iteration loop in the aforementioned S103 is to reach the maximum number of iterations. That is, when N is equal to the maximum number of iterations, the optimal candidate value referred to herein may correspond to, for example, a candidate value with an optimal performance value, or a candidate value corresponding to a performance value greater than a preset performance threshold.
After the target value of the parameter to be configured is obtained, the electronic device may output the target value of the parameter to be configured, so that a user may perform parameter configuration on the distributed storage system based on the target value. Or, the electronic device may use the target value of the parameter to be configured to perform parameter configuration on the distributed storage system. For example, the electronic device may use the target value of the parameter to be configured to re-assign the parameter to be configured in the distributed storage system, and send a control instruction to the distributed storage system to control the distributed storage system to restart, so that the distributed storage system operates based on the value of the parameter, thereby automatically configuring the parameter for the distributed storage system, and further improving the efficiency of configuring the parameter for the distributed storage system.
The target value of the parameter to be configured is obtained based on a test cluster, which may be a dedicated cluster for obtaining a candidate value of the parameter to be configured, that is, the test cluster is a cluster other than a cluster independent from the distributed system, and the test cluster does not need to provide a storage service for a user. The electronic equipment obtains the performance value under a certain candidate value of the parameter to be configured by using the test cluster, so that frequent restart of the distributed storage system due to testing of the performance value corresponding to the value by using the distributed storage system can be avoided, and further, the distributed storage system is prevented from being blocked to continuously provide services.
In addition, because the scale of the test cluster is smaller than that of the online cluster, the time consumed for performance test is relatively short, that is, the time consumed for parameter configuration of the distributed storage system is further reduced, and the efficiency of parameter configuration of the distributed storage system is further improved.
Alternatively, the test cluster may also be part of a cluster (a cluster providing services for users, referred to as an online cluster) of the distributed storage system, that is, a part of devices in the online cluster is used for testing. By the method, the cost for configuring the parameters of the distributed storage system can be reduced, and in addition, the continuous service provision of other equipment in the distributed storage system cannot be influenced.
In this embodiment, according to the constraint condition of the parameter value, the parameter to be configured and the value space of the parameter to be configured are obtained from the parameter of the distributed storage system, so that the electronic device can search the candidate value of the parameter to be configured in the value space of the parameter to be configured, and the efficiency and the accuracy of searching the candidate value of the parameter to be configured are improved. And then acquiring candidate values of the parameters to be configured from the value space of the parameters to be configured according to the target performance of the distributed storage system under the current hardware configuration and the current load. And determining a target value of the parameter to be configured from the candidate values of the parameter to be configured. The target value of the parameter to be configured of the distributed storage system is automatically acquired. According to the process, the parameter configuration of the distributed storage system is not needed manually, the time spent on the parameter configuration of the distributed storage system is reduced, and the parameter configuration efficiency of the distributed storage system is improved. And the target value of the parameter to be configured is determined based on the target performance of the distributed storage system which needs to be achieved under the current hardware configuration and the current load, so that the technical effect of improving the accuracy of the parameter configuration of the distributed storage system is achieved.
For example, the parameter to be configured may be all configurable parameters obtained from the parameters of the distributed storage system based on the above-mentioned constraint condition of the value, or may be a part of parameters that have a larger influence on the performance of the distributed storage system and are further selected from the all configurable parameters. In this implementation manner, the process of finding the candidate values of the parameters to be configured may be optimized only for the values of this part of the parameters to be configured, so as to achieve the technical effect of further improving the parameter configuration efficiency of the distributed storage system.
In view of the step S102, fig. 4 is a schematic flow chart of a method for obtaining a parameter to be configured and a value space thereof according to the present invention. As shown in fig. 4, the method comprises the steps of:
s201, according to the constraint condition of parameter dereferencing, X initial parameters to be configured and the dereferencing space of each initial parameter to be configured are obtained from the parameters of the distributed storage system.
For example, the electronic device may obtain X initial parameters to be configured and a value space of each initial parameter to be configured from the parameters of the distributed storage system according to the constraint condition of the at least one parameter value in step S102.
In the following, a distributed storage system whose parameter values satisfy the five constraint conditions is taken as an example, and how to obtain X initial parameters to be configured from parameters of the distributed storage system according to the constraint conditions of the parameter values, and a value space of each initial parameter to be configured are described as an example. In the following examples, the parameters of the distributed storage system may refer to parameters in a preset parameter configuration file. It should be understood that the method is equally applicable in a distributed storage system that satisfies the constraints of at least one of the parameter values described above.
For example, according to the first constraint condition, if the value of the parameter is not modifiable, that is, the parameter itself is a fixed value, and the value thereof does not need to be optimized. The electronic device may delete the parameters whose values may not be modified from the preset configuration parameter file. For example, parameters that are not modifiable in the Ceph distributed storage system, such as globally unique identifiers, network addresses, and the like, are deleted from a preset configuration parameter file.
According to the third constraint condition, taking the parameter in the fileStore module as an example, whether the parameter value in the fileStore module is valid or not is related to the OSD _ objectstore parameter value (which is used to determine the local storage module used by the OSD, such as a blueStore module or a fileStore module). For example, when the OSD _ object store parameter takes the value of blueStore, it indicates that the OSD uses the blueStore module to perform local storage, and the fileStore module is not used.
According to the fourth constraint condition, the electronic device may make the value of one of the parameters equal to the sum of the value of the parameter and the values of the other parameters, subtract the values of the other parameters, and delete the parameter from the preset configuration parameter file. For example, the sum of the blue _ cache _ kv _ ratio parameter and the blue _ cache _ meta _ ratio parameter always needs to be equal to 1. Then, the value of the bluestore _ cache _ kv _ ratio parameter may be determined by the value of the bluestore _ cache _ meta _ ratio parameter. For example, the value of the bluestore _ cache _ kv _ ratio parameter is equal to 1 minus the value of the bluestore _ cache _ meta _ ratio parameter, and the bluestore _ cache _ kv _ ratio parameter is deleted from the preset configuration parameter file. Or the value of the bluestore _ cache _ meta _ ratio parameter is equal to or equal to 1 minus the value of the bluestore _ cache _ kv _ ratio parameter, and the bluestore _ cache _ meta _ ratio parameter is deleted from the preset configuration parameter file.
According to the fifth constraint condition, because the parameters for defining the value boundaries of the other parameters have no actual influence on the performance of the distributed storage system, and only influence the value boundaries of the defined parameters, the electronic device can make the values of the parameters for defining the value boundaries of the other parameters equal to the values of the defined parameters, and delete the parameters for defining the value boundaries of the other parameters from the preset configuration parameter file. For example, the value of ms _ async _ max _ op _ reads (used to define the value boundary of ms _ async _ op _ reads) is equal to the value of ms _ async _ op _ reads, and the ms _ async _ max _ op _ reads parameter is deleted from the preset configuration parameter file.
Therefore, based on the first constraint condition, the third constraint condition, the fourth constraint condition and the fifth constraint condition, the electronic device deletes the parameters which do not need to be subjected to value optimization from the preset configuration parameter file, and further obtains X initial parameters to be configured. Wherein X is a positive integer.
It is considered that a part of the parameters (first type of parameters for short) which take values as character strings may exist in the initial parameters to be configured. The process of searching the candidate value of the parameter to be configured is to search the next candidate value in the value space of the value of the parameter. For example, the electronic device may set the value space of the type of parameter to be from the first valueOpening deviceThe first continuous integer so that the electronic device can obtain the candidate value of the first type parameter. Wherein the first value can be any integer. For example, the osd _ objectstore parameter in the Ceph distributed storage system belongs to a first type of parameter, and its value may be blueStore, fileStore, memstor, or kstore, and the electronic device may set the value space of the parameter to 1 to 4, that is, 1 is a first value. The osd _ objectstore parameter value can be converted into a numerical value according to a rule shown in table 1, for example. It should be understood that the value space of the parameter may have other ranges, and the electronic device may also convert the value of the first type parameter into a numerical value through other conversion rules.
TABLE 1
Serial number osd _ objectstore parameter value Transformed value
1 blueStore 1
2 fileStore 2
3 memstor 3
4 kstore 4
It is considered that some parameters (for short, second-class parameters) lacking value boundaries may also exist in the initial parameters to be configured. For a parameter lacking a value boundary, since the value space has no boundary, the electronic device cannot perform an action of searching for the value of the parameter in the process of searching for the candidate value of the parameter to be configured. For such parameters, the electronic device may set the boundary of the value space of the second type of parameter as a boundary dynamically expanded based on the value of the second type of parameter. For example, the electronic device may first set an initial value space of the second type of parameter according to a default value of the second type of parameter. For example, the electronic device may set an initial value space of the second type of parameter according to a first preset rule. For example, the first preset rule may be that 0.5 times of the default value of the second type of parameter is used as the lower value limit of the second type of parameter, and 1.5 times of the default value of the second type of parameter is used as the lower value limit of the second type of parameter. For example, assuming that the default value of the second type parameter is 1, the electronic device may set the value space to [0.5,1.5] according to the first preset rule. It should be understood that the first preset rule may also be other rules.
In the subsequent process of searching for candidate values of the parameter to be configured, if an upper bound or a lower bound of the value space of the second type of parameter is found (or a difference value from the upper bound is smaller than a value of a first preset threshold, or a difference value from the lower bound is smaller than a second preset threshold, where the first preset threshold and the second preset threshold may be the same or different in size), the electronic device may expand the value space of the parameter according to a second preset rule. The second preset rule may be, for example, setting 0.5 times the lower bound value as the new lower bound value, or setting 1.5 times the upper bound value as the new upper bound value. For example, if the electronic device sets the value space of the second type of parameter to be [0.5,1.5], and in the subsequent process of searching for a candidate value of the parameter to be configured, finds that the candidate value is 0.5, the value space of the parameter may be extended to be [0.25,1.5]. It should be understood that the second preset rule may also be other rules, for example, the value space of the parameter is expanded according to a certain step size.
Considering that a third type of parameter, except the first type of parameter and the second type of parameter, in the initial parameter to be configured may be a parameter whose value is a numerical value and whose value space is already set when the distributed storage system is developed, that is, a parameter whose value has a boundary in the second constraint condition, the electronic device may directly use the value space set during development as the value space of the third type of parameter. For the third type of parameters meeting the fourth constraint condition, when setting the value space, the value of the third type of parameters cannot exceed the sum of the values of other parameters. Therefore, for example, the electronic device may obtain the value space of the third type of parameter according to the second constraint condition and/or the fourth constraint condition, that is, the electronic device may obtain the value space of the third type of parameter according to the value space of the parameter itself and/or the sum of the value of the parameter and the value of the other parameter.
It should be understood that, the electronic device may also obtain, through other manners, X initial parameters to be configured from parameters of the distributed storage system according to a constraint condition of parameter values, and a value space of each initial parameter to be configured.
Optionally, the electronic device may directly use all the determined initial parameters to be configured as parameters to be configured, and use a value space of the initial parameters to be configured of the distributed storage system as a value space of each parameter to be configured. That is, all configurable parameters obtained from the parameters of the distributed storage system are subjected to subsequent value obtaining. Alternatively, the electronic device may further perform subsequent steps S202-203 to further screen out some of these configurable parameters that have a greater impact on the performance of the distributed storage system.
S202, obtaining coefficients of the initial parameters to be configured based on the initial parameters to be configured and the value space of the initial parameters to be configured.
The coefficient of the initial parameter to be configured is positively correlated with the influence of the initial parameter to be configured on the performance of the distributed storage system. I.e. the larger the coefficient, the larger the initial parameter to be configured indicates that it has a greater impact on the performance of the distributed storage system.
Exemplarily, after acquiring the initial parameters to be configured and the value space of each initial parameter to be configured, the electronic device may randomly acquire Z initial values of each initial parameter to be configured from the value space of the initial parameter to be configured, where Z is a positive integer. And then testing the performance of the test cluster under the current hardware configuration and the current load by using each initial value of each initial parameter to be configured, and acquiring corresponding performance values, namely acquiring Z initial values of each initial parameter to be configured and the performance value corresponding to each initial value. And finally, acquiring the coefficient of each initial parameter to be configured according to the Z initial values of each initial parameter to be configured, each initial parameter to be configured and the performance value of the test cluster when each initial value of each initial parameter to be configured is used.
For example, the specific implementation manner of obtaining the coefficient of each initial parameter to be configured according to the Z initial values of each initial parameter to be configured, and the performance value of the distributed storage system when each initial value of each initial parameter to be configured is, for example, to use a LASSO Algorithm (LASSO) to process the Z initial values of each initial parameter to be configured, and to process the performance value of the distributed storage system when each initial value of each initial parameter to be configured is used, and then obtain the coefficient of each initial parameter to be configured.
It should be understood that, the present invention does not limit how the electronic device obtains the coefficient of each initial parameter to be configured based on each initial parameter to be configured and the value space of each initial parameter to be configured. The above method for obtaining the coefficient of each initial parameter to be configured is only one possible implementation manner proposed by the present invention. In specific implementation, the electronic device may further obtain the coefficient of each initial parameter to be configured by other methods based on each initial parameter to be configured and the value space of each initial parameter to be configured.
Optionally, before obtaining the coefficient of each initial parameter to be configured, the electronic device may further convert a value of a first type parameter (that is, a parameter whose value is a string) in the initial parameter to be configured into a plurality of dummy variables (dummy variables) to quantize a value of the first type parameter that cannot be quantitatively processed, and ensure that there is no attribute of a size relationship between values of the first type parameter, thereby improving accuracy of obtaining the coefficient of each initial parameter to be configured. The value of the dummy variable is a second value or a third value, for example, 0 or 1. Taking the osd _ objectstore parameter as an example, the osd _ objectstore parameter takes the values of blueStore, fileStore, memstor and kstore, the corresponding dummy variables are is _ blueStore, is _ fileStore, is _ memstor and is _ kstore, and the value of each dummy variable is 0 or 1.
Optionally, before obtaining the coefficient of each initial parameter to be configured, the electronic device may further perform normalization processing on Z initial values of the initial parameter to be configured and a performance value of the distributed storage system when each initial value of the initial parameter to be configured is used, so as to convert the initial values and the performance values of each initial parameter to be configured to the same order of magnitude, so that the values are more amenable to gaussian distribution, and thus, the accuracy of obtaining the coefficient of each initial parameter to be configured can be improved. In specific implementation, the electronic device may perform logarithmic conversion on Z initial values of the initial parameter to be configured and a value of which skewness is greater than a third preset threshold in performance values of the distributed storage system when each initial value of the initial parameter to be configured is used, by using log1 p. And then acquiring Z initial values of the initial parameter to be configured after the normalization processing, and performance values of the distributed storage system when each initial value of the initial parameter to be configured is used, and taking the values as a basis for acquiring the coefficient of the initial parameter to be configured.
S203, according to the sequence of the coefficients from high to low, taking the first Y initial parameters to be configured as the parameters to be configured, and taking the value space of the first Y initial parameters to be configured as the value space of the parameters to be configured.
After obtaining the coefficients of the initial parameters to be configured, the electronic device may sort the initial parameters to be configured according to the order of the coefficients from high to low. And according to the sorting result, taking the first Y initial parameters to be configured as the parameters to be configured, and taking the value space of the first Y initial parameters to be configured as the value space of the parameters to be configured. Therefore, the subsequent optimization process of the values of the parameters to be configured can be realized only by optimizing the values of Y parameters to be configured, and the time spent on the parameter configuration of the distributed storage system can be further reduced. Optionally, the electronic device may further use an initial parameter to be configured whose coefficient is greater than a fourth preset threshold as the parameter to be configured, and use a value space of the initial parameter to be configured whose coefficient is greater than the fourth preset threshold as the value space of the parameter to be configured.
In this embodiment, according to the constraint condition of the parameter value, the electronic device deletes the parameter whose value does not need to be found as the candidate value, obtains X initial parameters to be configured, and can reduce the time required for the candidate value finding process of the parameters to be configured. And then, setting the value space of each initial parameter to be configured so that the electronic equipment can search a candidate value of the parameter to be configured in the parameter value space to avoid the condition that the parameter value exceeds the value space to cause the distributed storage system to work abnormally. And then, based on each initial parameter to be configured and the value space of each initial parameter to be configured, obtaining the coefficient of each initial parameter to be configured, wherein the size of the coefficient can represent the influence of the initial parameter to be configured on the performance of the distributed storage system. And then, taking the first Y initial parameters to be configured as the parameters to be configured to find candidate values. Through the steps, the number of the parameters needing to be processed in the process of searching the candidate value of the parameter to be configured is further reduced, so that the time spent on parameter configuration of the distributed storage system is further reduced, and the parameter configuration efficiency of the distributed storage system is further improved.
The following describes an exemplary parameter configuration method provided by the present invention with a specific example. Fig. 5 is a schematic flow chart of another parameter configuration method provided by the present invention. As shown in fig. 5, the method comprises the steps of:
s301, receiving a parameter configuration request of the distributed storage system of the user. The parameter configuration request includes a parameter configuration target, where the parameter configuration target is used to indicate a target performance that the distributed storage system needs to achieve under a current hardware configuration and a current load.
S302, according to the constraint condition of parameter value, deleting the parameters which do not need to be subjected to value optimization from the parameters of the distributed storage system from a preset configuration parameter file to obtain X initial parameters to be configured and the value space of the initial parameters to be configured.
S303, randomly acquiring Z initial values of each initial parameter to be configured from the value space of the initial parameter to be configured.
S304, testing the performance of the test cluster under the current hardware configuration and the current load by using each initial value of each initial parameter to be configured, and acquiring a corresponding performance value.
S305, obtaining the coefficient of each initial parameter to be configured according to the Z initial values of each initial parameter to be configured, and the performance value of the test cluster when each initial value of each initial parameter to be configured is used.
S306, according to the sequence of the coefficients from high to low, taking the first Y initial parameters to be configured as the parameters to be configured, and taking the value space of the first Y initial parameters to be configured as the value space of the parameters to be configured.
It should be understood that, the electronic device executes the above step S301, and the sequence of the steps S302 to S306 is not limited in this application. That is, the electronic device may first perform step S301 and then perform steps S302 to S306, or may first perform steps S302 to S306 and then perform step S301.
And S307, in the value space of the parameter to be configured, assigning an initial value to the parameter to be configured in a random assignment mode to serve as a candidate value of the parameter to be configured.
S308, based on the test cluster, under the current hardware configuration and the current load, the candidate value of the parameter to be configured is used for carrying out performance test on the test cluster, and a corresponding performance value is obtained.
S309, acquiring candidate values of the parameters to be configured from the value space of the parameters to be configured according to the performance values under the current hardware configuration and the current load based on the test cluster.
S310, judging whether candidate values meeting the parameter configuration target are obtained or not. If the electronic device determines that the candidate value meeting the parameter configuration target is obtained, step S312 is executed. If the electronic device determines that the candidate value meeting the parameter configuration target is not obtained, step S311 is executed.
And S311, judging whether the maximum iteration number is reached. If the electronic device determines that the maximum number of iterations is reached, step S312 is executed. If the electronic device determines that the maximum number of iterations has not been reached, step S308 is executed.
And S312, determining a target value of the parameter to be configured. If the step is triggered by the step S310, that is, the end of the iteration loop is before the maximum iteration number is reached, the electronic device obtains the candidate value that satisfies the parameter configuration target. The target value of the parameter to be configured may be, for example, a candidate value that enables the distributed storage system to achieve the target performance, or a candidate value whose difference from the target performance is within a preset threshold range. If the step is triggered by the step S311, the end of the iteration loop reaches the maximum number of iterations. The target value of the parameter to be configured may be, for example, a candidate value corresponding to the optimal performance value, or a candidate value corresponding to a performance value greater than a preset performance threshold.
And S313, using the target value of the parameter to be configured to configure the parameter of the online cluster.
In this example, the parameters that do not need to be subjected to value optimization are deleted from the preset configuration parameter file, so that the time required for searching for candidate values of the parameters to be configured is reduced, and the efficiency of parameter configuration of the distributed storage system is further improved. And then, in the initial parameters to be configured, the parameters which have higher influence on the performance of the distributed storage system are used as the parameters to be configured, so that the efficiency and the accuracy of parameter configuration of the distributed storage system are further improved. And then according to the target performance of the distributed storage system, which needs to be achieved under the current hardware configuration and the current load, obtaining the candidate value of the parameter to be configured from the value space of the parameter to be configured. And determining a target value of the parameter to be configured from the candidate values of the parameter to be configured. The target value of the parameter to be configured of the distributed storage system is automatically acquired. And the target value of the parameter to be configured is obtained based on the test cluster, so that frequent restart of the distributed storage system caused by testing the performance values corresponding to the values by using the distributed storage system can be avoided, and further, the distributed storage system is prevented from being blocked to continuously provide services. In addition, the scale of the test cluster is smaller than that of the online cluster, so that the time consumed for performance test is relatively short, namely, the time consumed for parameter configuration of the distributed storage system is further reduced, and the efficiency of parameter configuration of the distributed storage system is further improved.
Fig. 6 is a schematic structural diagram of a parameter configuration apparatus provided in the present invention. As shown in fig. 6, the apparatus includes:
a receiving module 401, configured to receive a parameter configuration request of a distributed storage system, where the parameter configuration request includes a parameter configuration target, and the parameter configuration target is used to indicate a target performance that the distributed storage system needs to achieve under a current hardware configuration and a current load.
A first obtaining module 402, configured to obtain a parameter to be configured and a value space of the parameter to be configured from parameters of the distributed storage system according to a constraint condition of parameter value; wherein the constraint condition of the parameter value is related to the influence of the parameter on the operation of the distributed storage system.
A second obtaining module 403, configured to obtain, according to the parameter configuration target, N candidate values of the parameter to be configured from the value space of the parameter to be configured; in the Mth iteration, based on a performance value of a test cluster under the current hardware configuration and the current load and under the value of a parameter to be configured obtained by the M-1 th iteration, the Mth candidate value of the parameter to be configured is obtained from the value space of the parameter to be configured; wherein M is an integer greater than 1 and less than or equal to N.
A determining module 404, configured to determine a target value of the parameter to be configured among the N candidate values of the parameter to be configured.
Optionally, the determining module 404 is further configured to perform parameter configuration on the distributed storage system by using the target value of the parameter to be configured after determining the target value of the parameter to be configured.
Optionally, the first obtaining module 402 is specifically configured to obtain, according to a constraint condition of parameter value, X initial parameters to be configured from parameters of the distributed storage system, and a value space of each initial parameter to be configured; acquiring coefficients of the initial parameters to be configured based on the initial parameters to be configured and the value space of the initial parameters to be configured; according to the sequence of coefficients from high to low, taking Y initial parameters to be configured before the sequencing as the parameters to be configured, and taking the value space of the Y initial parameters to be configured before the sequencing as the value space of the parameters to be configured; wherein X is a positive integer; the coefficient is positively correlated with the influence of the initial parameter to be configured on the performance of the distributed storage system; y is an integer greater than 1 and less than or equal to X.
Optionally, the constraint condition of the parameter value includes at least one of the following: the first constraint condition is that whether the value of the constraint parameter is modifiable, the second constraint condition is that whether the value of the constraint parameter has a boundary, the third constraint condition is that whether the value of the constraint parameter is affected by the values of other parameters, the fourth constraint condition is that whether the sum of the value of the constraint parameter and the values of other parameters is equal to a preset value, and the fifth constraint condition is that whether the value of the constraint parameter is used for limiting the boundary of the values of other parameters.
Optionally, the first obtaining module 402 is specifically configured to obtain the X initial parameters to be configured from a preset configuration parameter file according to the first constraint condition, the third constraint condition, the fourth constraint condition, and the fifth constraint condition; setting the value space of a first type of parameters in the X initial parameters to be configured as continuous integers starting from a first value; setting the boundary of the value space of a second type of parameters in the X initial parameters to be configured as a boundary based on the dynamic expansion of the values of the second type of parameters; for a third type of parameters except the first type of parameters and the second type of parameters in the X initial parameters to be configured, acquiring a value space of the third type of parameters according to the second constraint condition and/or the fourth constraint condition; the first type of parameters are parameters taking values as character strings; the second type of parameter is a parameter lacking a value boundary.
Optionally, the first obtaining module 402 is specifically configured to randomly obtain Z initial values of each initial parameter to be configured from the value space of the initial parameter to be configured; for the Z initial values of each initial parameter to be configured, obtaining a performance value of the test cluster when each initial value of each initial parameter to be configured is used under the current hardware configuration and the current load; acquiring coefficients of the initial parameters to be configured according to the Z initial values of the initial parameters to be configured, the initial parameters to be configured and performance values of the distributed storage system when each initial value of the initial parameters to be configured is used; wherein Z is a positive integer.
Optionally, the first obtaining module 402 is further configured to, before the obtaining of the coefficient of each initial parameter to be configured, convert a first type parameter in the initial parameter to be configured into a dummy variable; and/or performing normalization processing on the Z initial values of the initial parameters to be configured and the performance value of the distributed storage system when each initial value of the initial parameters to be configured is used; the value of the dummy variable is a second value or a third value, and the first type parameter is a parameter with a value of a character string.
The parameter configuration apparatus provided by the present invention is used for executing the foregoing method embodiments, and the implementation principle and technical effect thereof are similar, which are not described again.
Fig. 7 is a schematic structural diagram of an electronic device according to the present invention. As shown in fig. 7, the electronic device 500 may include: at least one processor 501 and memory 502.
The memory 502 stores programs. In particular, the program may include program code comprising computer operating instructions.
Memory 502 may include high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The processor 501 is configured to execute computer-executable instructions stored in the memory 502 to implement the parameter configuration method. The processor 501 may be a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement the embodiments of the present invention.
Optionally, the electronic device 500 may further include a communication interface 503. In a specific implementation, if the communication interface 503, the memory 502 and the processor 501 are implemented independently, the communication interface 503, the memory 502 and the processor 501 may be connected to each other through a bus and perform communication with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (EISA) bus, or the like. Buses may be divided into address buses, data buses, control buses, etc., but do not represent only one bus or type of bus.
Optionally, in a specific implementation, if the communication interface 503, the memory 502, and the processor 501 are integrated into a chip, the communication interface 503, the memory 502, and the processor 501 may complete communication through an internal interface.
The invention also provides a parameter configuration system, which comprises the electronic device shown in fig. 7 and a test cluster.
As an example, the parameter configuration system includes: the electronic device in the embodiments described above in connection with any of fig. 3, 4 and 5, and the test cluster.
As another example, the parameter configuration system includes: the parameter configuration apparatus described above in connection with fig. 6.
The present invention also provides a computer-readable storage medium, which may include: various media that can store program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and in particular, the computer readable storage medium stores program instructions, and the program instructions are used for the method in the foregoing embodiments.
The present invention also provides a program product comprising execution instructions stored in a readable storage medium. The at least one processor of the electronic device may read the execution instruction from the readable storage medium, and the at least one processor executes the execution instruction to enable the electronic device to implement the parameter configuration method provided in the foregoing various embodiments.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A method of parameter configuration, the method comprising:
receiving a parameter configuration request of a distributed storage system, wherein the parameter configuration request comprises a parameter configuration target, and the parameter configuration target is used for indicating target performance to be achieved by the distributed storage system under the current hardware configuration and the current load;
acquiring a parameter to be configured from the parameters of the distributed storage system according to the constraint condition of parameter value, and obtaining the value space of the parameter to be configured; wherein the constraint condition of the parameter value is related to the influence of the parameter on the operation of the distributed storage system;
acquiring N candidate values of the parameter to be configured from the value space of the parameter to be configured according to the parameter configuration target; in the M iteration, based on a performance value of a test cluster under the current hardware configuration and the current load and under the value of a parameter to be configured obtained by the M-1 iteration, the M candidate value of the parameter to be configured is obtained from the value space of the parameter to be configured; m is an integer greater than 1 and less than or equal to N;
and determining a target value of the parameter to be configured in the N candidate values of the parameter to be configured.
2. The method according to claim 1, wherein after determining the target value of the parameter to be configured, the method further comprises:
and using the target value of the parameter to be configured to configure the parameter of the distributed storage system.
3. The method according to claim 1 or 2, wherein the obtaining a parameter to be configured from a parameter of the distributed storage system according to a constraint condition of a parameter value, and a value space of the parameter to be configured includes:
acquiring X initial parameters to be configured and a value space of each initial parameter to be configured from parameters of the distributed storage system according to constraint conditions of parameter values, wherein X is a positive integer;
acquiring coefficients of the initial parameters to be configured based on the initial parameters to be configured and the value space of the initial parameters to be configured; wherein the coefficient is positively correlated with the influence of the initial parameter to be configured on the performance of the distributed storage system;
according to the sequence of coefficients from high to low, taking Y initial parameters to be configured before the sequencing as the parameters to be configured, and taking the value space of the Y initial parameters to be configured before the sequencing as the value space of the parameters to be configured; wherein Y is an integer greater than 1 and less than or equal to X.
4. The method of claim 3, wherein the constraint on the value of the parameter comprises at least one of: the first constraint condition is that whether the value of the constraint parameter is modifiable, the second constraint condition is that whether the value of the constraint parameter has a boundary, the third constraint condition is that whether the value of the constraint parameter is affected by the values of other parameters, the fourth constraint condition is that whether the sum of the value of the constraint parameter and the values of other parameters is equal to a preset value, and the fifth constraint condition is that whether the value of the constraint parameter is used for limiting the boundary of the values of other parameters.
5. The method according to claim 4, wherein the obtaining X initial parameters to be configured from the parameters of the distributed storage system according to the constraint condition of parameter values, and the value space of each initial parameter to be configured includes:
obtaining the X initial parameters to be configured from a preset configuration parameter file according to the first constraint condition, the third constraint condition, the fourth constraint condition and the fifth constraint condition;
setting the value space of a first type of parameters in the X initial parameters to be configured as continuous integers starting from a first value; the first type of parameters are parameters taking values as character strings;
setting the boundary of the value space of a second type parameter in the X initial parameters to be configured as a boundary based on the value dynamic expansion of the second type parameter; wherein the second type of parameter is a parameter lacking a value boundary;
and for a third type of parameters except the first type of parameters and the second type of parameters in the X initial parameters to be configured, acquiring a value space of the third type of parameters according to the second constraint condition and/or the fourth constraint condition.
6. The method according to claim 3, wherein the obtaining the coefficient of each initial parameter to be configured based on each initial parameter to be configured and a value space of each initial parameter to be configured comprises:
randomly acquiring Z initial values of each initial parameter to be configured from the value space of the initial parameter to be configured, wherein Z is a positive integer;
for the Z initial values of each initial parameter to be configured, obtaining a performance value of the test cluster when each initial value of each initial parameter to be configured is used under the current hardware configuration and the current load;
and acquiring coefficients of the initial parameters to be configured according to the Z initial values of the initial parameters to be configured, the initial parameters to be configured and performance values of the distributed storage system when each initial value of the initial parameters to be configured is used.
7. The method according to claim 6, wherein before obtaining the coefficient of each of the initial parameters to be configured, the method further comprises:
converting the first type of parameters in the initial parameters to be configured into dummy variables; the value of the dummy variable is a second value or a third value, and the first type parameter is a parameter with a value of a character string; and/or the presence of a gas in the gas,
and normalizing the Z initial values of the initial parameter to be configured and the performance value of the distributed storage system when each initial value of the initial parameter to be configured is used.
8. An apparatus for parameter configuration, the apparatus comprising:
the system comprises a receiving module, a parameter configuration module and a load module, wherein the receiving module is used for receiving a parameter configuration request of the distributed storage system, the parameter configuration request comprises a parameter configuration target, and the parameter configuration target is used for indicating target performance of the distributed storage system to be achieved under current hardware configuration and current load;
the first acquisition module is used for acquiring a parameter to be configured and a value space of the parameter to be configured from the parameter of the distributed storage system according to a constraint condition of parameter value; wherein the constraint condition of the parameter value is related to the influence of the parameter on the operation of the distributed storage system;
a second obtaining module, configured to obtain, according to the parameter configuration target, N candidate values of the parameter to be configured from the value space of the parameter to be configured; in the M iteration, based on a performance value of a test cluster under the current hardware configuration and the current load and under the value of a parameter to be configured obtained by the M-1 iteration, the M candidate value of the parameter to be configured is obtained from the value space of the parameter to be configured; m is an integer greater than 1 and less than or equal to N;
and the determining module is used for determining a target value of the parameter to be configured in the N candidate values of the parameter to be configured.
9. An electronic device, comprising: at least one processor, a memory;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the electronic device to perform the method of any of claims 1-7.
10. A computer-readable storage medium having computer-executable instructions stored thereon which, when executed by a processor, implement the method of any one of claims 1-7.
CN202011301312.8A 2020-11-19 2020-11-19 Parameter configuration method and device, electronic equipment and storage medium Active CN112417041B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011301312.8A CN112417041B (en) 2020-11-19 2020-11-19 Parameter configuration method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011301312.8A CN112417041B (en) 2020-11-19 2020-11-19 Parameter configuration method and device, electronic equipment and storage medium

Publications (2)

Publication Number Publication Date
CN112417041A CN112417041A (en) 2021-02-26
CN112417041B true CN112417041B (en) 2022-12-02

Family

ID=74774775

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011301312.8A Active CN112417041B (en) 2020-11-19 2020-11-19 Parameter configuration method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN112417041B (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109857451A (en) * 2019-02-13 2019-06-07 广东高云半导体科技股份有限公司 IP kernel document generating method, device, equipment and medium
CN110413340A (en) * 2019-06-28 2019-11-05 三一重机有限公司 Method for parameter configuration, device, equipment and the readable storage medium storing program for executing of engineering equipment
CN110472889A (en) * 2019-08-22 2019-11-19 泰康保险集团股份有限公司 Resource allocation method, device for allocating resources, storage medium and electronic equipment

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109857451A (en) * 2019-02-13 2019-06-07 广东高云半导体科技股份有限公司 IP kernel document generating method, device, equipment and medium
CN110413340A (en) * 2019-06-28 2019-11-05 三一重机有限公司 Method for parameter configuration, device, equipment and the readable storage medium storing program for executing of engineering equipment
CN110472889A (en) * 2019-08-22 2019-11-19 泰康保险集团股份有限公司 Resource allocation method, device for allocating resources, storage medium and electronic equipment

Also Published As

Publication number Publication date
CN112417041A (en) 2021-02-26

Similar Documents

Publication Publication Date Title
CN106294120B (en) Method, apparatus and computer program product for testing code
CN111562884B (en) Data storage method and device and electronic equipment
CN112417041B (en) Parameter configuration method and device, electronic equipment and storage medium
CN112165505B (en) Decentralized data processing method, electronic device and storage medium
CN108463813B (en) Method and device for processing data
CN111966490A (en) Spark partition load balancing method
US7756594B2 (en) Systems and methods for parsing flexible audio codec topologies
CN116702668A (en) Regression testing method and device, electronic equipment and storage medium
CN113849160B (en) Dynamic migration method and device for compiling tasks, electronic equipment and storage medium
CN113795032B (en) Method and device for judging invisible faults of indoor division, storage medium and equipment
CN115509931A (en) System-based performance test method and device, electronic equipment and storage medium
CN109344049B (en) Method and apparatus for testing a data processing system
CN111800446B (en) Scheduling processing method, device, equipment and storage medium
CN113010310A (en) Job data processing method and device and server
CN112231236A (en) Method for testing database performance
CN111885159A (en) Data acquisition method and device, electronic equipment and storage medium
CN111162963A (en) Method and system for testing performance of network communication equipment based on binary iteration method
CN110795251A (en) Method and equipment for determining terminal operation parameters
CN111324532B (en) MPI parameter determination method, device and equipment of parallel computing software
CN115389911B (en) Chip scheduler fault judgment method and device, electronic equipment and storage medium
CN116414714A (en) Interface test parameter optimization method, equipment and medium
CN116501405A (en) Server configuration method, device, equipment and medium
CN113886059A (en) Database data processing method, device, equipment and storage medium
CN115333924A (en) Broadband network repairing method, device, electronic equipment and medium
CN117834530A (en) Method, system, equipment and medium for processing enterprise batch service flow

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant