CN113687780B - QoS optimization method, system, terminal and storage medium for distributed storage server - Google Patents
QoS optimization method, system, terminal and storage medium for distributed storage server Download PDFInfo
- Publication number
- CN113687780B CN113687780B CN202110872204.4A CN202110872204A CN113687780B CN 113687780 B CN113687780 B CN 113687780B CN 202110872204 A CN202110872204 A CN 202110872204A CN 113687780 B CN113687780 B CN 113687780B
- Authority
- CN
- China
- Prior art keywords
- processing speed
- request processing
- client
- population
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/004—Artificial life, i.e. computing arrangements simulating life
- G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Computer And Data Communications (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a QoS (quality of service) optimization method, a QoS optimization system, a QoS optimization terminal and a QoS optimization storage medium for a distributed storage server, wherein the QoS optimization method comprises the following steps: presetting a value range of each parameter item of a service quality template; generating an initial population of the multi-objective optimization algorithm according to the value range of each parameter item by utilizing a random algorithm; pre-constructing a client request processing speed target function and an internal request processing speed target function based on parameter item values of a service quality template; calculating population fitness by using a client request processing speed objective function and an internal request processing speed objective function; and selecting a population with high fitness to perform optimization iteration according to a non-dominant sorting method to obtain an optimal scheme set of parameter item values of the service quality template. The invention can effectively process the I/O requests inside and outside the dispatching server to obtain the optimal configuration parameters, and the user can configure QoS template parameters according to the optimal solution sets and the actual needs thereof, thereby improving the service quality of the clusters.
Description
Technical Field
The invention relates to the technical field of distributed storage, in particular to a distributed storage server QoS (quality of service) optimization method, a system, a terminal and a storage medium.
Background
Ceph is a new generation of free software distributed file system designed specifically for doctor papers by Sage Weil (the joint creator of DreamHost) at Santa Cruz division of California university. After the graduation of 2007, sage began to be put into Ceph development in full time, making it suitable for use in a production environment. The main goal of Ceph is to design a POSIX-based distributed file system without a single point of failure, enabling fault-tolerant and seamless replication of data. Month 3 2010, linus Torvalds incorporated Ceph clients into kernel 2.6.34. An article by IBM developer in the garden discusses the architecture of Ceph, fault tolerant implementation, and simplified functions of mass data management.
In Ceph clusters, efficient IO scheduling is crucial as an object storage OSD module that handles client I/O requests and I/O operations generated internally to the system. Currently, the same type of external I/O request is usually regarded as a client, including client_op and osd_subtop, and different types of I/O such as snaptrim, recovery and scrub are generated internally by the object storage device OSD. To schedule these different types of I/O requests, the dmClock algorithm is introduced, simply by setting up QoS templates in the server for each type of I/O. Taking two important I/O operations of client_op and recovery as an example, each operation has reservation, weight, limit reserved, weighted and upper limit parameters, namely: osd_op_queue_mclock_client_op_res, osd_op_queue_mclock_client_op_wgt, osd_op_queue_mclock_client_op_lim, osd_op_queue_mclock_recovery_res, osd_op_queue_mclock_recovery_wgt, osd_op_queue_mclock_recovery_lim.
In Ceph cluster, adopting dm clock has at least two purposes, firstly, realizing unified I/O scheduling, solving the contention problem of external client side and internal I/O of cluster; and secondly, the I/O processing in different scenes is better satisfied. Of course, on the premise of a certain server processing speed, the improvement of the I/O of the external client end inevitably reduces the processing of the I/O in the cluster, and how to set the QoS template parameters is worth the important attention of the staff. The current QoS template parameter setting is generally set according to experience of a user, and is difficult to meet the complex and diverse requirements of the current large-scale storage cluster.
Disclosure of Invention
Aiming at the fact that the prior art relies heavily on experience of operation and maintenance personnel, parameters of each Ceph cluster are different, parameter setting is not common, and the manual setting of QoS template parameters is not optimal value in most cases, so that the I/O request processing speed is not optimal, and the QoS template parameter setting becomes a bottleneck for improving the I/O request processing speed. The invention provides a QoS (quality of service) optimization method, a QoS optimization system, a QoS optimization terminal and a QoS optimization storage medium for a distributed storage server, so as to solve the technical problems.
In a first aspect, the present invention provides a QoS optimization method for a distributed storage server, including:
presetting a value range of each parameter item of a service quality template;
generating an initial population of the multi-objective optimization algorithm according to the value range of each parameter item by utilizing a random algorithm;
pre-constructing a client request processing speed target function and an internal request processing speed target function based on parameter item values of a service quality template;
calculating population fitness by using a client request processing speed objective function and an internal request processing speed objective function;
and selecting a population with high fitness to perform optimization iteration according to a non-dominant sorting method to obtain an optimal scheme set of parameter item values of the service quality template.
Further, presetting a value range of each parameter item of the service quality template, which comprises the following steps:
and respectively inputting the upper limit value and the lower limit value of each parameter item according to the cluster limiting rule.
Further, the parameter items of the qos template include:
client reservation parameters, client weight parameters, client upper limit parameters, internal reservation parameters, internal weight parameters, and internal upper limit parameters.
Further, generating an initial population of the multi-objective optimization algorithm according to the value range of each parameter item by using a random algorithm, wherein the initial population comprises the following components:
presetting the number of individuals of an initial population;
generating a plurality of individuals according to the value ranges of the parameter items by utilizing a random algorithm, wherein the value of the parameter item in each individual is a value randomly selected in the corresponding value range;
the number of the individuals is equal to the number of individuals of a preset initial population, and the individuals form the initial population.
Further, pre-constructing a client request processing speed objective function and an internal request processing speed objective function based on the parameter item value of the service quality template, including:
using the formulaCalculating an initial value of a client request processing speed, wherein cr is reserved for a client, cw is a client weight, rr is reserved internally, rw is an internal weight, and M is a cluster maximum performance value; comparing the initial value of the client request processing speed with the upper limit of the client, and taking a smaller value from the initial value and the upper limit as the client request processing speed;
using the formulaCalculating an initial value of an internal request processing speed, wherein cr is reserved for a client, cw is weight of the client, rr is reserved internally, rw is weight internally, and M is the maximum performance value of the cluster; comparing the initial value of the internal request processing speed with the internal upper limit, and taking the smaller value from the initial value and the internal upper limit as the internal request processing speed.
Further, selecting a population with high fitness for optimization iteration according to a non-dominant sorting method to obtain an optimal scheme set of parameter item values of a service quality template, wherein the optimal scheme set comprises the following steps:
non-dominant ordering individuals in the existing population according to fitness, and selecting individuals with the half population quantity which is ranked at the front as preferred individuals;
performing mutation operation and crossover operation on the preferred individuals to generate a secondary population;
sorting the secondary population and the previous generation population of the secondary population according to a non-dominant rule, selecting individuals with the number of preset populations which are ranked in front as the secondary population, and iterating;
and monitoring the iteration times, stopping iteration if the actual iteration times reach the preset iteration times, and outputting the non-dominant solution set in the latest population as an optimal solution set.
In a second aspect, the present invention provides a distributed storage server QoS optimization system, including:
a range setting unit, configured to preset a value range of each parameter item of the quality of service template;
the initial generation unit is used for generating an initial population of the multi-objective optimization algorithm according to the value range of each parameter item by utilizing a random algorithm;
the function construction unit is used for constructing a client request processing speed target function and an internal request processing speed target function based on the parameter item value of the service quality template in advance;
the adaptation calculation unit is used for calculating population adaptation degree by utilizing the client request processing speed objective function and the internal request processing speed objective function;
and the iteration screening unit is used for selecting a population with high fitness for optimization iteration according to a non-dominant sorting method to obtain a parameter item value optimal scheme set of the service quality template.
Further, the range setting unit is configured to input an upper limit value and a lower limit value of each parameter item according to the cluster restriction rule. The parameter items of the quality of service template include: client reservation parameters, client weight parameters, client upper limit parameters, internal reservation parameters, internal weight parameters, and internal upper limit parameters.
Further, the initial generation unit is used for presetting the number of individuals of the initial population;
generating a plurality of individuals according to the value ranges of the parameter items by utilizing a random algorithm, wherein the value of the parameter item in each individual is a value randomly selected in the corresponding value range;
the number of the individuals is equal to the number of individuals of a preset initial population, and the individuals form the initial population.
Further, the function construction unit is configured to utilize the formulaCalculating an initial value of a client request processing speed, wherein cr is reserved for a client, cw is a client weight, rr is reserved internally, rw is an internal weight, and M is a cluster maximum performance value; comparing the initial value of the processing speed of the client request with the upper limit of the client, and taking the smaller value from the initial value and the upper limit as the client requestSolving the processing speed;
using the formulaCalculating an initial value of an internal request processing speed, wherein cr is reserved for a client, cw is weight of the client, rr is reserved internally, rw is weight internally, and M is the maximum performance value of the cluster; comparing the initial value of the internal request processing speed with the internal upper limit, and taking the smaller value from the initial value and the internal upper limit as the internal request processing speed.
Further, the iterative screening unit is used for non-dominantly sorting individuals in the existing population according to the fitness, and selecting individuals with the half population quantity which is ranked at the front as preferred individuals;
performing mutation operation and crossover operation on the preferred individuals to generate a secondary population;
sorting the secondary population and the previous generation population of the secondary population according to a non-dominant rule, selecting individuals with the number of preset populations which are ranked in front as the secondary population, and iterating;
and monitoring the iteration times, stopping iteration if the actual iteration times reach the preset iteration times, and outputting the non-dominant solution set in the latest population as an optimal solution set.
In a third aspect, a terminal is provided, including:
a processor, a memory, wherein,
the memory is used for storing a computer program,
the processor is configured to call and run the computer program from the memory, so that the terminal performs the method of the terminal as described above.
In a fourth aspect, there is provided a computer storage medium having instructions stored therein which, when run on a computer, cause the computer to perform the method of the above aspects.
The invention has the advantages that,
the QoS optimization method of the distributed storage server provided by the invention comprises the steps of presetting the value ranges of all parameter items of a service quality template, generating an initial population of a multi-objective optimization algorithm according to the value ranges of all parameter items by utilizing a random algorithm, constructing a client request processing speed objective function and an internal request processing speed objective function based on the value of the parameter items of the service quality template, calculating population fitness by utilizing the client request processing speed objective function and the internal request processing speed objective function, and finally selecting a population with high fitness for optimization iteration according to a non-dominant sorting method to obtain an optimal scheme set of the value of the parameter items of the service quality template. The invention can effectively process the I/O requests inside and outside the dispatching server to obtain the optimal configuration parameters, and the user can configure QoS template parameters according to the optimal solution sets and the actual needs thereof, thereby improving the service quality of the clusters.
The distributed storage server QoS optimization system provided by the invention has the advantages that the range setting unit is used for presetting the value ranges of all parameter items of the service quality template, then the initial generation unit is used for generating an initial population of the multi-objective optimization algorithm according to the value ranges of all parameter items by utilizing the random algorithm, meanwhile, the function construction unit is used for constructing a client request processing speed objective function and an internal request processing speed objective function based on the value of the parameter items of the service quality template, the adaptation calculation unit is used for calculating the population fitness by utilizing the client request processing speed objective function and the internal request processing speed objective function, and finally the iteration screening unit is used for selecting the population with high fitness according to the non-dominant sorting method to carry out optimization iteration, so that the parameter item value optimal scheme set of the service quality template is obtained. The invention can effectively process the I/O requests inside and outside the dispatching server to obtain the optimal configuration parameters, and the user can configure QoS template parameters according to the optimal solution sets and the actual needs thereof, thereby improving the service quality of the clusters.
The terminal provided by the invention comprises the processor for running the QoS optimization method of the distributed storage server, the invention can effectively process the I/O requests inside and outside the scheduling server to obtain the optimal configuration parameters, and a user can configure QoS template parameters according to the optimal solution sets and the actual needs of the user, thereby improving the service quality of the cluster.
The storage medium provided by the invention stores a program for executing the QoS optimization method of the distributed storage server, the invention can effectively process the I/O requests inside and outside the scheduling server to obtain the optimal configuration parameters, and a user can configure QoS template parameters according to the optimal solution sets and the actual requirements of the user, thereby improving the service quality of the cluster.
In addition, the invention has reliable design principle, simple structure and very wide application prospect.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the description of the embodiments or the prior art will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.
FIG. 1 is a schematic flow chart of a method of one embodiment of the invention.
FIG. 2 is a schematic block diagram of a system of one embodiment of the present invention.
Fig. 3 is a schematic structural diagram of a terminal according to an embodiment of the present invention.
Detailed Description
In order to make the technical solution of the present invention better understood by those skilled in the art, the technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
The following explains key terms appearing in the present invention.
QOS (Quality of Service ) refers to a network that can utilize various basic technologies to provide better service capability for specified network communications, and is a security mechanism of the network, which is a technology for solving the problems of network delay and congestion. The guarantee of QoS is important for networks with limited capacity, in particular for streaming multimedia applications, such as VoIP and IPTV, since these applications often require a fixed transmission rate and are also relatively delay sensitive.
Five classes of operations for client_op, osd_ subop, snaptrim, recovery, and scrub OSD
dmClock-distributed mclock algorithm and I/O scheduling algorithm based on time labels
Pareto solution set Pareto optimal solution set, optimal solution set of multi-objective optimization problem
reservation, weight, limit reservation, weight, upper limit
FIG. 1 is a schematic flow chart of a method of one embodiment of the invention. The execution body of fig. 1 may be a distributed storage server QoS optimization system.
As shown in fig. 1, the method includes:
step 110, presetting the value range of each parameter item of the service quality template;
step 120, generating an initial population of the multi-objective optimization algorithm according to the value range of each parameter item by using a random algorithm;
step 130, constructing a client request processing speed target function and an internal request processing speed target function based on the parameter item value of the service quality template in advance;
step 140, calculating population fitness by using the client request processing speed objective function and the internal request processing speed objective function;
and 150, selecting a population with high fitness for optimization iteration according to a non-dominant sorting method to obtain an optimal scheme set of parameter item values of the service quality template.
In order to facilitate understanding of the present invention, the distributed storage server QoS optimization method provided by the present invention is further described below by using the principle of the distributed storage server QoS optimization method according to the present invention, and combining with the process of optimizing the distributed storage server QoS in the embodiment.
There are many multi-objective optimization problems in storage systems, that is, multiple objectives are in conflict with each other, and the result of one objective becomes better and the result of the other objective becomes worse. As researchers have conducted intensive research into solving multi-objective problems, many excellent solving algorithms have emerged. The multi-objective evolutionary algorithm is the most representative processing means, and can find a non-dominant Pareto solution set corresponding to the approximate Pareto front in a decision space, namely an optimal solution set. The decision maker may select a desired solution from the solution set depending on the actual application.
Specifically, in this embodiment, the I/O request speed of the external client and the I/O operation speed generated in the processing cluster are used as optimization targets, so that the QoS optimization problem of the server in the distributed storage is converted into a multi-target problem, and the multi-target evolutionary algorithm is adopted to obtain the optimal configuration parameters. The QoS optimization method for the distributed storage server provided by the embodiment comprises the following steps:
s1, presetting a value range of each parameter item of a service quality template.
And respectively inputting the upper limit value and the lower limit value of each parameter item according to the cluster limiting rule. The parameter items of the quality of service template in this embodiment include: client reservation parameters, client weight parameters, client upper limit parameters, internal reservation parameters, internal weight parameters, and internal upper limit parameters.
In other embodiments of the present invention, the quality of service template may contain multiple types of I/O request parameters, but each I/O request parameter includes a reservation parameter, a weight parameter, and an upper limit parameter.
Therefore, the present embodiment mainly uses, as optimized parameters, the osd_op_queue_mcyclock_client_op_res (cr), the osd_op_queue_mcyclock_client_op_wgt (cw), the osd_op_queue_mcyclock_client_op_lim (Cl), the osd_op_queue_mcyclock_recov_res (rr), and the osd_op_queue_mcyclock_recov_lim (rw), the osd_op_queue_mcyclock_lim (rl), and the I/O request speed (Cl) of the external client and the I/O operation speed (In) generated inside the processing cluster.
First, 6 search fields of parameters, namely a feasible solution field, namely a population individual number P and a maximum iteration number N, of osd_op_queue_mclock_op_res, osd_op_queue_mclock_mclock_client_op_wgt, osd_op_queue_mclock_mclock_client_op_lim, osd_op_queue_mclock_recovery_whgt and osd_op_queue_mclock_recovery_lim are input.
S2, generating an initial population of the multi-objective optimization algorithm according to the value range of each parameter item by using a random algorithm.
Generating a plurality of individuals according to the value ranges of the parameter items by utilizing a random algorithm, wherein the value of the parameter item in each individual is a value randomly selected in the corresponding value range; the number of the individuals is equal to the number of individuals of a preset initial population, and the individuals form the initial population.
I.e. using a random algorithm, P particles are generated in the feasible solution domain, each particle representing a feasible solution, where the solution is a parameter value scheme of a quality of service template.
S3, constructing a client request processing speed target function and an internal request processing speed target function based on the parameter item value of the service quality template in advance.
Using the formulaCalculating an initial value of a client request processing speed, wherein cr is reserved for a client, cw is a client weight, rr is reserved internally, rw is an internal weight, and M is a cluster maximum performance value; comparing the initial value of the client request processing speed with the upper limit of the client, and taking a smaller value from the initial value and the upper limit as the client request processing speed; using the formula->Calculating an initial value of an internal request processing speed, wherein cr is reserved for a client, cw is weight of the client, rr is reserved internally, rw is weight internally, and M is the maximum performance value of the cluster; comparing the initial value of the internal request processing speed with the internal upper limit, and taking the smaller value from the initial value and the internal upper limit as the internal request processing speed.
S4, calculating population fitness by using the client request processing speed objective function and the internal request processing speed objective function.
Combining the client request processing speed objective function and the internal request processing speed objective function into one objective function:
where CL is the client request processing speed objective function and In is the internal request processing speed objective function. cr is reserved for the client, cw is the weight of the client, rr is reserved internally, rw is the weight internally, M is the maximum performance value of the cluster, cl is the upper limit of the client, and rl is the upper limit internally.
And S5, selecting a population with high fitness for optimization iteration according to a non-dominant sorting method, and obtaining a parameter item value optimal scheme set of the service quality template.
Non-dominant ordering individuals in the existing population according to fitness, and selecting individuals with the half population quantity which is ranked at the front as preferred individuals; performing mutation operation and crossover operation on the preferred individuals to generate a secondary population; sorting the secondary population and the previous generation population of the secondary population according to a non-dominant rule, selecting individuals with the number of preset populations which are ranked in front as the secondary population, and iterating; and monitoring the iteration times, stopping iteration if the actual iteration times reach the preset iteration times, and outputting the non-dominant solution set in the latest population as an optimal solution set.
The specific iteration flow is as follows:
the particles are screened from the current population, the P particles are ordered according to the fitness of each particle and the non-dominant rule, and the particles ranked in the top 50% are selected to enter the next step.
Performing mutation, crossing and selection operation on the P/2 particles to generate a secondary population, and calculating the fitness value of each particle according to the objective function of the step S4; wherein the compiling operation is performed by multiplying each parameter value in the individual scheme by a coefficient; the cross operation is the random combination of parameter values of different individual schemes.
Sequencing the secondary population and the primary population together according to a non-dominant rule, and taking the first P particles as the population of the next generation; "non-dominant solution" is defined as: assuming that any two solutions S1 and S2 are better than S2 for all targets, we call S1 dominant S2, and S1 non-dominant if the solution of S1 is not dominant by the other solutions.
And outputting the Pareto solution set in the population when the maximum iteration number is reached, otherwise, continuing to perform fitness screening and iteration.
The obtained Pareto solution set is the optimal parameter model pool.
The QoS optimization method of the distributed storage server provided by the embodiment can effectively process the I/O requests inside and outside the scheduling server to obtain the optimal configuration parameters, and a user can configure QoS template parameters according to the optimal solution sets and the actual requirements of the user, so that the service quality of the cluster is improved.
As shown in fig. 2, the system 200 includes:
a range setting unit 210, configured to preset a value range of each parameter item of the qos template;
an initial generation unit 220, configured to generate an initial population of the multi-objective optimization algorithm according to the value range of each parameter item by using a random algorithm;
a function construction unit 230, configured to construct in advance a client request processing speed objective function and an internal request processing speed objective function based on the values of the parameter items of the quality of service template;
an adaptation calculation unit 240 for calculating population adaptation degree using the client request processing speed objective function and the internal request processing speed objective function;
and the iteration screening unit 250 is used for selecting a population with high fitness for optimization iteration according to a non-dominant sorting method to obtain a parameter item value optimal scheme set of the service quality template.
Alternatively, as an embodiment of the present invention, the range setting unit is configured to input the upper limit value and the lower limit value of each parameter item according to the cluster restriction rule, respectively. The parameter items of the quality of service template include: client reservation parameters, client weight parameters, client upper limit parameters, internal reservation parameters, internal weight parameters, and internal upper limit parameters.
Optionally, as an embodiment of the present invention, the initial generating unit is configured to preset the number of individuals in the initial population; generating a plurality of individuals according to the value ranges of the parameter items by utilizing a random algorithm, wherein the value of the parameter item in each individual is a value randomly selected in the corresponding value range; the number of the individuals is equal to the number of individuals of a preset initial population, and the individuals form the initial population.
Alternatively, as an embodiment of the present invention, the function construction unit is configured to use a formulaCalculating an initial value of a client request processing speed, wherein cr is reserved for a client, cw is a client weight, rr is reserved internally, rw is an internal weight, and M is a cluster maximum performance value; comparing the initial value of the client request processing speed with the upper limit of the client, and taking a smaller value from the initial value and the upper limit as the client request processing speed; using the formula->Calculating an initial value of an internal request processing speed, wherein cr is reserved for a client, cw is weight of the client, rr is reserved internally, rw is weight internally, and M is the maximum performance value of the cluster; comparing the initial value of the internal request processing speed with the internal upper limit, and taking the smaller value from the initial value and the internal upper limit as the internal request processing speed.
Optionally, as an embodiment of the present invention, the iterative screening unit is configured to non-dominantly sort the individuals in the existing population according to fitness, and select, as the preferred individuals, the individuals in the half population number of the first sorted individuals; performing mutation operation and crossover operation on the preferred individuals to generate a secondary population; sorting the secondary population and the previous generation population of the secondary population according to a non-dominant rule, selecting individuals with the number of preset populations which are ranked in front as the secondary population, and iterating; and monitoring the iteration times, stopping iteration if the actual iteration times reach the preset iteration times, and outputting the non-dominant solution set in the latest population as an optimal solution set.
Fig. 3 is a schematic structural diagram of a terminal 300 according to an embodiment of the present invention, where the terminal 300 may be used to execute the QoS optimization method of the distributed storage server according to the embodiment of the present invention.
The terminal 300 may include: a processor 310, a memory 320 and a communication unit 330. The components may communicate via one or more buses, and it will be appreciated by those skilled in the art that the configuration of the server as shown in the drawings is not limiting of the invention, as it may be a bus-like structure, a star-like structure, or include more or fewer components than shown, or may be a combination of certain components or a different arrangement of components.
The memory 320 may be used to store instructions for execution by the processor 310, and the memory 320 may be implemented by any type of volatile or non-volatile memory terminal or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. The execution of the instructions in memory 320, when executed by processor 310, enables terminal 300 to perform some or all of the steps in the method embodiments described below.
The processor 310 is a control center of the storage terminal, connects various parts of the entire electronic terminal using various interfaces and lines, and performs various functions of the electronic terminal and/or processes data by running or executing software programs and/or modules stored in the memory 320, and invoking data stored in the memory. The processor may be comprised of an integrated circuit (Integrated Circuit, simply referred to as an IC), for example, a single packaged IC, or may be comprised of a plurality of packaged ICs connected to the same function or different functions. For example, the processor 310 may include only a central processing unit (Central Processing Unit, simply CPU). In the embodiment of the invention, the CPU can be a single operation core or can comprise multiple operation cores.
And a communication unit 330 for establishing a communication channel so that the storage terminal can communicate with other terminals. Receiving user data sent by other terminals or sending the user data to other terminals.
The present invention also provides a computer storage medium in which a program may be stored, which program may include some or all of the steps in the embodiments provided by the present invention when executed. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random-access memory (random access memory, RAM), or the like.
Therefore, the invention sets the value range of each parameter item of the service quality template in advance, then generates the initial population of the multi-objective optimization algorithm according to the value range of each parameter item by utilizing a random algorithm, simultaneously constructs the client request processing speed objective function and the internal request processing speed objective function based on the value of the parameter item of the service quality template, calculates the population fitness by utilizing the client request processing speed objective function and the internal request processing speed objective function, and finally selects the population with high fitness for optimization iteration according to a non-dominant sorting method to obtain the parameter item value optimal scheme set of the service quality template. The invention can effectively process the I/O requests inside and outside the scheduling server to obtain the optimal configuration parameters, and the user can configure QoS template parameters according to the optimal solution sets and the actual needs of the user, so that the service quality of the cluster is improved, and the technical effects achieved by the embodiment can be seen from the description above and are not repeated here.
It will be apparent to those skilled in the art that the techniques of embodiments of the present invention may be implemented in software plus a necessary general purpose hardware platform. Based on such understanding, the technical solution in the embodiments of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium such as a U-disc, a mobile hard disc, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk or an optical disk, etc. various media capable of storing program codes, including several instructions for causing a computer terminal (which may be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the method described in the embodiments of the present invention.
The same or similar parts between the various embodiments in this specification are referred to each other. In particular, for the terminal embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and reference should be made to the description in the method embodiment for relevant points.
In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods may be implemented in other ways. For example, the system embodiments described above are merely illustrative, e.g., the division of the elements is merely a logical functional division, and there may be additional divisions when actually implemented, e.g., multiple elements or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interface, system or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
Although the present invention has been described in detail by way of preferred embodiments with reference to the accompanying drawings, the present invention is not limited thereto. Various equivalent modifications and substitutions may be made in the embodiments of the present invention by those skilled in the art without departing from the spirit and scope of the present invention, and it is intended that all such modifications and substitutions be within the scope of the present invention/be within the scope of the present invention as defined by the appended claims. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (7)
1. The QoS optimization method for the distributed storage service end is characterized by comprising the following steps:
presetting a value range of each parameter item of a service quality template;
generating an initial population of the multi-objective optimization algorithm according to the value range of each parameter item by utilizing a random algorithm;
pre-constructing a client request processing speed target function and an internal request processing speed target function based on parameter item values of a service quality template;
calculating population fitness by using a client request processing speed objective function and an internal request processing speed objective function;
selecting a population with high fitness for optimization iteration according to a non-dominant sorting method to obtain an optimal scheme set of parameter item values of a service quality template;
presetting a value range of each parameter item of a service quality template, wherein the value range comprises:
respectively inputting an upper limit value and a lower limit value of each parameter item according to a cluster limiting rule;
the parameter items of the quality of service template include:
client reservation parameters, client weight parameters, client upper limit parameters, internal reservation parameters, internal weight parameters and internal upper limit parameters;
pre-constructing a client request processing speed target function and an internal request processing speed target function based on parameter item values of a service quality template, wherein the client request processing speed target function and the internal request processing speed target function comprise the following steps:
using the formulaCalculating the processing speed of a client requestA value, wherein cr is reserved for a client, cw is a client weight, rr is reserved internally, rw is an internal weight, and M is a cluster maximum performance value; comparing the initial value of the client request processing speed with the upper limit of the client, and taking a smaller value from the initial value and the upper limit as the client request processing speed;
using the formulaCalculating an initial value of an internal request processing speed, wherein cr is reserved for a client, cw is weight of the client, rr is reserved internally, rw is weight internally, and M is the maximum performance value of the cluster; comparing the initial value of the internal request processing speed with the internal upper limit, and taking the smaller value from the initial value and the internal upper limit as the internal request processing speed.
2. The method of claim 1, wherein generating an initial population of the multi-objective optimization algorithm based on the range of values for each parameter item using a random algorithm comprises:
presetting the number of individuals of an initial population;
generating a plurality of individuals according to the value ranges of the parameter items by utilizing a random algorithm, wherein the value of the parameter item in each individual is a value randomly selected in the corresponding value range;
the number of the individuals is equal to the number of individuals of a preset initial population, and the individuals form the initial population.
3. The method of claim 1, wherein selecting a population with high fitness for optimization iteration according to a non-dominant ranking method to obtain a set of optimal solution for parameter term values of a quality of service template, comprises:
non-dominant ordering individuals in the existing population according to fitness, and selecting individuals with the half population quantity which is ranked at the front as preferred individuals;
performing mutation operation and crossover operation on the preferred individuals to generate a secondary population;
sorting the secondary population and the previous generation population of the secondary population according to a non-dominant rule, selecting individuals with the number of preset populations which are ranked in front as the secondary population, and iterating;
and monitoring the iteration times, stopping iteration if the actual iteration times reach the preset iteration times, and outputting the non-dominant solution set in the latest population as an optimal solution set.
4. A distributed storage server QoS optimization system, comprising:
a range setting unit, configured to preset a value range of each parameter item of the quality of service template;
the initial generation unit is used for generating an initial population of the multi-objective optimization algorithm according to the value range of each parameter item by utilizing a random algorithm;
the function construction unit is used for constructing a client request processing speed target function and an internal request processing speed target function based on the parameter item value of the service quality template in advance;
the adaptation calculation unit is used for calculating population adaptation degree by utilizing the client request processing speed objective function and the internal request processing speed objective function;
the iteration screening unit is used for selecting a population with high fitness for optimization iteration according to a non-dominant sorting method to obtain a parameter item value optimal scheme set of the service quality template;
presetting a value range of each parameter item of a service quality template, wherein the value range comprises:
respectively inputting an upper limit value and a lower limit value of each parameter item according to a cluster limiting rule;
the parameter items of the quality of service template include:
client reservation parameters, client weight parameters, client upper limit parameters, internal reservation parameters, internal weight parameters and internal upper limit parameters;
pre-constructing a client request processing speed target function and an internal request processing speed target function based on parameter item values of a service quality template, wherein the client request processing speed target function and the internal request processing speed target function comprise the following steps:
using the formulaCalculating an initial value of a client request processing speed, wherein cr is reserved for a client, cw is a client weight, rr is reserved internally, rw is an internal weight, and M is a cluster maximum performance value; comparing the initial value of the client request processing speed with the upper limit of the client, and taking a smaller value from the initial value and the upper limit as the client request processing speed;
using the formulaCalculating an initial value of an internal request processing speed, wherein cr is reserved for a client, cw is weight of the client, rr is reserved internally, rw is weight internally, and M is the maximum performance value of the cluster; comparing the initial value of the internal request processing speed with the internal upper limit, and taking the smaller value from the initial value and the internal upper limit as the internal request processing speed.
5. The system of claim 4, wherein the iterative screening unit is configured to:
non-dominant ordering individuals in the existing population according to fitness, and selecting individuals with the half population quantity which is ranked at the front as preferred individuals;
performing mutation operation and crossover operation on the preferred individuals to generate a secondary population;
sorting the secondary population and the previous generation population of the secondary population according to a non-dominant rule, selecting individuals with the number of preset populations which are ranked in front as the secondary population, and iterating;
and monitoring the iteration times, stopping iteration if the actual iteration times reach the preset iteration times, and outputting the non-dominant solution set in the latest population as an optimal solution set.
6. A terminal, comprising:
a processor;
a memory for storing execution instructions of the processor;
wherein the processor is configured to perform the method of any of claims 1-3.
7. A computer readable storage medium storing a computer program, which when executed by a processor implements the method of any one of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110872204.4A CN113687780B (en) | 2021-07-30 | 2021-07-30 | QoS optimization method, system, terminal and storage medium for distributed storage server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110872204.4A CN113687780B (en) | 2021-07-30 | 2021-07-30 | QoS optimization method, system, terminal and storage medium for distributed storage server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113687780A CN113687780A (en) | 2021-11-23 |
| CN113687780B true CN113687780B (en) | 2024-02-23 |
Family
ID=78578394
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110872204.4A Active CN113687780B (en) | 2021-07-30 | 2021-07-30 | QoS optimization method, system, terminal and storage medium for distributed storage server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113687780B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114153397B (en) * | 2021-12-03 | 2023-08-22 | 苏州浪潮智能科技有限公司 | Distributed storage multi-site synchronization optimization method, device, equipment, storage medium |
| CN114415945B (en) * | 2021-12-28 | 2025-04-29 | 航天科工网络信息发展有限公司 | A parameter tuning method, system, device, and medium for a distributed storage system |
| CN116225311B (en) * | 2022-12-12 | 2023-11-21 | 荣耀终端有限公司 | Configuration method, device and server for terminal equipment storage system parameters |
| CN118381708B (en) * | 2024-04-10 | 2024-12-27 | 山东省计算中心（国家超级计算济南中心） | Management scheduling method and system for cross-domain heterogeneous storage cluster |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104954278A (en) * | 2015-06-23 | 2015-09-30 | 南京邮电大学 | Bee colony optimization based network traffic scheduling method under multiple QoS (quality of service) constraints |
| WO2020228143A1 (en) * | 2019-05-15 | 2020-11-19 | 福州大学 | Cloud software service resource allocation method based on qos model self-correction |
| CN112511346A (en) * | 2020-11-23 | 2021-03-16 | 大连理工大学 | Web service combination method based on credibility screening |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140164048A1 (en) * | 2012-12-07 | 2014-06-12 | Xerox Corporation | Scalable weight-agnostic multi-objective qos optimization for workflow planning |
-
2021
- 2021-07-30 CN CN202110872204.4A patent/CN113687780B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104954278A (en) * | 2015-06-23 | 2015-09-30 | 南京邮电大学 | Bee colony optimization based network traffic scheduling method under multiple QoS (quality of service) constraints |
| WO2020228143A1 (en) * | 2019-05-15 | 2020-11-19 | 福州大学 | Cloud software service resource allocation method based on qos model self-correction |
| CN112511346A (en) * | 2020-11-23 | 2021-03-16 | 大连理工大学 | Web service combination method based on credibility screening |
Non-Patent Citations (3)
| Title |
|---|
| 基于多目标遗传算法的云服务部署优化方法;颉斌;杨扬;旷毅;;华中科技大学学报(自然科学版)(S1);全文 * |
| 基于服务质量的Web服务优化选择算法及仿真;王俊丽;柳先辉;卫刚;;同济大学学报(自然科学版)(04);全文 * |
| 机载公共设备管理系统中的自适应调度算法;刘亭;王占林;裘丽华;;计算机工程(09);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113687780A (en) | 2021-11-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113687780B (en) | QoS optimization method, system, terminal and storage medium for distributed storage server | |
| Liu et al. | Deadline‐constrained coevolutionary genetic algorithm for scientific workflow scheduling in cloud computing | |
| Shi et al. | Location-aware and budget-constrained service deployment for composite applications in multi-cloud environment | |
| Yang et al. | A spatiotemporal compression based approach for efficient big data processing on cloud | |
| Ji et al. | Hierarchical reinforcement learning for energy-efficient API traffic optimization in large-scale advertising systems | |
| US9380107B2 (en) | Migration event scheduling management | |
| Hamzaoui et al. | A survey on the current challenges of energy-efficient cloud resources management | |
| Chen et al. | Scheduling independent tasks in cloud environment based on modified differential evolution | |
| Zhang et al. | UCPSO: A uniform initialized particle swarm optimization algorithm with cosine inertia weight | |
| Peng et al. | Genetic algorithm‐based task scheduling in cloud computing using MapReduce framework | |
| Pallewatta et al. | Reliability-aware proactive placement of microservices-based IoT applications in fog computing environments | |
| CN108416465A (en) | A kind of Workflow optimization method under mobile cloud environment | |
| Velasquez et al. | A rank-based mechanism for service placement in the fog | |
| Ivashko et al. | A survey of desktop grid scheduling | |
| Qi et al. | SHIELD: Sustainable hybrid evolutionary learning framework for carbon, wastewater, and energy-aware data center management | |
| Sundara Kumar et al. | RETRACTED: Improving big data analytics data processing speed through map reduce scheduling and replica placement with HDFS using genetic optimization techniques | |
| Amini Motlagh et al. | A new reliability‐based task scheduling algorithm in cloud computing | |
| Zhang et al. | Survey on Task Scheduling Optimization Strategy under Multi-Cloud Environment. | |
| Wang et al. | An improved genetic algorithm for cost-effective data-intensive service composition | |
| Ji et al. | Adaptive workflow scheduling for diverse objectives in cloud environments | |
| Li et al. | Data analytics for fog computing by distributed online learning with asynchronous update | |
| Yang et al. | Trust-based scheduling strategy for cloud workflow applications | |
| Li et al. | Efficient multi-attribute precedence-based task scheduling for edge computing in geo-distributed cloud environment | |
| Wang et al. | FaaSConf: QoS-aware Hybrid Resources Configuration for Serverless Workflows | |
| Zhang et al. | GAI based Resource and QoE Aware Service Placement in Next-Generation Multi-domain IoT Networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |