CN113687780A - Distributed storage server QoS optimization method, system, terminal and storage medium - Google Patents
Distributed storage server QoS optimization method, system, terminal and storage medium Download PDFInfo
- Publication number
- CN113687780A CN113687780A CN202110872204.4A CN202110872204A CN113687780A CN 113687780 A CN113687780 A CN 113687780A CN 202110872204 A CN202110872204 A CN 202110872204A CN 113687780 A CN113687780 A CN 113687780A
- Authority
- CN
- China
- Prior art keywords
- population
- value
- processing speed
- parameter
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000005457 optimization Methods 0.000 title claims abstract description 49
- 238000003860 storage Methods 0.000 title claims abstract description 42
- 238000012545 processing Methods 0.000 claims abstract description 93
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 38
- 230000006870 function Effects 0.000 claims description 59
- 238000012216 screening Methods 0.000 claims description 8
- 238000012544 monitoring process Methods 0.000 claims description 6
- 230000035772 mutation Effects 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 5
- 230000003044 adaptive effect Effects 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims 1
- 230000008569 process Effects 0.000 abstract description 10
- 239000002245 particle Substances 0.000 description 9
- 238000004891 communication Methods 0.000 description 6
- 238000011084 recovery Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000006467 substitution reaction Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 235000002020 sage Nutrition 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/004—Artificial life, i.e. computing arrangements simulating life
- G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Computer And Data Communications (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a QoS optimization method, a system, a terminal and a storage medium for a distributed storage server, comprising the following steps: presetting the value ranges of all parameter items of the service quality template; generating an initial population of the multi-objective optimization algorithm according to the value range of each parameter item by using a random algorithm; a client request processing speed objective function and an internal request processing speed objective function which are based on the parameter item value of the service quality template are constructed in advance; calculating population fitness by using a client request processing speed objective function and an internal request processing speed objective function; and selecting the population with high fitness according to a non-dominated sorting method to perform optimization iteration to obtain the optimal parameter item value scheme set of the service quality template. The invention can effectively process I/O requests inside and outside the dispatching server to obtain optimal configuration parameters, and a user can configure QoS template parameters according to the optimal solution sets and the actual needs of the user, thereby improving the service quality of the cluster.
Description
Technical Field
The invention relates to the technical field of distributed storage, in particular to a QoS (quality of service) optimization method, a system, a terminal and a storage medium for a distributed storage server.
Background
Ceph is a new generation of free software distributed file system designed specifically for the Ph paper by Sage Weil (the Union founder of DreamHost) at Santa Cruz, Calif. Since the end of 2007, Sage began full time into development of Ceph, making it suitable for use in a production environment. The main goal of Ceph is to design a POSIX-based distributed file system without a single point of failure, enabling fault-tolerant and seamless replication of data. In 3 2010, Linus Torvalds incorporated Ceph client into kernel 2.6.34. An article in the IBM developer campus discusses architecture, fault tolerance implementation, and simplified mass data management functions of Ceph.
In a Ceph cluster, an OSD (on screen display) of an object storage device serves as a module for processing client I/O requests and I/O operations generated inside a system, and effective IO scheduling is crucial. Currently, the external I/O request of the same type is generally regarded as a client, including client _ op and OSD _ subop, and different types of I/O such as snaptrim, recovery and scrub are generated internally by the object storage device OSD. To schedule these different types of I/O requests, the dmClock algorithm is introduced, requiring only a QoS template in the server for each type of I/O. Taking two important I/O operations, namely client _ op and recovery, as an example, the configurable parameters of each operation include reservation, weight, limit reservation, weight and upper limit, namely: the system comprises a main controller, an osd _ op _ queue _ mclock _ client _ op _ res, an osd _ op _ queue _ mclock _ client _ op _ wgt, an osd _ op _ queue _ mclock _ client _ op _ lim, an osd _ op _ queue _ mclock _ recovery _ res, an osd _ op _ queue _ mclock _ recovery _ wgt and an osd _ op _ queue _ mclock _ recovery _ lim.
In the Ceph cluster, dmClock is adopted for at least two purposes, one is to realize uniform I/O scheduling and solve the contention problem of external clients and I/O in the cluster; and the second is better meeting the I/O processing in different scenes. Of course, on the premise that the processing speed of the server is constant, the improvement of the I/O of the external client inevitably reduces the processing of the I/O inside the cluster, and how to set the QoS template parameters is worth focusing by the staff. Currently, the setting of the QoS template parameters is generally set according to the experience of a user, and the complex and various requirements of the current large-scale storage cluster are difficult to meet.
Disclosure of Invention
Aiming at the problem that the prior art depends heavily on the experience of operation and maintenance personnel, the parameters of each Ceph cluster are different, the parameter setting has no commonality, and the QoS template parameter is not the optimal value mostly by manual setting, so that the I/O request processing speed cannot reach the best, and the QoS template parameter setting becomes the bottleneck for improving the I/O request processing speed. The invention provides a distributed storage server QoS optimization method, a system, a terminal and a storage medium, which aim to solve the technical problems.
In a first aspect, the present invention provides a QoS optimization method for a distributed storage service, including:
presetting the value ranges of all parameter items of the service quality template;
generating an initial population of the multi-objective optimization algorithm according to the value range of each parameter item by using a random algorithm;
a client request processing speed objective function and an internal request processing speed objective function which are based on the parameter item value of the service quality template are constructed in advance;
calculating population fitness by using a client request processing speed objective function and an internal request processing speed objective function;
and selecting the population with high fitness according to a non-dominated sorting method to perform optimization iteration to obtain the optimal parameter item value scheme set of the service quality template.
Further, presetting the value ranges of the parameter items of the service quality template, including:
and respectively inputting the upper limit value and the lower limit value of each parameter item according to the cluster restriction rule.
Further, the parameter items of the quality of service template include:
the client reservation parameter, the client weight parameter, the client upper limit parameter, the internal reservation parameter, the internal weight parameter and the internal upper limit parameter.
Further, generating an initial population of the multi-objective optimization algorithm according to the value ranges of the parameter items by using a random algorithm, wherein the method comprises the following steps:
presetting the number of individuals of an initial population;
generating a plurality of individuals according to the value ranges of the parameter items by using a random algorithm, wherein the value of the parameter item in each individual is a value randomly selected in the corresponding value range;
the number of the plurality of individuals is equal to the number of the preset initial population, and the initial population is formed by the plurality of individuals.
Further, a client request processing speed objective function and an internal request processing speed objective function which are based on the parameter item value of the service quality template are pre-constructed, and the method comprises the following steps:
using formulas
Calculating an initial value of the processing speed requested by the client, wherein cr is reserved for the client, cw is the weight of the client, rr is reserved in the interior, rw is the weight of the interior, and M is the maximum performance value of the cluster; comparing the initial value of the client request processing speed with the upper limit of the client, and taking the smaller value from the initial value and the upper limit as the client request processing speed;
using formulas
Calculating an initial value of the processing speed of the internal request, wherein cr is reserved for the client, cw is the weight of the client, rr is reserved for the inside, rw is the weight of the inside, and M is the maximum performance value of the cluster; and comparing the initial value of the internal request processing speed with the internal upper limit, and taking the smaller value from the initial value and the internal upper limit as the internal request processing speed.
Further, selecting a population with high fitness according to a non-dominated sorting method to perform optimization iteration to obtain an optimal parameter item value scheme set of the service quality template, wherein the optimal parameter item value scheme set comprises the following steps:
carrying out non-dominant sorting on individuals in the existing population according to fitness, and selecting the individuals with half of the population quantity at the top of the sorting as preferred individuals;
performing mutation operation and cross operation on the preferred individuals to generate a secondary population;
sorting the secondary population and the previous generation population of the secondary population according to a non-dominant rule, selecting individuals with a preset population quantity in the front sorting as the secondary population, and performing iteration;
and monitoring the iteration times, stopping iteration and outputting a non-dominated solution set in the latest population as an optimal solution set if the actual iteration times reach a preset iteration time.
In a second aspect, the present invention provides a distributed storage server QoS optimization system, including:
the range setting unit is used for presetting the value ranges of all the parameter items of the service quality template;
the initial generation unit is used for generating an initial population of the multi-objective optimization algorithm according to the value range of each parameter item by using a random algorithm;
the function construction unit is used for constructing a client request processing speed objective function and an internal request processing speed objective function based on the parameter item value of the service quality template in advance;
the adaptive computing unit is used for computing population fitness by utilizing a client request processing speed objective function and an internal request processing speed objective function;
and the iterative screening unit is used for selecting the population with high fitness according to a non-dominated sorting method to perform optimization iteration so as to obtain the optimal scheme set of the parameter item value of the service quality template.
Further, the range setting unit is used for respectively inputting an upper limit value and a lower limit value of each parameter item according to the cluster restriction rule. The parameter items of the service quality template comprise: the client reservation parameter, the client weight parameter, the client upper limit parameter, the internal reservation parameter, the internal weight parameter and the internal upper limit parameter.
Further, the initial generation unit is used for presetting the number of individuals of the initial population;
generating a plurality of individuals according to the value ranges of the parameter items by using a random algorithm, wherein the value of the parameter item in each individual is a value randomly selected in the corresponding value range;
the number of the plurality of individuals is equal to the number of the preset initial population, and the initial population is formed by the plurality of individuals.
Further, the method can be used for preparing a novel materialThe function construction unit is used for utilizing the formula
Calculating an initial value of the processing speed requested by the client, wherein cr is reserved for the client, cw is the weight of the client, rr is reserved in the interior, rw is the weight of the interior, and M is the maximum performance value of the cluster; comparing the initial value of the client request processing speed with the upper limit of the client, and taking the smaller value from the initial value and the upper limit as the client request processing speed;
using formulas
Calculating an initial value of the processing speed of the internal request, wherein cr is reserved for the client, cw is the weight of the client, rr is reserved for the inside, rw is the weight of the inside, and M is the maximum performance value of the cluster; and comparing the initial value of the internal request processing speed with the internal upper limit, and taking the smaller value from the initial value and the internal upper limit as the internal request processing speed.
Further, the iterative screening unit is used for carrying out non-dominant sorting on the individuals in the existing population according to the fitness, and selecting the individuals with half of the population quantity at the top of the sorting as preferred individuals;
performing mutation operation and cross operation on the preferred individuals to generate a secondary population;
sorting the secondary population and the previous generation population of the secondary population according to a non-dominant rule, selecting individuals with a preset population quantity in the front sorting as the secondary population, and performing iteration;
and monitoring the iteration times, stopping iteration and outputting a non-dominated solution set in the latest population as an optimal solution set if the actual iteration times reach a preset iteration time.
In a third aspect, a terminal is provided, including:
a processor, a memory, wherein,
the memory is used for storing a computer program which,
the processor is used for calling and running the computer program from the memory so as to make the terminal execute the method of the terminal.
In a fourth aspect, a computer storage medium is provided having stored therein instructions that, when executed on a computer, cause the computer to perform the method of the above aspects.
The beneficial effect of the invention is that,
the QoS optimization method for the distributed storage server side comprises the steps of presetting the value ranges of all parameter items of a service quality template, then generating an initial population of a multi-objective optimization algorithm according to the value ranges of all parameter items by using a random algorithm, simultaneously constructing a client request processing speed objective function and an internal request processing speed objective function based on the value of the parameter items of the service quality template, calculating population fitness by using the client request processing speed objective function and the internal request processing speed objective function, and finally selecting a population with high fitness according to a non-dominated sorting method to perform optimization iteration to obtain an optimal scheme set of the parameter item value of the service quality template. The invention can effectively process I/O requests inside and outside the dispatching server to obtain optimal configuration parameters, and a user can configure QoS template parameters according to the optimal solution sets and the actual needs of the user, thereby improving the service quality of the cluster.
The QoS optimization system of the distributed storage service end presets the value ranges of all parameter items of a service quality template through a range setting unit, then an initial generation unit generates an initial population of a multi-objective optimization algorithm according to the value ranges of all parameter items by using a random algorithm, a function construction unit constructs a client request processing speed objective function and an internal request processing speed objective function based on the value of the parameter items of the service quality template, an adaptive calculation unit calculates population fitness by using the client request processing speed objective function and the internal request processing speed objective function, and finally an iterative screening unit selects a population with high fitness according to a non-domination sorting method to perform optimization iteration to obtain a parameter item value optimal scheme set of the service quality template. The invention can effectively process I/O requests inside and outside the dispatching server to obtain optimal configuration parameters, and a user can configure QoS template parameters according to the optimal solution sets and the actual needs of the user, thereby improving the service quality of the cluster.
The terminal provided by the invention comprises a processor for operating the QoS optimization method of the distributed storage server, can effectively process I/O requests inside and outside the scheduling server to obtain optimal configuration parameters, and a user can configure QoS template parameters according to the optimal solution sets and the actual needs of the user, thereby improving the service quality of the cluster.
The storage medium provided by the invention stores a program for executing the QoS optimization method of the distributed storage server, the I/O requests inside and outside the dispatching server can be effectively processed by the invention, the optimal configuration parameters are obtained, and a user can configure the QoS template parameters according to the optimal solution sets and the actual needs of the user, thereby improving the service quality of the cluster.
In addition, the invention has reliable design principle, simple structure and very wide application prospect.
Drawings
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present invention, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
FIG. 1 is a schematic flow diagram of a method of one embodiment of the invention.
FIG. 2 is a schematic block diagram of a system of one embodiment of the present invention.
Fig. 3 is a schematic structural diagram of a terminal according to an embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention, and it is obvious that the described embodiment is only a part of the embodiment of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The following explains key terms appearing in the present invention.
QOS (Quality of Service) refers to a network that can provide better Service capability for specified network communication by using various basic technologies, and is a security mechanism of the network, which is a technology for solving the problems of network delay and congestion. QoS guarantees are important for capacity-limited networks, especially for streaming multimedia applications such as VoIP and IPTV, which often require fixed transmission rates and are sensitive to delay.
Five types of operation for client _ op, OSD _ subop, snaptrim, recovery, and scrub OSD
dmClock is a distributed mclock algorithm, I/O scheduling algorithm based on time tags
Pareto solution set Pareto optimal solution set and optimal solution set of multi-objective optimization problem
reservation, weight, limit reservation, weight, upper limit
FIG. 1 is a schematic flow diagram of a method of one embodiment of the invention. The execution subject in fig. 1 may be a distributed storage service QoS optimization system.
As shown in fig. 1, the method includes:
and 150, selecting the population with high fitness according to a non-dominated sorting method to perform optimization iteration to obtain an optimal scheme set of parameter item values of the service quality template.
In order to facilitate understanding of the present invention, the following further describes the distributed storage server QoS optimization method provided by the present invention with reference to the principle of the distributed storage server QoS optimization method of the present invention and the process of optimizing the distributed storage server QoS in the embodiment.
There are many multi-objective optimization problems in storage systems, i.e., multiple objectives are conflicting, and a good result for one objective may degrade the results for the other objectives. As researchers have conducted intensive research into solving multi-objective problems, many excellent solution algorithms have emerged. The multi-objective evolutionary algorithm is the most representative processing means, and can find a non-dominated Pareto solution set, namely an optimal solution set, corresponding to an approximate Pareto frontier in a decision space. The decision maker can select a desired solution from the solution set depending on the actual application.
Specifically, in the embodiment, the speed of processing the I/O request of the external client and the speed of processing the I/O operation generated inside the cluster are used as optimization targets, the QoS optimization problem of the server in distributed storage is converted into a multi-target problem, and an optimal configuration parameter is obtained by using a multi-target evolutionary algorithm. The QoS optimization method for the distributed storage server provided in this embodiment includes:
and S1, presetting the value range of each parameter item of the service quality template.
And respectively inputting the upper limit value and the lower limit value of each parameter item according to the cluster restriction rule. The parameter items of the service quality template in this embodiment include: the client reservation parameter, the client weight parameter, the client upper limit parameter, the internal reservation parameter, the internal weight parameter and the internal upper limit parameter.
In other embodiments of the present invention, the qos template may contain multiple types of I/O request parameters, but each type of I/O request parameter includes a reservation parameter, a weight parameter, and an upper limit parameter.
Therefore, the present embodiment mainly uses osd _ op _ queue _ mclk _ client _ op _ res (cr), osd _ op _ queue _ mclk _ client _ op _ wgt (cw), osd _ op _ queue _ mclk _ client _ im (Cl), osd _ op _ queue _ mclk _ resume _ res (rr), and osd _ op _ queue _ mclk _ resume _ wgt (rw), and osd _ op _ queue _ mclk _ resume _ list (rl) as optimized parameters, and processes the I/O request speed (Cl) of the external client and the I/O operation speed (In) generated inside the cluster as two optimized targets.
Firstly, search fields of 6 parameters, namely, a feasible solution field, a population number P and a maximum iteration number N, are input, namely, an osd _ op _ queue _ mclock _ client _ op _ res, an osd _ op _ queue _ client _ op _ wgt, an osd _ op _ queue _ mclock _ client _ op _ lim, an osd _ op _ queue _ mclock _ event _ resume _ res, an osd _ op _ queue _ mclock _ resume _ wgt and an osd _ op _ queue _ mclock _ resume _ lim.
And S2, generating an initial population of the multi-objective optimization algorithm according to the value range of each parameter item by using a random algorithm.
Generating a plurality of individuals according to the value ranges of the parameter items by using a random algorithm, wherein the value of the parameter item in each individual is a value randomly selected in the corresponding value range; the number of the plurality of individuals is equal to the number of the preset initial population, and the initial population is formed by the plurality of individuals.
That is, a random algorithm is used to generate P particles in a feasible solution domain, each particle represents a feasible solution, and the solution here is a parameter value-taking scheme of a quality of service template.
S3, pre-constructing a client request processing speed objective function and an internal request processing speed objective function based on the parameter item value of the service quality template.
Using formulas
Calculating an initial value of the processing speed requested by the client, wherein cr is reserved for the client, cw is the weight of the client, rr is reserved in the interior, rw is the weight of the interior, and M is the maximum performance value of the cluster; comparing the initial value of the client request processing speed with the upper limit of the client, and taking the smaller value from the initial value and the upper limit as the client request processing speed; using formulas
Calculating an initial value of the processing speed of the internal request, wherein cr is reserved for the client, cw is the weight of the client, rr is reserved for the inside, rw is the weight of the inside, and M is the maximum performance value of the cluster; and comparing the initial value of the internal request processing speed with the internal upper limit, and taking the smaller value from the initial value and the internal upper limit as the internal request processing speed.
And S4, calculating the population fitness by using the client request processing speed objective function and the internal request processing speed objective function.
Merging the client request processing speed objective function and the internal request processing speed objective function into one objective function:
wherein, CL is the objective function of the processing speed of the client request, and In is the objective function of the processing speed of the internal request. cr is reserved for the client, cw is the client weight, rr is reserved internally, rw is the internal weight, M is the cluster maximum performance value, cl is the client upper limit, and rl is the internal upper limit.
And S5, selecting the population with high fitness according to a non-dominated sorting method to perform optimization iteration, and obtaining the parameter item value optimal scheme set of the service quality template.
Carrying out non-dominant sorting on individuals in the existing population according to fitness, and selecting the individuals with half of the population quantity at the top of the sorting as preferred individuals; performing mutation operation and cross operation on the preferred individuals to generate a secondary population; sorting the secondary population and the previous generation population of the secondary population according to a non-dominant rule, selecting individuals with a preset population quantity in the front sorting as the secondary population, and performing iteration; and monitoring the iteration times, stopping iteration and outputting a non-dominated solution set in the latest population as an optimal solution set if the actual iteration times reach a preset iteration time.
The specific iteration process is as follows:
and screening particles from the current population, sorting the P particles according to a non-dominant rule according to the fitness of each particle, and selecting the particles with the top 50% rank to enter the next step.
Performing variation, intersection and selection operations on the P/2 particles to generate a secondary population, and calculating the fitness value of each particle according to the objective function of the step S4; wherein the compiling operation is performed by multiplying a coefficient for each parameter value in the individual schemes; the cross operation is a random combination of parameter values of different individual schemes.
Sequencing the secondary population and the original population together according to a non-domination rule, and taking the first P particles as the population of the next generation; the definition of "non-dominant solution" is: assuming that S1 is better than S2 for all targets for any two solutions S1 and S2, we call S1 dominates S2, and if the solution of S1 is not dominated by other solutions, then S1 is called non-dominated solution.
And outputting the Pareto solution set in the population when the maximum iteration times are reached, and otherwise, continuing fitness screening and iteration.
And obtaining a Pareto solution set, namely an optimal parameter model pool.
The distributed storage server-side QoS optimization method provided by this embodiment can effectively process I/O requests inside and outside the scheduling server to obtain optimal configuration parameters, and a user can configure QoS template parameters according to the optimal solution sets and the actual needs of the user, thereby improving the service quality of a cluster.
As shown in fig. 2, the system 200 includes:
a range setting unit 210, configured to preset a value range of each parameter item of the qos template;
an initial generating unit 220, configured to generate an initial population of the multi-objective optimization algorithm according to the value ranges of the parameter items by using a random algorithm;
a function constructing unit 230, configured to pre-construct a client request processing speed objective function and an internal request processing speed objective function based on a parameter item value of the quality of service template;
an adaptation calculation unit 240 for calculating population fitness using the client request processing speed objective function and the internal request processing speed objective function;
and the iterative screening unit 250 is configured to select a population with high fitness according to a non-dominated sorting method to perform optimization iteration, so as to obtain an optimal scheme set of parameter item values of the service quality template.
Optionally, as an embodiment of the present invention, the range setting unit is configured to input an upper limit value and a lower limit value of each parameter item according to a cluster restriction rule. The parameter items of the service quality template comprise: the client reservation parameter, the client weight parameter, the client upper limit parameter, the internal reservation parameter, the internal weight parameter and the internal upper limit parameter.
Optionally, as an embodiment of the present invention, the initial generating unit is configured to preset an individual number of the initial population; generating a plurality of individuals according to the value ranges of the parameter items by using a random algorithm, wherein the value of the parameter item in each individual is a value randomly selected in the corresponding value range; the number of the plurality of individuals is equal to the number of the preset initial population, and the initial population is formed by the plurality of individuals.
Optionally, as an embodiment of the invention, the function construction unit is used for utilizing a formula
Calculating an initial value of the processing speed requested by the client, wherein cr is reserved for the client, cw is the weight of the client, rr is reserved in the interior, rw is the weight of the interior, and M is the maximum performance value of the cluster; comparing the initial value of the client request processing speed with the upper limit of the client, and taking the smaller value from the initial value and the upper limit as the client request processing speed; using formulas
Calculating an initial value of the processing speed of the internal request, wherein cr is reserved for the client, cw is the weight of the client, rr is reserved for the inside, rw is the weight of the inside, and M is the maximum performance value of the cluster; and comparing the initial value of the internal request processing speed with the internal upper limit, and taking the smaller value from the initial value and the internal upper limit as the internal request processing speed.
Optionally, as an embodiment of the present invention, the iterative screening unit is configured to perform non-dominant sorting on the individuals in the existing population according to fitness, and select an individual with half of the population number in the top sorting as a preferred individual; performing mutation operation and cross operation on the preferred individuals to generate a secondary population; sorting the secondary population and the previous generation population of the secondary population according to a non-dominant rule, selecting individuals with a preset population quantity in the front sorting as the secondary population, and performing iteration; and monitoring the iteration times, stopping iteration and outputting a non-dominated solution set in the latest population as an optimal solution set if the actual iteration times reach a preset iteration time.
Fig. 3 is a schematic structural diagram of a terminal 300 according to an embodiment of the present invention, where the terminal 300 may be used to execute a QoS optimization method for a distributed storage service end according to the embodiment of the present invention.
Among them, the terminal 300 may include: a processor 310, a memory 320, and a communication unit 330. The components communicate via one or more buses, and those skilled in the art will appreciate that the architecture of the servers shown in the figures is not intended to be limiting, and may be a bus architecture, a star architecture, a combination of more or less components than those shown, or a different arrangement of components.
The memory 320 may be used for storing instructions executed by the processor 310, and the memory 320 may be implemented by any type of volatile or non-volatile storage terminal or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The executable instructions in memory 320, when executed by processor 310, enable terminal 300 to perform some or all of the steps in the method embodiments described below.
The processor 310 is a control center of the storage terminal, connects various parts of the entire electronic terminal using various interfaces and lines, and performs various functions of the electronic terminal and/or processes data by operating or executing software programs and/or modules stored in the memory 320 and calling data stored in the memory. The processor may be composed of an Integrated Circuit (IC), for example, a single packaged IC, or a plurality of packaged ICs connected with the same or different functions. For example, the processor 310 may include only a Central Processing Unit (CPU). In the embodiment of the present invention, the CPU may be a single operation core, or may include multiple operation cores.
A communication unit 330, configured to establish a communication channel so that the storage terminal can communicate with other terminals. And receiving user data sent by other terminals or sending the user data to other terminals.
The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and the program may include some or all of the steps in the embodiments provided by the present invention when executed. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a Random Access Memory (RAM).
Therefore, the invention presets the value ranges of all parameter items of the service quality template, then generates an initial population of the multi-objective optimization algorithm according to the value ranges of all parameter items by using a random algorithm, simultaneously constructs a client request processing speed objective function and an internal request processing speed objective function based on the value of the parameter items of the service quality template, calculates population fitness by using the client request processing speed objective function and the internal request processing speed objective function, and finally selects a population with high fitness according to a non-dominated sorting method to carry out optimization iteration to obtain an optimal scheme set of the parameter item value of the service quality template. The invention can effectively process I/O requests inside and outside the scheduling server to obtain optimal configuration parameters, and a user can configure QoS template parameters according to the optimal solution sets and the actual needs of the user, thereby improving the service quality of the cluster.
Those skilled in the art will readily appreciate that the techniques of the embodiments of the present invention may be implemented as software plus a required general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied in the form of a software product, where the computer software product is stored in a storage medium, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like, and the storage medium can store program codes, and includes instructions for enabling a computer terminal (which may be a personal computer, a server, or a second terminal, a network terminal, and the like) to perform all or part of the steps of the method in the embodiments of the present invention.
The same and similar parts in the various embodiments in this specification may be referred to each other. Especially, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant points can be referred to the description in the method embodiment.
In the embodiments provided in the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the above-described system embodiments are merely illustrative, and for example, the division of the units is only one logical functional division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, systems or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
Although the present invention has been described in detail by referring to the drawings in connection with the preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made on the embodiments of the present invention by those skilled in the art without departing from the spirit and scope of the present invention, and these modifications or substitutions are within the scope of the present invention/any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A distributed storage server QoS optimization method is characterized by comprising the following steps:
presetting the value ranges of all parameter items of the service quality template;
generating an initial population of the multi-objective optimization algorithm according to the value range of each parameter item by using a random algorithm;
a client request processing speed objective function and an internal request processing speed objective function which are based on the parameter item value of the service quality template are constructed in advance;
calculating population fitness by using a client request processing speed objective function and an internal request processing speed objective function;
and selecting the population with high fitness according to a non-dominated sorting method to perform optimization iteration to obtain the optimal parameter item value scheme set of the service quality template.
2. The method of claim 1, wherein presetting a value range of each parameter item of the quality of service template comprises:
and respectively inputting the upper limit value and the lower limit value of each parameter item according to the cluster restriction rule.
3. The method of claim 2, wherein the parameter items of the quality of service template comprise:
the client reservation parameter, the client weight parameter, the client upper limit parameter, the internal reservation parameter, the internal weight parameter and the internal upper limit parameter.
4. The method of claim 1, wherein generating an initial population of the multi-objective optimization algorithm from the value ranges of the parameter items using a stochastic algorithm comprises:
presetting the number of individuals of an initial population;
generating a plurality of individuals according to the value ranges of the parameter items by using a random algorithm, wherein the value of the parameter item in each individual is a value randomly selected in the corresponding value range;
the number of the plurality of individuals is equal to the number of the preset initial population, and the initial population is formed by the plurality of individuals.
5. The method of claim 3, wherein pre-constructing a client request processing speed objective function and an internal request processing speed objective function based on the parameter item values of the quality of service template comprises:
using formulas
Calculating an initial value of the processing speed requested by the client, wherein cr is reserved for the client, cw is the weight of the client, rr is reserved in the interior, rw is the weight of the interior, and M is the maximum performance value of the cluster; comparing the initial value of the client request processing speed with the upper limit of the client, and taking the smaller value from the initial value and the upper limit as the client request processing speed;
using formulas
Calculating an initial value of the processing speed of the internal request, wherein cr is reserved for the client, cw is the weight of the client, rr is reserved for the inside, rw is the weight of the inside, and M is the maximum performance value of the cluster; ratio ofThe smaller value of the initial value of the internal request processing speed and the internal upper limit is taken as the internal request processing speed.
6. The method of claim 1, wherein the population with high fitness is selected according to a non-dominated sorting method for optimization iteration to obtain an optimal scheme set of parameter item values of the quality of service template, and the method comprises the following steps:
carrying out non-dominant sorting on individuals in the existing population according to fitness, and selecting the individuals with half of the population quantity at the top of the sorting as preferred individuals;
performing mutation operation and cross operation on the preferred individuals to generate a secondary population;
sorting the secondary population and the previous generation population of the secondary population according to a non-dominant rule, selecting individuals with a preset population quantity in the front sorting as the secondary population, and performing iteration;
and monitoring the iteration times, stopping iteration and outputting a non-dominated solution set in the latest population as an optimal solution set if the actual iteration times reach a preset iteration time.
7. A distributed storage server QoS optimization system, comprising:
the range setting unit is used for presetting the value ranges of all the parameter items of the service quality template;
the initial generation unit is used for generating an initial population of the multi-objective optimization algorithm according to the value range of each parameter item by using a random algorithm;
the function construction unit is used for constructing a client request processing speed objective function and an internal request processing speed objective function based on the parameter item value of the service quality template in advance;
the adaptive computing unit is used for computing population fitness by utilizing a client request processing speed objective function and an internal request processing speed objective function;
and the iterative screening unit is used for selecting the population with high fitness according to a non-dominated sorting method to perform optimization iteration so as to obtain the optimal scheme set of the parameter item value of the service quality template.
8. The system of claim 7, wherein the iterative filtering unit is configured to:
carrying out non-dominant sorting on individuals in the existing population according to fitness, and selecting the individuals with half of the population quantity at the top of the sorting as preferred individuals;
performing mutation operation and cross operation on the preferred individuals to generate a secondary population;
sorting the secondary population and the previous generation population of the secondary population according to a non-dominant rule, selecting individuals with a preset population quantity in the front sorting as the secondary population, and performing iteration;
and monitoring the iteration times, stopping iteration and outputting a non-dominated solution set in the latest population as an optimal solution set if the actual iteration times reach a preset iteration time.
9. A terminal, comprising:
a processor;
a memory for storing instructions for execution by the processor;
wherein the processor is configured to perform the method of any one of claims 1-6.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110872204.4A CN113687780B (en) | 2021-07-30 | 2021-07-30 | QoS optimization method, system, terminal and storage medium for distributed storage server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110872204.4A CN113687780B (en) | 2021-07-30 | 2021-07-30 | QoS optimization method, system, terminal and storage medium for distributed storage server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113687780A true CN113687780A (en) | 2021-11-23 |
| CN113687780B CN113687780B (en) | 2024-02-23 |
Family
ID=78578394
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110872204.4A Active CN113687780B (en) | 2021-07-30 | 2021-07-30 | QoS optimization method, system, terminal and storage medium for distributed storage server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113687780B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114153397A (en) * | 2021-12-03 | 2022-03-08 | 苏州浪潮智能科技有限公司 | Distributed storage multi-site synchronous optimization method, device, equipment and storage medium |
| CN116225311A (en) * | 2022-12-12 | 2023-06-06 | 荣耀终端有限公司 | Configuration method, device and server for terminal equipment storage system parameters |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140164048A1 (en) * | 2012-12-07 | 2014-06-12 | Xerox Corporation | Scalable weight-agnostic multi-objective qos optimization for workflow planning |
| CN104954278A (en) * | 2015-06-23 | 2015-09-30 | 南京邮电大学 | Bee colony optimization based network traffic scheduling method under multiple QoS (quality of service) constraints |
| WO2020228143A1 (en) * | 2019-05-15 | 2020-11-19 | 福州大学 | Cloud software service resource allocation method based on qos model self-correction |
| CN112511346A (en) * | 2020-11-23 | 2021-03-16 | 大连理工大学 | Web service combination method based on credibility screening |
-
2021
- 2021-07-30 CN CN202110872204.4A patent/CN113687780B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140164048A1 (en) * | 2012-12-07 | 2014-06-12 | Xerox Corporation | Scalable weight-agnostic multi-objective qos optimization for workflow planning |
| CN104954278A (en) * | 2015-06-23 | 2015-09-30 | 南京邮电大学 | Bee colony optimization based network traffic scheduling method under multiple QoS (quality of service) constraints |
| WO2020228143A1 (en) * | 2019-05-15 | 2020-11-19 | 福州大学 | Cloud software service resource allocation method based on qos model self-correction |
| CN112511346A (en) * | 2020-11-23 | 2021-03-16 | 大连理工大学 | Web service combination method based on credibility screening |
Non-Patent Citations (3)
| Title |
|---|
| 刘亭;王占林;裘丽华;: "机载公共设备管理系统中的自适应调度算法", 计算机工程, no. 09 * |
| 王俊丽;柳先辉;卫刚;: "基于服务质量的Web服务优化选择算法及仿真", 同济大学学报(自然科学版), no. 04 * |
| 颉斌;杨扬;旷毅;: "基于多目标遗传算法的云服务部署优化方法", 华中科技大学学报(自然科学版), no. 1 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114153397A (en) * | 2021-12-03 | 2022-03-08 | 苏州浪潮智能科技有限公司 | Distributed storage multi-site synchronous optimization method, device, equipment and storage medium |
| CN114153397B (en) * | 2021-12-03 | 2023-08-22 | 苏州浪潮智能科技有限公司 | Distributed storage multi-site synchronous optimization method, device, equipment and storage medium |
| CN116225311A (en) * | 2022-12-12 | 2023-06-06 | 荣耀终端有限公司 | Configuration method, device and server for terminal equipment storage system parameters |
| CN116225311B (en) * | 2022-12-12 | 2023-11-21 | 荣耀终端有限公司 | Configuration method, device and server for terminal equipment storage system parameters |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113687780B (en) | 2024-02-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Yu et al. | Location-aware associated data placement for geo-distributed data-intensive applications | |
| Yu et al. | Deadline scheduling as restless bandits | |
| Wang et al. | Predictive scheduling framework for electric vehicles with uncertainties of user behaviors | |
| Jiang et al. | Optimal cloud resource auto-scaling for web applications | |
| Kumar et al. | Hybrid user-item based collaborative filtering | |
| CN113687780A (en) | Distributed storage server QoS optimization method, system, terminal and storage medium | |
| Alizadeh et al. | Reduced-order load models for large populations of flexible appliances | |
| Hardiansyah et al. | Solving economic load dispatch problem using particle swarm optimization technique | |
| Nguyen et al. | Two-stage robust edge service placement and sizing under demand uncertainty | |
| Wang et al. | Immune scheduling network based method for task scheduling in decentralized fog computing | |
| Golshani et al. | Proactive auto-scaling for cloud environments using temporal convolutional neural networks | |
| Peng et al. | Genetic Algorithm‐Based Task Scheduling in Cloud Computing Using MapReduce Framework | |
| Liang et al. | An improved genetic algorithm for service selection under temporal constraints in cloud computing | |
| Amini Motlagh et al. | A new reliability‐based task scheduling algorithm in cloud computing | |
| Sun et al. | Aers: An autonomic and elastic resource scheduling framework for cloud applications | |
| CN112631612B (en) | Optimization method for kubernetes cloud platform configuration based on genetic algorithm | |
| Qi et al. | SHIELD: Sustainable hybrid evolutionary learning framework for carbon, wastewater, and energy-aware data center management | |
| Millar et al. | Constrained coordinated distributed control of smart grid with asynchronous information exchange | |
| CN113761856A (en) | Data processing method and device | |
| CN111475251A (en) | Cluster container scheduling method, system, terminal and storage medium | |
| CN113450042A (en) | Method and device for determining replenishment quantity | |
| Prado et al. | On providing quality of service in grid computing through multi-objective swarm-based knowledge acquisition in fuzzy schedulers | |
| CN115269176A (en) | Task allocation method, device, computer equipment, storage medium and product | |
| CN113760550A (en) | Resource allocation method and resource allocation device | |
| Chau et al. | Competitive online algorithms for geographical load balancing in data centers with energy storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |