CN115599296A - Automatic node expansion method and system for distributed storage system - Google Patents
Automatic node expansion method and system for distributed storage system Download PDFInfo
- Publication number
- CN115599296A CN115599296A CN202211188692.8A CN202211188692A CN115599296A CN 115599296 A CN115599296 A CN 115599296A CN 202211188692 A CN202211188692 A CN 202211188692A CN 115599296 A CN115599296 A CN 115599296A
- Authority
- CN
- China
- Prior art keywords
- wolf
- hadoop
- nodes
- layer
- deep learning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000013135 deep learning Methods 0.000 claims abstract description 57
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 28
- 238000012549 training Methods 0.000 claims abstract description 20
- 241000282461 Canis lupus Species 0.000 claims description 110
- 239000013598 vector Substances 0.000 claims description 29
- 230000006870 function Effects 0.000 claims description 24
- 238000005457 optimization Methods 0.000 claims description 23
- 238000000605 extraction Methods 0.000 claims description 20
- 238000004364 calculation method Methods 0.000 claims description 14
- 230000006978 adaptation Effects 0.000 claims description 7
- 210000002569 neuron Anatomy 0.000 claims description 7
- 241000282326 Felis catus Species 0.000 claims description 6
- 230000009466 transformation Effects 0.000 claims description 6
- 238000011176 pooling Methods 0.000 claims description 4
- 241000282421 Canidae Species 0.000 claims description 3
- 238000006243 chemical reaction Methods 0.000 claims 1
- 238000012545 processing Methods 0.000 abstract description 5
- 230000008859 change Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 238000004590 computer program Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 230000015654 memory Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 239000008358 core component Substances 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000002068 genetic effect Effects 0.000 description 1
- 230000007787 long-term memory Effects 0.000 description 1
- 230000005291 magnetic effect Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
- 230000003936 working memory Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0625—Power saving in storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/004—Artificial life, i.e. computing arrangements simulating life
- G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Biophysics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Human Computer Interaction (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to a method and a system for automatically stretching and retracting nodes of a distributed storage system, wherein the method comprises the following steps: acquiring network flow data and making the network flow data into a data set; building a deep learning network model for regression prediction; training a deep learning network model by using a data set to obtain a traffic prediction model for predicting network traffic; the method comprises the steps of obtaining real-time network flow, obtaining future network flow through a flow prediction model, calculating storage data requirements corresponding to the future network flow, calculating the number of nodes required by the hadoop according to the storage data requirements, keeping the number of the nodes of the hadoop unchanged if the number of the current nodes of the hadoop is matched with the number of the required nodes, and adjusting the number of the current nodes of the hadoop to the number of the matched required nodes if the number of the current nodes of the hadoop is not matched with the number of the required nodes. Compared with the prior art, the distributed storage and processing are carried out based on hadoop, the elastic expansion of the number of the nodes is automatically carried out by learning and predicting the flow trend through a deep learning algorithm, the storage service quality is ensured, and the storage resources of a distributed storage system are effectively saved.
Description
Technical Field
The invention relates to the field of data analysis and distributed storage, in particular to a method and a system for automatically stretching and retracting nodes of a distributed storage system.
Background
The distributed File System is one of Hadoop core components and exists as a distributed storage service at the bottom layer. The problem solved by distributed file systems is large data storage. They are storage systems that span across multiple computers. Distributed file systems have a wide range of applications in the big data age, providing the needed scalability for storing and processing very large scale data.
The Hadoop cluster size and processing capacity are usually set according to the demand of cluster resources in the peak period, so that the cluster resources are inevitably wasted in the low peak period. At present, research on automatic stretching of distributed nodes exists, but most of the existing research at present sets an elastic stretching threshold value by depending on experience and requirements, dynamic stretching of the dynamic distributed nodes according to requirements is not really realized, and the stretching method causes resource waste to a certain extent.
Disclosure of Invention
The invention aims to overcome the defects in the prior art and provide a method and a system for automatically stretching nodes of a distributed storage system.
The purpose of the invention can be realized by the following technical scheme:
a node automatic scaling method of a distributed storage system comprises the following steps:
acquiring network flow data and making the network flow data into a data set;
building a deep learning network model for regression prediction;
training a deep learning network model by using the data set to obtain a traffic prediction model for predicting network traffic;
acquiring real-time network flow, obtaining future network flow through the flow prediction model, calculating a storage data requirement corresponding to the future network flow, calculating the number of nodes required by hadoop according to the storage data requirement, if the number of nodes required by hadoop is matched with the number of nodes required by hadoop, keeping the number of nodes of hadoop unchanged, repeating the step, otherwise, adjusting the number of nodes required by hadoop to the number of nodes required by matching, and repeating the step.
Further, the deep learning network model comprises an input layer, a first feature extraction layer, a second feature extraction layer, a first GRU layer, a second GRU layer, a loss function layer and an output layer, wherein the input layer is used for acquiring input data, the first feature extraction layer and the second feature extraction layer respectively obtain a first feature value and a second feature value after performing feature extraction on the input data, the first feature value and the second feature value are input into the first GRU layer after dimension addition and dimension transformation, the output of the first GRU layer is sent into the second GRU layer, the output of the second GRU layer is sent into the output layer, and the loss function layer is used for performing parameter optimization on the deep learning network model.
Further, the principle of the deep learning network model is as follows:
x j2 ＝f(p(x i ))
f(*)＝max(0，*)
x j ＝cat(x j1 ，x j2 )
x t ＝reshape(x j )
y′ i ＝GRU1(x t )
y i ＝GRU2(x t )
wherein m is the number of neurons in the input layer, x i Representing input data, w i，j Representing weights in the first feature extraction layer, b j Representing a threshold value, x, in a first feature extraction layer j1 Representing a first characteristic value, x j2 Representing a second characteristic value, p () representing a max pooling operation, x j Denotes a value obtained by dimension-adding the first characteristic value and the second characteristic value, cat () denotes addition by dimension, x t Represents values of the first and second feature values after dimension transformation, reshape () represents dimension transformation, GRU1 () and GRU2 () represent operations of the first and second GRU layers, y' i Denotes the output of the first GRU layer, y i Representing the output of the second GRU layer and MAE the loss function.
Further, the parameters of the deep learning network model are optimally determined by a modified gray wolf algorithm as follows:
(1) initializing wolf clusters
Wherein N represents the scale of the wolf colony,
represents the position vector of the ith wolf, i is more than or equal to 1 and less than or equal to N,
the dimensionality number of the initial coefficient vector is the same as the number of parameters to be optimized in the deep learning network model and corresponds to the parameters one by one, and the coefficient vector is initialized
Coefficient vector
And attenuation coefficient
Determining the maximum iteration times and the upper and lower bounds of the parameters to be optimized;
(2) calculating the fitness value of each wolf in the wolf group, sequentially setting three wolfs with the optimal fitness value as alpha wolfs, beta wolfs and delta wolfs, and setting the rest wolfs as omega wolfs, if the preset convergence condition is met, executing the step (5), otherwise, executing the step (3);
(3) updating the position of the omega wolf, wherein the position updating formula is as follows:
wherein the content of the first and second substances,
indicating the location of the tth iteration of the individual wolf,
coefficient vectors representing alpha wolf, beta wolf and delta wolf,
indicating the position of the t-th iteration of alpha wolf, beta wolf and delta wolf,
represents the distance between the individual grey wolf and alpha wolf at the t-th iteration,
represents the distance between the individual grey wolf and beta wolf at the t-th iteration,
represents the distance between the grey wolf individual and the delta wolf at the t-th iteration, subscripts i =1, 2, 3 correspond to alpha wolf, beta wolf and delta wolf, respectively, f (×) represents the fitness function;
(4) updating a coefficient vector
Coefficient vector
And (3) executing the step (2), and updating the formula as follows:
wherein iter represents the current number of iterations, max _ iter represents the maximum number of iterations,
and
each represents [0,1]A randomly generated vector;
(5) and taking the position of the alpha wolf as an optimal parameter value of the deep learning network model.
Further, the first GRU layer and the second GRU layer have the same structure, and the parameters to be optimized in the deep learning network model include: the number of neurons in the first GRU layer and the second GRU layer and the size of the convolution kernel.
Further, the calculation method of the fitness value is as follows: substituting parameters corresponding to the position vectors of the wolfs into the deep learning network model, training the deep learning network model by using a data set, calculating a loss function in each training, and solving a fitness value based on the loss function, wherein the calculation formula of the fitness value is as follows:
wherein epsilon is a set hyper-parameter, K represents the final training times of the deep learning network model, MAE k Representing the loss function for the kth iteration.
Further, the output of the traffic prediction model is a network traffic prediction value { f) in a time period T 1 ，f 2 ，...，f N N represents the traffic number, and the future network traffic is determined based on the network traffic predicted value in the time period T as follows:
taking the peak value f of the flow in the time period T max Taking the minimum flow value f in the time period T min Taking the average value f of the flow in the time period T avg Then, the calculation formula of the future network traffic f is as follows:
f＝w 1 f max +w 2 f min +w 3 f avg
wherein, w 1 、w 2 And w 3 Representing the weight coefficients.
Further, w 3 ＝0.5，w 1 、w 2 The calculation formula of (2) is as follows:
wherein n is h Is { f 1 ，f 2 ，...，f N Higher than f avg Number of flows of n l Is { f 1 ，f 2 ，...，f N Lower than f in avg The number of flows of (c).
Further, the "number of nodes currently adapted to the required number of nodes by hadoop" is specifically:
acquiring the current node number D of hadoop 1 Obtaining the number of required nodes D 2 If D is 1 -λ≤D 2 ≤D 1 + lambda, the number of hadoop current nodes is adapted to the number of required nodes, wherein lambda is a preset elastic expansion threshold;
the "adjusting the current node number of hadoop to the adaptation requirement node number" specifically includes:
obtaining an adjusting step length;
if the current node number of the hadoop is smaller than the required node number, increasing the current node number according to the adjusting step length, judging whether the increased current node number is matched with the required node number, if so, finishing the adjustment, otherwise, repeating the step;
if the current node number of the hadoop is larger than the required node number, reducing the current node number according to the adjustment step length, judging whether the reduced current node number is matched with the required node number, if so, finishing the adjustment, otherwise, repeating the step.
A node auto-scaling system for a distributed storage system, comprising:
the traffic prediction module is loaded with a trained traffic prediction model and used for obtaining future network traffic according to the real-time network traffic;
the demand node calculation module is used for calculating the storage data demand corresponding to the future network flow and calculating the demand node number of hadoop according to the storage data demand;
the adaptation judging module is used for judging whether the current node number of the hadoop is adapted to the required node number;
the node number adjusting module is used for adjusting the current node number of the hadoop;
the flow prediction model is trained as follows:
acquiring network flow data and making the network flow data into a data set;
building a deep learning network model for regression prediction;
and training a deep learning network model by using the data set to obtain a flow prediction model for network flow prediction.
Compared with the prior art, the invention has the following beneficial effects:
(1) The change trend of the network flow data is automatically sensed through a deep learning algorithm, and the number of nodes required by the distributed file system is automatically adjusted according to the change trend, so that the real dynamic expansion of the nodes along with the network flow data is realized, the storage service quality is ensured, and the storage resources of the distributed storage system are effectively saved.
(2) The improved wolf algorithm is utilized to carry out parameter optimization of the deep learning network model, the problems of long time and large required storage space during model training are effectively solved, and the prediction precision of the model can be effectively improved after the neuron number and the convolution kernel size of the GRU are optimized.
(3) When the future network flow is calculated, the maximum value, the minimum value and the average value of the flow are comprehensively considered, and the weight is determined according to the number of nodes which are higher than the average value of the flow and lower than the average value of the flow, so that the future network flow is more accurate.
Drawings
FIG. 1 is a flow chart of a method for node auto-scaling;
FIG. 2 is a schematic structural diagram of a deep learning network model;
FIG. 3 is a flow chart of the improved Grey wolf algorithm optimization parameters.
Detailed Description
The invention is described in detail below with reference to the figures and the specific embodiments. The present invention is implemented on the premise of the technical solution of the present invention, and a detailed implementation manner and a specific operation process are given, it is obvious that the described embodiment is only a part of the embodiment of the present invention, not all embodiments, and the protection scope of the present invention is not limited to the following embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic may be included in at least one implementation of the invention. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus. Furthermore, the terms "first," "second," "third," and the like are used solely to distinguish one from another, and are not to be construed as indicating or implying relative importance.
The present specification provides method steps as in the examples or flow diagrams, but may include more or fewer steps based on routine or non-inventive labor. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. In actual system or server product execution, the method shown in the embodiment or the figures can be executed sequentially or in parallel (for example, in the environment of parallel processors or multi-thread processing), or the execution sequence of steps without timing limitation can be adjusted.
Example 1:
the application provides a node automatic telescoping method of a distributed storage system, as shown in fig. 1, including the following steps:
s1, acquiring network flow data and making the network flow data into a data set;
acquiring network flow data in a period of time, recording related information including year, month, day, week, hour, minute, second, millisecond, flow number at the moment and the like, determining the length of a training sample, determining a predicted time step, intercepting the network flow data according to a certain time length, and carrying out preprocessing operations such as normalization, vacancy value processing and the like, thereby completing construction of a data set
S2, building a deep learning network model for regression prediction;
in this application, the deep learning network model that constructs includes the input layer, first characteristic extraction layer, the second characteristic extraction layer, first GRU layer, the second GRU layer, loss function layer and output layer, as shown in fig. 2, wherein, the input layer is used for acquireing the input data, first characteristic extraction layer and second characteristic layer obtain first eigenvalue and second eigenvalue respectively after carrying out the characteristic extraction to the input data, first eigenvalue and second eigenvalue input first GRU layer after dimension addition and dimension transform, the output of first GRU layer is sent into the second GRU layer, the output of second GRU layer is sent into the output layer, loss function layer is used for carrying out parameter optimization to the deep learning network model.
In the application, the first feature extraction layer is a maximum pooling layer, the second feature extraction layer is a convolution layer, the first GRU layer and the second GRU layer have the same structure, and a GRU (Gate recovery Unit) is one of the Recurrent neural networks, and is provided for solving the problems of Long-Term Memory, gradient in back propagation and the like, like an LSTM (Long-Short Term Memory). The input and output structure of the GRU is the same as that of a common RNN, with inputs: input x at time t t Hidden layer State H at time t-1 t-1 And (3) outputting: output y of hidden node at time t t Hidden state h passed to the next node t 。
r t ＝σ(x t W xr +H t-1 W hr +b r )
z t ＝σ(x t W xz +H t-1 W hz +b z )
y t ＝W o ·H t
GRU has only two doors, r t The updating gate is obtained by combining an input gate and a forgetting gate in the LSTM for the GRU, and is used for controlling the data size of the previous memory information which can be continuously reserved to the current moment, or determining how much information of the previous time step and the current time step is to be continuously transmitted to the future; z is a radical of formula t The gate is reset for the other gate, which controls how much past information is to be forgotten.
Represents a candidate hidden state, H t Representing the final hidden state.
Specifically, the principle of the deep learning network model is as follows:
x j2 ＝f(p(x i ))
f(*)＝max(0，*)
x j ＝cat(x j1 ，x j2 )
x t ＝reshape(x j )
y′ i ＝GRU1(x t )
y i ＝GRU2(x t )
wherein m is the number of neurons in the input layer, x i Representing input data, w i，j Representing weights in the first feature extraction layer, b j Representing a threshold value, x, in a first feature extraction layer j1 Representing a first characteristic value, x j2 Representing a second characteristic value, p () representing a max pooling operation, x j Representing the value of the first and second eigenvalues after dimensional addition, cat () representing the addition by dimension, x t Representing the first and second eigenvalues passing throughA dimension-transformed value, reshape () represents a dimension transformation, and GRU1 () and GRU2 () represent operations of the first and second GRU layers, y' i Denotes the output of the first GRU layer, y i Denotes the output of the second GRU layer, MAE denotes the loss function, f (x) i ) Indicates the result of the prediction, y i Representing the real result of the recording.
S3, training the deep learning network model by using the data set to obtain a traffic prediction model for predicting network traffic;
generally, parameters of a deep learning network model are set empirically or randomly, and often need to be tried for finding optimal parameters many times, and in order to find optimal parameters faster, many researchers have proposed that optimization algorithms such as particle swarm and genetic algorithms can be used for parameter optimization, so as to determine optimal network parameters. The gray wolf optimization (GWOO) is defined as a group intelligent optimization model which has the advantages of flexibility, simplicity and a non-derivative mechanism. Some studies have found that GWO has better numerical properties than other traditional optimization models, can prevent local optimization, and is considered a convenient stochastic approach to solving highly nonlinear, multivariate, and multimodality optimization problems. GWO is a new metaheuristic technique originally proposed by milr gialy et al and inspired by the hunting and social class of grey wolves. The gray wolf group is essentially divided into four classes, named alpha (α), beta (β), delta (σ), and omega (ω), respectively. The top level of the hierarchy is the alpha wolf. In addition, alpha wolf also makes critical decisions. The second level in the grey wolf's scale is the beta wolf, which acts as a mentor and provides feedback to authorize and assist the alpha wolf. On the other hand, the deltoid wolf obeys the commands of the alpha wolf and the beta wolf, and governs the omega wolf. The major stages of the grayish water hunting process include killing, surrounding and attacking the game. In the GWO algorithm, the hierarchy is the most suitable solution, the second best solution, the third best solution, and the remaining candidates.
However, the inventor finds that, in actual operation, when the traditional gray wolf algorithm is directly applied to perform parameter optimization on the deep learning network model designed by the application, the parameter optimization is easy to fall into a local optimal solution, and the optimization effect is poor. Aiming at the problems, the inventor improves the gray wolf algorithm, and utilizes the improved gray wolf algorithm to automatically perform parameter optimization, so that the parameter optimization is more reliable and accurate, and as shown in fig. 3, the parameter optimization of the deep learning network model by using the improved gray wolf algorithm is as follows:
(1) initializing wolf clusters
Wherein, N represents the scale of the wolf colony,
represents the position vector of the ith wolf, i is more than or equal to 1 and less than or equal to N,
the dimensionality number of the initial coefficient vector is the same as the number of parameters to be optimized in the deep learning network model and corresponds to the parameters one by one, and the coefficient vector is initialized
Coefficient vector
And attenuation coefficient
Determining the maximum iteration times and the upper and lower bounds of the parameters to be optimized; in this embodiment, in the deep learning network model, the parameters to be optimized include the number of neurons in the first GRU layer and the second GRU layer and the size of a convolution kernel, the number of graybugs is set to 50, the maximum iteration time is set to 100, the lower bound is set to-20, the upper bound is set to 20, and the number of dimensions is set to 3;
(2) calculating the fitness value of each wolf in the wolf group, sequentially setting three wolfs with the optimal fitness value as alpha wolfs, beta wolfs and delta wolfs, and setting the rest wolfs as omega wolfs, if the preset convergence condition is met, executing the step (5), otherwise, executing the step (3);
the calculation method of the fitness value is as follows: substituting parameters corresponding to the position vectors of the wolfs into the deep learning network model, training the deep learning network model by using a data set, calculating a loss function in each training, and solving a fitness value based on the loss function, wherein the calculation formula of the fitness value is as follows:
wherein epsilon is a set hyper-parameter, K represents the final training times of the deep learning network model, MAE k Representing the loss function for the k-th iteration.
(3) Updating the position of the omega wolf, wherein the position updating formula is as follows:
in the traditional gray wolf optimization algorithm, the position of omega wolf
Is updated by the formula
In the present application, the new location of the ω wolf in the population is determined according to the fitness scores of the alpha, beta, and delta wolf as follows:
wherein, the first and the second end of the pipe are connected with each other,
indicating the location of the tth iteration of the individual wolf,
coefficient vectors representing alpha wolf, beta wolf and delta wolf,
indicating the position of the t-th iteration of the alpha, beta and delta wolves,
represents the distance between the individual grey wolf and alpha wolf at the t-th iteration,
represents the distance between the individual grey wolf and beta wolf at the t-th iteration,
denotes the distance between the individual grey wolfs and the delta wolf at the t-th iteration, the indices i =1, 2, 3 correspond to alpha wolfs, beta wolfs and delta wolfs, respectively, f (—) denotes the fitness function;
(4) updating a coefficient vector
Coefficient vector
And (3) executing the step (2), and updating the formula as follows:
wherein, the first and the second end of the pipe are connected with each other,
and
each represents [0,1]In the traditional gray wolf optimization algorithm, attenuation coefficient
Is a linearly decreasing vector from 2 to 0, albeit with parameters
In the traditional GWO algorithm, linear decline exists, but in many problems, the exploration and development behaviors of the algorithm need to change in a non-linear way to be far away from the local optimal solution, and therefore the attenuation coefficient is adjusted by the method
The following:
wherein iter represents the current number of iterations, max _ iter represents the maximum number of iterations,
(5) and taking the position of the alpha wolf as an optimal parameter value of the deep learning network model.
The improvement of the grey wolf algorithm in the application is as follows:
1. when the position of each gray wolf is updated, the test vectors are not averaged, but the weighted sum of the test vectors is used for updating each position according to the scoring weights of the three leading gray wolfs, and the updating mechanism is favorable for improving the exploration and development capacity of the algorithm and the capacity of searching for the optimal GUR parameters, so that the accuracy of the deep learning network model provided by the application is improved.
2. Coefficient of attenuation
Adjusted, although this parameter drops linearly in the original GWO algorithm, in many problems the exploration and development behavior of the algorithm needs to change non-linearly to get away from the locally optimal solution, in order to avoid the algorithm getting into the local optimal, the present application redesigns the algorithm
The update rule of (1).
Through the modification of the two points, the improved wolf algorithm is more suitable for optimizing the deep learning network model provided by the application. And a large number of experiments prove that the improved GWO algorithm has a good outstanding optimization effect in optimizing the deep learning network model provided by the application, the original GWO algorithm has a poor optimization effect, the allowable exploration duration of the transition process can be prolonged by 30% (the exploration time of the original GWO process accounts for about 35% of the total time, and the exploration time accounts for 65% after improvement), the mean square error of about 1% of the prediction result is finally reduced, and the optimal solution can be found only by 90% of iterations of the original GWO.
In the prior art, although a traffic prediction algorithm based on deep learning is researched, the traffic prediction algorithm is usually limited to the prediction capability of a deep learning model, the accuracy is low, and the optimal parameters are difficult to train.
The network traffic data change contains a large amount of information, namely whether the network is attacked or not can be understood, the task amount needing to be processed in the network can also be understood, and the characteristic that the network traffic change is more and more complex along with the continuous development of the network is more and more difficult to understand, so that the traffic prediction is carried out by means of deep learning according to the deep mining of the potential information related to the traffic task of the traffic data change.
It can be understood that, in deep learning network model training, whether the deep learning network model is valid or not may be determined by using a difference between a predicted traffic sequence and a real traffic sequence, and a data set may be further divided into a training set, a verification set, a test set, and the like, which are not described herein again.
And S4, acquiring real-time network flow, obtaining future network flow through a flow prediction model, calculating a storage data requirement corresponding to the future network flow, calculating the number of nodes required by the hadoop according to the storage data requirement, if the number of the current nodes of the hadoop is matched with the number of the required nodes, keeping the number of the nodes of the hadoop unchanged, repeating the step, otherwise, adjusting the number of the current nodes of the hadoop to the number of the matched required nodes, and repeating the step.
Wherein, the output of the flow prediction model is a network flow prediction value { f) in a time period T 1 ，f 2 ，...，f N And N represents the flow number, if T can take 1s, and the time granularity is set to 1ms, the flow number N is 1000, and each time granularity has a flow predicted value. Determining the future network flow based on the network flow predicted value in the time period T as follows:
taking the peak value f of the flow in the time period T max Taking the minimum flow value f in the time period T min Taking the average value f of the flow in the time period T avg Then, the calculation formula of the future network traffic f is as follows:
f＝w 1 f max +w 2 f min +w 3 f avg
wherein w 1 、w 2 And w 3 Representing the weight coefficients.
w 3 ＝0.5，w 1 、w 2 The calculation formula of (2) is as follows:
wherein n is h Is { f 1 ，f 2 ，...，f N Higher than f avg Number of flows of n l Is { f 1 ，f 2 ，...，f N Lower than f in avg The number of flows of (c).
Wherein n is h Is { f 1 ，f 2 ，...，f N Higher than f avg Number of flows of n l Is { f 1 ，f 2 ，...，f N Lower than f in avg The number of flows of (c). When the future network flow is calculated, the maximum value, the minimum value and the mean value of the flow are comprehensively considered, and the weight is determined according to the number of nodes higher than the mean value of the flow and lower than the mean value of the flow, so that the future network flow is more accurate
The specific judgment of the adaptive required node number of the current node number of hadoop is as follows: acquiring the current node number D of hadoop 1 Obtaining the number of required nodes D 2 If D is 1 -λ≤D 2 ≤D 1 + lambda, the number of hadoop current nodes is adapted to the number of required nodes, where lambda is a preset elastic expansion threshold, and in this embodiment, lambda is set to be 3/4 of the number of hadoop current nodes;
the "adjusting the current number of nodes of hadoop to the number of nodes required for adaptation" specifically includes: acquiring an adjusting step length; if the current node number of the hadoop is smaller than the required node number, increasing the current node number according to the adjusting step length, judging whether the increased current node number is matched with the required node number, if so, finishing the adjustment, otherwise, repeating the step; if the current node number of hadoop is larger than the required node number, reducing the current node number according to the adjustment step length, judging whether the reduced current node number is matched with the required node number, if so, finishing the adjustment, otherwise, repeating the step.
Wherein, the adjusting step length can be set to be a proper size, so that the number of the nodes is gradually increased or decreased until the requirement is met.
The invention analyzes the network flow for the first time to predict the demand of data on the distributed storage nodes, automatically expands the nodes, and simultaneously carries out secondary adjustment on partial expansion results in order to save resources and not influence the storage requirement, thereby really realizing dynamic expansion of the distributed nodes and avoiding the waste of node resources as much as possible.
It should be noted that the present application may be implemented in software and/or a combination of software and hardware, for example, implemented using Application Specific Integrated Circuits (ASICs), general purpose computers or any other similar hardware devices. In one embodiment, the software programs of the present application may be executed by a processor to implement the steps or functions described above. As such, the software programs (including associated data structures) of the present application can be stored in a computer readable recording medium, such as RAM memory, magnetic or optical drive or diskette and the like. Additionally, some of the steps or functions of the present application may be implemented in hardware, for example, as circuitry that cooperates with the processor to perform various steps or functions.
In addition, some of the present application may be implemented as a computer program product, such as computer program instructions, which when executed by a computer, may invoke or provide methods and/or techniques in accordance with the present application through the operation of the computer. Program instructions which invoke the methods of the present application may be stored on a fixed or removable recording medium and/or transmitted via a data stream on a broadcast or other signal bearing medium and/or stored in a working memory of a computer device operating in accordance with the program instructions. An embodiment according to the present application comprises an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein the computer program instructions, when executed by the processor, trigger the apparatus to perform a method and/or a solution according to the aforementioned embodiments of the present application.
The present application further provides an automatic node expansion system of a distributed storage system, including:
the traffic prediction module is loaded with a trained traffic prediction model and used for obtaining future network traffic according to the real-time network traffic;
the demand node calculation module is used for calculating the storage data demand corresponding to the future network flow and calculating the demand node number of hadoop according to the storage data demand;
the adaptation judging module is used for judging whether the current node number of the hadoop is adapted to the required node number;
the node number adjusting module is used for adjusting the current node number of the hadoop;
the flow prediction model is trained as follows:
acquiring network flow data and making the network flow data into a data set;
building a deep learning network model for regression prediction;
and training the deep learning network model by using the data set to obtain a traffic prediction model for predicting network traffic.
The foregoing detailed description of the preferred embodiments of the invention has been presented. It should be understood that numerous modifications and variations could be devised by those skilled in the art in light of the present teachings without departing from the inventive concepts. Therefore, the technical solutions available to those skilled in the art through logic analysis, reasoning and limited experiments based on the prior art according to the concept of the present invention should be within the scope of protection defined by the claims.
Claims (10)
1. A node automatic scaling method of a distributed storage system is characterized by comprising the following steps:
acquiring network flow data and making the network flow data into a data set;
building a deep learning network model for regression prediction;
training a deep learning network model by using the data set to obtain a flow prediction model for network flow prediction;
acquiring real-time network flow, obtaining future network flow through the flow prediction model, calculating a storage data requirement corresponding to the future network flow, calculating the number of nodes required by the hadoop according to the storage data requirement, if the number of the current nodes of the hadoop is matched with the number of the required nodes, keeping the number of the nodes of the hadoop unchanged, repeating the step, otherwise, adjusting the number of the current nodes of the hadoop to the number of the matched required nodes, and repeating the step.
2. The method according to claim 1, wherein the deep learning network model comprises an input layer, a first feature extraction layer, a second feature extraction layer, a first GRU layer, a second GRU layer, a loss function layer and an output layer, wherein the input layer is used for acquiring input data, the first feature extraction layer and the second feature layer perform feature extraction on the input data to respectively obtain a first feature value and a second feature value, the first feature value and the second feature value are input into the first GRU layer after dimension addition and dimension conversion, the output of the first GRU layer is sent into the second GRU layer, the output of the second GRU layer is sent into the output layer, and the loss function layer is used for performing parameter optimization on the deep learning network model.
3. The method of claim 2, wherein the deep learning network model is based on the following principles:
x j2 ＝f(p(x i ))
f(*)＝max(0，*)
x j ＝cat(x j1 ，x j2 )
x t ＝reshape(x j )
y′ i ＝GRU1(x t )
y i ＝GRU2(x t )
wherein m is the number of neurons in the input layer, x i Representing input data, w i，j Representing weights in the first feature extraction layer, b j Representing a first feature extractionThreshold in layer, x j1 Representing a first characteristic value, x j2 Representing a second characteristic value, p () representing a max pooling operation, x j Representing the value of the first and second eigenvalues after dimensional addition, cat () representing the addition by dimension, x t Represents values of the first and second feature values after dimension transformation, reshape () represents dimension transformation, GRU1 () and GRU2 () represent operations of the first and second GRU layers, y' i Denotes the output of the first GRU layer, y i The output of the second GRU layer is represented and MAE represents the penalty function.
4. The method of claim 3, wherein the parameters of the deep learning network model are determined by an improved Hurrill algorithm optimization as follows:
(1) initializing wolf clusters
Wherein, N represents the scale of the wolf colony,
represents the position vector of the ith wolf, i is more than or equal to 1 and less than or equal to N,
the dimensionality number of the initial coefficient vector is the same as the number of parameters to be optimized in the deep learning network model and corresponds to the parameters one by one, and the coefficient vector is initialized
Coefficient vector
And attenuation coefficient
Determining the maximum iteration times and the upper and lower bounds of the parameters to be optimized;
(2) calculating the fitness value of each wolf in the wolf group, sequentially setting three wolfs with the optimal fitness value as alpha wolfs, beta wolfs and delta wolfs, and setting the rest wolfs as omega wolfs, if the preset convergence condition is met, executing the step (5), otherwise, executing the step (3);
(3) updating the position of the omega wolf, wherein the position updating formula is as follows:
wherein, the first and the second end of the pipe are connected with each other,
indicating the position of the t-th iteration of the wolf individual,
coefficient vectors representing alpha wolf, beta wolf and delta wolf,
indicating the position of the t-th iteration of the alpha, beta and delta wolves,
represents the distance between the individual grey wolf and alpha wolf at the t-th iteration,
represents the distance between the individual grey wolf and beta wolf at the t-th iteration,
denotes the distance between the individual grey wolfs and the delta wolf at the t-th iteration, the indices i =1, 2, 3 correspond to alpha wolfs, beta wolfs and delta wolfs, respectively, f (—) denotes the fitness function;
(4) updating a coefficient vector
Coefficient vector
And (3) executing the step (2), and updating the formula as follows:
wherein iter represents the current number of iterations, max _ iter represents the maximum number of iterations,
and
each represents [0,1 ]]Medium randomA generated vector;
(5) and taking the position of the alpha wolf as an optimal parameter value of the deep learning network model.
5. The method of claim 4, wherein the first GRU layer and the second GRU layer have the same structure, and the deep learning of the parameters to be optimized in the network model comprises: the number of neurons in the first GRU layer and the second GRU layer and the size of the convolution kernel.
6. The method according to claim 5, wherein the fitness value is calculated in a manner that: substituting parameters corresponding to the position vectors of the wolfs into the deep learning network model, training the deep learning network model by using a data set, calculating a loss function in each training, and solving a fitness value based on the loss function, wherein the calculation formula of the fitness value is as follows:
wherein epsilon is a set hyper-parameter, K represents the final training times of the deep learning network model, MAE k Representing the loss function for the k-th iteration.
7. The method according to claim 1, wherein the output of the traffic prediction model is a predicted value { f } of network traffic in a time period T 1 ，f 2 ，...，f N And N represents the flow number, and the future network flow is determined based on the network flow predicted value in the time period T as follows:
taking the peak value f of the flow in the time period T max Taking the minimum flow value f in the time period T min Taking the average value f of the flow in the time period T avg Then, the calculation formula of the future network traffic f is as follows:
f＝w 1 f max +w 2 f min +w 3 f avg
wherein, w 1 、w 2 And w 3 Representing the weight coefficients.
8. The method of claim 7, wherein w is the number of nodes in the distributed storage system 3 ＝0.5，w 1 、w 2 The calculation formula of (c) is:
wherein n is h Is { f 1 ，f 2 ，...，f N Higher than f avg Number of flows of n l Is { f 1 ，f 2 ，...，f N Lower than f in avg The number of flows of (c).
9. The method according to claim 1, wherein the "hadoop current node number adaptation required node number" is specifically:
acquiring the current node number D of hadoop 1 Obtaining the number D of required nodes 2 If D is 1 -λ≤D 2 ≤D 1 + lambda, the number of hadoop current nodes is adapted to the number of required nodes, wherein lambda is a preset elastic expansion threshold;
the "adjusting the current node number of hadoop to the adaptation requirement node number" specifically includes:
obtaining an adjusting step length;
if the current node number of the hadoop is smaller than the required node number, increasing the current node number according to the adjusting step length, judging whether the increased current node number is matched with the required node number, if so, finishing the adjustment, otherwise, repeating the step;
if the current node number of hadoop is larger than the required node number, reducing the current node number according to the adjustment step length, judging whether the reduced current node number is matched with the required node number, if so, finishing the adjustment, otherwise, repeating the step.
10. A node automatic telescoping system of a distributed storage system, comprising:
the traffic prediction module is loaded with a trained traffic prediction model and used for obtaining future network traffic according to the real-time network traffic;
the demand node calculation module is used for calculating the storage data demand corresponding to the future network flow and calculating the demand node number of hadoop according to the storage data demand;
the adaptation judging module is used for judging whether the current node number of the hadoop is adapted to the required node number;
the node number adjusting module is used for adjusting the current node number of the hadoop;
the flow prediction model is trained as follows:
acquiring network flow data and making the network flow data into a data set;
building a deep learning network model for regression prediction;
and training a deep learning network model by using the data set to obtain a flow prediction model for network flow prediction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211188692.8A CN115599296A (en) | 2022-09-28 | 2022-09-28 | Automatic node expansion method and system for distributed storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211188692.8A CN115599296A (en) | 2022-09-28 | 2022-09-28 | Automatic node expansion method and system for distributed storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115599296A true CN115599296A (en) | 2023-01-13 |
Family
ID=84844321
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211188692.8A Pending CN115599296A (en) | 2022-09-28 | 2022-09-28 | Automatic node expansion method and system for distributed storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115599296A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116074209A (en) * | 2023-02-20 | 2023-05-05 | 中移动信息技术有限公司 | Data prediction method, device, equipment and computer storage medium |
-
2022
- 2022-09-28 CN CN202211188692.8A patent/CN115599296A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116074209A (en) * | 2023-02-20 | 2023-05-05 | 中移动信息技术有限公司 | Data prediction method, device, equipment and computer storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Mattos et al. | A stochastic variational framework for recurrent Gaussian processes models | |
| Chakravarty et al. | A PSO based integrated functional link net and interval type-2 fuzzy logic system for predicting stock market indices | |
| CN110909926A (en) | TCN-LSTM-based solar photovoltaic power generation prediction method | |
| Han et al. | Hierarchical extreme learning machine for feedforward neural network | |
| CN113302605A (en) | Robust and data efficient black box optimization | |
| Guo et al. | A fully-pipelined expectation-maximization engine for Gaussian mixture models | |
| CN116595356B (en) | Time sequence signal prediction method and device, electronic equipment and storage medium | |
| CN115599296A (en) | Automatic node expansion method and system for distributed storage system | |
| Jarvenpaa et al. | Batch simulations and uncertainty quantification in Gaussian process surrogate approximate Bayesian computation | |
| CN113407820B (en) | Method for processing data by using model, related system and storage medium | |
| Bo et al. | Developing real-time scheduling policy by deep reinforcement learning | |
| WO2023202484A1 (en) | Neural network model repair method and related device | |
| Zhang et al. | A hierarchical multivariate denoising diffusion model | |
| Guan et al. | Reduce the difficulty of incremental learning with self-supervised learning | |
| Wu et al. | Improved saddle point prediction in stochastic two-player zero-sum games with a deep learning approach | |
| Bick | Towards delivering a coherent self-contained explanation of proximal policy optimization | |
| Cheng et al. | Robust Actor-Critic With Relative Entropy Regulating Actor | |
| CN115242428A (en) | Network security situation prediction method based on optimized CW-RNN | |
| CN114386565A (en) | Providing a neural network | |
| Nastac | An adaptive retraining technique to predict the critical process variables | |
| Neves et al. | When less may be more: Exploring similarity to improve experience replay | |
| CN116452320B (en) | Credit risk prediction method based on continuous learning | |
| CN117152588B (en) | Data optimization method, system, device and medium | |
| Urban et al. | Gaussian process neurons | |
| Wang et al. | Acceleration techniques for optimization over trained neural network ensembles |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |