CN107229693A - The method and system of big data system configuration parameter tuning based on deep learning - Google Patents
The method and system of big data system configuration parameter tuning based on deep learning Download PDFInfo
- Publication number
- CN107229693A CN107229693A CN201710361578.3A CN201710361578A CN107229693A CN 107229693 A CN107229693 A CN 107229693A CN 201710361578 A CN201710361578 A CN 201710361578A CN 107229693 A CN107229693 A CN 107229693A
- Authority
- CN
- China
- Prior art keywords
- parameter
- layer
- stipulations
- output
- configuration parameter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a kind of method and system of the big data system configuration parameter tuning based on deep learning, wherein method includes：Neural metwork training step, Primary Construction deep neural network, using at least one mapping stipulations parameter as input parameter, using it is to be predicted go out allocation optimum parameter as output parameter, training sample set is used as using the historical data of big data system；Again to map the stipulations time as the criterion of the deep neural network, the parameter learning rule based on backpropagation thought is adjusted to the weights of every layer of neuron, until the mapping stipulations time meets time cost requirement；Configuration parameter prediction steps, set the initial value of at least one mapping stipulations parameter, and read current test data, are input in the deep neural network obtained via neural metwork training step, obtain configuration parameter.The present invention carries out tuning by deep neural network to the configuration parameter in mapping stipulations framework, it is to avoid manual adjustment, and the parameter application effect of prediction is good.
Description
Technical field
The present invention relates to field of computer technology, more particularly to a kind of big data system configuration parameter based on deep learning
The method and system of tuning.
Background technology
In recent years, big data is explored and analyzed and flourished in every field.Big data system can be divided into 3 levels：(1)
Basal layer：I.e. basic data machined layer, hardware resource is distributed to the execution podium level for supporting calculating task；(2) podium level：I.e.
Kernel business tier, both provided an interface for being easily handled data set for application layer, and can management infrastructure Layer assignment
Resource；(3) application layer：Predict the outcome output layer, predicts expert decision-making, provides big data analysis result.
Podium level serves the effect formed a connecting link in big data system, is also the core of a big data system
Point.MapReduce (mapping stipulations) in Hadoop system is exactly a kind of model in podium level.Hadoop is a distribution
System infrastructure.User can develop distributed program in the case where not knowing about distributed low-level details.Make full use of collection
The power of group carries out high-speed computation and storage.MapReduce is a kind of programming model under Hadoop, for large-scale dataset
The concurrent operation of (being more than 1TB).He be very easy to programming personnel will not distributed parallel programming in the case of, by oneself
Program operate in distributed system.Hadoop MapReduce functions, which are realized, smashes individual task, and mapping is appointed
Business (Map) is sent on multiple nodes, afterwards again to load stipulations (Reduce) in the form of individual data collection in data warehouse.
Configuration parameter is set to have a great impact to MapReduce service behaviours.The configuration parameter of high-quality makes MapReduce
Outstanding work, and configuration parameter mistake is Hadoop Map Reduce system performance degradation and causes the main original of thrashing
Cause.In order to which helpdesk keeper optimizes systematic function, it is necessary to adjust the characteristics of configuration parameter processing is different, different program and
Different input data, to pursue faster work performance.In conventional method, keeper is adjusted one by one to configuration parameter,
Or linear regression is utilized, parameter is configured, extracting parameter feature, showed according to MapReduce transaction capabilities, so as to provide
Approximate optimal solution, predicted configuration parameter is to reach more preferable service behaviour.
However, there are two hang-ups during Admin Administration's Hadoop system：(1) because the behavior of large scale distributed system
It is excessively complicated with feature, it is difficult to find appropriate configuration parameter；(2) there are hundreds of parameters, main influence systems performance in system
Configuration parameter have tens, configuration parameter tuning is become trouble.In conventional method, manual method or automatic using returning
Ginseng is adjusted, extremely complex cumbersome, parameter regulation is needed to consume the plenty of time, and income effect is not fine, and system overall work is needed
Consume for a long time.
The content of the invention
The technical problem to be solved in the present invention is, is automatically adjusted for manual method in the prior art or using recurrence
There is provided a kind of big data system configuration parameter tuning based on deep learning for the defect that the efficiency of configuration parameter is low and effect is poor
Method and system.
First aspect present invention there is provided a kind of method of the big data system configuration parameter tuning based on deep learning,
Including neural metwork training step and configuration parameter prediction steps；Wherein,
The neural metwork training step comprises the following steps：
Step 1-1, Primary Construction deep neural network, wherein using at least one mapping stipulations parameter as input parameter,
Using it is to be predicted go out allocation optimum parameter as output parameter, training sample set is used as using the historical data of big data system；
Step 1-2, to map the stipulations time as the criterion of the deep neural network, based on backpropagation thought
Parameter learning rule is adjusted to the weights of every layer of neuron, until the mapping stipulations time meets time cost requirement；
The configuration parameter prediction steps comprise the following steps：
Step 2-1, setting at least one mapping stipulations parameter initial value, and read current test data；
Step 2-2, by it is described at least one mapping stipulations parameter initial value and current test data be input to via god
In the deep neural network obtained through network training step, the configuration ginseng for the big data system based on deep learning is obtained
Number.
In the method according to the big data system configuration parameter tuning of the present invention based on deep learning, it is described extremely
The quantity of few mapping stipulations parameter is 2~20.
Second aspect of the present invention there is provided a kind of system of the big data system configuration parameter tuning based on deep learning,
Including neural metwork training module and configuration parameter prediction module；Wherein,
The neural metwork training module is used for Primary Construction deep neural network, wherein with least one mapping stipulations ginseng
Number as input parameter, using it is to be predicted go out allocation optimum parameter as output parameter, using the historical data of big data system as
Training sample set；And to map the stipulations time as the criterion of the deep neural network, the ginseng based on backpropagation thought
Number learning rules are adjusted to the weights of every layer of neuron, until the mapping stipulations time meets time cost requirement；
The configuration parameter prediction module is used for the initial value of at least one mapping stipulations parameter described in setting and worked as
Preceding test data is input in the deep neural network obtained via neural metwork training step, obtains being used to be based on deep learning
Big data system configuration parameter.
In the system according to the big data system configuration parameter tuning of the present invention based on deep learning, it is described extremely
The quantity of few mapping stipulations parameter is 2~20.
Implement the method and system of the big data system configuration parameter tuning based on deep learning of the present invention, with following
Beneficial effect：The present invention carries out tuning by deep neural network to the configuration parameter in mapping stipulations framework, it is to avoid artificial
Regulation, finds the problem of optimized parameter, by the study to history parameters, each configuration parameter itself can be obtained to a deeper level
Feature, and relation each other, by the multiple study of depth network, right value update, neural network forecast obtains being best suitable for application layer
The parameter configuration of application demand.The present invention not only saves the time of parameter regulation, when the parameter of appropriate system makes the system work
Between distribute to compressed and decompressed data so that substantially reduce write-in and transmission time, make total system work can be rapidly completed,
More preferable working effect can be reached again.
Brief description of the drawings
Fig. 1 is the method for the big data system configuration parameter tuning based on deep learning according to the preferred embodiment of the present invention
Flow chart；
Fig. 2 is the schematic flow sheet of neural metwork training step in the method according to the preferred embodiment of the present invention；
Fig. 3 is the system of the big data system configuration parameter tuning based on deep learning according to the preferred embodiment of the present invention
Module frame chart.
Embodiment
To make the purpose, technical scheme and advantage of the embodiment of the present invention clearer, below in conjunction with the embodiment of the present invention
In accompanying drawing, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is
A part of embodiment of the present invention, rather than whole embodiments.Based on the embodiment in the present invention, ordinary skill people
The every other embodiment that member is obtained on the premise of creative work is not made, belongs to the scope of protection of the invention.
, will be deep the invention provides a kind of method that big data network configuration parameters tuning is carried out using deep neural network
Degree neural network framework is incorporated into configuration parameter link, both saves time cost, good working effect can be reached again.This hair
Parameter is learnt in bright mapping tasks (Map task) and stipulations task (Reduce task) mainly for big data system
With distribute rationally.It is a complicated flow to map stipulations (MapReduce), and first the workflow for mapping stipulations is carried out below
Simple to introduce, the key step of stipulations mapping is as follows：
(1) Map ends (mapping end) course of work
(1) each input burst can allow a map task (mapping tasks) to handle, in a distributed manner file system (HDFS)
The size (initial value is 64M) of a block be burst.The result of Map outputs can be placed on a circulating memory buffering for the time being
Qu Zhong, the size initial value of the buffering area is 100M, by io.sort.mb property controls.When the buffering area soon overflows (just
Beginning is set to the 80% of buffer size, by io.sort.spill.percent property controls), can be in local file system
A spill file is created, the data in the buffering area are write into this file.
(2) before write-in disk, data are divided into by thread according to the number of reduce tasks (stipulations task) first
Equal number of subregion, that is, a reduce task correspond to the data of a subregion.Then to the data in each subregion
It is ranked up, Combiner is set, the result after sequence is carried out into Combia (merging) operates.
(3) when map tasks export last record, many spill files is might have, are at this moment needed these
Piece file mergence.Can constantly it be ranked up during merging and Combia (merging) operations.
(4) data in subregion are transferred to corresponding reduce tasks.
(2) Reduce ends (stipulations end) course of work
(1) Reduce can receive the data that different map tasks are transmitted, and the data that each map is transmitted are ordered
's.If the data volume that reduce terminations are received is fairly small, be stored directly in internal memory (buffer size, by
Mapred.job.shuffle.input.buffer.percent property controls, the percentage of the heap space of expression used as said purpose
Than).If data volume has exceeded the certain proportion of the buffer size (by mapred.job.shuffle.merge.percent
Determine), then overflow and write in disk after merging to data.
(2) the abbreviation program that application layer is defined, final output data are performed.Compress on demand, write final output
To HDFS.
Referring to Fig. 1, being adjusted for the big data system configuration parameter based on deep learning according to the preferred embodiment of the present invention
Excellent method flow diagram.As shown in figure 1, the big data system configuration parameter tuning based on deep learning that the embodiment is provided
Method mainly includes neural metwork training step and configuration parameter prediction steps：
First, neural metwork training step, construction depth neutral net, with keeper are performed into S102 in step S101
The history working condition of offer is training set, using the allocation optimum parameter that predicts to export.And with (mapping stipulations)
MapReduce time cost is the final criterion of network structure, and continuous feedback adjustment structure obtains ultimate depth nerve net
Network structure.It is specific as follows：
Step S101, Primary Construction deep neural network, wherein using at least one mapping stipulations parameter as input parameter,
Using it is to be predicted go out allocation optimum parameter as output parameter, training sample set is used as using the historical data of big data system.This is big
The historical data of data system is specially the history working condition that keeper provides.Preferably, at least one described mapping stipulations
Parameter can choose one or more from following important parameter form 1.In a particular application, according to different situations, from system
Obtained at keeper and input/output list is added to 20 maximum parameters of systematic influence, Selecting All Parameters are as shown in table 1 below.This is extremely
The quantity of few mapping stipulations parameter is preferably 2~20.
The important parameter table of form 1
Step S102, to map the stipulations time as the criterion of the deep neural network, based on backpropagation thought
Parameter learning rule the weights of every layer of neuron are adjusted, until the mapping stipulations time meets time cost requirement.Should
Using MapReduce time cost as the final criterion of network structure in step, continuous feedback adjustment structure obtains final depth
Spend the structure of neutral net.
Then, configuration parameter prediction steps are performed into S104 in step S103, it is pre- using obtained deep neural network
Measure the configuration parameter for making working effect optimal.It is specific as follows：
Step S103, the initial value of setting at least one mapping stipulations parameter, and read current test data.
Step S104, the initial value and current test data of at least one mapping stipulations parameter are input to via god
In the deep neural network obtained through network training step, the configuration ginseng for the big data system based on deep learning is obtained
Number.
As can be seen here, the present invention introduces depth after initialization mapping (Map) task and stipulations (Reduce) task parameters
Neutral net, training set source is historic task daily record, history parameters is learnt, semi-supervised learning, gone through by known
History working condition, to the feedback of service behaviour, draws the parameter inside deep neural network, so that predict and optimization, for
Different programs and different input datas are attained by the configuration parameter of optimal work performance.
Fig. 2 is please referred to, is that the flow of neural metwork training step in the method according to the preferred embodiment of the present invention is shown
It is intended to.As shown in Fig. 2 the neural metwork training step includes：
First, in step s 201, flow starts；
Then, in step S202, Primary Construction deep neural network.The deep neural network is to utilize backpropagation
Common deep-neural-network.Specifically, built in the step to map five layer depth nerve nets of the stipulations parameter as input parameter
Network, using it is to be predicted go out allocation optimum parameter as output parameter, foregoing five layer network includes input layer, output layer and three respectively
Hidden layer.
Then, in step S203, depth nerve net is inputted using the historical data of big data system as training sample set
Network.Training sample x is inputted, hidden layer is output as xl=f (ul), wherein ul=Wlxl-1+bl, wherein, function f represents output activation letter
Number, W represents weights, and b represents bias term, and l represents the 1st layer.Because parameter can not infinitely expand during map and reduce, have
Certain limit, it is therefore desirable to which fixed b is parameter upper limit.
Then, in step S204, judge whether the mapping stipulations time meets time cost requirement.Use square error generation
Valency function weighs error, it is assumed that output parameter classification is c, and training sample concentrates N number of training sample altogether, then maps stipulations time
With the error E between stipulated time cost tNFor：Wherein,For n-th training sample
The kth dimension of target output,Tieed up for the kth of the corresponding reality output of n-th of sample, c=20.Calculate the mistake between each layer network
Difference, goes to step S206 and preserves the deep neural network, otherwise go to step S205 and adjust every layer of god when error is less than predetermined threshold value
Weights through member.
In step S205, the weights of every layer of neuron are adjusted.Specifically, by the sensitivity δ of neuron in the step
Zoomed in and out come the weights W to every layer of neuron, it is the minimum weights of E to finally give：
Wherein,And l layers of sensitivity：δl=(Wl+1)Tδl+1οf'(ul)；The neuron of output layer
Sensitivity be：δL=f'(uL)·(yn-tn), wherein L represents total number of plies, ynFor the reality output of n-th of neuron, tnFor
The target output of n neuron.
In step S206, the deep neural network is preserved；
Finally, in step S207, the flow of the neural metwork training step terminates.
Using stipulations mapping time and configuration parameter as output in the present invention, stipulations mapping time can be understood as centre
During output, configuration parameter, which is that we are most important, to be recorded and the output that uses, is compared according to output time and ideal time
Weights are adjusted after error, the output of time is not only changed after adjustment weights, the output of configuration parameter is also changed, therefore can
To obtain configuration parameter during time optimal.
Referring to Fig. 3, being adjusted for the big data system configuration parameter based on deep learning according to the preferred embodiment of the present invention
The module frame chart of excellent system.As shown in figure 3, the big data system configuration parameter based on deep learning that the embodiment is provided is adjusted
Excellent system 300 includes neural metwork training module 301 and configuration parameter prediction module 302.
Wherein, neural metwork training module 301 is used for Primary Construction deep neural network, wherein being advised with least one mapping
About parameter as input parameter, using it is to be predicted go out allocation optimum parameter as output parameter, with the historical data of big data system
It is used as training sample set；And to map the stipulations time as the criterion of the deep neural network, based on backpropagation thought
Parameter learning rule the weights of every layer of neuron are adjusted, until the mapping stipulations time meets time cost requirement.It is excellent
Selection of land, at least one described mapping stipulations parameter can choose one or more from weight form.At least one mapping stipulations
The quantity of parameter is preferably 2~20.
Specifically, the neural metwork training module 301 is built to map five layer depths god of the stipulations parameter as input parameter
Through network, using it is to be predicted go out allocation optimum parameter as output parameter, five layer network include respectively input layer, output layer and
Three hidden layers, input training sample x, and hidden layer is output as xl=f (ul), wherein ul=Wlxl-1+bl, function f, which is represented, exports activation letter
Number, W represents weights, and b represents bias term, and l represents the 1st layer.
The neural metwork training module 301 also weighs error using square error cost function, it is assumed that output parameter class
Not Wei c, training sample concentrate altogether N number of training sample, then map stipulations time and the stipulated time cost t between error ENFor：Wherein,Kth for the target output of n-th of training sample is tieed up,For n-th of sample
The kth dimension of corresponding reality output.
The error between each layer network is then calculated, the deep neural network is preserved when error is less than predetermined threshold value, it is no
Then zoomed in and out by the sensitivity δ of neuron come the weights W to every layer of neuron：
Wherein,And l layers of sensitivity：δl=(Wl+1)Tδl+1οf'(ul)；The neuron of output layer
Sensitivity be：δL=f'(uL)·(yn-tn), wherein L represents total number of plies, ynFor the reality output of n-th of neuron, tnFor
The target output of n neuron.
Configuration parameter prediction module 302 is connected with neural metwork training module 301, for by described in setting at least one
The initial value and current test data for mapping stipulations parameter are input to the depth nerve net obtained via neural metwork training step
In network, the configuration parameter for the big data system based on deep learning is obtained.
In summary, deep neural network is used to the configuration in mapping stipulations (MapReduce) framework using the present invention
Parameter carries out tuning, it is to avoid manual adjustment, finds the problem of optimized parameter, by the study to history parameters, can be deeper
Obtain each configuration parameter own characteristic, and relation each other level, by the multiple study of depth network, right value update,
Neural network forecast obtains being best suitable for the parameter configuration of application layer applications demand.The present invention not only saves the time of parameter regulation, closes
The parameter of suitable system makes System production time distribute to compressed and decompressed data, so as to substantially reduce write-in and transmission time, makes
Total system work can be rapidly completed, and more preferable working effect can be reached again.Simultaneously for different basal layer input datas and
Application layer propose application requirement, can autonomous learning, with stronger adaptability.
Finally it should be noted that：The above embodiments are merely illustrative of the technical solutions of the present invention, rather than its limitations；Although
The present invention is described in detail with reference to the foregoing embodiments, it will be understood by those within the art that：It still may be used
To be modified to the technical scheme described in foregoing embodiments, or equivalent substitution is carried out to which part technical characteristic；
And these modification or replace, do not make appropriate technical solution essence depart from various embodiments of the present invention technical scheme spirit and
Scope.
Claims (8)
1. a kind of method of the big data system configuration parameter tuning based on deep learning, it is characterised in that including neutral net
Training step and configuration parameter prediction steps；Wherein,
The neural metwork training step comprises the following steps：
Step 1-1, Primary Construction deep neural network, wherein using at least one mapping stipulations parameter as input parameter, to treat
Allocation optimum parameter is predicted as output parameter, training sample set is used as using the historical data of big data system；
Step 1-2, to map the stipulations time as the criterion of the deep neural network, the parameter based on backpropagation thought
Learning rules are adjusted to the weights of every layer of neuron, until the mapping stipulations time meets time cost requirement；
The configuration parameter prediction steps comprise the following steps：
Step 2-1, setting at least one mapping stipulations parameter initial value, and read current test data；
Step 2-2, by it is described at least one mapping stipulations parameter initial value and current test data be input to via nerve net
In the deep neural network that network training step is obtained, the configuration parameter for the big data system based on deep learning is obtained.
2. the method for the big data system configuration parameter tuning according to claim 1 based on deep learning, its feature exists
In the quantity of at least one mapping stipulations parameter is 2~20.
3. the method for the big data system configuration parameter tuning according to claim 1 or 2 based on deep learning, its feature
It is, in the step 1-1：
Build to map five layer depth neutral nets of the stipulations parameter as input parameter, using it is to be predicted go out allocation optimum parameter as
Output parameter, five layer network includes input layer, output layer and three hidden layers respectively, inputs training sample x, and hidden layer is output as
Y=xl=f (ul), wherein ul=Wlxl-1+bl, function f, which is represented, exports activation primitive, and W represents weights, and b represents bias term, and l is represented
1st layer.
4. the method for the big data system configuration parameter tuning according to claim 3 based on deep learning, its feature exists
In in the step 1-2：
Error is weighed using square error cost function, it is assumed that output parameter classification is c, training sample concentrates N number of training altogether
Sample, then map the error E between stipulations time and the stipulated time cost tNFor：Its
In,Kth for the target output of n-th of training sample is tieed up,Tieed up for the kth of the corresponding reality output of n-th of sample；
The error between each layer network is calculated, the deep neural network is preserved when error is less than predetermined threshold value, otherwise passes through god
Sensitivity δ through member zooms in and out come the weights W to every layer of neuron：
<mrow>
<msup>
<mi>&Delta;W</mi>
<mi>l</mi>
</msup>
<mo>=</mo>
<mo>-</mo>
<mi>&eta;</mi>
<mfrac>
<mrow>
<mo>&part;</mo>
<mi>E</mi>
</mrow>
<mrow>
<mo>&part;</mo>
<msup>
<mi>W</mi>
<mi>l</mi>
</msup>
</mrow>
</mfrac>
<mo>;</mo>
</mrow>
Wherein,And l layers of sensitivity：The spirit of the neuron of output layer
Sensitivity is：δL=f'(uL)·(yn-tn), wherein L represents total number of plies, ynFor the reality output of n-th of neuron, tnFor n-th
The target output of neuron.
5. a kind of system of the big data system configuration parameter tuning based on deep learning, it is characterised in that including neutral net
Training module and configuration parameter prediction module；Wherein,
The neural metwork training module is used for Primary Construction deep neural network, wherein being made with least one mapping stipulations parameter
For input parameter, using it is to be predicted go out allocation optimum parameter as output parameter, training is used as using the historical data of big data system
Sample set；And to map the stipulations time as the criterion of the deep neural network, the parametrics based on backpropagation thought
Practise rule to be adjusted the weights of every layer of neuron, until the mapping stipulations time meets time cost requirement；
The configuration parameter prediction module is used for the initial value of at least one mapping stipulations parameter described in setting and current survey
Data input is tried into the deep neural network obtained via neural metwork training step, is obtained for based on the big of deep learning
The configuration parameter of data system.
6. the system of the big data system configuration parameter tuning according to claim 5 based on deep learning, its feature exists
In the quantity of at least one mapping stipulations parameter is 2~20.
7. the system of the big data system configuration parameter tuning based on deep learning according to claim 5 or 6, its feature
It is, the neural metwork training module is used to build to map five layer depth neutral nets of the stipulations parameter as input parameter,
Using it is to be predicted go out allocation optimum parameter as output parameter, five layer network respectively include input layer, output layer and three it is hidden
Layer, inputs training sample x, and hidden layer is output as xl=f (ul), wherein ul=Wlxl-1+bl, wherein, function f represents output activation letter
Number, W represents weights, and b represents bias term, and l represents the 1st layer.
8. the system of the big data system configuration parameter tuning according to claim 7 based on deep learning, its feature exists
In the neural metwork training module weighs error using square error cost function, it is assumed that output parameter classification is c, instruction
Practice common N number of training sample in sample set, then map the error E between stipulations time and the stipulated time cost tNFor：Wherein,Kth for the target output of n-th of training sample is tieed up,For n-th of sample
The kth dimension of corresponding reality output；
The error between each layer network is calculated, the deep neural network is preserved when error is less than predetermined threshold value, otherwise passes through god
Sensitivity δ through member zooms in and out come the weights W to every layer of neuron：
<mrow>
<msup>
<mi>&Delta;W</mi>
<mi>l</mi>
</msup>
<mo>=</mo>
<mo>-</mo>
<mi>&eta;</mi>
<mfrac>
<mrow>
<mo>&part;</mo>
<mi>E</mi>
</mrow>
<mrow>
<mo>&part;</mo>
<msup>
<mi>W</mi>
<mi>l</mi>
</msup>
</mrow>
</mfrac>
<mo>;</mo>
</mrow>
Wherein,And l layers of sensitivity：The spirit of the neuron of output layer
Sensitivity is：δL=f'(uL)·(yn-tn), wherein L represents total number of plies, ynFor the reality output of n-th of neuron, tnFor n-th
The target output of neuron.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710361578.3A CN107229693B (en) | 2017-05-22 | 2017-05-22 | The method and system of big data system configuration parameter tuning based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710361578.3A CN107229693B (en) | 2017-05-22 | 2017-05-22 | The method and system of big data system configuration parameter tuning based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107229693A true CN107229693A (en) | 2017-10-03 |
| CN107229693B CN107229693B (en) | 2018-05-01 |
Family
ID=59933231
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710361578.3A Active CN107229693B (en) | 2017-05-22 | 2017-05-22 | The method and system of big data system configuration parameter tuning based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107229693B (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108363478A (en) * | 2018-01-09 | 2018-08-03 | 北京大学 | For wearable device deep learning application model load sharing system and method |
| CN108764568A (en) * | 2018-05-28 | 2018-11-06 | 哈尔滨工业大学 | A kind of data prediction model tuning method and device based on LSTM networks |
| CN108990141A (en) * | 2018-07-19 | 2018-12-11 | 浙江工业大学 | A kind of energy-collecting type wireless relay network througput maximization approach based on the study of depth Multi net voting |
| CN109041195A (en) * | 2018-07-19 | 2018-12-18 | 浙江工业大学 | A kind of energy-collecting type wireless relay network througput maximization approach based on semi-supervised learning |
| CN109445935A (en) * | 2018-10-10 | 2019-03-08 | 杭州电子科技大学 | A kind of high-performance big data analysis system self-adaption configuration method under cloud computing environment |
| CN109739950A (en) * | 2018-12-25 | 2019-05-10 | 中国政法大学 | Screen the method and device of applicable law provision |
| CN109992404A (en) * | 2017-12-31 | 2019-07-09 | 中国移动通信集团湖北有限公司 | PC cluster resource regulating method, device, equipment and medium |
| CN110134697A (en) * | 2019-05-22 | 2019-08-16 | 南京大学 | A kind of parameter automated tuning method, apparatus, system towards key-value pair storage engines |
| CN110427356A (en) * | 2018-04-26 | 2019-11-08 | 中移(苏州)软件技术有限公司 | One parameter configuration method and equipment |
| WO2020125079A1 (en) * | 2018-12-19 | 2020-06-25 | 清华大学 | High-throughput material simulation calculation optimisation method based on time prediction |
| CN113254472A (en) * | 2021-06-17 | 2021-08-13 | 浙江大华技术股份有限公司 | Parameter configuration method, device, equipment and readable storage medium |
| TWI752614B (en) * | 2020-09-03 | 2022-01-11 | 國立陽明交通大學 | Multiple telecommunication endpoints system and testing method thereof based on ai decision |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104504460A (en) * | 2014-12-09 | 2015-04-08 | 北京嘀嘀无限科技发展有限公司 | Method and device for predicating user loss of car calling platform |
| US20150227393A1 (en) * | 2014-02-10 | 2015-08-13 | International Business Machines Corporation | Dynamic Resource Allocation in Mapreduce |
| CN106022521A (en) * | 2016-05-19 | 2016-10-12 | 四川大学 | Hadoop framework-based short-term load prediction method for distributed BP neural network |
| CN106202431A (en) * | 2016-07-13 | 2016-12-07 | 华中科技大学 | A kind of Hadoop parameter automated tuning method and system based on machine learning |
| CN106648654A (en) * | 2016-12-20 | 2017-05-10 | 深圳先进技术研究院 | Data sensing-based Spark configuration parameter automatic optimization method |
-
2017
- 2017-05-22 CN CN201710361578.3A patent/CN107229693B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150227393A1 (en) * | 2014-02-10 | 2015-08-13 | International Business Machines Corporation | Dynamic Resource Allocation in Mapreduce |
| CN104504460A (en) * | 2014-12-09 | 2015-04-08 | 北京嘀嘀无限科技发展有限公司 | Method and device for predicating user loss of car calling platform |
| CN106022521A (en) * | 2016-05-19 | 2016-10-12 | 四川大学 | Hadoop framework-based short-term load prediction method for distributed BP neural network |
| CN106202431A (en) * | 2016-07-13 | 2016-12-07 | 华中科技大学 | A kind of Hadoop parameter automated tuning method and system based on machine learning |
| CN106648654A (en) * | 2016-12-20 | 2017-05-10 | 深圳先进技术研究院 | Data sensing-based Spark configuration parameter automatic optimization method |
Non-Patent Citations (1)
| Title |
|---|
| 吕琼帅: "BP神经网络的优化与研究", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109992404B (en) * | 2017-12-31 | 2022-06-10 | 中国移动通信集团湖北有限公司 | Cluster computing resource scheduling method, device, equipment and medium |
| CN109992404A (en) * | 2017-12-31 | 2019-07-09 | 中国移动通信集团湖北有限公司 | PC cluster resource regulating method, device, equipment and medium |
| CN108363478A (en) * | 2018-01-09 | 2018-08-03 | 北京大学 | For wearable device deep learning application model load sharing system and method |
| CN108363478B (en) * | 2018-01-09 | 2019-07-12 | 北京大学 | For wearable device deep learning application model load sharing system and method |
| CN110427356B (en) * | 2018-04-26 | 2021-08-13 | 中移(苏州)软件技术有限公司 | Parameter configuration method and equipment |
| CN110427356A (en) * | 2018-04-26 | 2019-11-08 | 中移(苏州)软件技术有限公司 | One parameter configuration method and equipment |
| CN108764568B (en) * | 2018-05-28 | 2020-10-23 | 哈尔滨工业大学 | Data prediction model tuning method and device based on LSTM network |
| CN108764568A (en) * | 2018-05-28 | 2018-11-06 | 哈尔滨工业大学 | A kind of data prediction model tuning method and device based on LSTM networks |
| CN108990141A (en) * | 2018-07-19 | 2018-12-11 | 浙江工业大学 | A kind of energy-collecting type wireless relay network througput maximization approach based on the study of depth Multi net voting |
| CN109041195A (en) * | 2018-07-19 | 2018-12-18 | 浙江工业大学 | A kind of energy-collecting type wireless relay network througput maximization approach based on semi-supervised learning |
| CN108990141B (en) * | 2018-07-19 | 2021-08-03 | 浙江工业大学 | Energy-collecting wireless relay network throughput maximization method based on deep multi-network learning |
| CN109445935A (en) * | 2018-10-10 | 2019-03-08 | 杭州电子科技大学 | A kind of high-performance big data analysis system self-adaption configuration method under cloud computing environment |
| CN109445935B (en) * | 2018-10-10 | 2021-08-10 | 杭州电子科技大学 | Self-adaptive configuration method of high-performance big data analysis system in cloud computing environment |
| WO2020125079A1 (en) * | 2018-12-19 | 2020-06-25 | 清华大学 | High-throughput material simulation calculation optimisation method based on time prediction |
| CN109739950A (en) * | 2018-12-25 | 2019-05-10 | 中国政法大学 | Screen the method and device of applicable law provision |
| CN110134697A (en) * | 2019-05-22 | 2019-08-16 | 南京大学 | A kind of parameter automated tuning method, apparatus, system towards key-value pair storage engines |
| TWI752614B (en) * | 2020-09-03 | 2022-01-11 | 國立陽明交通大學 | Multiple telecommunication endpoints system and testing method thereof based on ai decision |
| CN113254472A (en) * | 2021-06-17 | 2021-08-13 | 浙江大华技术股份有限公司 | Parameter configuration method, device, equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107229693B (en) | 2018-05-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107229693B (en) | The method and system of big data system configuration parameter tuning based on deep learning | |
| Madani et al. | A Monte-Carlo game theoretic approach for multi-criteria decision making under uncertainty | |
| CN102567391B (en) | Method and device for building classification forecasting mixed model | |
| CN103631657B (en) | A kind of method for scheduling task based on MapReduce | |
| CN108491969A (en) | Spatial Load Forecasting model building method based on big data | |
| CN102541858B (en) | Based on mapping and the data balancing processing method of stipulations, Apparatus and system | |
| CN111008693B (en) | Network model construction method, system and medium based on data compression | |
| CN104573106A (en) | Intelligent urban construction examining and approving method based on case-based reasoning technology | |
| CN104750780B (en) | A kind of Hadoop configuration parameter optimization methods based on statistical analysis | |
| CN103473598A (en) | Extreme learning machine based on length-changing particle swarm optimization algorithm | |
| Zhao et al. | QoS-aware web service selection with negative selection algorithm | |
| CN107734052A (en) | The load balancing container dispatching method that facing assembly relies on | |
| Papageorgiou et al. | Application of fuzzy cognitive maps to water demand prediction | |
| CN105654196A (en) | Adaptive load prediction selection method based on electric power big data | |
| CN104077634B (en) | active-reactive type dynamic project scheduling method based on multi-objective optimization | |
| CN106371924B (en) | A kind of method for scheduling task minimizing MapReduce cluster energy consumption | |
| CN110489749B (en) | Business process optimization method of intelligent office automation system | |
| CN108363643A (en) | A kind of HDFS copy management methods based on file access temperature | |
| Khmeleva et al. | Fuzzy-logic controlled genetic algorithm for the rail-freight crew-scheduling problem | |
| CN102624865A (en) | Cluster load prediction method and distributed cluster management system | |
| CN109711707B (en) | Comprehensive state evaluation method for ship power device | |
| CN107886160A (en) | A kind of BP neural network section water demand prediction method | |
| CN105808582A (en) | Parallel generation method and device of decision tree on the basis of layered strategy | |
| CN108829846A (en) | A kind of business recommended platform data cluster optimization system and method based on user characteristics | |
| CN105847358A (en) | Method for realizing big data node distribution in cloud computing environment and system thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |