CN107229693B - The method and system of big data system configuration parameter tuning based on deep learning - Google Patents

The method and system of big data system configuration parameter tuning based on deep learning Download PDF

Info

Publication number
CN107229693B
CN107229693B CN201710361578.3A CN201710361578A CN107229693B CN 107229693 B CN107229693 B CN 107229693B CN 201710361578 A CN201710361578 A CN 201710361578A CN 107229693 B CN107229693 B CN 107229693B
Authority
CN
China
Prior art keywords
parameter
stipulations
layer
output
neural network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710361578.3A
Other languages
Chinese (zh)
Other versions
CN107229693A (en
Inventor
王宏志
王艺蒙
赵志强
孙旭冉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Da Da Data Industry Co Ltd
Original Assignee
Da Da Data Industry Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Da Da Data Industry Co Ltd filed Critical Da Da Data Industry Co Ltd
Priority to CN201710361578.3A priority Critical patent/CN107229693B/en
Publication of CN107229693A publication Critical patent/CN107229693A/en
Application granted granted Critical
Publication of CN107229693B publication Critical patent/CN107229693B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The present invention provides a kind of method and system of the big data system configuration parameter tuning based on deep learning, wherein method includes：Neural metwork training step, Primary Construction deep neural network, using at least one mapping stipulations parameter as input parameter, using it is to be predicted go out allocation optimum parameter as output parameter, training sample set is used as using the historical data of big data system；Again to map the stipulations time as the measurement standard of the deep neural network, the parameter learning rule based on backpropagation thought is adjusted the weights of every layer of neuron, until the mapping stipulations time meets time cost requirement；Parameter prediction step is configured, sets the initial value of at least one mapping stipulations parameter, and reads current test data, is input in the deep neural network obtained via neural metwork training step, obtains configuration parameter.The present invention carries out tuning by deep neural network to the configuration parameter in mapping stipulations frame, avoids manual adjustment, and the parameter good application effect predicted.

Description

The method and system of big data system configuration parameter tuning based on deep learning
Technical field
The present invention relates to field of computer technology, more particularly to a kind of big data system configuration parameter based on deep learning The method and system of tuning.
Background technology
In recent years, big data is explored and analysis flourishes in every field.Big data system can be divided into 3 levels：(1) Basal layer：That is hardware resource, is distributed to the execution podium level for supporting calculating task by basic data machined layer；(2) podium level：I.e. Kernel business tier, not only provided an interface for being easily handled data set for application layer, but also can management infrastructure Layer assignment Resource；(3) application layer：That is prediction result output layer, predicts expert decision-making, provides big data analysis result.
Podium level plays the role of forming a connecting link in big data system, and the core of a big data system Point.MapReduce (mapping stipulations) in Hadoop system is exactly a kind of model in podium level.Hadoop is a distribution System infrastructure.User can develop distributed program in the case where not knowing about distributed low-level details.Make full use of collection The power of group carries out high-speed computation and storage.MapReduce is a kind of programming model under Hadoop, for large-scale dataset The concurrent operation of (being more than 1TB).He greatly facilitate programming personnel will not distributed parallel programming in the case of, by oneself Program operate in distributed system.The MapReduce functions of Hadoop, which realize, smashes individual task, and mapping is appointed Business (Map) is sent on multiple nodes, loads stipulations (Reduce) in the form of individual data collection again afterwards in data warehouse.
Configuration parameter setting has a great impact MapReduce working performances.Good configuration parameter makes MapReduce Outstanding work, and configure Map Reduce system performance degradation and cause the main original of thrashing that parameter error is Hadoop Cause.For helpdesk administrator's optimization system performance, it is necessary to adjust configuration parameter processing it is different the characteristics of, different programs and Different input data, to pursue faster work performance.In conventional method, administrator adjusts configuration parameter one by one, Or using linear regression, parameter is configured, extracting parameter feature, showed according to MapReduce transaction capabilities, so as to provide Approximate optimal solution, predicted configuration parameter is to reach more preferable working performance.
However, there are two hang-ups during Admin Administration's Hadoop system：(1) because the behavior of large scale distributed system It is excessively complicated with feature, it is difficult to find appropriate configuration parameter；(2) there are hundreds of parameters, main influence systems performance in system Configuration parameter have tens, configuration arameter optimization is become trouble.In conventional method, manual method or automatic using returning Ginseng is adjusted, extremely complex cumbersome, parameter regulation needs to consume the plenty of time, and income effect is not fine, and system overall work needs Consume for a long time.
The content of the invention
The technical problem to be solved in the present invention is, is automatically adjusted for manual method in the prior art or using returning The efficiency for configuring parameter is low and the defects of effect is poor, there is provided a kind of big data system configuration parameter tuning based on deep learning Method and system.
First aspect present invention, there is provided a kind of method of the big data system configuration parameter tuning based on deep learning, Including neural metwork training step and configuration parameter prediction step；Wherein,
The neural metwork training step comprises the following steps：
Step 1-1, Primary Construction deep neural network, wherein using at least one mapping stipulations parameter as input parameter, Using it is to be predicted go out allocation optimum parameter as output parameter, training sample set is used as using the historical data of big data system；
Step 1-2, to map the stipulations time as the measurement standard of the deep neural network, based on backpropagation thought Parameter learning rule is adjusted the weights of every layer of neuron, until the mapping stipulations time meets time cost requirement；
The configuration parameter prediction step comprises the following steps：
Step 2-1, the initial value of at least one mapping stipulations parameter is set, and reads current test data；
Step 2-2, the initial value of at least one mapping stipulations parameter and current test data are input to via god In the deep neural network obtained through network training step, the configuration ginseng for the big data system based on deep learning is obtained Number.
In the method for the big data system configuration parameter tuning according to the present invention based on deep learning, it is described extremely The quantity of few mapping stipulations parameter is 2~20.
Second aspect of the present invention, there is provided a kind of system of the big data system configuration parameter tuning based on deep learning, Including neural metwork training module and configuration parameter prediction module；Wherein,
The neural metwork training module is used for Primary Construction deep neural network, wherein with least one mapping stipulations ginseng Number is used as input parameter, using it is to be predicted go out allocation optimum parameter as output parameter, using the historical data of big data system as Training sample set；And to map the stipulations time as the measurement standard of the deep neural network, the ginseng based on backpropagation thought Number learning rules are adjusted the weights of every layer of neuron, until the mapping stipulations time meets time cost requirement；
The configuration parameter prediction module is used for the initial value of at least one mapping stipulations parameter of setting and works as Preceding test data is input in the deep neural network obtained via neural metwork training step, obtains being used to be based on deep learning Big data system configuration parameter.
In the system of the big data system configuration parameter tuning according to the present invention based on deep learning, it is described extremely The quantity of few mapping stipulations parameter is 2~20.
Implement the method and system of the big data system configuration parameter tuning based on deep learning of the present invention, have following Beneficial effect：The present invention carries out tuning by deep neural network to the configuration parameter in mapping stipulations frame, avoids manually Adjust, find the problem of optimized parameter, by the study to history parameters, each configuration parameter itself can be obtained to a deeper level Feature, and mutual relation, obtain being most suitable for application layer by the multiple study of depth network, right value update, neural network forecast The parameter configuration of application demand.The present invention not only saves the time of parameter regulation, when the parameter of appropriate system makes the system work Between distribute to compressed and decompressed data, so as to substantially reduce write-in and transmission time, total system work is rapidly completed, More preferable working effect can be reached again.
Brief description of the drawings
Fig. 1 is the method according to the big data system configuration parameter tuning based on deep learning of the preferred embodiment of the present invention Flow chart；
Fig. 2 is the flow diagram of neural metwork training step in the method according to the preferred embodiment of the present invention；
Fig. 3 is the system according to the big data system configuration parameter tuning based on deep learning of the preferred embodiment of the present invention Module frame chart.
Embodiment
To make the purpose, technical scheme and advantage of the embodiment of the present invention clearer, below in conjunction with the embodiment of the present invention In attached drawing, the technical solution in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is The part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, ordinary skill people Member's all other embodiments obtained on the premise of creative work is not made, belong to the scope of protection of the invention.
, will be deep the present invention provides a kind of method that big data network configuration parameters tuning is carried out using deep neural network Degree neural network framework is incorporated into configuration parameter link, not only saves time cost, but also can reach good working effect.This hair Parameter is learnt in bright mapping tasks (Map task) and stipulations task (Reduce task) mainly for big data system With distribute rationally.It is a complicated flow to map stipulations (MapReduce), and first the workflow for mapping stipulations is carried out below Simple to introduce, the key step of stipulations mapping is as follows：
(1) Map ends (mapping end) course of work
(1) each input burst can allow a map task (mapping tasks) to handle, in a distributed manner file system (HDFS) The size (initial value 64M) of a block be a burst.The result of Map outputs can be placed on a circulating memory buffering for the time being Qu Zhong, the size initial value of the buffering area is 100M, by io.sort.mb property controls.When the buffering area soon overflows (just Beginning is arranged to the 80% of buffer size, by io.sort.spill.percent property controls), can be in local file system A spill file is created, the data in the buffering area are write into this file.
(2) before disk is write, data are divided into by thread according to the number of reduce tasks (stipulations task) first Equal number of subregion, that is, a reduce task correspond to the data of a subregion.Then to the data in each subregion It is ranked up, Combiner is set, the result after sequence is subjected to Combia (merging) operations.
(3) when map tasks export last record, many spill files is might have, are at this moment needed these Piece file mergence.It can constantly be ranked up during merging and be operated with Combia (merging).
(4) data in subregion are transferred to corresponding reduce tasks.
(2) Reduce ends (stipulations end) course of work
(1) Reduce can receive the data that different map tasks transmit, and the data that each map is transmitted are ordered 's.If the data volume that reduce terminations are received is fairly small, be stored directly in memory (buffer size, by Mapred.job.shuffle.input.buffer.percent property controls, the percentage of the heap space of expression used as said purpose Than).If data volume has exceeded the certain proportion of the buffer size (by mapred.job.shuffle.merge.percent Determine), then overflow and write in disk after merging to data.
(2) the abbreviation program that application layer defines, final output data are performed.Compress on demand, write final output To HDFS.
Referring to Fig. 1, it is the big data system configuration parameter tune based on deep learning according to the preferred embodiment of the present invention Excellent method flow diagram.As shown in Figure 1, the big data system configuration parameter tuning based on deep learning that the embodiment provides Method mainly includes neural metwork training step and configuration parameter prediction step：
First, neural metwork training step, construction depth neutral net, with administrator are performed into S102 in step S101 The history working status of offer is training set, using the allocation optimum parameter predicted as output.And with (mapping stipulations) The time cost of MapReduce is the final measurement standard of network structure, and continuous feedback adjustment structure, obtains ultimate depth nerve net Network structure.It is specific as follows：
Step S101, Primary Construction deep neural network, wherein using at least one mapping stipulations parameter as input parameter, Using it is to be predicted go out allocation optimum parameter as output parameter, training sample set is used as using the historical data of big data system.This is big The historical data of data system is specially the history working status that administrator provides.Preferably, at least one mapping stipulations Parameter can choose one or more from following important parameter form 1.In a particular application, according to different situations, from system Obtained at administrator and input/output list is added to 20 parameters of systematic influence maximum, Selecting All Parameters are as shown in table 1 below.This is extremely The quantity of few mapping stipulations parameter is preferably 2~20.
1 important parameter table of form
Step S102, to map the stipulations time as the measurement standard of the deep neural network, based on backpropagation thought Parameter learning rule the weights of every layer of neuron are adjusted, until mapping the stipulations time meet time cost requirement.Should Using the time cost of MapReduce as the final measurement standard of network structure in step, continuous feedback adjustment structure, obtains final depth Spend the structure of neutral net.
Then, configuration parameter prediction step is performed into S104 in step S103, it is pre- using obtained deep neural network Measure the configuration parameter for making working effect optimal.It is specific as follows：
Step S103, the initial value of setting at least one mapping stipulations parameter, and read current test data.
Step S104, the initial value of at least one mapping stipulations parameter and current test data are input to via god In the deep neural network obtained through network training step, the configuration ginseng for the big data system based on deep learning is obtained Number.
It can be seen from the above that the present invention introduces depth after initialization mapping (Map) task and stipulations (Reduce) task parameters Neutral net, training set source are historic task daily records, history parameters are learnt, semi-supervised learning, are gone through by known History working status, to the feedback of working performance, draws the parameter inside deep neural network, so that predict simultaneously optimization, for Different programs and different input datas are attained by the configuration parameter of optimal work performance.
Fig. 2 is please referred to, is that the flow of neural metwork training step in the method according to the preferred embodiment of the present invention is shown It is intended to.As shown in Fig. 2, the neural metwork training step includes：
First, in step s 201, flow starts；
Then, in step S202, Primary Construction deep neural network.The deep neural network is to utilize backpropagation Common deep-neural-network.Specifically, built in the step to map five layer depth nerve nets of the stipulations parameter as input parameter Network, using it is to be predicted go out allocation optimum parameter as output parameter, foregoing five layer network includes input layer, output layer and three respectively Hidden layer.
Then, in step S203, depth nerve net is inputted using the historical data of big data system as training sample set Network.Training sample x is inputted, hidden layer output is xl=f (ul), wherein ul=Wlxl-1+bl, wherein, function f represents output activation letter Number, W represent weights, and b represents bias term, and l represents the 1st layer.Because parameter cannot infinitely expand during map and reduce, have A certain range, it is therefore desirable to which fixed b is parameter upper limit.
Then, in step S204, judge to map whether the stipulations time meets time cost requirement.Use square error generation Valency function weighs error, it is assumed that output parameter classification is c, and training sample concentrates N number of training sample altogether, then maps the stipulations time With the error E between stipulated time cost tNFor：Wherein,For n-th training sample The kth dimension of target output,Tieed up for the kth of the corresponding reality output of n-th of sample, c=20.Calculate the mistake between each layer network Difference, when error is less than predetermined threshold value going to step S206 preserves the deep neural network, otherwise goes to step S205 and adjusts every layer of god Weights through member.
In step S205, the weights of every layer of neuron are adjusted.Specifically, the sensitivity δ of neuron is passed through in the step To be zoomed in and out to the weights W of every layer of neuron, finally obtain be E minimums weights：
Wherein,And l layers of sensitivity：δl=(Wl+1)Tδl+1οf'(ul)；The neuron of output layer Sensitivity be：δL=f'(uL)·(yn-tn), wherein L represents total number of plies, ynFor the reality output of n-th of neuron, tnFor The target output of n neuron.
In step S206, the deep neural network is preserved；
Finally, in step S207, the flow of the neural metwork training step terminates.
Centre can be understood as output, stipulations mapping time using stipulations mapping time and configuration parameter in the present invention During output, configuration parameter, which is that we are most important, to be recorded and the output that uses, is compared according to output time and ideal time Weights are adjusted after error, the output of time is not only changed after weights are adjusted, also change the output of configuration parameter, therefore can Configuration parameter during obtaining time optimal.
Referring to Fig. 3, it is the big data system configuration parameter tune based on deep learning according to the preferred embodiment of the present invention The module frame chart of excellent system.As shown in figure 3, the big data system configuration parameter tune based on deep learning that the embodiment provides Excellent system 300 includes neural metwork training module 301 and configuration parameter prediction module 302.
Wherein, neural metwork training module 301 is used for Primary Construction deep neural network, wherein being advised with least one mapping About parameter as input parameter, using it is to be predicted go out allocation optimum parameter as output parameter, with the historical data of big data system As training sample set；And to map the stipulations time as the measurement standard of the deep neural network, based on backpropagation thought Parameter learning rule the weights of every layer of neuron are adjusted, until mapping the stipulations time meet time cost requirement.It is excellent Selection of land, at least one mapping stipulations parameter can be chosen one or more from weight form.At least one mapping stipulations The quantity of parameter is preferably 2~20.
Specifically, which builds to map five layer depths god of the stipulations parameter as input parameter Through network, using it is to be predicted go out allocation optimum parameter as output parameter, five layer network include respectively input layer, output layer and Three hidden layers, input training sample x, hidden layer output is xl=f (ul), wherein ul=Wlxl-1+bl, function f, which is represented, exports activation letter Number, W represent weights, and b represents bias term, and l represents the 1st layer.
The neural metwork training module 301 also weighs error using square error cost function, it is assumed that output parameter class Not Wei c, training sample concentrate altogether N number of training sample, then map stipulations time and the stipulated time cost t between error ENFor：Wherein,Kth for the target output of n-th of training sample is tieed up,For n-th of sample The kth dimension of corresponding reality output.
The error between each layer network is then calculated, the deep neural network is preserved when error is less than predetermined threshold value, it is no Then the weights W of every layer of neuron is zoomed in and out by the sensitivity δ of neuron：
Wherein,And l layers of sensitivity：δl=(Wl+1)Tδl+1οf'(ul)；The neuron of output layer Sensitivity be：δL=f'(uL)·(yn-tn), wherein L represents total number of plies, ynFor the reality output of n-th of neuron, tnFor The target output of n neuron.
Configuration parameter prediction module 302 is connected with neural metwork training module 301, for by the described at least one of setting The initial value and current test data for mapping stipulations parameter are input to the depth nerve net obtained via neural metwork training step In network, the configuration parameter for the big data system based on deep learning is obtained.
In conclusion using the present invention using deep neural network to the configuration in mapping stipulations (MapReduce) frame Parameter carries out tuning, avoids manual adjustment, finds the problem of optimized parameter, can be deeper by the study to history parameters Obtain each configuration parameter own characteristic level, and mutual relation, multiple by depth network learn, right value update, Neural network forecast obtains being most suitable for the parameter configuration of application layer applications demand.The present invention not only saves the time of parameter regulation, closes The parameter of suitable system makes System production time distribute to compressed and decompressed data, so as to substantially reduce write-in and transmission time, makes Total system work can be rapidly completed and reach more preferable working effect.At the same time for different basal layer input datas and Application layer propose application requirement, can autonomous learning, there is stronger adaptability.
Finally it should be noted that：The above embodiments are merely illustrative of the technical solutions of the present invention, rather than its limitations；Although The present invention is described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that：It still may be used To modify to the technical solution described in foregoing embodiments, or equivalent substitution is carried out to which part technical characteristic； And these modification or replace, do not make appropriate technical solution essence depart from various embodiments of the present invention technical solution spirit and Scope.

Claims (4)

  1. A kind of 1. method of the big data system configuration parameter tuning based on deep learning, it is characterised in that including neutral net Training step and configuration parameter prediction step；Wherein,
    The neural metwork training step comprises the following steps：
    Step 1-1, Primary Construction deep neural network, wherein using at least one mapping stipulations parameter as input parameter, to treat Allocation optimum parameter is predicted as output parameter, training sample set is used as using the historical data of big data system；
    Step 1-2, to map the stipulations time as the measurement standard of the deep neural network, the parameter based on backpropagation thought Learning rules are adjusted the weights of every layer of neuron, until the mapping stipulations time meets time cost requirement；
    The configuration parameter prediction step comprises the following steps：
    Step 2-1, the initial value of at least one mapping stipulations parameter is set, and reads current test data；
    Step 2-2, the initial value of at least one mapping stipulations parameter and current test data are input to via nerve net In the deep neural network that network training step obtains, the configuration parameter for the big data system based on deep learning is obtained；
    In the step 1-1：
    Structure to map five layer depth neutral nets of the stipulations parameter as input parameter, using it is to be predicted go out allocation optimum parameter as Output parameter, five layer network include input layer, output layer and three hidden layers respectively, input training sample x, and hidden layer output is Y=xl=f (ul), wherein ul=Wlxl-1+bl, function f, which is represented, exports activation primitive, and W represents weights, and b represents bias term, and l is represented Thel Layer；
    In the step 1-2：
    Error is weighed using square error cost function, it is assumed that output parameter classification is c, and training sample concentrates N number of training altogether Sample, then map the error E between stipulations time and the stipulated time cost tNFor：Its In,Kth for the target output of n-th of training sample is tieed up,Tieed up for the kth of the corresponding reality output of n-th of sample；
    The error between each layer network is calculated, the deep neural network is preserved when error is less than predetermined threshold value, otherwise passes through god The weights W of every layer of neuron is zoomed in and out through first sensitivity δ：
    Wherein,And l layers of sensitivity：The spirit of the neuron of output layer Sensitivity is：Wherein L represents total number of plies, ynFor the reality output of n-th of neuron, tnFor n-th The target output of neuron, symbolRepresent convolution.
  2. 2. the method for the big data system configuration parameter tuning according to claim 1 based on deep learning, its feature exist In the quantity of at least one mapping stipulations parameter is 2~20.
  3. 3. a kind of system of the big data system configuration parameter tuning based on deep learning, it is characterised in that including neutral net Training module and configuration parameter prediction module；Wherein,
    The neural metwork training module is used for Primary Construction deep neural network, wherein being made with least one mapping stipulations parameter For input parameter, using it is to be predicted go out allocation optimum parameter as output parameter, training is used as using the historical data of big data system Sample set；And to map the stipulations time as the measurement standard of the deep neural network, the parametrics based on backpropagation thought Practise rule to be adjusted the weights of every layer of neuron, until the mapping stipulations time meets time cost requirement；
    The configuration parameter prediction module is used for the initial value of at least one mapping stipulations parameter of setting and currently surveys Examination data are input in the deep neural network obtained via neural metwork training step, are obtained for based on the big of deep learning The configuration parameter of data system；
    Wherein, the neural metwork training module is used to build to map five layer depth nerve nets of the stipulations parameter as input parameter Network, using it is to be predicted go out allocation optimum parameter as output parameter, five layer network includes input layer, output layer and three respectively Hidden layer, inputs training sample x, hidden layer output is xl=f (ul), wherein ul=Wlxl-1+bl, wherein, function f represents output activation Function, W represent weights, and b represents bias term, and l represents l layers；
    The neural metwork training module weighs error using square error cost function, it is assumed that output parameter classification is c, instruction Practice common N number of training sample in sample set, then map the error E between stipulations time and the stipulated time cost tNFor：Wherein,Kth for the target output of n-th of training sample is tieed up,For n-th of sample The kth dimension of corresponding reality output；
    The error between each layer network is calculated, the deep neural network is preserved when error is less than predetermined threshold value, otherwise passes through god The weights W of every layer of neuron is zoomed in and out through first sensitivity δ：
    Wherein,And l layers of sensitivity：The spirit of the neuron of output layer Sensitivity is：Wherein L represents total number of plies, ynFor the reality output of n-th of neuron, tnFor n-th The target output of neuron, symbolRepresent convolution.
  4. 4. the system of the big data system configuration parameter tuning according to claim 3 based on deep learning, its feature exist In the quantity of at least one mapping stipulations parameter is 2~20.
CN201710361578.3A 2017-05-22 2017-05-22 The method and system of big data system configuration parameter tuning based on deep learning Active CN107229693B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710361578.3A CN107229693B (en) 2017-05-22 2017-05-22 The method and system of big data system configuration parameter tuning based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710361578.3A CN107229693B (en) 2017-05-22 2017-05-22 The method and system of big data system configuration parameter tuning based on deep learning

Publications (2)

Publication Number Publication Date
CN107229693A CN107229693A (en) 2017-10-03
CN107229693B true CN107229693B (en) 2018-05-01

Family

ID=59933231

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710361578.3A Active CN107229693B (en) 2017-05-22 2017-05-22 The method and system of big data system configuration parameter tuning based on deep learning

Country Status (1)

Country Link
CN (1) CN107229693B (en)

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109992404B (en) * 2017-12-31 2022-06-10 中国移动通信集团湖北有限公司 Cluster computing resource scheduling method, device, equipment and medium
CN108363478B (en) * 2018-01-09 2019-07-12 北京大学 For wearable device deep learning application model load sharing system and method
CN110427356B (en) * 2018-04-26 2021-08-13 中移(苏州)软件技术有限公司 Parameter configuration method and equipment
CN108764568B (en) * 2018-05-28 2020-10-23 哈尔滨工业大学 Data prediction model tuning method and device based on LSTM network
CN109041195A (en) * 2018-07-19 2018-12-18 浙江工业大学 A kind of energy-collecting type wireless relay network througput maximization approach based on semi-supervised learning
CN108990141B (en) * 2018-07-19 2021-08-03 浙江工业大学 Energy-collecting wireless relay network throughput maximization method based on deep multi-network learning
CN109445935B (en) * 2018-10-10 2021-08-10 杭州电子科技大学 Self-adaptive configuration method of high-performance big data analysis system in cloud computing environment
CN109815537B (en) * 2018-12-19 2020-10-27 清华大学 High-flux material simulation calculation optimization method based on time prediction
CN109739950B (en) * 2018-12-25 2020-03-31 中国政法大学 Method and device for screening applicable legal provision
CN110134697B (en) * 2019-05-22 2024-01-16 南京大学 Method, device and system for automatically adjusting parameters of storage engine for key value
TWI752614B (en) * 2020-09-03 2022-01-11 國立陽明交通大學 Multiple telecommunication endpoints system and testing method thereof based on ai decision
CN113254472B (en) * 2021-06-17 2021-11-16 浙江大华技术股份有限公司 Parameter configuration method, device, equipment and readable storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104504460A (en) * 2014-12-09 2015-04-08 北京嘀嘀无限科技发展有限公司 Method and device for predicating user loss of car calling platform
CN106022521A (en) * 2016-05-19 2016-10-12 四川大学 Hadoop framework-based short-term load prediction method for distributed BP neural network
CN106202431A (en) * 2016-07-13 2016-12-07 华中科技大学 A kind of Hadoop parameter automated tuning method and system based on machine learning
CN106648654A (en) * 2016-12-20 2017-05-10 深圳先进技术研究院 Data sensing-based Spark configuration parameter automatic optimization method

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9886310B2 (en) * 2014-02-10 2018-02-06 International Business Machines Corporation Dynamic resource allocation in MapReduce

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104504460A (en) * 2014-12-09 2015-04-08 北京嘀嘀无限科技发展有限公司 Method and device for predicating user loss of car calling platform
CN106022521A (en) * 2016-05-19 2016-10-12 四川大学 Hadoop framework-based short-term load prediction method for distributed BP neural network
CN106202431A (en) * 2016-07-13 2016-12-07 华中科技大学 A kind of Hadoop parameter automated tuning method and system based on machine learning
CN106648654A (en) * 2016-12-20 2017-05-10 深圳先进技术研究院 Data sensing-based Spark configuration parameter automatic optimization method

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
BP神经网络的优化与研究;吕琼帅;《中国优秀硕士学位论文全文数据库信息科技辑》;20120415(第4期);第I140-69页 *

Also Published As

Publication number Publication date
CN107229693A (en) 2017-10-03

Similar Documents

Publication Publication Date Title
CN107229693B (en) The method and system of big data system configuration parameter tuning based on deep learning
CN103631657B (en) A kind of method for scheduling task based on MapReduce
CN101493864B (en) Method and system for analyzing performance of a wind farm
DE602004011890T2 (en) Method for redistributing objects to arithmetic units
CN111797928A (en) Method and system for generating combined features of machine learning samples
CN104750780B (en) A kind of Hadoop configuration parameter optimization methods based on statistical analysis
Jablonsky MS Excel based software support tools for decision problems with multiple criteria
Baloui Jamkhaneh et al. Inspection error and its effects on single sampling plans with fuzzy parameters
CN107734052A (en) The load balancing container dispatching method that facing assembly relies on
CN109165081B (en) Web application self-adaptive resource allocation method based on machine learning
CN106371924B (en) A kind of method for scheduling task minimizing MapReduce cluster energy consumption
CN110489749B (en) Business process optimization method of intelligent office automation system
CN106534302A (en) Multi-task demand service combination method and system
CN113821332B (en) Method, device, equipment and medium for optimizing efficiency of automatic machine learning system
CN111783893A (en) Method and system for generating combined features of machine learning samples
CN105373432A (en) Cloud computing resource scheduling method based on virtual resource state prediction
CN105808582A (en) Parallel generation method and device of decision tree on the basis of layered strategy
CN108418708A (en) A kind of cloudy management scheduling system for supporting FT and X86 mixed architectures
CN109409746A (en) A kind of production scheduling method and device
Wang* et al. A hybrid knowledge discovery model using decision tree and neural network for selecting dispatching rules of a semiconductor final testing factory
CN106503271A (en) The intelligent shop site selection system of subspace Skyline inquiry under mobile Internet and cloud computing environment
CN105847358A (en) Method for realizing big data node distribution in cloud computing environment and system thereof
CN104008011A (en) Method for balancing resource load of entity machine in cluster environment
JP2020107185A (en) Image recognition device, image recognition method and program
CN108376180A (en) Influence the key parameter lookup method and device of big data system performance

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant