CN110222923A - Dynamically configurable big data analysis system - Google Patents
Dynamically configurable big data analysis system Download PDFInfo
- Publication number
- CN110222923A CN110222923A CN201910332409.6A CN201910332409A CN110222923A CN 110222923 A CN110222923 A CN 110222923A CN 201910332409 A CN201910332409 A CN 201910332409A CN 110222923 A CN110222923 A CN 110222923A
- Authority
- CN
- China
- Prior art keywords
- real
- time
- data
- analysis
- line
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007405 data analysis Methods 0.000 title claims abstract description 47
- 238000004458 analytical method Methods 0.000 claims abstract description 73
- 238000004364 calculation method Methods 0.000 claims abstract description 60
- 238000007726 management method Methods 0.000 claims abstract description 33
- 238000005457 optimization Methods 0.000 claims abstract description 25
- 238000012800 visualization Methods 0.000 claims abstract description 15
- 238000012545 processing Methods 0.000 claims description 19
- 230000004044 response Effects 0.000 claims description 11
- 238000010801 machine learning Methods 0.000 claims description 7
- 230000002688 persistence Effects 0.000 claims description 6
- 238000001514 detection method Methods 0.000 claims description 4
- 238000007619 statistical method Methods 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000013497 data interchange Methods 0.000 claims description 3
- 238000000034 method Methods 0.000 abstract description 27
- 230000008569 process Effects 0.000 abstract description 21
- 238000013523 data management Methods 0.000 abstract description 2
- 238000012423 maintenance Methods 0.000 description 14
- 238000013500 data storage Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 6
- 238000010223 real-time analysis Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000012217 deletion Methods 0.000 description 4
- 230000037430 deletion Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000004913 activation Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 241000251730 Chondrichthyes Species 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000012954 risk control Methods 0.000 description 2
- TVZRAEYQIKYCPH-UHFFFAOYSA-N 3-(trimethylsilyl)propane-1-sulfonic acid Chemical compound C[Si](C)(C)CCCS(O)(=O)=O TVZRAEYQIKYCPH-UHFFFAOYSA-N 0.000 description 1
- 241000208340 Araliaceae Species 0.000 description 1
- 241001269238 Data Species 0.000 description 1
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 1
- 235000003140 Panax quinquefolius Nutrition 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 230000003213 activating effect Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000000712 assembly Effects 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000001276 controlling effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005315 distribution function Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 235000008434 ginseng Nutrition 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 239000007858 starting material Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000002195 synergetic effect Effects 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a kind of dynamically configurable big data analysis systems, the system comprises the four modules such as real-time data memory management module, real-time streams analytical calculation module, off-line analysis module, visualization model, at least one is devised in each module can carry out the component of dynamic configuration management, as data management configuration component, real-time streams analytical calculation configuration component, off-line analysis calculate configuration component, dynamic configuration component.The invention also provides a kind of Dynamic Configurations of big data analysis system, devise the data structure and message structure of each comprising modules, pass through the dynamic configuration of the status information drive system of the warning data structure in dynamic configuration manager, propose the calculation method and Dynamic Configuration of warning redundancy, through the above way, the present invention can make system run on an efficient big data analysis calculating level, efficiently solve the optimization process of big data analysis platform management.
Description
This case is so that application No. is 201510577285.X, the applying date is on September 11st, 2015, entitled " dynamically to match
The big data analysis system and method set " patent application be female case divisional application.
Technical field
The present invention relates to big data analysis application fields, more particularly, to a kind of dynamically configurable big data analysis system
System.
Background technique
Present Business Intelligence system, DSS etc. increasingly require support large data sets at analysis, due to big
The data volume of data analytical calculation is big, process is complicated, the processing time is long, thus big data analysis and application are also faced with one kind
New challenge: system must have high reliability, it is desirable that software systems have adaptivity to variation, these systems needs have
The ability of configuration is updated under the premise of not interrupting system service, how fault-tolerant management problem is handled in the case where updating failure
It is abnormal, so that system is kept the operation of normal table.I.e. Dynamic Reconfiguration is to realize big data platform software adaptive reliability
A kind of important means.
The big data parallel processing frame Hadoop of early stage is limited to Single Point of Faliure and calculating mode is relatively single,
Hadoop2.0 introduces this universal resource management system of YARN, improves the resource utilization of system reliability and entire cluster,
Becoming can run including a variety of big data processing frames such as real-time streams processing frame Storm, Spark and programming mode,
But the fault-tolerant ability of big data analysis application system is improved, the reliability for further having had system is still a problem.
The big data engine Spark technology currently just risen extensively is initially tested by the AMPLab of UC Berkeley university
Room exploitation, is by the open source projects of Apache fund management now.The target of Spark be meet it is most according to data processing with
And the application excavated, make data analysis program operation faster, the fault-tolerance mould that preferably the general support memory of one kind calculates
Type.Spark introduces elasticity distribution formula data set (Resilient Distributed DataSets) RDD model, with abundant
Computational efficiency is promoted using memory source.From other big datas processing frame unlike, Spark can Shark,
It is efficiently handled from ETL to SQL using an engine to engineering on the basis of MLlib, GraphX and Spark Streaming
Practise the processing for arriving flow data again.Using Spark plus Spark Streaming (or Shark, B1inkDB) in real time and batch at
Reason；Add MLlib for stream process and machine learning using Spark Streaming；Figure flowing water is used for using Spark plus GraphX
Line etc..But this new real-time stream calculation frame has been although real-time performance and error resilience performance have obtained big improvement, system
High reliability and high availability are still a challenge.
As distributed system scale is more and more huger in big data platform, behavior becomes increasingly complex, occur in system
Various failures also exponentially increase, and bring harm and loss very serious to industry, government department, system once occurs
Shutdown event, it will bring massive losses and puzzlement, therefore these big data analysis systems need to have and do not interrupting system clothes
There is the ability automatically configured under the premise of business, to improve the reliability of system, enhance systematic risk controlling ability, it is flat to improve software
The overall operation efficiency of platform.For the problems in the relevant technologies, currently no effective solution has been proposed.
Summary of the invention
Match the technical problems to be solved by the present invention are: providing the dynamic optimization for calculating the runtime for big data analysis
It sets, to improve the reliability of system, enhances risk control ability.
In order to solve the above-mentioned technical problem, the technical solution adopted by the present invention are as follows: a kind of dynamically configurable big number is provided
According to analysis system, comprising:
Real-time data memory management module, for obtaining real-time streaming data, and dynamic configuration in Distributed Services cluster
Associated control parameters, and store；
Real-time streams analytical calculation module obtains real-time calculated result, and to real-time analysis for statisticalling analyze real time data
Algorithmic load carries out task adjustment；
Off-line analysis module obtains off-line calculation as a result, and negative to off-line analysis algorithm for statisticalling analyze off-line data
It is loaded into the adjustment of row task；
Visualization model, for being visualized to real-time calculated result and off-line calculation result, and in setting
Dynamic chart is provided in time delay range, shows cluster service operation state and response condition in time, is carried out to more than threshold data
Alert process.
To solve the above problems, the present invention also provides a kind of Dynamic Configuration of big data analysis system, including it is as follows
Step:
S1: preset time window by dynamic configuration manager predetermined warning data structure, and initializes；
S2: early warning redundancy lower bound and the upper bound of object instance are set according to the task type of object instance in node
Experience initial value and a parameter adjusting step constant；
S3: the early warning redundancy angle value of computing object example；
S4: it determines that the early warning redundancy angle value is located between lower bound and the experience initial value in the upper bound, and generates random number；
S5: according to the experience initial value of step-length, random number, the upper bound and lower bound, the upper dividing value of optimization and optimization lower bound are calculated
Value；
S6: determine that the early warning redundancy angle value is located at optimization floor value and optimizes between upper dividing value；
S7: the information warning list in preset time window, in the management of poll dynamic configuration；
S8: for the information warning list of node state, node state is modified, to realize the Dynamic Maintenance of node.
The beneficial effects of the present invention are: it is different from the prior art, the present invention is realized by the coordinated of above-mentioned module
The configuration optimization of system performance, while by calculating warning redundancy, so that system can run on an efficient big data
On analytical calculation platform, the reliability of system, but also enhancing risk control ability had not only been improved.
Detailed description of the invention
Fig. 1 is the structural schematic diagram of present system；
Fig. 2 is the dynamic dimension of types of objects example redundancy when each node of system is in NORMAL state in the method for the present invention
The flow diagram of shield；
Fig. 3 is the dynamic configuration maintenance process schematic diagram based on information warning list interior joint state in the method for the present invention.
Specific embodiment
To explain the technical content, the achieved purpose and the effect of the present invention in detail, below in conjunction with embodiment and cooperate attached
Figure is explained.
The most critical design of the present invention is: the synergistic effect of the modules by system, realizes matching for system performance
Optimization is set, and then obtains efficient big data analysis computing platform.
Fig. 1 is please referred to, the embodiment of the present invention provides a kind of dynamically configurable big data analysis system, comprising:
Real-time data memory management module, for obtaining real-time streaming data, and dynamic configuration in Distributed Services cluster
Associated control parameters, and store；
Real-time streams analytical calculation module obtains real-time calculated result, and to real-time analysis for statisticalling analyze real time data
Algorithmic load carries out task adjustment；
Off-line analysis module obtains off-line calculation as a result, and negative to off-line analysis algorithm for statisticalling analyze off-line data
It is loaded into the adjustment of row task；
Visualization model, for being visualized to real-time calculated result and off-line calculation result, and in setting
Dynamic chart is provided in time delay range, shows cluster service operation state and response condition in time, is carried out to more than threshold data
Alert process.
Wherein, the real-time data memory management module includes:
Real-time streaming data securing component, for obtaining the real-time streaming data in Distributed Services cluster, and be formatted,
Filtering is collected, and during collection, completes the batch work of flow data；
Real-time storage component, for the data interchange format asynchronous transmission after formatting to HDFS, by data batch into
Row storage.
Storage management configuration component is used for real-time data memory management module dynamic configuration associated control parameters.
Wherein, the real-time streams analytical calculation module includes:
Real-time streams processing component obtains real-time calculated result for obtaining data from HDFS to be analyzed in real time；And
By real-time calculated result persistence, it is sent to visualization model, and storage is into HDFS；
Real-time data analysis component, for real time data intelligence for statistical analysis and based on machine learning point
Analysis, and real-time parser is loaded and carries out task schedule, realize load balancing；
Real-time streams analytical calculation configuration component is used for real-time streams analytical calculation module dynamic configuration associated control parameters.
Wherein, the off-line analysis module includes:
Off-line data processing component obtains off-line calculation for obtaining off-line data from HDFS to carry out off-line analysis
As a result, and by off-line calculation result persistence, be sent to visualization model, and storage is into HDFS and NoSQL.
Off line data analysis component, for off-line data intelligence for statistical analysis and based on machine learning point
Analysis, and task schedule is carried out to off-line analysis algorithmic load, realize load balancing；
Off line data analysis calculates configuration component, is used for off-line analysis module dynamic configuration associated control parameters.
Wherein, the visualization model includes:
Dynamic configuration component realizes the configuration optimization of system performance for cooperateing with above-mentioned module；
Analysis view component in real time, for visualizing real-time calculated result, including System, real-time statistic analysis
And the data of intelligent predicting.
Off-line analysis view component, for visualizing off-line calculation as a result, including the summarizing of theme message, state point
Analysis and the displaying of intelligent predicting result and the statistics of location service request summarize.
Dynamic configuration view component for showing configuration data, and is associated with and shows real-time calculated result and off-line calculation
As a result detection accuracy.
A kind of Dynamic Configuration of big data analysis system, includes the following steps:
S1: preset time window by dynamic configuration manager predetermined warning data structure, and initializes；
S2: early warning redundancy lower bound and the upper bound of object instance are set according to the task type of object instance in node
Experience initial value and a parameter adjusting step constant；
S3: the early warning redundancy angle value of computing object example；
S4: it determines that the early warning redundancy angle value is located between lower bound and the experience initial value in the upper bound, and generates random number；
S5: according to the experience initial value of step-length, random number, the upper bound and lower bound, the upper dividing value of optimization and optimization lower bound are calculated
Value；
S6: determine that the early warning redundancy angle value is located at optimization floor value and optimizes between upper dividing value；
S7: the information warning list in preset time window, in the management of poll dynamic configuration；
S8: for the information warning list of node state, node state is modified, to realize the Dynamic Maintenance of node.
Wherein step S4 specifically:
S41: judge whether the early warning redundancy angle value is more than or equal to lower bound experience initial value；
If so, executing S42: updating the object instance；
If it is not, then executing S411: judging whether the object instance is in ready state；
If so, executing S412: activating the object instance, and return step S41；
If it is not, then executing S413: creation task instances, and return step S41；
Further include S43 wherein after step S42: judging whether the early warning redundancy angle value is less than or equal to upper bound experience
Initial value；
If so, executing S44: updating the object instance, and generate random number；
If it is not, then executing S431: judging whether the object instance is in ready state or heavy condition；
If so, executing S432: deleting the object instance, and return step S43；
If it is not, then executing S433: the parameter of regulating object example, and return step S43.
Wherein, after step S412, further includes:
S414: judge whether to activate successfully；
If so, return step S41；
Conversely, then executing S415: setting the node state of warning nodal information list as heavy duty；
Further include S416 after step S413: judging whether to create successfully；
If so, return step S41；
Conversely, then executing S415.
Wherein, further include S434 after step S432: judging whether to delete successfully；
If so, return step S43；
Conversely, then executing S415；
Further include S436 after step S433: judging whether to adjust successfully；
If so, return step S43；
Conversely, then executing S415.
Wherein, step S5 specifically:
S51: the optimization floor value is calculated: optimization floor value=lower bound experience initial value+step-length * random number；
S52: dividing value in the optimization is calculated: dividing value=upper bound experience initial value-step-length * random number in optimization.
Understand above-mentioned technical proposal in order to facilitate understanding, present invention combination FIG. 1 to FIG. 3 provide a specific embodiment into
Row illustrates.
Firstly, it is necessary to explanation, in big data analysis calculating, large-scale distributed calculating service needs to carry out system
Optimization, the error resilience performance for improving system are only inadequate to be ensured of from systems development process.Because big data analysis calculates system
The parameter of system performance involved in system is various, it is difficult to regulate and control, this is a very difficult job.For asking for this challenge
Topic, the present invention propose a kind of dynamically configurable big data analysis System and method for, which includes real-time data memory management
The four modules such as module, real-time streams analytical calculation module, off-line analysis module, visualization model design in each module
One can carry out the component of dynamic configuration management, such as data management configuration component, real-time streams analytical calculation configuration component, offline
Analytical calculation configuration component, dynamic configuration component.Wherein, dynamic configuration component is the core of system dynamic configuration management, same
When with each module cooperative realize system performance configuration optimization.And system can using current newest big data platform technology into
Row is realized, such as Hadoop, Kafka, Spark Streaming, Hive, and current system provided by the invention passes through product line portion
Administration's detection, operation conditions are good.
The overall structure of dynamically configurable big data analysis system proposed by the present invention, as shown in Figure 1.This system uses
Modularized design mainly includes real-time data memory management module, real-time streams analytical calculation module, off-line analysis module, visual
Change the four modules such as module.The major function of modules is as follows:
(1) real-time data memory management module
The module is made of three components, comprising: real-time streaming data securing component, real-time streaming data storage assembly, in real time
Storage management configuration component.
Real-time streaming data securing component be mainly responsible for the real-time streaming data in existing large-scale distributed service cluster into
Row obtains, and completes to format, filter and collect by the component, during collection, completes the batch work of flow data
(batching module)。
Data are criticized batch by the data interchange format JSON asynchronous transmission after formatting to HDFS by real-time storage component
It is stored, while also data being sent in batch queue (batch queue) by the component and are supplied to real-time computation module.
Storage management configuration component is mainly responsible for this module dynamic configuration associated control parameters.
(2) real-time streams analytical calculation module
The module is made of three components, comprising: real-time streams processing component, real-time data analysis component, real-time streams analysis
Calculate configuration component.
Real-time streams processing component is mainly that real-time analytic unit provides service.On the one hand, it is responsible for pulling from HDFS offline
The related data of calculated result provides analytic unit and does analysis reference, this is to belong to the precomputation analyzed in real time；On the other hand, will
Result persistence is analyzed, upper layer visualization had both been supplied to and data source is provided, and also to have stored data into HDFS.
Real-time data analysis component be mainly responsible for classical statistics analysis and the intellectual analysis based on machine learning, and to point
It analyses algorithmic load and carries out task schedule, realize load balancing.
Real-time streams analytical calculation configuration component is mainly responsible for this module dynamic configuration associated control parameters.
(3) off-line analysis module
The module is made of three components, comprising: off-line data processing component, off line data analysis component, off-line data
Analytical calculation configuration component.
Off-line data processing component is mainly that off-line analysis component provides service.On the one hand, be responsible for pulled from HDFS from
Related data carries out precomputation for off-line analysis；On the other hand, by off line data analysis result persistence, both it is supplied to upper layer
Visualization provides data source, also stores calculation result data to HDFS and NoSQL.
Off line data analysis component is mainly responsible for the global statistics analysis of classics and the overall situation intelligence based on machine learning
Analysis, and task schedule is carried out to off-line analysis algorithmic load, realize load balancing.
Off line data analysis calculates configuration component and is mainly responsible for this module dynamic configuration associated control parameters.
It is mainly to carry out offline classical statistics to data in distributed type assemblies to analyze for off-line calculation analysis module.
Off line data analysis task is scheduled by setting time window, is generated report according to calculated result, is opened for service
Hair and operation maintenance personnel carry out resource allocation and later period optimization reference to service.
(4) visualization model
The module is made of four components, comprising: dynamic configuration component, analyzes view at dynamic configuration view component in real time
Component, off-line analysis view component.
The module mainly ties calculating caused by real-time streams analytical calculation module and off line data analysis computing module
Fruit is visualized, and permission provides dynamic chart in the time delay range of setting, shows cluster service operation state in time
And response condition, alert process is carried out to more than threshold data.
The data of this modules exhibit are divided into three classes:
A, analysis view component shows analysis data in real time in real time
The part mainly includes the data of each analysis result System and real-time statistic analysis and intelligent predicting.
B, off-line analysis view component shows off-line analysis data
The part mainly includes the displaying of various theme messages summarized with state analysis and intelligent predicting result, including institute
On ground, the statistics of service request summarizes.
C, dynamic configuration view component shows configuration data, and is associated with the detection accuracy for showing analysis result.
In order to adapt to current large-scale distributed service system service state effective analysis, promoted analyze in real time i.e.
When analyze benefit, the analysis task to note abnormalities in time is generally required, so that the availability requirement of this real-time analyzer mentions
Height, the present invention construct redundant configuration technology to this real-time streams computing system, realize the dynamic configuration of real-time streams computing system,
The performance that real-time big data analysis system is improved under the premise of guaranteeing system availability, promotes the timeliness of instant analysis.
For convenient for discuss, wherein being made the following instructions to system of the present invention:
(1) system has N number of node, provides the analysis of M class data or statistics calculating task altogether；
It (2) is loose couplings between the component of one generic task of system completion of the present invention, i.e., system can be between node
Reliable asynchronous communication mechanism is provided, at the same it is asynchronous between communication-cost it is identical.
A kind of Dynamic Configuration is proposed below for the timeliness analyzed in real time, first to the various configuration pipes in system
The basic syntax form for the data structure BNF form that science and engineering is made indicates.
One, the data structure of example tasks
A time window is set as timeWindow, to entire big data in given timeWindow time span
Analysis system instance objects request of the present invention is defined as:
Task::=<Td, Load, λ Arrive, λ Cur>
Wherein Td indicates to judge the time-out time of calculation and object task failure, and Load is being averaged for object instance task requests
Task amount, λ Arrive are a kind of arrays of storage object example request arrival rate, and λ cur, which is that existing object request is average, to be reached
Rate is set when initial: λ Cur=λ Array [0].
Two, the relevant data structure of node
Unique identification is carried out with NodeID to system interior joint, indicates node name, system interior joint information with NodeName
The object instance list ObjectList of list NodeList and system.
NodeList [NodeID]: :=< NodeName, NodeCapacity, ActiveInstNum, ObjectList,
NodeStatus,ObjTypeSet>
NodeCapacity therein indicates the task amount that node NodeID can be handled within the unit time,
ActiveInstNum indicates the node activity example number, and ObjTypeSet indicates the set of the node object type, Ke Yiwei
INADMIN, RTADMIN or OLADMIN.
ObjectList [ObjID]: :=<ObjectName, ObjInstList, Task>
ObjID therein is the unique identification of service object's class in system, and ObjectName is service object's class in system
Title, ObjInstList indicates the example list that such service object is managed, and Task indicates the task of such service object
Information model.
ObjInstList [ObjInstID]: :=<NodeID, InstStatus, InstLoad>
ObjInstID therein is the unique identification of service object's example in system, and NodeID is host's section of the example
Point identification, InstStatus are service object's example state mark in system (normal, heavy duties), and InstLoad indicates object instance
ObjInstID current load.
InstStatus::=<NORMAL | OVERRIDE>, NORMAL therein indicates that example is in normal condition,
OVERRIDE indicates that example is in heavy condition.
NodeStatus::=<NORMAL | READY | OVERRIDE>, NORMAL therein indicates that node is in normal shape
State, READY indicate that node is in ready state, and OVERRIDE indicates that node is in heavy condition.
Three, the data structure of data storage management configuration component
The configuration data structure of the storage management configuration component indicates that mainly setting flow data divides with InConfAdmin
Criticize the module parameter and storage state modulator of work.The size for how constituting the data volume of a processing batch, is generally divided into quiet
Two kinds of strategy patterns of state setting and dynamic setting, for being easy to be suitble to fixed size in the case of being evenly supplied of data source
Analysis in real time and real-time results displaying are with dynamic time window timeWindow when being that random supply is generated for data source
It is suitable for.Data mode is defined as follows:
InConfAdmin::={<NodeList, BatchingRef, StoringRef>}
NodeList is the corresponding definition in 3.2 sections, is set as INADMIN for the ObjTypeSet in NodeList,
It is expressed as the node of data storage management type, BatchingRef indicates (BatchingRef) in batches ginseng in stream process component
Number control indicates storage parameter StoringRef control in stream process component.For the batch processing in data storage management component
It is defined as follows with the information warning for storage:
AlarmIn::={<BatchingAlarm, StoringAlarm>}, BatchingAlarm are that the data criticized are easy
Very few or excessive, StoringAlarm refers to that storage delay warns.
Four, the data structure of real-time streams analytical calculation configuration component
Management of the real-time streams analytical calculation configuration component RTConfAdmin as real-time analytical calculation module relevant parameter
Person participates in nodal information the list NodeList, real-time analysis task list TaskList of analytical calculation in major maintenance system；
Meanwhile the load balancing parameter (RTLoad) undertaken to analytic unit in real-time streams analytical calculation module is set.Accordingly
Data structure definition it is as follows:
RTConfAdmin::=<NodeList, RTAnalysisReferenceList, RTTaskList, RTLoad>
NodeList is the corresponding definition in 3.2 sections, is set as RTADMIN for the ObjTypeSet in NodeList,
It is correspondingly the information list for analyzing object in system in real time for the ObjectList in NodeList,
RTAnalysisReferenceList is real-time analysis parameter list, and RTTaskList is real-time analysis task list, list
Element type is the task model of analysis example as defined in Task, and RTLoad is the real-time load for calculating configuration component node.
Five, off-line analysis calculates the data structure of configuration component
Off-line analysis calculates manager of the configuration component OLConfAdmin as off-line analysis computing module relevant parameter,
The nodal information list NodeList that off-line analysis calculates, off-line analysis task list are participated in major maintenance system
OLTaskList, meanwhile, the load balancing parameter (OLLoad) undertaken to analytic unit in off-line analysis computing module carries out
Setting.Corresponding data structure definition is as follows:
OLConfAdmin::=<NodeList, RTAnalysisReferenceList, RTTaskList, RTLoad>
NodeList is the corresponding definition in 3.2 sections, is set as OLADMIN for the ObjTypeSet in NodeList,
It is the information list of off-line calculation analysis object in system accordingly for the ObjectList in NodeList,
OLAnalysisReferenceList is off-line calculation analysis parameter list, and OLTaskList is off-line calculation analysis task column
Table, list element type are the task models of analysis example as defined in Task, and OLLoad is off-line calculation configuration component node
Load.
Six, the data structure of dynamic configuration view component
Dynamic configuration view (being denoted as ViewConf_Admin) is located at the visualization level on system upper layer of the present invention, dimension
Configuration information list in protecting system totality.
ViewConfAdmin::={<AlarmAdmin, RTConfAdmin, OLConfAdmin, InConfAdmin>}
The Dynamic Configuration is made of four configuration components: dynamic configuration view ViewConf_Admin is for global pipe
Reason；AlarmAdmin indicates warning configuration management, and RTConfAdmin indicates real-time streams analytical calculation configuration component, as real-time
The manager of analytical calculation module relevant parameter；OLConfAdmin indicates that off line data analysis calculates configuration component, is mainly responsible for
The off-line calculation module dynamic configuration associated control parameters；InConfAdmin indicates storage management configuration component, is mainly responsible for this
Module dynamic configuration associated control parameters.
Seven, the data structure of dynamic configuration manager
Warning manager AlarmAdmin in the dynamic configuration view ViewConfAdmin of system upper layer has the ability to tie up
Configuration information list in shield view structure mainly includes node for the operation that the information warning of system must be done and right
As example needs to do corresponding operation, such as over-loading operation, parameter adjustment operation, delete operation, activation operation, updates and operate, set
Do-nothing operation etc..Warning configuration management data structure is indicated with AlarmAdmin, is shown to execute in ViewConfAdmin
The dynamic configuration node inter-related task.This, corresponding structure is defined as follows:
AlarmAdmin::=<AlarmNodeList, AlarmObjectInstList, AlarmTaskList>
AlarmNodeList [NodeID]: :={<NodeStatus, NodeLoad>}
AlarmObjectInstList [ObjectInstID]: :=< NodeID, AlarmObjID, AlarmInstID,
InstLoad>}
AlarmTaskList [TaskID]: :={<TaskName>}
Wherein, AlarmNodeList is the node listing in information warning, and AlarmObjectInstList is warning letter
The object instance list of related operation in breath, AlarmTaskList are warning task information lists.
Eight, the message in Dynamic Configuration
To Dynamic Configuration is used in big data analysis system, message structure is the configuration of runtime dynamic debugging system
Communication infrastructure.Several type of messages necessary to formal definition below, AlarmObjectInstList therein indicate warning
Object instance information when operation, AlarmNodeID are the exclusive node marks of warning.
(1) Alarm (<[NodeID]>,<[ObjInstList]>): Data Storage, real-time computing module with
And off-line calculation module sends the list information of warning node and object instance to dynamic configuration view ViewConfAdmin.
(2)Adjust(<[NodeList],[ReferenceList]>|<[NodeID],[ObjInstList],
[ReferenceList] >): dynamic configuration view component ViewConfAdmin passes through dynamic configuration manager to data storage tube
Manage the parameter adjustment information of module, real-time computing module and off-line calculation module sending node and object instance.
(3) OverLoad ([NodeList] |<[NodeID], [ObjInstList]>): dynamic configuration view component
ViewConfAdmin passes through dynamic configuration manager to Data Storage, real-time computing module and off-line calculation mould
The heavily loaded information of block sending node and object instance.
(4) Active ([NodeList] |<[NodeID], [ObjInstList]>): dynamic configuration view component
ViewConfAdmin passes through dynamic configuration manager to Data Storage, real-time computing module and off-line calculation mould
The active information of block sending object example.
(5) Delete ([NodeList] |<[NodeID], [ObjInstList]>): dynamic configuration view component
ViewConfAdmin passes through dynamic configuration manager to Data Storage, real-time computing module and off-line calculation mould
The deletion information of block sending node or object instance.
(6)Update(<[NodeList],[ReferenceList]>|<[NodeID],[ObjInstList],
[ReferenceList] >): each Module nodes (Data Storage, real-time computing module and off-line calculation module) are logical
Cross state or parameter of the dynamic configuration manager to dynamic configuration view component ViewConfAdmin sending node or object instance
More new information.
(7) GetLoad ([NodeList] |<[NodeID], [ObjInstList]>): dynamic configuration view component
ViewConfAdmin passes through dynamic configuration manager to Data Storage, real-time computing module and off-line calculation mould
The information of the acquisition load entropy of block sending node or object instance.
(8) SetNull ([NodeList] |<[NodeID], [ObjInstList]>): dynamic configuration view component
ViewConfAdmin passes through dynamic configuration manager to Data Storage, real-time computing module and off-line calculation mould
Block sending node zero load information.
(9) Create ([NodeList] |<[NodeID], [ObjInstList]>): dynamic configuration view component
ViewConfAdmin passes through dynamic configuration manager to Data Storage, real-time computing module and off-line calculation mould
The creation information of block sending node or object instance, and initialization related information.
After above-mentioned data structure definition, Fig. 2~Fig. 3 is please referred to, understands dynamic configuration side of the present invention to facilitate
Method.
Wherein, it should be appreciated that big data analysis computing system body is exactly a kind of large-scale distributed application of complexity,
Appearance when configuration driven operation is carried out by a dynamic configuration manager (Dynamic Configuration Manager, DCM)
Fault reason and corresponding resource distribution function dynamic implementation, make relevant parameter and configuration should be able to dynamically be adapted to environment,
The variation of application demand and system resource.It simultaneously in systems, can by dynamic adjusting analysis calculated examples relevant configured parameter
To guarantee to improve the performance of system under the premise of system availability is constant.The invention proposes one kind to be based on big data analysis system
Dynamic Configuration in system, target are optimization system performances, improve the efficiency of system.
1, the load entropy in dynamic configuration, early warning redundancy
During system operation, the average arrival rate of analysis processing request increases or all may to the unreasonable of request scheduling
Cause each node heavy duty.
Load entropy function is set as delta (), is ∝, user couple by the operating lag that instance objects failure is defined as request
The requirement quantization of system performance and availability is the penalties function of the response time about request, and therefore, the response time is shorter, is used
The penalty value that family defines is lower.Delta (<[NodeID], [ObjInstID]>) definition such as (1) formula.
Wherein, delta () is the task load entropy function of the object instance ObjInstID of node NodeID, and P (t) is real
The probability that the response time of example object ObjInstID request is t, w (t) are the punishment value functions defined the response time of request,
Penalty value when wf is instance objects failure, F (u) they are the probability of instance objects failure, and u is the static threshold of instance objects, this
Average arrival rate of the calculating of a threshold value u dependent on instance objects request.Rule of thumb formula can get the response of example request
Functional relation of the time with load.
The redundancy of service object is that a kind of measurement increase similar service object can in actual time window to undertake
The degree in load Load in object instance task task, has with current average arrival time and the response time of example request
It closes, the early warning redundancy of the object instance of specified class column is expressed as AlarmRedundancy ().
Usual λ Cur is instant example object requests average arrival rate, and AlarmRedundancy () is defined as follows:
AlarmRedundancy (<[NodeID], [ObjInstID]>): :=(λ Cur- λ Array) * K+ (λ
RespondTime-Td)*H+delta(<[NodeID],[ObjInstID]>)*L(2)
K, H and L therein are empirical constants, and delta () is appointing for the object instance ObjInstID of node NodeID
Business load entropy function.λ RespondTime is the average response time of existing object class, is defined as follows:
λ RespondTime (<[NodeID], [ObjInstID]>): :=TExecutive+TDesignWait+TWait
(3)
TExecutive is the average performance times of example tasks, TDesignWait be to be designated in node
The waiting time of business, TWait are the average latency of the example of the object type.
Wherein Load () is the task load (see 3.1) that the example of object is specified in specified node NodeID, E
(NodeCapacity ()) be the task amount that specified node NodeID can be handled within the unit time mathematical expectation (see
3.2) it, is defined as follows:
Sum (ObjInstNum) therein is to realize the example that object ObjInstID is specified in specified node NodeID
Number.
TQueueLength () therein is the length of the queue to be allocated of specified node, and λ Cur () is in specified node
Request average arrival rate in specified object type.
TWait (<[NodeID], [ObjInstID]>) is the average latency of the example of the object type.
(2) formula early warning redundancy can be calculated in conjunction with (1) above, (3), (4), (5), (6).
The early warning redundancy of object instance should have a reasonable floor value LowerBound and a reasonable upper bound
Value SupperBound, so that LowerBound≤AlarmRedundancy≤SupperBound, shows node when lower than lower bound
Object instance is in loose condition (of surface), can increase task amount；Show that the object instance in node is in numerous when if being higher than the upper bound
Busy condition can suitably reduce task amount.
The process of dynamic configuration is as shown in Figures 2 and 3 in the big data analysis system, and Dynamic Configuration Process is mainly divided to two
Point, one is driving each module in each node of system in the case where node state NodeStatus is in normal NORMAL state
All kinds of example redundancies carry out Dynamic Maintenance, do the operations such as heavy duty, deletion, update, creation according to example state, make in node
Example is maintained at an efficient horizontality when operation, as shown in Figure 2；The second is carrying out redundancy by basic object of node
Maintenance does over-loading operation according to the NodeStatus value of node state, updates the operations such as operation, deletion, creation, so that entire system
System is maintained at a preferable runnability state, as shown in Figure 3.
When specifically being safeguarded to the dynamic redundancy of object type example in node, i.e. the state variable to each node of system
When NodeStatus is in normal NORMAL state, the process of Dynamic Maintenance is carried out to the types of objects example redundancy in node
Figure is as shown in Figure 2.
Its main process is described as follows:
(1) a time window timeWindow is preset, number is warned by the default dynamic configuration of dynamic configuration manager
It according to structure AlarmAdmin, and initializes, the system maintenance in time window in starter node.
(2) early warning for setting object instance according to the task type of object instance ObjInstID in node NodeID is superfluous
The experience initial value and a parameter adjusting step constant S of remaining lower bound LowerBound and upper bound SupperBound；
(3) corresponding AR=AlarmRedundancy (<[NodeID], [ObjInstID]>) value is calculated；
(4) if AR≤LowerBound, the state letter that alert messages line up the similar task instances of middle present node is searched
Breath,
A, if there is the object instance for being in READY state, then activation operation Active is executed for the object instance
([NodeID],[ObjInstList])；
If B, activation operation is unsuccessful, modify in dynamic configuration warning data structure AlarmAdmin
The NodeStatus of AlarmNodeList [NodeID] is OVERRIDE；
C, if there is no the object instance of READY state, then task instances can be created, executes Create
([NodeID], [ObjInstList]) operation needs modification dynamic configuration warning data structure if the operation is unsuccessful
The NodeStatus of AlarmNodeList [NodeID] in AlarmAdmin is OVERRIDE.
(5) step (4) are repeated, until AR > LowerBound.
(6) it executes and updates Update ([NodeID], [ObjInstList]) operation.
(7) if AR >=SupperBound, the state letter that alert messages line up the similar task instances of middle present node is searched
Breath:
A, if there is the object instance for being in READY OVERRIDE state, then delete operation Delete is executed
([NodeID], [ObjInstList]) needs modification dynamic configuration warning data structure if deletion is unsuccessful
The NodeStatus of AlarmNodeList [NodeID] in AlarmAdmin is OVERRIDE.
B, step A is repeated, until AR≤SupperBound or all examples all above-mentioned delete operations.
C, if there is no the object instance for being in READY OVERRIDE state, then parameter adjustment operation is executed
Adjust([NodeID],[ObjInstList],[ReferenceList])；
If D, parameter adjustment operates successfully, step is repeated 3., until AR≤SupperBound.
If E, parameter adjustment operation is unsuccessful, modify in dynamic configuration warning data structure AlarmAdmin
The NodeStatus of AlarmNodeList [NodeID] is OVERRIDE.
(8) it executes and updates Update ([NodeID], [ObjInstList]) operation.
(9) if LowerBound≤AR≤SupperBound, a random number random is generated,
A, LowerBound=LowerBound+random*S,
B, BestLowerBound:=LowerBound；
If C, LowerBound≤SupperBound goes to step A；
D, SupperBound=SupperBound-random*S
E, BestSupperBound:=BestSupperBound；；
If F, AR≤SupperBound goes to step D；
(10) the bound BestLowerBound and BestSupperBound of the object instance redundancy of output optimization.
(11) the next example ObjInstID for starting the same class object in the same node NodeID, goes to step
(3)。
After iteration, object instance is maintained at a high-caliber efficient state when operation in node.
As shown in figure 3, when being directed to the Dynamic Maintenance of node, i.e., it is alert for the AlarmAdmin in dynamic configuration manager
Show that the status information of NodeStatus in the node information warning list AlarmNodeList in configuration management data structure carries out
Dynamic Maintenance, main flow are as shown in Figure 3.
Its main process is described as follows:
(1) the information warning column in predetermined timeWindow, in the AlarmAdmin in poll dynamic configuration manager
Table A larmNodeList starts the Dynamic Maintenance based on node for the alert messages queue of NodeStatus.
(2) if the NodeStatus in AlarmNodeList [NodeID] is NORMAL, it is real to execute object in node
The Dynamic Maintenance of example redundancy.
It (3), can be with new Object example if the NodeStatus in AlarmNodeList [NodeID] is READY
Task executes Create ([NodeID], [ObjInstList]) operation.
If A, created successfully, modifying corresponding NodeStatus is NORMAL.
If B, creation is unsuccessful, modifying corresponding NodeStatus is OVERRIDE.
C, it executes and updates Update ([NodeID]) operation
It (4), can the heavily loaded section if the NodeStatus in AlarmNodeList [NodeID] is OVERRIDE
Point executes OverLoad ([NodeID]) operation.
If A, heavy duty success, modifying corresponding NodeStatus is READY.
If B, heavy duty is unsuccessful, execution empties SetNull ([NodeID]) operation, executes delete operation Delete later
([NodeID])。
C, it executes and updates Update ([NodeID]) operation.
It is different from the prior art, when dynamic configuration of the present invention, has following advantage in performance:
(1) when node is in NORMAL, the object instance in node needs to execute heavy duty in optimization process respectively, deletes
It removes, activate operation.But when executing these operations, it is not polled traversal, reduces the cost on network communication in system.?
During calculating warning redundancy, time complexity is higher, but for the computing capability of big data analysis platform, this
Time complexity is acceptable.
For in the searching process of the up-and-down boundary of the object instance redundancy in system node, have in first operation compared with
Big expense, but subsequent optimization process can empirically be worth by the best boundary value in the optimization process of front, Ke Yi great
The expense in optimization process is saved greatly, achievees the effect that quickly to distribute rationally.
(2) when node is in READY or OVERRIDE state, because there are the over-loading operation of system node, systematicness
The expense of energy is larger, and communication overhead is relatively fewer.
In conclusion the system performance in the present invention has preferable performance advantage compared to conventional arrangement process.
The above description is only an embodiment of the present invention, is not intended to limit the scope of the invention, all to utilize this hair
Equivalents made by bright specification and accompanying drawing content are applied directly or indirectly in relevant technical field, similarly include
In scope of patent protection of the invention.
Claims (5)
1. a kind of dynamically configurable big data analysis system characterized by comprising
Real-time data memory management module, for obtaining real-time streaming data in Distributed Services cluster, and dynamic configuration is related
Control parameter, and store；
Real-time streams analytical calculation module obtains real-time calculated result, and to real-time parser for statisticalling analyze real time data
Load carry out task adjustment；
Off-line analysis module, for statisticalling analyze off-line data, obtain off-line calculation as a result, and to off-line analysis algorithmic load into
The adjustment of row task；
Visualization model, for being visualized to real-time calculated result and off-line calculation result, and in the time delay of setting
Dynamic chart is provided in range, shows cluster service operation state and response condition in time, is alarmed more than threshold data
Processing.
2. dynamically configurable big data analysis system according to claim 1, which is characterized in that the real-time data memory
Management module includes:
Real-time streaming data securing component for obtaining the real-time streaming data in Distributed Services cluster, and is formatted, mistake
Filter is collected, and during collection, completes the batch work of flow data；
Real-time storage component deposits data batch for the data interchange format asynchronous transmission after formatting to HDFS
Storage；
Storage management configuration component is used for real-time data memory management module dynamic configuration associated control parameters.
3. dynamically configurable big data analysis system according to claim 1, which is characterized in that the real-time streams analysis meter
Calculating module includes:
Real-time streams processing component obtains real-time calculated result for obtaining data from HDFS to be analyzed in real time；And it will be real
When calculated result persistence, be sent to visualization model, and storage is into HDFS；
Real-time data analysis component, for real time data is for statistical analysis and intellectual analysis based on machine learning, and
Real-time parser is loaded and carries out task schedule, realizes load balancing；
Real-time streams analytical calculation configuration component is used for real-time streams analytical calculation module dynamic configuration associated control parameters.
4. dynamically configurable big data analysis system according to claim 1, which is characterized in that the off-line analysis module
Include:
Off-line data processing component, for off-line data being obtained from HDFS to carry out off-line analysis, obtain off-line calculation as a result,
And by off-line calculation result persistence, it is sent to visualization model, and storage is into HDFS and NoSQL；
Off line data analysis component, for off-line data is for statistical analysis and intellectual analysis based on machine learning, and
Task schedule is carried out to off-line analysis algorithmic load, realizes load balancing；
Off line data analysis calculates configuration component, is used for off-line analysis module dynamic configuration associated control parameters.
5. dynamically configurable big data analysis system according to claim 1, which is characterized in that the visualization model packet
It includes:
Dynamic configuration component realizes the configuration optimization of system performance for cooperateing with above-mentioned module；
Analysis view component in real time, for visualizing real-time calculated result, including System, real-time statistic analysis and intelligence
Foreseeable data；
Off-line analysis view component, for visualize off-line calculation as a result, include the summarizing of theme message, state analysis and
The displaying of intelligent predicting result and the statistics of location service request summarize；
Dynamic configuration view component for showing configuration data, and is associated with and shows real-time calculated result and off-line calculation result
Detection accuracy.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910332409.6A CN110222923A (en) | 2015-09-11 | 2015-09-11 | Dynamically configurable big data analysis system |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510577285.XA CN105279603B (en) | 2015-09-11 | 2015-09-11 | Dynamically configurable big data analysis system and method |
| CN201910332409.6A CN110222923A (en) | 2015-09-11 | 2015-09-11 | Dynamically configurable big data analysis system |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510577285.XA Division CN105279603B (en) | 2015-09-11 | 2015-09-11 | Dynamically configurable big data analysis system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110222923A true CN110222923A (en) | 2019-09-10 |
Family
ID=55148575
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510577285.XA Active CN105279603B (en) | 2015-09-11 | 2015-09-11 | Dynamically configurable big data analysis system and method |
| CN201910332409.6A Pending CN110222923A (en) | 2015-09-11 | 2015-09-11 | Dynamically configurable big data analysis system |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510577285.XA Active CN105279603B (en) | 2015-09-11 | 2015-09-11 | Dynamically configurable big data analysis system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN105279603B (en) |
Families Citing this family (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105760459B (en) * | 2016-02-04 | 2019-04-30 | 四川蓝光嘉宝服务集团股份有限公司 | A kind of distributed data processing system and method |
| CN107220261B (en) * | 2016-03-22 | 2020-10-30 | 中国移动通信集团山西有限公司 | Real-time mining method and device based on distributed data |
| CN105912582A (en) * | 2016-03-31 | 2016-08-31 | 畅捷通信息技术股份有限公司 | Control method for users' behavior analyses and control system for users' behavior analyses |
| EP3458959B1 (en) * | 2016-05-17 | 2023-03-22 | AB Initio Technology LLC | Reconfigurable distributed processing |
| CN107451147B (en) * | 2016-05-31 | 2020-07-31 | 北京京东尚科信息技术有限公司 | Method and device for dynamically switching kafka clusters |
| CN106126641B (en) * | 2016-06-24 | 2019-02-05 | 中国科学技术大学 | A kind of real-time recommendation system and method based on Spark |
| CN107621972A (en) * | 2016-07-15 | 2018-01-23 | 中兴通讯股份有限公司 | Big data task dynamic management approach, device and server |
| CN107918579B (en) * | 2016-10-09 | 2021-08-06 | 北京神州泰岳软件股份有限公司 | Method and device for generating baseline data in batches |
| CN106407472B (en) * | 2016-11-01 | 2019-08-20 | 广西电网有限责任公司电力科学研究院 | A kind of the big data calculating analysis task visual edit and management system of order form mode |
| WO2018098670A1 (en) * | 2016-11-30 | 2018-06-07 | 华为技术有限公司 | Method and apparatus for performing data processing |
| CN106776984B (en) * | 2016-12-02 | 2018-09-25 | 航天星图科技（北京）有限公司 | A kind of cleaning method of distributed system mining data |
| CN107294801B (en) * | 2016-12-30 | 2020-03-31 | 江苏号百信息服务有限公司 | Streaming processing method and system based on massive real-time internet DPI data |
| CN107145789B (en) * | 2017-05-22 | 2019-08-23 | 国网江苏省电力公司电力科学研究院 | A kind of Visual Interactive method of big data safety analysis |
| CN110019189A (en) * | 2017-09-18 | 2019-07-16 | 飞狐信息技术（天津）有限公司 | A kind of generation method and generation system of chart |
| CN107623737A (en) * | 2017-09-28 | 2018-01-23 | 南京轨道交通系统工程有限公司 | A kind of track traffic radio communication scheduling system and its design method |
| US10579943B2 (en) * | 2017-10-30 | 2020-03-03 | Accenture Global Solutions Limited | Engineering data analytics platforms using machine learning |
| CN108959954B (en) * | 2018-03-30 | 2021-11-12 | 努比亚技术有限公司 | Storm authority control method, device, server and storage medium |
| CN108494600B (en) * | 2018-03-30 | 2022-12-23 | 大唐丘北风电有限责任公司 | Topology creation control method, device and storage medium |
| CN109144508A (en) * | 2018-07-23 | 2019-01-04 | 北京科东电力控制系统有限责任公司 | It generates, the method and device of customization alarm picture |
| CN109343138B (en) * | 2018-09-29 | 2020-09-25 | 深圳市华讯方舟太赫兹科技有限公司 | Load balancing method of security inspection system and security inspection equipment |
| US10922139B2 (en) * | 2018-10-11 | 2021-02-16 | Visa International Service Association | System, method, and computer program product for processing large data sets by balancing entropy between distributed data segments |
| CN111245559B (en) * | 2018-11-29 | 2023-04-18 | 阿里巴巴集团控股有限公司 | Information determination method, information judgment method and device and computing equipment |
| CN109739925A (en) * | 2019-01-07 | 2019-05-10 | 北京云基数技术有限公司 | A kind of data processing system and method based on big data |
| CN112693502A (en) * | 2019-10-23 | 2021-04-23 | 上海宝信软件股份有限公司 | Urban rail transit monitoring system and method based on big data architecture |
| CN114285891B (en) * | 2021-12-15 | 2024-01-23 | 北京天融信网络安全技术有限公司 | SSLVPN-based session reconstruction method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102497292A (en) * | 2011-11-30 | 2012-06-13 | 中国科学院微电子研究所 | Computer cluster monitoring method and system thereof |
| JP2013030927A (en) * | 2011-07-27 | 2013-02-07 | Oki Electric Ind Co Ltd | Network analysis system, and network analysis device and network analysis program, and data processing module and data processing program |
| CN103618644A (en) * | 2013-11-26 | 2014-03-05 | 曙光信息产业股份有限公司 | Distributed monitoring system based on hadoop cluster and method thereof |
| CN104579761A (en) * | 2014-12-24 | 2015-04-29 | 西安工程大学 | Automatic nosql cluster configuration system and method based on cloud computing |
| CN104615526A (en) * | 2014-12-05 | 2015-05-13 | 北京航空航天大学 | Monitoring system of large data platform |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2766592B1 (en) * | 1997-07-23 | 1999-08-27 | Bull Sa | DEVICE AND METHOD FOR DYNAMICALLY REGULATING THE ALLOCATION OF RESOURCES ON A COMPUTER SYSTEM |
| US8769541B2 (en) * | 2009-12-31 | 2014-07-01 | Facebook, Inc. | Load balancing web service by rejecting connections |
| CN102043675B (en) * | 2010-12-06 | 2012-11-14 | 北京华证普惠信息股份有限公司 | Thread pool management method based on task quantity of task processing request |
| CN103903455B (en) * | 2014-04-14 | 2016-04-13 | 东南大学 | Controlling Traffic Signals in Urban Roads optimization system |
| CN104317658B (en) * | 2014-10-17 | 2018-06-12 | 华中科技大学 | A kind of loaded self-adaptive method for scheduling task based on MapReduce |
| CN104375621A (en) * | 2014-11-28 | 2015-02-25 | 广东石油化工学院 | Dynamic weighting load assessment method based on self-adaptive threshold values in cloud computing |
-
2015
- 2015-09-11 CN CN201510577285.XA patent/CN105279603B/en active Active
- 2015-09-11 CN CN201910332409.6A patent/CN110222923A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2013030927A (en) * | 2011-07-27 | 2013-02-07 | Oki Electric Ind Co Ltd | Network analysis system, and network analysis device and network analysis program, and data processing module and data processing program |
| CN102497292A (en) * | 2011-11-30 | 2012-06-13 | 中国科学院微电子研究所 | Computer cluster monitoring method and system thereof |
| CN103618644A (en) * | 2013-11-26 | 2014-03-05 | 曙光信息产业股份有限公司 | Distributed monitoring system based on hadoop cluster and method thereof |
| CN104615526A (en) * | 2014-12-05 | 2015-05-13 | 北京航空航天大学 | Monitoring system of large data platform |
| CN104579761A (en) * | 2014-12-24 | 2015-04-29 | 西安工程大学 | Automatic nosql cluster configuration system and method based on cloud computing |
Non-Patent Citations (1)
| Title |
|---|
| 赵妍: "《面向大数据的挖掘方法与研究》", 30 June 2015 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105279603B (en) | 2020-02-07 |
| CN105279603A (en) | 2016-01-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110222923A (en) | Dynamically configurable big data analysis system | |
| CN107734035B (en) | Virtual cluster automatic scaling method in cloud computing environment | |
| CN110794800B (en) | Intelligent factory information management monitoring system | |
| CN104317658B (en) | A kind of loaded self-adaptive method for scheduling task based on MapReduce | |
| EP3229150B1 (en) | Streaming graph optimization method and apparatus | |
| CN106209482A (en) | A kind of data center monitoring method and system | |
| CN105227488A (en) | A kind of network flow group scheduling method for distributed computer platforms | |
| WO2015094269A1 (en) | Hybrid flows containing a continuous flow | |
| CN112347636A (en) | Equipment guarantee simulation modeling method based on Multi-Agent technology | |
| CN107291550A (en) | A kind of Spark platform resources dynamic allocation method and system for iterated application | |
| CN111124806A (en) | Equipment state real-time monitoring method and system based on distributed scheduling task | |
| CN108737566B (en) | Distributed real-time message filtering system | |
| CN115373835A (en) | Task resource adjusting method and device for Flink cluster and electronic equipment | |
| CN111324460B (en) | Power monitoring control system and method based on cloud computing platform | |
| WO2023231398A1 (en) | Monitoring method and device for distributed processing system | |
| CN106951351A (en) | A kind of database loads tendency monitoring method | |
| CN103442087B (en) | A kind of Web service system visit capacity based on response time trend analysis controls apparatus and method | |
| CN110442446A (en) | The method of processing high-speed digital signal data flow in real time | |
| CN113225994A (en) | Intelligent air conditioner control method facing data center | |
| CN107948001B (en) | Intelligent monitoring system and method for industrial network of ship manufacturing workshop | |
| CN111814981A (en) | Distributed real-time rule inference scheduling method | |
| CN110990059A (en) | Stream type calculation engine operation method and system for tilt data | |
| CN109213105A (en) | A kind of reconfigurable device realizes restructural method and dcs | |
| CN110347461B (en) | Airplane cockpit human-computer interface information scheduling system and method based on migration workflow | |
| CN113742125A (en) | Lightweight high-throughput computing mode and fault-tolerant method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| AD01 | Patent right deemed abandoned | ||
| AD01 | Patent right deemed abandoned |
Effective date of abandoning: 20240112 |