CN105279603B - Dynamically configurable big data analysis system and method - Google Patents
Dynamically configurable big data analysis system and method Download PDFInfo
- Publication number
- CN105279603B CN105279603B CN201510577285.XA CN201510577285A CN105279603B CN 105279603 B CN105279603 B CN 105279603B CN 201510577285 A CN201510577285 A CN 201510577285A CN 105279603 B CN105279603 B CN 105279603B
- Authority
- CN
- China
- Prior art keywords
- analysis
- value
- real
- dynamic configuration
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention discloses a big data analysis system and a method capable of being dynamically configured, wherein the system comprises a real-time data storage management module, a real-time flow analysis and calculation module, an off-line analysis module, a visualization module and the like, wherein each module is provided with at least one component capable of carrying out dynamic configuration management, such as a data management configuration component, a real-time flow analysis and calculation configuration component, an off-line analysis and calculation configuration component and a dynamic configuration component. The invention also provides a dynamic configuration method of the big data analysis system, which designs the data structure and the message structure of each component module, drives the dynamic configuration of the system through the state information of the warning data structure in the dynamic configuration manager, and provides a calculation method and a dynamic configuration method of warning redundancy.
Description
Technical Field
The invention relates to the field of big data analysis application, in particular to a big data analysis system and method capable of being dynamically configured.
Background
The existing business intelligent systems, decision support systems and the like increasingly require to support big data integration and analysis, and because the big data analysis and calculation has large data volume, complex process and long processing time, big data analysis and application also face a new challenge: the systems must have high reliability, require the software system to have adaptability to changes, these systems need to have the ability to update the configuration on the premise of not interrupting the system service, fault-tolerant management problem, how to handle the abnormality under the condition of updating failure, make the system keep normal and stable operation. Namely, the dynamic configuration technology is an important means for realizing the self-adaptive reliability of the software of the big data platform.
The Hadoop of the early big data parallel processing frame is limited by single point fault and the calculation mode is relatively single, and the Hadoop2.0 introduces a YARN (unified resource management system), so that the system reliability and the resource utilization rate of the whole cluster are improved, the system becomes a big data processing frame and a programming mode capable of running various big data processing frames including a real-time stream processing frame Storm, Spark and the like, the fault tolerance of a big data analysis application system is improved, and the system has good reliability and is still a difficult problem.
The big data engine Spark technology, which is currently emerging widely, was originally developed by the AMPLab laboratory at UC Berkeley university and is now an open source project managed by the Apache fund. The Spark is a general model supporting memory calculation, which can meet the requirements of most data processing and mining applications, and enables a data analysis program to run faster and have better fault tolerance. The Spark introduces an elastic Distributed data set (RDD) model to fully utilize memory resources to improve the calculation efficiency. Unlike other big data processing frameworks, Spark can efficiently handle the processing from ETL to SQL to machine learning to Streaming data with one engine on the basis of Shark, MLlib, GraphX and Spark Streaming. Spark plus Spark Streaming (or Shark, B1inkDB) was used for real-time and batch processing; using Spark Streaming plus MLlib for Streaming and machine learning; spark plus GraphX was used for graph pipelines, etc. However, although the real-time performance and fault-tolerant performance of the new real-time flow computing framework are greatly improved, the high reliability and high availability of the system still remain a challenge.
Along with the increasingly large scale and complex behaviors of distributed systems in large data platforms, various faults occurring in the systems also increase exponentially, so that very serious damage and loss are brought to the industrial and government departments, and once a shutdown event occurs to the systems, great loss and trouble are brought, so that the large data analysis systems need to have the capability of automatic configuration on the premise of not interrupting system service, so that the reliability of the systems is improved, the risk control capability of the systems is enhanced, and the overall operating efficiency of the software platforms is improved. An effective solution to the problems in the related art has not been proposed yet.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: and dynamic optimization configuration for big data analysis and calculation operation is provided, so that the reliability of the system is improved, and the risk control capability is enhanced.
In order to solve the technical problems, the invention adopts the technical scheme that: there is provided a dynamically configurable big data analytics system comprising:
the real-time data storage management module is used for acquiring real-time streaming data from the distributed service cluster, dynamically configuring relevant control parameters and storing the relevant control parameters;
the real-time flow analysis and calculation module is used for counting and analyzing real-time data, obtaining a real-time calculation result and carrying out task adjustment on a real-time analysis algorithm load;
the off-line analysis module is used for counting and analyzing off-line data, obtaining an off-line calculation result and carrying out task adjustment on the off-line analysis algorithm load;
and the visualization module is used for visually displaying the real-time calculation result and the off-line calculation result, providing a dynamic chart within a set time delay range, timely displaying the running state and the response condition of the cluster service and alarming the data exceeding the threshold value.
In order to solve the above problem, the present invention further provides a dynamic configuration method for a big data analysis system, which includes the following steps:
s1: presetting a time window, presetting an alarm data structure by a dynamic configuration manager, and initializing;
s2: setting an experience initial value of an early warning redundancy lower bound and an early warning redundancy upper bound of an object instance and a parameter adjustment step constant according to the task type of the object instance in a node;
s3: calculating an early warning redundancy value of the object instance;
s4: determining that the early warning redundancy value is between the empirical initial values of a lower bound and an upper bound, and generating a random number;
s5: calculating an optimized upper bound value and an optimized lower bound value according to the step length, the random number, the empirical initial values of the upper bound and the lower bound;
s6: determining that the early warning redundancy value is located between an optimized lower bound value and an optimized upper bound value;
s7: polling an alarm information list in dynamic configuration management in a preset time window;
s8: and modifying the node state aiming at the warning information list of the node state so as to realize the dynamic maintenance of the node.
The invention has the beneficial effects that: different from the prior art, the invention realizes the configuration optimization of the system performance through the cooperative cooperation of the modules, and simultaneously enables the system to run on a high-efficiency big data analysis and calculation platform through calculating the warning redundancy, thereby not only improving the reliability of the system, but also enhancing the risk control capability.
Drawings
FIG. 1 is a schematic diagram of the system of the present invention;
FIG. 2 is a schematic flow chart illustrating the dynamic maintenance of redundancy of object instances of various types when nodes of the system are in a NORMAL state according to the method of the present invention;
fig. 3 is a schematic diagram illustrating a dynamic configuration maintenance process based on node states in the warning information list in the method of the present invention.
Detailed Description
In order to explain technical contents, achieved objects, and effects of the present invention in detail, the following description is made with reference to the accompanying drawings in combination with the embodiments.
The most key concept of the invention is as follows: the configuration optimization of the system performance is realized through the synergistic effect of all the modules of the system, and a high-efficiency big data analysis and calculation platform is further obtained.
Referring to fig. 1, an embodiment of the present invention provides a dynamically configurable big data analysis system, including:
the real-time data storage management module is used for acquiring real-time streaming data from the distributed service cluster, dynamically configuring relevant control parameters and storing the relevant control parameters;
the real-time flow analysis and calculation module is used for counting and analyzing real-time data, obtaining a real-time calculation result and carrying out task adjustment on a real-time analysis algorithm load;
the off-line analysis module is used for counting and analyzing off-line data, obtaining an off-line calculation result and carrying out task adjustment on the off-line analysis algorithm load;
and the visualization module is used for visually displaying the real-time calculation result and the off-line calculation result, providing a dynamic chart within a set time delay range, timely displaying the running state and the response condition of the cluster service and alarming the data exceeding the threshold value.
Wherein the real-time data storage management module comprises:
the real-time streaming data acquisition component is used for acquiring real-time streaming data in the distributed service cluster, formatting, filtering and collecting the real-time streaming data, and completing the batch work of the streaming data in the collection process;
and the real-time storage component is used for asynchronously sending the formatted data exchange format to the HDFS and storing the data batch.
And the storage management configuration component is used for dynamically configuring the related control parameters by the real-time data storage management module.
Wherein, the real-time flow analysis and calculation module comprises:
the real-time stream processing assembly is used for acquiring data from the HDFS for real-time analysis and acquiring a real-time calculation result; the real-time calculation result is persisted, sent to a visualization module and stored in an HDFS;
the real-time data analysis component is used for carrying out statistical analysis and intelligent analysis based on machine learning on the real-time data, and carrying out task scheduling on the real-time analysis algorithm load to realize load balancing;
and the real-time flow analysis calculation configuration component is used for dynamically configuring the related control parameters by the real-time flow analysis calculation module.
Wherein the offline analysis module comprises:
and the offline data processing component is used for acquiring offline data from the HDFS for offline analysis, acquiring an offline calculation result, persisting the offline calculation result, sending the offline calculation result to the visualization module, and storing the offline calculation result in the HDFS and the NoSQL.
The off-line data analysis component is used for carrying out statistical analysis and intelligent analysis based on machine learning on off-line data, and carrying out task scheduling on off-line analysis algorithm loads to realize load balancing;
and the offline data analysis calculation configuration component is used for dynamically configuring the related control parameters by the offline analysis module.
Wherein the visualization module comprises:
the dynamic configuration component is used for cooperating with the modules to realize the configuration optimization of the system performance;
and the real-time analysis view component is used for visually displaying real-time calculation results, including data of real-time summarization, real-time statistical analysis and intelligent prediction.
And the offline analysis view component is used for visually displaying offline calculation results, including summary of theme messages, state analysis and intelligent prediction result display, and statistical summary of location service requests.
And the dynamic configuration view component is used for displaying the configuration data and correlating the configuration data with the detection precision of the real-time calculation result and the off-line calculation result.
A dynamic configuration method of a big data analysis system comprises the following steps:
s1: presetting a time window, presetting an alarm data structure by a dynamic configuration manager, and initializing;
s2: setting an experience initial value of an early warning redundancy lower bound and an early warning redundancy upper bound of an object instance and a parameter adjustment step constant according to the task type of the object instance in a node;
s3: calculating an early warning redundancy value of the object instance;
s4: determining that the early warning redundancy value is between the empirical initial values of a lower bound and an upper bound, and generating a random number;
s5: calculating an optimized upper bound value and an optimized lower bound value according to the step length, the random number, the empirical initial values of the upper bound and the lower bound;
s6: determining that the early warning redundancy value is located between an optimized lower bound value and an optimized upper bound value;
s7: polling an alarm information list in dynamic configuration management in a preset time window;
s8: and modifying the node state aiming at the warning information list of the node state so as to realize the dynamic maintenance of the node.
Wherein the step S4 specifically includes:
s41: judging whether the early warning redundancy value is larger than or equal to a lower-bound experience initial value or not;
if yes, go to S42: updating the object instance;
if not, executing S411: judging whether the object instance is in a ready state;
if yes, go to step S412: activating the object instance and returning to step S41;
if not, go to S413: creating a task instance and returning to step S41;
wherein after the step S42, it further comprises S43: judging whether the early warning redundancy value is smaller than or equal to an upper-bound empirical initial value or not;
if yes, go to S44: updating the object instance and generating a random number;
if not, executing S431: judging whether the object instance is in a ready state or a heavy load state;
if yes, go to S432: deleting the object instance and returning to step S43;
if not, executing S433: adjusts the parameters of the object instance, and returns to step S43.
After step S412, the method further includes:
s414: judging whether the activation is successful;
if yes, return to step S41;
otherwise, executing S415: setting the node state of the warning node information list as a heavy load;
after the step S413, the method further includes S416, determining whether the creation is successful;
if yes, return to step S41;
otherwise, S415 is executed.
After step S432, step S434 is further included: judging whether the deletion is successful;
if yes, return to step S43;
otherwise, executing S415;
after step S433, the method further includes step S436: judging whether the adjustment is successful;
if yes, return to step S43;
otherwise, S415 is executed.
Wherein, step S5 specifically includes:
s51: calculating the optimized lower bound value: optimizing a lower bound value which is an empirical initial value of the lower bound and + step size which is a random number;
s52: calculating the optimized upper bound value: and optimizing an upper bound value, namely an upper bound empirical initial value, namely step size, random number.
For the convenience of understanding the technical solution, the present invention is described with reference to fig. 1 to 3, which provide a specific embodiment.
Firstly, it should be noted that, in big data analysis and calculation, the large-scale distributed computing service needs to perform system optimization, and it is not enough to improve the fault tolerance of the system only from the system development process. Because the parameters related to the system performance in the big data analysis computing system are various and are difficult to regulate, the big data analysis computing system is very difficult to work. Aiming at the challenging problem, the invention provides a dynamically configurable big data analysis system and a method, wherein the system comprises a real-time data storage management module, a real-time flow analysis and calculation module, an off-line analysis module, a visualization module and the like, and each module is provided with a component capable of carrying out dynamic configuration management, such as a data management configuration component, a real-time flow analysis and calculation configuration component, an off-line analysis and calculation configuration component and a dynamic configuration component. The dynamic configuration component is the core of the system dynamic configuration management, and simultaneously realizes the configuration optimization of the system performance in cooperation with each module. The system can be realized by adopting the current latest big data platform technology, such as Hadoop, Kafka, Spark Streaming, Hive and the like, and the system provided by the invention has good running condition by product line deployment detection at present.
The general structure of the dynamically configurable big data analysis system proposed by the present invention is shown in fig. 1. The system adopts a modular design and mainly comprises a real-time data storage management module, a real-time flow analysis and calculation module, an off-line analysis module, a visualization module and the like. The main functions of the various modules are as follows:
⑴ real-time data storage management module
The module is made up of three components, including: the system comprises a real-time stream data acquisition component, a real-time stream data storage component and a real-time storage management configuration component.
The real-time streaming data acquisition component is mainly responsible for acquiring real-time streaming data in the existing large-scale distributed service cluster, formatting, filtering and collecting are completed by the component, and batch modules (batching modules) of the streaming data are completed in the collecting process.
The real-time storage component asynchronously sends the formatted data exchange format JSON to the HDFS to store the data batch, and meanwhile, the real-time storage component sends the data to a batch queue (batch queue) to be provided for the real-time computing component.
The storage management configuration component is mainly responsible for dynamically configuring relevant control parameters of the module.
⑵ real-time flow analysis and calculation module
The module is made up of three components, including: the real-time flow analysis and calculation configuration component comprises a real-time flow processing component, a real-time data analysis component and a real-time flow analysis and calculation configuration component.
The real-time stream processing component mainly provides services for the real-time analysis component. On one hand, relevant data which are responsible for pulling off-line calculation results from the HDFS are provided for an analysis component to be used as analysis reference, and the analysis reference is pre-calculation belonging to real-time analysis; on the other hand, the analysis result is persisted, and the data source is provided for the upper layer visualization and the data is stored in the HDFS.
The real-time data analysis component is mainly responsible for classical statistical analysis and intelligent analysis based on machine learning, and carries out task scheduling on analysis algorithm loads to realize load balancing.
The real-time flow analysis calculation configuration component is mainly responsible for dynamically configuring relevant control parameters of the module.
⑶ offline analysis module
The module is made up of three components, including: the device comprises an offline data processing component, an offline data analysis component and an offline data analysis, calculation and configuration component.
The offline data processing component mainly provides services for the offline analysis component. On one hand, the method is responsible for pulling and separating related data from the HDFS and carrying out pre-calculation for offline analysis; on the other hand, the offline data analysis result is persisted, and the persisted offline data analysis result is provided for an upper layer visualization to provide a data source, and the calculation result data is stored in the HDFS and the NoSQL.
The offline data analysis component is mainly responsible for classical global statistical analysis and global intelligent analysis based on machine learning, and carries out task scheduling on offline analysis algorithm loads to realize load balancing.
The off-line data analysis calculation configuration component is mainly responsible for dynamically configuring relevant control parameters of the module.
The offline calculation analysis module mainly performs offline classical statistical analysis on data in the distributed cluster.
And scheduling the offline data analysis task according to a set time window, and generating a report according to a calculation result for resource allocation and later-stage optimization reference of service development and operation and maintenance personnel.
⑷ visualization module
The module is made up of four components, including: a dynamic configuration component, a dynamic configuration view component, a real-time analysis view component, an offline analysis view component.
The module is mainly used for visually displaying calculation results generated by the real-time flow analysis and calculation module and the off-line data analysis and calculation module, allowing a dynamic chart to be provided within a set time delay range, displaying the running state and the response condition of the cluster service in time and alarming data exceeding a threshold value.
The data displayed by the module is divided into three types:
a. real-time analysis view component display real-time analysis data
The part mainly comprises data of real-time summarization of analysis results, real-time statistical analysis and intelligent prediction.
b. Offline analysis view component display offline analysis data
The part mainly comprises the collection of various theme messages, the display of state analysis and intelligent prediction results, and the statistical collection of location service requests.
c. The dynamic configuration view component displays configuration data and may correlate to the detection accuracy of the displayed analysis results.
In order to adapt to effective analysis of the service state of the current large-scale distributed service system and improve the instant analysis benefit of real-time analysis, abnormal analysis tasks are often required to be found in time, so that the availability requirement of the real-time analysis system is improved.
For ease of discussion, the system of the present invention is described as follows:
⑴ the system has N nodes, provides M kinds of data analysis or statistics calculation task;
⑵ the components of the system of the present invention that perform a class of tasks are loosely coupled, i.e., the system provides a reliable asynchronous communication mechanism between nodes with the same communication overhead between asynchronisms.
In the following, a dynamic configuration method is proposed for timeliness of real-time analysis, and first, data structures of various configuration management works in the system are expressed in a basic syntax form of a BNF paradigm.
Data structure of instance task
Setting a time window as timeWindow, and defining an example object request of the whole big data analysis system in a given timeWindow time span as follows:
Task::＝<Td,Load,λArrive,λCur>
wherein Td represents the timeout time for judging the invalidation of the object calculation task, Load is the average task amount of the object instance task request, λ Arrive is an array for storing the arrival rate of the object instance request, λ cur is the average arrival rate of the current object request, and is initially set as: λ Cur ═ λ Array [0 ].
Two, node-dependent data structure
The node in the system is uniquely identified by NodeID, the Nodename represents the node name, the node information list NodeList in the system and the object instance list ObjectList of the system.
NodeList[NodeID]::＝<NodeName,NodeCapacity,ActiveInstNum,ObjectList,NodeStatus,ObjTypeSet>
Wherein, NodeCapacity represents the task amount that the node NodeID can process in unit time, ActiveInstNum represents the number of active instances of the node, ObjTypeSet represents the collection of object types of the node, which can be INADMIN, RTADMIN or OLADMIN.
ObjectList[ObjID]::＝<ObjectName,ObjInstList,Task>
Wherein, ObjID is the unique identification of the service object class in the system, ObjectName is the name of the service object class in the system, ObjInstList represents the instance list managed by the service object, and Task represents the Task information model of the service object.
ObjInstList[ObjInstID]::＝<NodeID,InstStatus,InstLoad>
Wherein, objInstID is the unique identification of the service object instance in the system, NodeID is the host node identification of the instance, InstStatus is the status identification (normal, heavy load) of the service object instance in the system, and InstLoad represents the current load of the object instance objInstID.
InstStatus: < NORMAL | OVERRIDE >, wherein NORMAL indicates that the example is in a NORMAL state, and OVERRIDE indicates that the example is in a heavy load state.
The node status indicates that the node is in a NORMAL state, the node is in a READY state, and the overload indicates that the node is in a heavy-load state.
Data structure of data storage management configuration assembly
The configuration data structure of the storage management configuration component is represented by InConfAdmin, and is mainly used for setting module parameters of streaming data batch work and controlling storage parameters. How to form the size of the data volume of a processing batch is generally divided into two strategy modes of static setting and dynamic setting, under the condition of uniform supply of data sources, the data source is easy to be suitable for real-time analysis and real-time result display by using a fixed size, and when the data sources are randomly supplied and generated, a dynamic time window is suitable. The dataform is defined as follows:
InConfAdmin::＝{<NodeList,BatchingRef,StoringRef>}
NodeList is the corresponding definition in section 3.2, with ObjTypeSet in NodeList set to INADMIN, representing a node of data storage management type, and BatchinRef represents batch (BatchinRef) parameter control in a stream processing component and StoringRef control in a stream processing component. The alert information for the batch and store in the data store management component is defined as follows:
AlarmIn { < BathingAlarm, StoringAlarm > }, BathingAlarm is that the data of the batch is too little or too big easily, StoringAlarm refers to the storage delay warning.
Data structure of real-time flow analysis calculation configuration component
A real-time flow analysis calculation configuration component RTConfAdmin serves as a manager of relevant parameters of a real-time analysis calculation module, and mainly maintains a node information list NodeList and a real-time analysis task list TaskList which are involved in analysis calculation in a system; and simultaneously, setting a load balancing parameter (RTload) borne by the analysis component in the real-time flow analysis and calculation module. The corresponding data structure is defined as follows:
RTConfAdmin::＝<NodeList,RTAnalysisReferenceList,RTTaskList,RTLoad>
NodeList is the corresponding definition in section 3.2, with ObjTypeSet in NodeList set to RTADMIN, and with ObjectList in NodeList the information list of real-time analysis objects in the system, RTAnalysiReferenceList is the real-time analysis parameter list, RTTaskList is the real-time analysis Task list, whose list element type is the Task model of the analysis instance specified by Task, RTLoad is the load of the real-time computation configuration component node.
Fifthly, analyzing and calculating the data structure of the configuration component off line
The offline analysis and calculation configuration component olconfadamin is used as a manager of the parameters related to the offline analysis and calculation module, and mainly maintains a node information list NodeList and an offline analysis task list oltaskelst which participate in offline analysis and calculation in the system, and simultaneously sets a load balancing parameter (OLLoad) borne by the analysis component in the offline analysis and calculation module. The corresponding data structure is defined as follows:
OLConfAdmin::＝<NodeList,RTAnalysisReferenceList,RTTaskList,RTLoad>
NodeList is the corresponding definition in section 3.2, OLADMIN is set for ObjTypeSet in NodeList, and accordingly, olobjectlist in NodeList is the information list of the off-line calculation analysis object in the system, OLAnalysisReferenceList is the off-line calculation analysis parameter list, oltaskstlist is the off-line calculation analysis Task list, the list element type is the Task model of the analysis instance specified by Task, and OLLoad is the load of the off-line calculation configuration component node.
Sixth, data structure of dynamic configuration view component
The dynamic configuration view (marked as ViewConf _ Admin) is positioned at the visualization level of the upper layer of the system, and maintains a configuration information list in the system overall.
ViewConfAdmin::＝{<AlarmAdmin，RTConfAdmin,OLConfAdmin,InConfAdmin>}
The dynamic configuration method comprises four configuration components: dynamically configuring a view ViewConf _ Admin for global management; the AlarmAdmin represents warning configuration management, and the RTConfAdmin represents a real-time flow analysis and calculation configuration component, which serves as a manager of relevant parameters of a real-time analysis and calculation module; OLConfAdmin represents an offline data analysis calculation configuration component and is mainly responsible for dynamically configuring relevant control parameters of the offline calculation module; InConfAdmin represents a storage management configuration component, which is primarily responsible for dynamically configuring the relevant control parameters for the module.
Data structure of dynamic configuration manager
The alert manager alarmmaddmin in the system upper layer dynamic configuration view viewconfadamin has the capability of maintaining a configuration information list in a view structure, and operations that must be performed on the alert information of the system mainly include that nodes and object instances need to perform corresponding operations, such as reloading operations, parameter adjusting operations, deleting operations, activating operations, updating operations, emptying operations, and the like. The alert configuration management data structure is denoted by AlarmAdmin to perform the tasks associated with dynamically configuring the node as demonstrated in viewconfadamin. Thus, its corresponding structure is defined as follows:
AlarmAdmin::＝<AlarmNodeList,AlarmObjectInstList,AlarmTaskList>
AlarmNodeList[NodeID]::＝{<NodeStatus,NodeLoad>}
AlarmObjectInstList[ObjectInstID]::＝{<NodeID,AlarmObjID,AlarmInstID,InstLoad>}
AlarmTaskList[TaskID]::＝{<TaskName>}
the alarmmonodelist is a node list in the warning information, the alarmmobjectinstlist is a list of related running object instances in the warning information, and the AlarmTaskList is a list of warning task information.
Eight, message in dynamic configuration method
A dynamic configuration method is adopted in a big data analysis system, and a message structure is a communication basis for dynamically adjusting system configuration in a running period. The following forms define the necessary several message types, where AlarmObjectInstList represents the runtime object instance information of the alarm, and alarmeneid is the unique node identification of the alarm.
(1) And the Alarm (< [ NodeID ] >, [ ObjInstList ] >) data storage management module, the real-time calculation module and the off-line calculation module send the list information of the warning nodes and the object instances to the dynamic configuration view ViewConfAdmin.
(2) And Adjust (< [ NodeList ], [ ReferenceList ] > | < [ NodeID ], [ ObjInstList ], [ ReferenceList ] >) the dynamic configuration view component ViewConfAdmin sends parameter adjustment information of the nodes and the object instances to the data storage management module, the real-time calculation module and the off-line calculation module through the dynamic configuration manager.
(3) And the dynamic configuration view component ViewConfAdmin sends the reloading information of the nodes and the object instances to the data storage management module, the real-time calculation module and the off-line calculation module through a dynamic configuration manager.
(4) And Active ([ NodeList ] | < [ NodeID ], [ ObjInstList ] >) the dynamic configuration view component ViewConfAdmin sends the activation information of the object instance to the data storage management module, the real-time calculation module and the off-line calculation module through the dynamic configuration manager.
(5) Delete ([ NodeList ] | < [ NodeID ], [ ObjInstList ] >) dynamic configuration view component ViewConfAdmin sends the deletion information of the node or object instance to the data storage management module, the real-time computation module and the off-line computation module through the dynamic configuration manager.
(6) And each module node (a data storage management module, a real-time calculation module and an offline calculation module) sends the state or parameter updating information of the node or the object instance to the dynamic configuration view component ViewConfAdmin through a dynamic configuration manager.
(7) GetLoad ([ NodeList ] | < [ NodeID ], [ ObjInstList ] >) the dynamic configuration view component ViewConfAdmin sends the information of the acquired load entropy of the node or the object instance to the data storage management module, the real-time calculation module and the off-line calculation module through the dynamic configuration manager.
(8) SetNull ([ NodeList ] | < [ NodeID ], [ ObjInstList ] >) the dynamic configuration view component ViewConfAdmin sends node no-load information to the data storage management module, the real-time computation module and the off-line computation module through the dynamic configuration manager.
(9) And Create ([ NodeList ] | < [ NodeID ], [ ObjInstList ] >) the dynamic configuration view component ViewConfAdmin sends the creation information of the node or the object instance to the data storage management module, the real-time calculation module and the off-line calculation module through the dynamic configuration manager, and initializes the related information.
After the above data structure is defined, please refer to fig. 2 to fig. 3 to facilitate understanding of the dynamic configuration method according to the present invention.
It should be understood that the big data analysis computing system is a complex large-scale distributed application, and a Dynamic Configuration Manager (DCM) performs Configuration-driven fault-tolerant processing during runtime and Dynamic implementation of corresponding resource Configuration functions, so that relevant parameters and Configuration can be dynamically adjusted to adapt to changes in environment, application requirements, and system resources. Meanwhile, in the system, the performance of the system can be improved on the premise of unchanging the availability of the system by dynamically adjusting and analyzing the relevant configuration parameters of the calculation examples. The invention provides a dynamic configuration method based on a big data analysis system, aiming at optimizing the system performance and improving the system efficiency.
1. Load entropy, early warning redundancy in dynamic configuration
During system operation, node reloads may be caused by an increase in the average arrival rate of analysis processing requests or unreasonable scheduling of requests.
Setting the load entropy function to delta (), defining instance object failures as requests response delays of ∈ and quantifying user demand for system performance and availability as a penalty function with respect to request response time, so that the shorter the response time, the lower the user-defined penalty value. delta (< [ NodeID ], [ ObjInstID ] >) is defined as formula (1).
Wherein delta () is a task load entropy function of an object instance ObjInstID of a node NodeID, p (t) is a probability that a response time of an instance object ObjInstID request is t, w (t) is a penalty value function defined by the response time of the request, wf is a penalty value when the instance object fails, f (u) is a probability that the instance object fails, u is a static threshold value of the instance object, and the calculation of this threshold value u depends on an average arrival rate of the instance object requests. The response time of the instance request as a function of load may be obtained from empirical formulas.
The redundancy of the service object is a measure to enable the same kind of service object to increase the degree of assuming the Load in the Task of the object instance within the current time window, the early warning redundancy of the object instance of the specified class column is expressed as alarmrondansacy () in relation to the current average arrival time and response time of the instance request.
In general, λ Cur is the current instance object request average arrival rate, and alarmreducance () is defined as follows:
AlarmRedundancy(<[NodeID],[ObjInstID]>)::＝(λCur-λArray)*K+(λRespondTime-Td)*H+delta(<[NodeID],[ObjInstID]>)*L (2)
where K, H and L are both empirical constants, delta () is a task load entropy function of the object instance ObjInstID of the node NodeID. λ RespondTime is the average response time of the current object class, which is defined as follows:
λRespondTime(<[NodeID],[ObjInstID]>)::＝TExecutive+TDesignWait+TWait(3)
TExecutive is the average execution time of the task of an instance, TDesignWait is the latency of the task to be assigned in the node where it is located, and TWait is the average latency of the instance of the object type.
Where Load () is the task Load of the instance of the specified object in the specified node NodeID (see 3.1), E (nodecacity ()) is the mathematical expected value of the amount of tasks that the specified node NodeID can handle per unit time (see 3.2), which is defined as follows:
where sum (objinstnum) is the number of instances implementing the specified object ObjInstID in the specified node NodeID.
Wherein TQueLength () is the length of the queue to be allocated of a specified node, and λ Cur () is the average arrival rate of requests in a specified object type in the specified node.
TWait (< [ NodeID ], [ ObjInstID ] >) is the average latency of an instance of the object type.
The formula (2) early warning redundancy can be calculated by combining the above (1), (3), (4), (5) and (6).
The early warning redundancy of the object instance should have a reasonable lower bound value LowerBound and a reasonable upper bound value SuperBound, so that LowerBound is less than or equal to AlarmReduncyny and less than or equal to SuperBound, and when the lower bound value is lower than the lower bound value, the node object instance is in a loose state, and the workload can be increased; if the object instance is in a busy state when the object instance is higher than the upper bound, the workload can be properly reduced.
The flow of dynamic configuration in the big data analysis system is shown in fig. 2 and fig. 3, the dynamic configuration process mainly includes two parts, one is that under the condition that the node status is in a NORMAL state, each module is driven to dynamically maintain the redundancy of each type of instance in each node of the system, and the operation such as overloading, deleting, updating, creating and the like is carried out according to the instance state, so that the runtime instance in the node is kept in an efficient horizontal state, as shown in fig. 2; the other is to perform redundancy maintenance by using the node as a basic object, and perform operations such as reloading, updating, deleting, creating and the like according to the NodeStatus value of the node state, so as to keep the whole system in a better operation performance state, as shown in fig. 3.
Specifically, a flowchart for dynamically maintaining the redundancy of each type of object instance in a node when the dynamic redundancy of the object type instance in the node is maintained, that is, when the state variable NodeStatus of each node of the system is in a NORMAL state, is shown in fig. 2.
The main process is described as follows:
(1) presetting a time window timeWindow, presetting a dynamic configuration warning data structure AlarmAdmin by a dynamic configuration manager, initializing, and starting system maintenance in the node in the time window.
(2) Setting an empirical initial value of an early warning redundancy lower bound LowerBound and an upper bound SuperBound of an object instance and a parameter adjusting step length constant S in the node NodeID according to the task type of the object instance ObjInstID;
(3) calculating corresponding AR ═ AlarmRedundant (< [ NodeID ], [ ObjInstID ] >) values;
(4) if AR is less than or equal to LowerBound, searching the state information of the task instance of the current node in the warning message queue,
A. if the object instance in the READY state exists, executing an Active operation Active ([ NodeID ], [ obj instlist ]) aiming at the object instance;
B. if the activation operation is unsuccessful, modifying NodeStatus of AlarmNoList [ NodeID ] in the dynamic configuration warning data structure AlarmEdmin to OVERRIDE;
C. if there is no object instance in READY state, a task instance can be newly created, and Create ([ NodeID ], [ obj instlist ]) operation is executed, if the operation is unsuccessful, the NodeStatus of alarmnnodeld [ NodeID ] in dynamic configuration alarm data structure alarmmadmin needs to be modified to become over.
(5) And (5) repeating the step (4) until the AR > LowerBound.
(6) An Update ([ NodeID ], [ obj instlist ]) operation is performed.
(7) If the AR is more than or equal to SuperBound, searching the state information of the current node similar task instance in the warning message queue:
A. if there is an object instance in READY or OVERRIDE state, Delete operation Delete ([ NodeID ], [ obj instlist ]) is executed, if Delete is unsuccessful, then modify AlarmNodeList [ NodeID ] in dynamic configuration alarm data structure alarmmadmin, NodeStatus of which is OVERRIDE.
B. Repeating the step A until AR ≦ SuperBound or all the examples have the deletion operation.
C. If there is no object instance in READY or OVERRIDE state, performing parameter adjustment operation Adjust ([ NodeID ], [ obj instlist ], [ ReferenceList ]);
D. if the parameter adjustment operation is successful, step ③ is repeated until AR ≦ SuperBound.
E. If the parameter adjustment operation is unsuccessful, modifying the NodeStatus of AlarmNoList [ NodeID ] in the dynamic configuration warning data structure AlarmEdmin to OVERRIDE.
(8) An Update ([ NodeID ], [ obj instlist ]) operation is performed.
(9) If LowerBound is less than or equal to AR and less than or equal to SuperBound, a random number random is generated,
A、LowerBound＝LowerBound+random*S，
B、BestLowerBound:＝LowerBound；
C. if LowerBound is less than or equal to SuperBound, transferring to the step A;
D、SupperBound＝SupperBound-random*S
E、BestSupperBound:＝BestSupperBound；；
F. if AR is less than or equal to SuperBound, transferring to the step D;
(10) and outputting BestLowerBound and BestSuperBound of the upper and lower boundaries of the redundancy of the optimized object instance.
(11) And (4) starting the next instance ObjInstID of the same class object in the same node NodeID, and turning to the step (3).
After the iteration is finished, the runtime object instance in the node is kept in a high-level efficiency state.
As shown in fig. 3, in the dynamic maintenance for a node, that is, the status information of the node status in the node alert information list AlarmNodeList in the alarmdmin alert configuration management data structure in the dynamic configuration manager is dynamically maintained, and the main flow is as shown in fig. 3.
The main process is described as follows:
(1) and polling an alarm information list AlarmNodeList in Alarmadmin in the dynamic configuration manager in a preset timeWindow, and starting node-based dynamic maintenance aiming at an alarm message queue of NodeStatus.
(2) If NodeStatus in the AlarmNodeList [ NodeID ] is NORMAL, the dynamic maintenance of the redundancy of the object instance in the node is executed.
(3) If NodeStatus in AlarmNodeList [ NodeID ] is READY, object instance task can be created to execute Create ([ NodeID ], [ ObjInstList ]) operation.
A. If the creation is successful, the corresponding NodeStatus is modified to NORMAL.
B. If the creation is not successful, the corresponding NodeStatus is modified to OVERRIDE.
C. Perform Update ([ NodeID ]) operation
(4) If NodeStatus in AlarmNodeList [ NodeID ] is OVERRIDE, the node can be reloaded and the Overload ([ NodeID ]) operation is executed.
A. And if the overloading is successful, modifying the corresponding NodeStatus to READY.
B. If the reload is unsuccessful, a blanking SetNull ([ NodeID ]) operation is performed, followed by a Delete operation Delete ([ NodeID ]).
C. An Update ([ NodeID ]) operation is performed.
Different from the prior art, the invention has the following advantages in performance during dynamic configuration:
(1) when the node is in NORMAL, the object instance in the node needs to respectively execute reloading, deleting and activating operations in the optimization process. However, when the operations are executed, polling traversal is not performed, and network communication overhead in the system is reduced. In calculating the alert redundancy, the time complexity is high, but it is acceptable for the computing power of a large data analysis platform.
In the optimization process of the upper and lower boundaries of the object instance redundancy in the system node, a large cost is generated during the initial operation, but in the subsequent optimization process, the best boundary value in the previous optimization process can be used as an empirical value, so that the cost in the optimization process can be greatly saved, and the effect of rapid optimization configuration can be achieved.
(2) When the node is in a READY or overload state, because of the heavy load operation of the system node, the overhead of the system performance is large, and the communication overhead is relatively small.
In summary, the system performance in the present invention has better performance advantages compared to the conventional configuration process.
The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all equivalent changes made by using the contents of the present specification and the drawings, or applied directly or indirectly to the related technical fields, are included in the scope of the present invention.
Claims (5)
1. A dynamic configuration method of a big data analysis system is characterized by comprising the following steps:
s1: presetting a time window, presetting an alarm data structure by a dynamic configuration manager, and initializing;
s2: setting an experience initial value of an early warning redundancy lower bound and an early warning redundancy upper bound of an object instance and a parameter adjustment step constant according to the task type of the object instance in a node;
s3: calculating an early warning redundancy value of the object instance;
s4: determining that the early warning redundancy value is between the empirical initial values of a lower bound and an upper bound, and generating a random number;
s5: calculating an optimized upper bound value and an optimized lower bound value according to the step length, the random number, the empirical initial values of the upper bound and the lower bound;
s6: determining that the early warning redundancy value is located between an optimized lower bound value and an optimized upper bound value;
s7: polling an alarm information list in dynamic configuration management in a preset time window;
s8: and modifying the node state aiming at the warning information list of the node state so as to realize the dynamic maintenance of the node.
2. The dynamic configuration method of the big data analysis system according to claim 1, wherein the step S4 specifically comprises:
s41: judging whether the early warning redundancy value is larger than or equal to a lower-bound experience initial value or not;
if yes, go to S42: updating the object instance;
if not, executing S411: judging whether the object instance is in a ready state;
if yes, go to step S412: activating the object instance and returning to step S41;
if not, go to S413: creating a task instance and returning to step S41;
wherein after the step S42, it further comprises S43: judging whether the early warning redundancy value is smaller than or equal to an upper-bound empirical initial value or not;
if yes, go to S44: updating the object instance and generating a random number;
if not, executing S431: judging whether the object instance is in a ready state or a heavy load state;
if yes, go to S432: deleting the object instance and returning to step S43;
if not, executing S433: adjusts the parameters of the object instance, and returns to step S43.
3. The dynamic configuration method of big data analysis system according to claim 2, further comprising, after step S412:
s414: judging whether the activation is successful;
if yes, return to step S41;
otherwise, executing S415: setting the node state of the warning information list as a heavy load;
after step S413, the method further includes step S416: judging whether the creation is successful;
if yes, return to step S41;
otherwise, S415 is executed.
4. The dynamic configuration method of big data analysis system according to claim 2, further comprising, after step S432, step S434: judging whether the deletion is successful;
if yes, return to step S43;
otherwise, executing S415: setting the node state of the warning information list as a heavy load;
after step S433, the method further includes step S436: judging whether the adjustment is successful;
if yes, return to step S43;
otherwise, S415 is executed.
5. The dynamic configuration method of the big data analysis system according to claim 1, wherein step S5 specifically comprises:
s51: calculating the optimized lower bound value: optimizing a lower bound value which is an empirical initial value of the lower bound and + step size which is a random number;
s52: calculating the optimized upper bound value: and optimizing an upper bound value, namely an upper bound empirical initial value, namely step size, random number.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910332409.6A CN110222923A (en) | 2015-09-11 | 2015-09-11 | Dynamically configurable big data analysis system |
| CN201510577285.XA CN105279603B (en) | 2015-09-11 | 2015-09-11 | Dynamically configurable big data analysis system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510577285.XA CN105279603B (en) | 2015-09-11 | 2015-09-11 | Dynamically configurable big data analysis system and method |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910332409.6A Division CN110222923A (en) | 2015-09-11 | 2015-09-11 | Dynamically configurable big data analysis system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105279603A CN105279603A (en) | 2016-01-27 |
| CN105279603B true CN105279603B (en) | 2020-02-07 |
Family
ID=55148575
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510577285.XA Active CN105279603B (en) | 2015-09-11 | 2015-09-11 | Dynamically configurable big data analysis system and method |
| CN201910332409.6A Pending CN110222923A (en) | 2015-09-11 | 2015-09-11 | Dynamically configurable big data analysis system |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910332409.6A Pending CN110222923A (en) | 2015-09-11 | 2015-09-11 | Dynamically configurable big data analysis system |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN105279603B (en) |
Families Citing this family (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105760459B (en) * | 2016-02-04 | 2019-04-30 | 四川蓝光嘉宝服务集团股份有限公司 | A kind of distributed data processing system and method |
| CN107220261B (en) * | 2016-03-22 | 2020-10-30 | 中国移动通信集团山西有限公司 | Real-time mining method and device based on distributed data |
| CN105912582A (en) * | 2016-03-31 | 2016-08-31 | 畅捷通信息技术股份有限公司 | Control method for users' behavior analyses and control system for users' behavior analyses |
| WO2017201127A1 (en) * | 2016-05-17 | 2017-11-23 | Ab Initio Technology Llc | Reconfigurable distributed processing |
| CN107451147B (en) * | 2016-05-31 | 2020-07-31 | 北京京东尚科信息技术有限公司 | Method and device for dynamically switching kafka clusters |
| CN106126641B (en) * | 2016-06-24 | 2019-02-05 | 中国科学技术大学 | A kind of real-time recommendation system and method based on Spark |
| CN107621972A (en) * | 2016-07-15 | 2018-01-23 | 中兴通讯股份有限公司 | Big data task dynamic management approach, device and server |
| CN107918579B (en) * | 2016-10-09 | 2021-08-06 | 北京神州泰岳软件股份有限公司 | Method and device for generating baseline data in batches |
| CN106407472B (en) * | 2016-11-01 | 2019-08-20 | 广西电网有限责任公司电力科学研究院 | A kind of the big data calculating analysis task visual edit and management system of order form mode |
| WO2018098670A1 (en) * | 2016-11-30 | 2018-06-07 | 华为技术有限公司 | Method and apparatus for performing data processing |
| CN106776984B (en) * | 2016-12-02 | 2018-09-25 | 航天星图科技（北京）有限公司 | A kind of cleaning method of distributed system mining data |
| CN107294801B (en) * | 2016-12-30 | 2020-03-31 | 江苏号百信息服务有限公司 | Streaming processing method and system based on massive real-time internet DPI data |
| CN107145789B (en) * | 2017-05-22 | 2019-08-23 | 国网江苏省电力公司电力科学研究院 | A kind of Visual Interactive method of big data safety analysis |
| CN110019189A (en) * | 2017-09-18 | 2019-07-16 | 飞狐信息技术（天津）有限公司 | A kind of generation method and generation system of chart |
| CN107623737A (en) * | 2017-09-28 | 2018-01-23 | 南京轨道交通系统工程有限公司 | A kind of track traffic radio communication scheduling system and its design method |
| US10579943B2 (en) * | 2017-10-30 | 2020-03-03 | Accenture Global Solutions Limited | Engineering data analytics platforms using machine learning |
| CN108959954B (en) * | 2018-03-30 | 2021-11-12 | 努比亚技术有限公司 | Storm authority control method, device, server and storage medium |
| CN108494600B (en) * | 2018-03-30 | 2022-12-23 | 大唐丘北风电有限责任公司 | Topology creation control method, device and storage medium |
| CN109144508A (en) * | 2018-07-23 | 2019-01-04 | 北京科东电力控制系统有限责任公司 | It generates, the method and device of customization alarm picture |
| CN109343138B (en) * | 2018-09-29 | 2020-09-25 | 深圳市华讯方舟太赫兹科技有限公司 | Load balancing method of security inspection system and security inspection equipment |
| CN111245559B (en) * | 2018-11-29 | 2023-04-18 | 阿里巴巴集团控股有限公司 | Information determination method, information judgment method and device and computing equipment |
| CN109739925A (en) * | 2019-01-07 | 2019-05-10 | 北京云基数技术有限公司 | A kind of data processing system and method based on big data |
| CN112693502A (en) * | 2019-10-23 | 2021-04-23 | 上海宝信软件股份有限公司 | Urban rail transit monitoring system and method based on big data architecture |
| CN114285891B (en) * | 2021-12-15 | 2024-01-23 | 北京天融信网络安全技术有限公司 | SSLVPN-based session reconstruction method and system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6338072B1 (en) * | 1997-07-23 | 2002-01-08 | Bull S.A. | Device and process for dynamically controlling the allocation of resources in a data processing system |
| CN103903455A (en) * | 2014-04-14 | 2014-07-02 | 东南大学 | Urban road traffic signal control optimization system |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8769541B2 (en) * | 2009-12-31 | 2014-07-01 | Facebook, Inc. | Load balancing web service by rejecting connections |
| CN102043675B (en) * | 2010-12-06 | 2012-11-14 | 北京华证普惠信息股份有限公司 | Thread pool management method based on task quantity of task processing request |
| JP5853465B2 (en) * | 2011-07-27 | 2016-02-09 | 沖電気工業株式会社 | Network analysis system |
| CN102497292A (en) * | 2011-11-30 | 2012-06-13 | 中国科学院微电子研究所 | Computer cluster monitoring method and system thereof |
| CN103618644A (en) * | 2013-11-26 | 2014-03-05 | 曙光信息产业股份有限公司 | Distributed monitoring system based on hadoop cluster and method thereof |
| CN104317658B (en) * | 2014-10-17 | 2018-06-12 | 华中科技大学 | A kind of loaded self-adaptive method for scheduling task based on MapReduce |
| CN104375621A (en) * | 2014-11-28 | 2015-02-25 | 广东石油化工学院 | Dynamic weighting load assessment method based on self-adaptive threshold values in cloud computing |
| CN104615526A (en) * | 2014-12-05 | 2015-05-13 | 北京航空航天大学 | Monitoring system of large data platform |
| CN104579761B (en) * | 2014-12-24 | 2018-03-23 | 西安工程大学 | A kind of nosql clusters automatic configuration system and method for automatic configuration based on cloud computing |
-
2015
- 2015-09-11 CN CN201510577285.XA patent/CN105279603B/en active Active
- 2015-09-11 CN CN201910332409.6A patent/CN110222923A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6338072B1 (en) * | 1997-07-23 | 2002-01-08 | Bull S.A. | Device and process for dynamically controlling the allocation of resources in a data processing system |
| CN103903455A (en) * | 2014-04-14 | 2014-07-02 | 东南大学 | Urban road traffic signal control optimization system |
Non-Patent Citations (2)
| Title |
|---|
| 基于Xen虚拟机的内存资源实时监控与按需调整;胡耀，等;《计算机应用》;20130101;第33卷(第1期);全文 * |
| 基于分布式日志系统的数据云服务平台设计与实现;魏彬;《中国优秀硕士学位论文全文数据库 信息科技辑》;20140115(第01期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105279603A (en) | 2016-01-27 |
| CN110222923A (en) | 2019-09-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105279603B (en) | Dynamically configurable big data analysis system and method | |
| CN107734035B (en) | Virtual cluster automatic scaling method in cloud computing environment | |
| CN110794800B (en) | Intelligent factory information management monitoring system | |
| US11474874B2 (en) | Systems and methods for auto-scaling a big data system | |
| US8954971B2 (en) | Data collecting method, data collecting apparatus and network management device | |
| CN110287228B (en) | Method for realizing real-time data acquisition based on power grid dispatching domain equipment monitoring | |
| US10503636B2 (en) | Concurrent hierarchical dead actor collection in a distributed system | |
| CN101408861A (en) | Real time monitoring system and method of application program | |
| CN105159769A (en) | Distributed job scheduling method suitable for heterogeneous computational capability cluster | |
| CN113176948B (en) | Edge gateway, edge computing system and configuration method thereof | |
| CN108009258A (en) | It is a kind of can Configuration Online data collection and analysis platform | |
| CN103685309A (en) | Asynchronous request queue model oriented to map visualization tile service | |
| Zyrianoff et al. | Scalability of real-time iot-based applications for smart cities | |
| CN108737566B (en) | Distributed real-time message filtering system | |
| Bedini et al. | Modeling performance of a parallel streaming engine: bridging theory and costs | |
| CN102158545A (en) | Resource pool management method and device | |
| EP3422186A1 (en) | Method, device and system for preventing memory data loss | |
| CN109901948A (en) | Shared-nothing database cluster strange land dual-active disaster tolerance system | |
| CN101893876A (en) | Control network design for large-scale distributed control system | |
| CN112068940A (en) | Real-time task scheduling method, device, scheduling system and storage medium | |
| CN115964151A (en) | Flow calculation task scheduling system and method for big data processing | |
| CN116247816A (en) | Micro-grid automatic control system, method, device and computer equipment | |
| Zhou et al. | AHPA: adaptive horizontal pod autoscaling systems on alibaba cloud container service for kubernetes | |
| CN111538575B (en) | Resource scheduling system, method, device, equipment and medium | |
| CN113051061A (en) | Design of group-string matching stream type big data platform |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |