WO2016165471A1 - Hadoop集群组件指标的采集方法及装置 - Google Patents

Hadoop集群组件指标的采集方法及装置 Download PDF

Info

Publication number
WO2016165471A1
WO2016165471A1 PCT/CN2016/073955 CN2016073955W WO2016165471A1 WO 2016165471 A1 WO2016165471 A1 WO 2016165471A1 CN 2016073955 W CN2016073955 W CN 2016073955W WO 2016165471 A1 WO2016165471 A1 WO 2016165471A1
Authority
WO
WIPO (PCT)
Prior art keywords
indicator
value
node
time
component
Prior art date
Application number
PCT/CN2016/073955
Other languages
English (en)
French (fr)
Inventor
陈建伟
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2016165471A1 publication Critical patent/WO2016165471A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/14—Network analysis or design
    • H04L41/142—Network analysis or design using statistical or mathematical methods
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks

Definitions

  • the present invention relates to the field of communications, and in particular, to a method and an apparatus for collecting indicators of a Hadoop cluster component.
  • the big data management system based on Hadoop ecosystem (hereinafter referred to as big data management system) is one of the key systems of big data system.
  • resources management can be realized, such as cluster creation, HBase distributed database, HDFS file system, MR program resources, node resources, installation resources, user status, etc.; management of relationships can be realized through big data management system.
  • the status monitoring of Hadoop cluster operations and operations in the big data management system is an important guarantee for the normal operation of the big data management system.
  • the component monitoring in the cluster mainly involves the performance index collection technology.
  • the network management system performance indicators mainly include two common acquisition methods: passive collection and active polling acquisition.
  • the active polling acquisition algorithms in the related art mainly include: an equal time interval polling periodic acquisition algorithm.
  • the polling interval is determined to be difficult, the interval time is too long, and the real-time performance cannot be guaranteed. Short, increase the node host load and data warehousing summary pressure in the cluster, and even lead to the loss of collected data; based on the historical polling round-trip delay algorithm, the next polling interval is determined according to the historical polling request response time, the value and the network
  • the bandwidth state is related.
  • the polling time can be adjusted dynamically and dynamically, the network state information changing with time has a great influence on network performance and fault management, and the algorithm cannot dynamically represent the details of data changes.
  • the polling algorithm has also been improved.
  • a polling strategy based on a single prediction algorithm such as unary linear regression has emerged, which is based on the deviation and threshold of the predicted and actual values of the collected historical data. Contrast, dynamically adjust the polling time, can describe the data details and simple intelligent adjustment, but the single prediction algorithm will have problems such as applicable scene limitation and accuracy.
  • the one-dimensional linear regression algorithm is not ideal in the fluctuating data.
  • the index value of the calculation component such as Yarn fluctuates greatly, and the calculation will have a large error.
  • the metrics collection strategy ignores the load status of the nodes in the cluster and the Hadoop component characteristics. In the case of node overloading, there may even be cases where the component metrics collection task cannot be completed, thereby affecting the normal operation of the entire big data management system.
  • the embodiment of the invention provides a method and a device for collecting Hadoop cluster component indicators, so as to at least solve related technologies.
  • the problem of poor performance caused by node load and Hadoop component characteristics in the cluster is ignored.
  • a method for collecting an indicator of a Hadoop cluster component including: acquiring an index value of a first node of a Hadoop cluster component at a plurality of times before a specified time; And obtaining, by the indicator value of the multiple moments, a first indicator prediction value of the first node at the specified time; acquiring a plurality of load parameters of the first node, and the first indicator prediction value and the specified time Actually collecting the deviation value between the index values; performing weighting calculation on the plurality of load parameters and the deviation value according to the component type of the first node, and obtaining the collected result by weighting calculation according to the second predetermined rule
  • the collection time of the Hadoop cluster component indicator when the collection time is reached, the Hadoop cluster component indicator is collected and the Hadoop cluster component indicator is reported to the management system.
  • obtaining, by the first predetermined rule, the first indicator prediction value of the first node at the specified moment by using the indicator value of the multiple moments including: using an indicator value of the multiple moments according to the grayscale model a preliminary preliminary prediction value of the indicator at the specified time; according to the Markov chain model, the initial predicted deviation value of the indicator at the specified time is obtained by the preliminary predicted value of the indicator and the actual collected value at the specified time; The predicted deviation value and the preliminary predicted value of the index are used to calculate the predicted value of the first indicator.
  • obtaining the acquisition time of the Hadoop cluster component indicator according to the second predetermined rule includes: comparing the result of the weight calculation with a preset threshold; and obtaining a polling delay of the specified time according to the comparison result. Time, the sum of the polling delay time plus the preset polling cycle time is used as the acquisition time for collecting the Hadoop cluster component indicator values.
  • the load parameters of the first node include: CPU usage, memory usage, bandwidth usage, and transmission delay.
  • the component types of the first node include: a computing type and a storage type.
  • a device for collecting Hadoop cluster component indicators including: a first acquiring module, configured to acquire an index value of a first node of a Hadoop cluster component at a plurality of times before a specified time; a first processing module, configured to obtain, by the first predetermined rule, a first indicator predicted value of the first node at the specified time by using an indicator value of the multiple time points; and a second acquiring module configured to acquire the first a plurality of load parameters of a node, and a deviation value between the first indicator predicted value and an actual collected index value at the specified time; the second processing module is configured to perform a pair according to the component type of the first node Performing a weighting calculation on the plurality of load parameters and the deviation value, and obtaining an acquisition time of collecting the Hadoop cluster component indicators by using a result of the weighting calculation according to the second predetermined rule; and acquiring, when the collection time is reached, Collecting the Hadoop cluster component metrics and reporting the Hadoop cluster component metrics to the
  • the first processing module includes: a first processing unit, configured to obtain, by using an indicator value of the multiple moments according to the grayscale model, a preliminary preliminary prediction value of the specified time; the second processing unit is configured to Obtaining, by the Markov chain model, a preliminary prediction deviation value of the indicator at the specified time by using the preliminary predicted value of the indicator and the actual collected value at the specified time; the calculating unit is configured to initially predict the deviation value according to the indicator and the The preliminary predicted value of the indicator calculates the predicted value of the first indicator.
  • the second processing module includes: a comparing unit configured to compare the result of the weighting calculation with a preset threshold value; and the third processing unit is configured to obtain the specified time according to the comparison result
  • the polling delay time is the sum of the polling delay time plus the preset polling cycle time as the acquisition time for collecting the Hadoop cluster component indicator values.
  • the load parameters of the first node include: CPU usage, memory usage, bandwidth usage, and transmission delay.
  • the component types of the first node include: a computing type and a storage type.
  • the first indicator predicting value of the first node at the specified time is obtained by using the first predetermined rule by using the index value of the first node in the obtained Hadoop cluster component at a plurality of times before the specified time. And performing weighting calculation on the deviation value between the predicted value of the first indicator and the actual collected index value at the specified time, and the plurality of load parameters of the first node, and obtaining the result by using the second predetermined rule.
  • the acquisition time of the Hadoop cluster component metrics is collected, and the Hadoop cluster component metrics are collected and the Hadoop cluster component metrics are reported to the management system. It can be seen that in the invention, the node host load and the component to be collected are combined, and the cluster is dynamically adjusted.
  • the component metric collection time solves the problem of poor performance caused by ignoring the node load and Hadoop component characteristics in the cluster in the metric collection polling strategy in the related art.
  • FIG. 1 is a flowchart of a method for collecting Hadoop cluster component indicators according to an embodiment of the present invention
  • FIG. 2 is a structural block diagram of an apparatus for collecting HADOP cluster component indicators according to an embodiment of the present invention
  • FIG. 3 is a block diagram 1 of an optional structure of a device for collecting Hadoop cluster component indicators according to an embodiment of the present invention
  • FIG. 4 is a block diagram 2 of an optional structure of a device for collecting Hadoop cluster component indicators according to an embodiment of the present invention
  • FIG. 5 is a structural block diagram of a Hadoop component indicator polling acquisition module according to an alternative embodiment of the present invention.
  • FIG. 6 is a flow chart of a Hadoop component indicator polling delay time calculation in accordance with an alternate embodiment of the present invention.
  • FIG. 1 is a method according to an embodiment of the present invention.
  • a flowchart of a method for collecting Hadoop cluster component indicators is shown in FIG. 1 , and the process includes the following steps:
  • Step S102 Acquire an index value of the first node in the Hadoop cluster component at multiple times before the specified time
  • Step S104 Obtain a first indicator predicted value of the first node at a specified time according to the index value of the multiple time points according to the first predetermined rule;
  • Step S106 Acquire a plurality of load parameters of the first node, and a deviation value between the predicted value of the first indicator and the actual collected index value at the specified time;
  • Step S108 Perform weighting calculation on multiple load parameters and deviation values according to component types of the first node, and obtain acquisition time of collecting Hadoop cluster component indicators according to the second predetermined rule;
  • Step S110 When the collection time is reached, the Hadoop cluster component indicator is collected and the Hadoop cluster component indicator is reported to the management system.
  • the first indicator predicting value of the first node at the specified time is obtained by using the first predetermined rule by using the index value of the first node in the acquired Hadoop cluster component at a plurality of times before the specified time. And performing a weighting calculation on the deviation between the predicted value of the first indicator and the actual collected index value at the specified time, and the plurality of load parameters of the first node, and obtaining the result by using the second predetermined rule by the result of the weighting calculation
  • the collection time of the Hadoop cluster component indicator is collected, and the Hadoop cluster component indicator is collected and the Hadoop cluster component indicator is reported to the management system.
  • the node host load and the component to be collected are dynamically adjusted in this embodiment.
  • the collection time of the component indicators in the cluster solves the problem of poor performance caused by ignoring the node load and Hadoop component characteristics in the cluster in the metric collection and polling strategy.
  • the first indicator predicting value of the first node at the specified time is obtained by using the index value of the multiple time according to the first predetermined rule in the step S104. It is achieved as follows:
  • Step S11 obtaining a preliminary preliminary prediction value of the indicator at a specified time according to the indicator value of the plurality of times according to the grayscale model;
  • Step S12 obtaining a preliminary prediction deviation value of the indicator at the specified time according to the Markov chain model by using the preliminary predicted value of the index and the actual collected value at the specified time;
  • Step S13 Calculate the first index prediction value according to the preliminary prediction deviation value of the indicator and the preliminary prediction value of the index.
  • the first index prediction value of the specified time is obtained by the gray scale model and the Markov chain model, and the gray scale model and the Markov chain model can be applied to more scenes, so that The prediction value of the indicator is more accurate, that is, the solution of the present embodiment can solve the problem of different scene accuracy caused by using a single prediction algorithm in the indicator collection polling strategy in the related art.
  • the load parameter of the first node may be: CPU Usage, memory usage, bandwidth usage, and transmission delay; it should be noted that the load parameters involved in this embodiment are not limited to the above parameters, and the load parameters are preferred load parameters of this embodiment.
  • the component type package of the first node Includes: calculation and storage.
  • an application scenario in this embodiment may be: acquiring the load status of the first node host in the big data cluster at a specified time, such as the CPU usage rate ⁇ and the memory.
  • the method for obtaining the acquisition time of the Hadoop cluster component indicator according to the second predetermined rule in the step S108 in this embodiment may be implemented in the following manner in an optional implementation manner of the embodiment:
  • Step S21 comparing the result of the weighting calculation with a preset threshold value
  • Step S22 The polling delay time of the specified time is obtained according to the comparison result, and the sum of the polling delay time plus the preset polling cycle time is taken as the acquisition time of collecting the Hadoop cluster component index value.
  • the polling delay time can be 2s to 5s according to experience.
  • the algorithm for obtaining the acquisition time by using the preset threshold value and the polling delay time is more comprehensive, and the fixed period is reasonably avoided, that is, the related technology is solved.
  • the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases, the former is A better implementation.
  • the technical solution of the present invention which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk,
  • the optical disc includes a number of instructions for causing a terminal device (which may be a cell phone, a computer, a server, or a network device, etc.) to perform the methods of various embodiments of the present invention.
  • a device for collecting the components of the Hadoop cluster component is provided, and the device is used to implement the foregoing embodiments and preferred embodiments, and details are not described herein.
  • the term "module” may implement a combination of software and/or hardware of a predetermined function.
  • the apparatus described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware, is also possible and contemplated.
  • the device includes: a first obtaining module 202, configured to acquire a first node in a Hadoop cluster component before a specified time
  • the first processing module 204 is coupled to the first obtaining module 202, and configured to obtain, according to the first predetermined rule, the first index predicted value of the first node at the specified time by using the index value of the multiple time points
  • the second obtaining module 206 is coupled to the first processing module 204, and configured to acquire a plurality of load parameters of the first node, and a deviation value between the predicted value of the first indicator and the actual collected index value at the specified time
  • the module 208 is coupled to the second obtaining module 206, and is configured to perform weighted calculation on the plurality of load parameters and the deviation value according to the component type of the first node, and obtain the collected Hadoop cluster component index by using the result of the weighting calculation according to the second predetermined rule.
  • the acquisition module 208 is coupled to the second obtaining module 206, and is configured to perform weighted calculation on the plurality of
  • FIG. 3 is a block diagram of an optional structure of a device for collecting Hadoop cluster component indicators according to an embodiment of the present invention.
  • the first processing module 204 includes: a first processing unit 302, configured to pass according to a grayscale model.
  • the index value of the plurality of time points obtains the indicator preliminary prediction value at the specified time;
  • the second processing unit 304 is coupled to the first processing unit 302, and is set to obtain the actual acquisition value of the initial predicted value of the indicator and the specified time according to the Markov chain model.
  • the index to the specified time is preliminary predicted deviation value;
  • the calculation unit 306 is coupled to the second processing unit 304, and is configured to calculate the first indicator prediction value according to the indicator preliminary prediction deviation value and the indicator preliminary prediction value.
  • the second processing module 208 includes: a comparison unit 402 configured to calculate a weighted calculation result. The preset threshold value is compared; the third processing unit 404 is coupled to the comparison unit 402, and is configured to obtain a polling delay time at a specified time according to the comparison result, and add the polling delay time to the preset polling cycle time. And the acquisition time as the indicator value of the Hadoop cluster component.
  • the load parameters of the first node include: CPU usage, memory usage, bandwidth usage, and transmission delay.
  • the component types of the first node include: computational type and storage type.
  • FIG. 5 is a structural block diagram of a Hadoop component index polling collection module according to an alternative embodiment of the present invention. As shown in FIG. 5, the apparatus includes:
  • the master node summary module is configured to receive the indicator data collected by each node in the cluster in the cluster master node manager for aggregation and persistence processing;
  • the indicator collection module is configured to perform collection and transmission of indicator tasks on each node of the cluster.
  • the indicator prediction module is set to predict the new indicator value according to the component historical actual index by combining the gray model suitable for trend prediction and the Markov chain model suitable for large wave prediction on each node of the cluster.
  • the load collection module is configured to collect the current load status of the node host on each node of the cluster.
  • the polling calculation module is set to calculate the delay time required to perform the indicator collection task on each node of the cluster.
  • the metric prediction module When the node performs the component metric collection task, the metric prediction module first calculates the metric prediction value, and the load collection module collects the current node load, and the polling calculation module calculates the deviation according to the latest historical index prediction deviation, the node load, and the component type. The delay time required for the task to be executed, plus the set minimum polling collection period, is obtained by the execution of the indicator collection and transmission task time. After the time is reached, the indicator collection module collects component indicators and sends them to the master node summary module to complete an indicator collection and perform subsequent indicator collection tasks.
  • the optional embodiment further provides a Hadoop cluster component indicator polling collection method, and the steps of the method include:
  • Step S1 initializing
  • the polling acquisition strategy is set according to a fixed period, the minimum polling period of the component index is set, and the indicator collection task is started.
  • Step S2 predicting according to the component historical index value
  • the preliminary predicted value of the index obtained by using the gray GM (1, 1) model is used, and the percentage deviation between the predicted value and the collected actual value is calculated. Then, according to the historical index, the initial deviation value is predicted, and the Markov chain model is used to calculate the preliminary prediction deviation value, and the latest preliminary index prediction result of step 1 is corrected accordingly to obtain the final index prediction value.
  • Step S3 combining prediction value deviation, component type characteristic, and load weighting calculation
  • Step S4 comparing the threshold values to obtain the current polling time
  • the weighting calculation result is compared with the set threshold value, and according to the comparison result, the polling delay time of the secondary index acquisition is obtained, and the set minimum minimum polling period is obtained, and the execution time of the current index collection task is obtained.
  • Step S5 collecting component performance index values
  • the performance index value of the component is collected and sent to the big data management system.
  • the Hadoop cluster component index polling collection method and device in the alternative embodiment combined with the host load of the cluster in the cluster and the Hadoop component type to be collected in the big data management system, dynamically adjust the component index collection time in the Hadoop cluster, and the calculation algorithm is more comprehensive. Reasonable avoidance of fixed cycles brings many problems.
  • the polling algorithm combined with the grayscale model and the Markov chain model can be applied to more scenarios and is more accurate, improving the performance of the big data management system index monitoring.
  • FIG. 6 is a flowchart for calculating the polling delay time of the Hadoop component according to an optional embodiment of the present invention.
  • Figure, as shown in Figure 6, the steps of the method include:
  • Step S601 Acquire component historical indicator data
  • Step S602 Acquire a grayscale model prediction index
  • Step S603 Acquire a Markov chain model to correct the predicted value; then perform step S606;
  • Step S604 acquiring the component type; then performing step S606;
  • Step S605 Acquire a node load situation; then perform step S606;
  • Step S606 weighted addition and threshold comparison
  • Step S607 Polling the indicator collection delay time.
  • Steps S601 to S607 in Fig. 6 will be described in detail below in conjunction with the following steps.
  • the steps of the method include:
  • Step S41 The HDFS file system uses the capacity indicator (hereinafter referred to as the HDFS usage capacity) to collect the task, and sets the minimum fixed polling cycle time T1.
  • the capacity indicator hereinafter referred to as the HDFS usage capacity
  • Step S44 calculating the deviation prediction value y n at the time n according to the historical use capacity prediction value and the actual acquisition deviation value Y n-1 using the Markov chain model;
  • Step S45 Calculate the final capacity prediction value x n at the time n according to the preliminary predicted value X n ' of the HDFS use capacity and the n-time predicted deviation predicted value y n ;
  • Step S47 Acquire the load condition (CPU usage rate ⁇ , memory usage rate ⁇ , bandwidth usage rate ⁇ , transmission delay t) of the node changed in the big data cluster at time n, according to the characteristics of the HDFS storage type component, need to be reduced
  • the weight of memory usage and transmission delay, weighted calculation f a ⁇ z n-1 + b ⁇ ⁇ + c ⁇ ⁇ + d ⁇ t + k;
  • Step S48 comparing the calculated f value with each preset threshold value to obtain a polling delay time T at time n;
  • Step S49 After the T1+T time, collect the HDFS usage capacity index value, summarize it into the big data management system, and complete the big data indicator collection.
  • each of the above modules may be implemented by software or hardware.
  • the foregoing may be implemented by, but not limited to, the foregoing modules are all located in the same processor; or, the modules are located in multiple In the processor.
  • Embodiments of the present invention also provide a storage medium.
  • the foregoing storage medium may be Set to store the program code used to perform the following steps:
  • Step S1 Obtain an index value of the first node in the cluster component at multiple times before the specified time;
  • Step S2 obtaining, according to the first predetermined rule, the first indicator predicted value of the first node at the specified time by using the index value of the multiple moments;
  • Step S3 acquiring a plurality of load parameters of the first node, and a deviation value between the predicted value of the first indicator and the actual collected index value at the specified time;
  • Step S4 Perform weighting calculation on multiple load parameters and deviation values according to component types of the first node, and obtain acquisition time of the collection cluster component indicators according to the second predetermined rule;
  • Step S5 When the collection time is reached, the cluster component indicator is collected and the cluster component indicator is reported to the management system.
  • modules or steps of the present invention described above can be implemented by a general-purpose computing device that can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they may be implemented by program code executable by the computing device such that they may be stored in the storage device by the computing device and, in some cases, may be different from the order herein.
  • the steps shown or described are performed, or they are separately fabricated into individual integrated circuit modules, or a plurality of modules or steps thereof are fabricated as a single integrated circuit module.
  • the invention is not limited to any specific combination of hardware and software.
  • the first indicator predicting value of the first node at the specified time is obtained by using the first predetermined rule by using the index value of the first node in the obtained Hadoop cluster component at a plurality of times before the specified time. And performing weighting calculation on the deviation value between the predicted value of the first indicator and the actual collected index value at the specified time, and the plurality of load parameters of the first node, and obtaining the result by using the second predetermined rule.
  • the acquisition time of the Hadoop cluster component metrics is collected, and the Hadoop cluster component metrics are collected and the Hadoop cluster component metrics are reported to the management system. It can be seen that in the invention, the node host load and the component to be collected are combined, and the cluster is dynamically adjusted.
  • the component metric collection time solves the problem of poor performance caused by ignoring the node load and Hadoop component characteristics in the cluster in the metric collection polling strategy in the related art.

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Mathematical Physics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Pure & Applied Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Algebra (AREA)
  • Debugging And Monitoring (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

本发明提供了一种Hadoop集群组件指标的采集方法及装置，该方法包括：获取Hadoop集群组件中第一节点在指定时刻之前多个时刻的指标值；根据第一预定规则通过多个时刻的指标值得到第一节点在指定时刻的第一指标预测值；获取第一节点的多个负载参数，以及第一指标预测值与指定时刻的实际采集指标值之间的偏差值；根据第一节点的组件类型对多个负载参数以及偏差值进行加权计算，并根据第二预定规则通过加权计算的结果得到采集Hadoop集群组件指标的采集时间；在达到采集时间时，采集Hadoop集群组件指标并将Hadoop集群组件指标上报到管理系统。

Description

Hadoop集群组件指标的采集方法及装置 技术领域
本发明涉及通信领域，具体而言，涉及一种Hadoop集群组件指标的采集方法及装置。
背景技术
基于Hadoop生态圈的大数据管理系统(以下简称大数据管理系统)是大数据系统的关键系统之一。通过大数据管理系统可以实现对资源的管理，如集群创建、HBase分布式数据库、HDFS文件系统、MR程序资源、节点资源、安装资源、用户状态等；通过大数据管理系统可以实现对关系的管理，如用户与分布式文件目录的关系、MR程序资源与节点资源的关系、安装资源与节点资源的关系等；通过大数据管理系统可以实现对行为的管理和组件资源、性能指标的监控，如HDFS服务的DataNode的节点远程安装、服务的启动和停止和Yarn运行、作业状态监控等。
大数据管理系统中Hadoop集群的运行、作业等状态监控是大数据管理系统正常运行的重要保障。集群中组件监控主要涉及性能指标采集技术，目前，网络管理系统性能指标主要有被动收集和主动轮询采集两种常用采集方法。
相关技术中的主动轮询采集算法主要有：等时间间隔轮询周期性采集算法，虽然操作简单、实施方便，但是轮询间隔确定比较困难，间隔时间过长，无法保证实时性，间隔时间过短，增加集群中节点主机负载和数据入库汇总压力，甚至会导致采集数据丢失；基于历史轮询往返延迟的算法，主要根据历史轮询请求响应时间决定下一个轮询间隔，该值与网络带宽状态有关，虽然能够简单动态地调节轮询时间，但是随时间变化的网络状态信息对网络性能和故障管理有很大影响，而且该算法不能动态表示出数据变化的细节。随着人工智能、神经网络等技术的出现，轮询算法也有所改进，出现了基于一元线性回归等单一预测算法的轮询策略，都是通过将采集的历史数据预测值与实际值偏差与门限对比，动态调节轮询时间，能够刻画数据细节并且简单智能调节，但是单一预测算法会存在适用场景限制和准确性等问题，例如：一元线性回归算法在波动较大的数据中性能并不理想，而像Yarn这样计算型组件指标值波动较大，计算会出现较大误差。此外，相关技术中指标采集策略中忽略集群中节点的负载情况与Hadoop组件特性，在节点重载情况下，甚至会出现无法完成组件指标采集任务的情况，从而影响整个大数据管理系统正常运行。
针对相关技术中指标采集轮询策略中忽略集群中节点负载与Hadoop组件特性的导致性能不佳的问题，目前尚未存在有效的解决方案。
发明内容
本发明实施例提供了一种Hadoop集群组件指标的采集方法及装置，以至少解决相关技术 中指标采集轮询策略中忽略集群中节点负载与Hadoop组件特性的导致性能不佳的问题。
根据本发明实施例的一个方面，提供了一种Hadoop集群组件指标的采集方法，包括：获取Hadoop集群组件中第一节点在指定时刻之前多个时刻的指标值；根据第一预定规则通过所述多个时刻的指标值得到所述第一节点在所述指定时刻的第一指标预测值；获取所述第一节点的多个负载参数，以及所述第一指标预测值与所述指定时刻的实际采集指标值之间的偏差值；根据所述第一节点的组件类型对所述多个负载参数以及所述偏差值进行加权计算，并根据第二预定规则通过加权计算的结果得到采集所述Hadoop集群组件指标的采集时间；在达到所述采集时间时，采集所述Hadoop集群组件指标并将所述Hadoop集群组件指标上报到管理系统。
可选地，根据第一预定规则通过所述多个时刻的指标值得到所述第一节点在所述指定时刻的第一指标预测值包括：根据灰度模型通过所述多个时刻的指标值得到所述指定时刻的指标初步预测值；根据马尔科夫链模型通过所述指标初步预测值与所述指定时刻的实际采集值得到所述指定时刻的指标初步预测偏差值；根据所述指标初步预测偏差值与所述指标初步预测值计算出所述第一指标预测值。
可选地，根据第二预定规则得到所述Hadoop集群组件指标的采集时间包括：将加权计算得到的结果与预先设定的门限值进行比较；根据比较结果得到所述指定时刻的轮询延迟时间，将所述轮询延迟时间加上预设轮询周期时间的和作为采集Hadoop集群组件指标值的采集时间。
可选地，所述第一节点的所述负载参数包括：CPU使用率、内存使用率、带宽使用率、传输时延。
可选地，所述第一节点的组件类型包括：计算型和存储型。
根据本发明实施例的另一个方面，提供了一种Hadoop集群组件指标的采集装置，包括：第一获取模块，设置为获取Hadoop集群组件中第一节点在指定时刻之前多个时刻的指标值；第一处理模块，设置为根据第一预定规则通过所述多个时刻的指标值得到所述第一节点在所述指定时刻的第一指标预测值；第二获取模块，设置为获取所述第一节点的多个负载参数，以及所述第一指标预测值与所述指定时刻的实际采集指标值之间的偏差值；第二处理模块，设置为根据所述第一节点的组件类型对所述多个负载参数以及所述偏差值进行加权计算，并根据第二预定规则通过加权计算的结果得到采集所述Hadoop集群组件指标的采集时间；采集模块，设置为在达到所述采集时间时，采集所述Hadoop集群组件指标并将所述Hadoop集群组件指标上报到管理系统。
可选地，所述第一处理模块包括：第一处理单元，设置为根据灰度模型通过所述多个时刻的指标值得到所述指定时刻的指标初步预测值；第二处理单元，设置为根据马尔科夫链模型通过所述指标初步预测值与所述指定时刻的实际采集值得到所述指定时刻的指标初步预测偏差值；计算单元，设置为根据所述指标初步预测偏差值与所述指标初步预测值计算出所述第一指标预测值。
可选地，所述第二处理模块包括：比较单元，设置为将加权计算得到的结果与预先设定的门限值进行比较；第三处理单元，设置为根据比较结果得到所述指定时刻的轮询延迟时间，将所述轮询延迟时间加上预设轮询周期时间的和作为采集Hadoop集群组件指标值的采集时间。
可选地，所述第一节点的所述负载参数包括：CPU使用率、内存使用率、带宽使用率、传输时延。
可选地，所述第一节点的组件类型包括：计算型和存储型。
在本发明实施例中，采用将获取到的Hadoop集群组件中第一节点在指定时刻之前多个时刻的指标值通过第一预定规则得到该第一节点在该指定时刻的第一指标预测值，通该第一指标预测值与该指定时刻的实际采集指标值之间的偏差值与该第一节点的多个负载参数进行加权计算，将该加权计算的结果通过第二预定规则得到该采集该Hadoop集群组件指标的采集时间，在达到采集时间时，采集Hadoop集群组件指标并将Hadoop集群组件指标上报到管理系统；可见，在发明中中结合节点主机负载与待采集组件类型，动态调节集群中组件指标采集时间，从而解决了相关技术中指标采集轮询策略中忽略集群中节点负载与Hadoop组件特性的导致性能不佳的问题。
附图说明
此处所说明的附图用来提供对本发明的进一步理解，构成本申请的一部分，本发明的示意性实施例及其说明用于解释本发明，并不构成对本发明的不当限定。在附图中：
图1是根据本发明实施例的Hadoop集群组件指标的采集方法的流程图；
图2是根据本发明实施例的Hadoop集群组件指标的采集装置的结构框图；
图3是根据本发明实施例的Hadoop集群组件指标的采集装置的可选结构框图一；
图4是根据本发明实施例的Hadoop集群组件指标的采集装置的可选结构框图二；
图5是根据本发明可选实施例的Hadoop组件指标轮询采集模块结构框图；
图6是根据本发明可选实施例的Hadoop组件指标轮询延迟时间计算流程图。
具体实施方式
下文中将参考附图并结合实施例来详细说明本发明。需要说明的是，在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互组合。
需要说明的是，本发明的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。
在本实施例中提供了一种Hadoop集群组件指标的采集方法，图1是根据本发明实施例的 Hadoop集群组件指标的采集方法的流程图，如图1所示，该流程包括如下步骤：
步骤S102：获取Hadoop集群组件中第一节点在指定时刻之前多个时刻的指标值；
步骤S104：根据第一预定规则通过多个时刻的指标值得到第一节点在指定时刻的第一指标预测值；
步骤S106：获取第一节点的多个负载参数，以及第一指标预测值与指定时刻的实际采集指标值之间的偏差值；
步骤S108：根据第一节点的组件类型对多个负载参数以及偏差值进行加权计算，并根据第二预定规则得到采集Hadoop集群组件指标的采集时间；
步骤S110：在达到采集时间时，采集Hadoop集群组件指标并将Hadoop集群组件指标上报到管理系统。
通过上述步骤S102至步骤S110，采用将获取到的Hadoop集群组件中第一节点在指定时刻之前多个时刻的指标值通过第一预定规则得到该第一节点在该指定时刻的第一指标预测值，通该第一指标预测值与该指定时刻的实际采集指标值之间的偏差值与该第一节点的多个负载参数进行加权计算，将该加权计算的结果通过第二预定规则得到该采集该Hadoop集群组件指标的采集时间，在达到采集时间时，采集Hadoop集群组件指标并将Hadoop集群组件指标上报到管理系统；可见，在本实施例中结合节点主机负载与待采集组件类型，动态调节集群中组件指标采集时间，从而解决了相关技术中指标采集轮询策略中忽略集群中节点负载与Hadoop组件特性的导致性能不佳的问题。
对于本实施例中步骤S104涉及到的根据第一预定规则通过多个时刻的指标值得到第一节点在指定时刻的第一指标预测值的方式，在本实施例的一个可选实施方式中，通过如下方式来实现：
步骤S11：根据灰度模型通过多个时刻的指标值得到指定时刻的指标初步预测值；
步骤S12：根据马尔科夫链模型通过指标初步预测值与指定时刻的实际采集值得到指定时刻的指标初步预测偏差值；
步骤S13：根据指标初步预测偏差值与指标初步预测值计算出第一指标预测值。
由上述步骤S11至步骤S13可知，采用了通过灰度模型和马尔科夫链模型得到该指定时刻的第一指标预测值，而灰度模型和马尔科夫链模型能够适用更多场景，使得得到的指标预测值更加准确，也就是通过上述本实施例的方案，能够解决相关技术中的指标采集轮询策略中使用单一预测算法的导致的不同场景准确性问题。
对于本实施例步骤S108中的根据第一节点的组件类型对多个负载参数以及偏差值θ进行加权计算，在本实施例的可选实施方式中，该第一节点的负载参数可以是：CPU使用率、内存使用率、带宽使用率、传输时延；需要说明的是，本实施例中涉及到的负载参数并不仅仅限于上述参数，上述负载参数是本实施例的优选负载参数。此外，该第一节点的组件类型包 括：计算型和存储型。
对于本实施例中的步骤S108中涉及到的上述方式，在本实施例的一个应用场景可以是：在指定时刻获取大数据集群中该第一节点主机的负载情况，如CPU使用率α、内存使用率β、带宽使用率χ、传输时延t，根据组件类型中的存储型组件的特性，需要减少内存使用率和传输时延的权重，给定不同的加权系数加权计算f＝a·θ+b·α+c·β+d·χ+k·t。
而对于本实施例中的步骤S108中涉及到的根据第二预定规则得到Hadoop集群组件指标的采集时间的方式，在本实施例的一个可选实施方式中，可以通过如下方式来实现：
步骤S21：将加权计算得到的结果与预先设定的门限值进行比较；
步骤S22：根据比较结果得到指定时刻的轮询延迟时间，将轮询延迟时间加上预设轮询周期时间的和作为采集Hadoop集群组件指标值的采集时间。
其中，该轮询延迟时间可以根据经验取值为2s至5s。
由上述步骤S21和步骤S22中，采用了通过预先设定的门限值与轮询延迟时间使得得到采集时间的算法更加全面，合理避免了固定周期带来诸多问题，也就是说解决了相关技术中大数据管理系统组件指标采集中存在的固定周期性采集性能数据存在的实时性和效率问题。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到根据上述实施例的方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，或者网络设备等)执行本发明各个实施例的方法。
在本实施例中还提供了一种Hadoop集群组件指标的采集装置，该装置用于实现上述实施例及优选实施方式，已经进行过说明的不再赘述。如以下所使用的，术语“模块”可以实现预定功能的软件和/或硬件的组合。尽管以下实施例所描述的装置较佳地以软件来实现，但是硬件，或者软件和硬件的组合的实现也是可能并被构想的。
图2是根据本发明实施例的Hadoop集群组件指标的采集装置的结构框图，如图2所示，该装置包括：第一获取模块202，设置为获取Hadoop集群组件中第一节点在指定时刻之前多个时刻的指标值；第一处理模块204，与第一获取模块202耦合连接，设置为根据第一预定规则通过多个时刻的指标值得到第一节点在指定时刻的第一指标预测值；第二获取模块206，与第一处理模块204耦合连接，设置为获取第一节点的多个负载参数，以及第一指标预测值与指定时刻的实际采集指标值之间的偏差值；第二处理模块208，与第二获取模块206耦合连接，设置为根据第一节点的组件类型对多个负载参数以及偏差值进行加权计算，并根据第二预定规则通过加权计算的结果得到采集Hadoop集群组件指标的采集时间；采集模块210，与第二处理模块208耦合连接，设置为在达到采集时间时，采集Hadoop集群组件指标并将Hadoop 集群组件指标上报到管理系统。
图3是根据本发明实施例的Hadoop集群组件指标的采集装置的可选结构框图一，如图3所示，该第一处理模块204包括：第一处理单元302，设置为根据灰度模型通过多个时刻的指标值得到指定时刻的指标初步预测值；第二处理单元304，与第一处理单元302耦合连接，设置为根据马尔科夫链模型通过指标初步预测值与指定时刻的实际采集值得到指定时刻的指标初步预测偏差值；计算单元306，与第二处理单元304耦合连接，设置为根据指标初步预测偏差值与指标初步预测值计算出第一指标预测值。
图4是根据本发明实施例的Hadoop集群组件指标的采集装置的可选结构框图二，如图4所示，该第二处理模块208包括：比较单元402，设置为将加权计算得到的结果与预先设定的门限值进行比较；第三处理单元404，与比较单元402耦合连接，设置为根据比较结果得到指定时刻的轮询延迟时间，将轮询延迟时间加上预设轮询周期时间的和作为采集Hadoop集群组件指标值的采集时间。
可选地，第一节点的负载参数包括：CPU使用率、内存使用率、带宽使用率、传输时延。该第一节点的组件类型包括：计算型和存储型。
下面通过本发明的可选实施例对本发明进行举例说明；
本可选实施例提供了一种Hadoop集群组件指标轮询采集模块，图5是根据本发明可选实施例的Hadoop组件指标轮询采集模块结构框图，如图5所示，该装置包括：
主节点汇总模块，设置为在集群主节点Manager中，接收集群中各个节点主机轮询采集的指标数据，进行汇总，持久化处理；
指标采集模块，设置为在集群各个节点上，负责执行采集与发送指标任务。
指标预测模块，设置为在集群各个节点上，结合适合趋势预测的灰度模型和适合大波动预测的马尔科夫链模型，根据组件历史实际指标预测新的指标值。
负载采集模块，设置为在集群各个节点上，采集节点主机当前负载情况。
轮询计算模块，设置为在集群各个节点上，负责计算出执行指标采集任务需要的延迟时间。
当节点执行组件指标采集任务时，首先由指标预测模块计算指标预测值，同时负载采集模块会采集当前节点负载，轮询计算模块根据最近一次历史指标预测偏差、节点负载以及组件类型，加权计算出当次执行任务需要的延迟时间，加上设定的最小轮询采集周期，得到执行指标采集发送任务时间。到达该时间后，指标采集模块采集组件指标，并且向主节点汇总模块发送，完成一次指标采集，依次进行之后的指标采集任务。
此外，基于该Hadoop集群组件指标轮询采集模块本可选实施例还提供了一种Hadoop集群组件指标轮询采集方法，该方法的步骤包括：
步骤S1，初始化；
其中，按照固定周期轮询采集策略，设定组件指标采集最小轮询周期，开始指标采集任务。
步骤S2，根据组件历史指标值预测；
其中，首先根据组件历史指标值，使用灰度GM(1，1)模型得到的指标初步预测值，计算该预测值与采集的实际值偏差百分比。然后，根据历史指标初步预测偏差值，使用马尔可夫链模型，计算得到初步预测偏差值，并且据此对步骤1的最新的初步指标预测结果进行修正，得到最终指标预测值。
步骤S3，结合预测值偏差、组件类型特性、负载加权计算；
其中，计算上述最终预测值与实际采集值的偏差百分比，获取大数据集群中该节点主机当前负载情况(包括CPU使用率，内存使用率，带宽使用率，传输时延)，按照计算型、存储型的组件类型，给定不同的加权系数，加权计算。
步骤S4，门限值对比得到当次轮询时间；
其中，加权计算结果与设定的门限值比较，根据对比结果得到对应当次指标采集轮询延迟时间，加上设定的最小固定轮询周期，得到当次指标采集任务执行时间。
步骤S5，采集组件性能指标值；
其中，到达执行采集指标任务时间后，采集该组件性能指标值，发送汇总到大数据管理系统。
通过本可选实施例的Hadoop集群组件指标轮询采集方法及装置，结合大数据管理系统中集群中节点主机负载与待采集Hadoop组件类型，动态调节Hadoop集群中组件指标采集时间，计算算法更加全面，合理避免固定周期带来诸多问题，其中轮询算法结合灰度模型与马尔科夫链模型，能够适用更多场景，更加准确，提升了大数据管理系统指标监控的性能。
下面结合图6和HDFS文件系统使用容量这一指标采集样例和具体实施例对本发明可选实施例进行举例说明，图6是根据本发明可选实施例的Hadoop组件指标轮询延迟时间计算流程图，如图6所示，该方法的步骤包括：
步骤S601：获取组件历史指标数据；
步骤S602：获取灰度模型预测指标；
步骤S603：获取马尔科夫链模型对预测值修正；之后执行步骤S606；
步骤S604：获取组件类型；之后执行步骤S606；
步骤S605：获取节点负载情况；之后执行步骤S606；
步骤S606：加权相加与门限比较；
步骤S607：轮询指标采集延迟时间。
下面结合以下步骤对图6中的步骤S601至步骤S607进行详细的描述。
该方法的步骤包括：
步骤S41：HDFS文件系统使用容量指标(以下简称HDFS使用容量)采集任务，设定最小固定轮询周期时间T1。
步骤S42：根据HDFS使用容量历史指标值Xn-1＝{x1,,x2,...,xn-1}，使用经典灰度模型GM(1，1)，计算得到在n时刻的使用容量的初步预测值Xn'；
步骤S43：计算HDFS使用容量历史每个采集时刻经过灰度模型初步预测值与实际采集值偏差Yn-1＝Xn-1-Xn-1'/Xn-1；
步骤S44：根据历史使用容量预测值与实际采集偏差值Yn-1，使用马尔科夫链模型，计算在n时刻偏差预测值yn；
步骤S45：根据HDFS使用容量的初步预测值Xn'与n时刻预测偏差预测值yn，计算出n时刻最终的容量预测值xn；
步骤S46：计算最近一次即n-1时刻，经过灰度模型以及马尔科夫链模型预测的使用容量值与实际采集的偏差zn-1＝xn-1-xn-1’/xn-1；
步骤S47：获取在n时刻，大数据集群中改节点主机的负载情况(CPU使用率α，内存使用率β，带宽使用率χ，传输时延t)，根据HDFS存储型组件的特性，需要减少内存使用率和传输时延的权重，加权计算f＝a·zn-1+b·α+c·β+d·t+k；
步骤S48：将计算得到的f值，与预先设定的各个门限值对比，得到在n时刻轮询延迟时间T；
步骤S49：在T1+T时间后，采集HDFS使用容量指标值，汇总到大数据管理系统中，完成大数据指标采集。
需要说明的是，上述各个模块是可以通过软件或硬件来实现的，对于后者，可以通过以下方式实现，但不限于此：上述模块均位于同一处理器中；或者，上述模块分别位于多个处理器中。
本发明的实施例还提供了一种存储介质。可选地，在本实施例中，上述存储介质可以被 设置为存储用于执行以下步骤的程序代码：
步骤S1：获取集群组件中第一节点在指定时刻之前多个时刻的指标值；
步骤S2：根据第一预定规则通过多个时刻的指标值得到第一节点在指定时刻的第一指标预测值；
步骤S3：获取第一节点的多个负载参数，以及第一指标预测值与指定时刻的实际采集指标值之间的偏差值；
步骤S4：根据第一节点的组件类型对多个负载参数以及偏差值进行加权计算，并根据第二预定规则得到采集集群组件指标的采集时间；
步骤S5：在达到采集时间时，采集集群组件指标并将集群组件指标上报到管理系统。
可选地，本实施例中的具体示例可以参考上述实施例及可选实施方式中所描述的示例，本实施例在此不再赘述。
显然，本领域的技术人员应该明白，上述的本发明的各模块或各步骤可以用通用的计算装置来实现，它们可以集中在单个的计算装置上，或者分布在多个计算装置所组成的网络上，可选地，它们可以用计算装置可执行的程序代码来实现，从而，可以将它们存储在存储装置中由计算装置来执行，并且在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤，或者将它们分别制作成各个集成电路模块，或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。这样，本发明不限制于任何特定的硬件和软件结合。
以上所述仅为本发明的优选实施例而已，并不用于限制本发明，对于本领域的技术人员来说，本发明可以有各种更改和变化。凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。
工业实用性
在本发明实施例中，采用将获取到的Hadoop集群组件中第一节点在指定时刻之前多个时刻的指标值通过第一预定规则得到该第一节点在该指定时刻的第一指标预测值，通该第一指标预测值与该指定时刻的实际采集指标值之间的偏差值与该第一节点的多个负载参数进行加权计算，将该加权计算的结果通过第二预定规则得到该采集该Hadoop集群组件指标的采集时间，在达到采集时间时，采集Hadoop集群组件指标并将Hadoop集群组件指标上报到管理系统；可见，在发明中中结合节点主机负载与待采集组件类型，动态调节集群中组件指标采集时间，从而解决了相关技术中指标采集轮询策略中忽略集群中节点负载与Hadoop组件特性的导致性能不佳的问题。

Claims (10)

  1. 一种Hadoop集群组件指标的采集方法，包括：
    获取Hadoop集群组件中第一节点在指定时刻之前多个时刻的指标值；
    根据第一预定规则通过所述多个时刻的指标值得到所述第一节点在所述指定时刻的第一指标预测值；
    获取所述第一节点的多个负载参数，以及所述第一指标预测值与所述指定时刻的实际采集指标值之间的偏差值；
    根据所述第一节点的组件类型对所述多个负载参数以及所述偏差值进行加权计算，并根据第二预定规则通过加权计算的结果得到采集所述Hadoop集群组件指标的采集时间；
    在达到所述采集时间时，采集所述Hadoop集群组件指标并将所述Hadoop集群组件指标上报到管理系统。
  2. 根据权利要求1所述的方法，其中，根据第一预定规则通过所述多个时刻的指标值得到所述第一节点在所述指定时刻的第一指标预测值包括：
    根据灰度模型通过所述多个时刻的指标值得到所述指定时刻的指标初步预测值；
    根据马尔科夫链模型通过所述指标初步预测值与所述指定时刻的实际采集值得到所述指定时刻的指标初步预测偏差值；
    根据所述指标初步预测偏差值与所述指标初步预测值计算出所述第一指标预测值。
  3. 根据权利要求1所述的方法，其中，根据第二预定规则得到所述Hadoop集群组件指标的采集时间包括：
    将加权计算得到的结果与预先设定的门限值进行比较；
    根据比较结果得到所述指定时刻的轮询延迟时间，将所述轮询延迟时间加上预设轮询周期时间的和作为采集Hadoop集群组件指标值的采集时间。
  4. 根据权利要求1所述的方法，其中，所述第一节点的所述负载参数包括：CPU使用率、内存使用率、带宽使用率、传输时延。
  5. 根据权利要求1所述的方法，其中，所述第一节点的组件类型包括：计算型和存储型。
  6. 一种Hadoop集群组件指标的采集装置，包括：
    第一获取模块，设置为获取Hadoop集群组件中第一节点在指定时刻之前多个时刻的指标值；
    第一处理模块，设置为根据第一预定规则通过所述多个时刻的指标值得到所述第一节点在所述指定时刻的第一指标预测值；
    第二获取模块，设置为获取所述第一节点的多个负载参数，以及所述第一指标预测值与所述指定时刻的实际采集指标值之间的偏差值；
    第二处理模块，设置为根据所述第一节点的组件类型对所述多个负载参数以及所述偏差值进行加权计算，并根据第二预定规则通过加权计算的结果得到采集所述Hadoop集群组件指标的采集时间；
    采集模块，设置为在达到所述采集时间时，采集所述Hadoop集群组件指标并将所述Hadoop集群组件指标上报到管理系统。
  7. 根据权利要求6所述的装置，其中，所述第一处理模块包括：
    第一处理单元，设置为根据灰度模型通过所述多个时刻的指标值得到所述指定时刻的指标初步预测值；
    第二处理单元，设置为根据马尔科夫链模型通过所述指标初步预测值与所述指定时刻的实际采集值得到所述指定时刻的指标初步预测偏差值；
    计算单元，设置为根据所述指标初步预测偏差值与所述指标初步预测值计算出所述第一指标预测值。
  8. 根据权利要求6所述的装置，其中，所述第二处理模块包括：
    比较单元，设置为将加权计算得到的结果与预先设定的门限值进行比较；
    第三处理单元，设置为根据比较结果得到所述指定时刻的轮询延迟时间，将所述轮询延迟时间加上预设轮询周期时间的和作为采集Hadoop集群组件指标值的采集时间。
  9. 根据权利要求6所述的装置，其中，所述第一节点的所述负载参数包括：CPU使用率、内存使用率、带宽使用率、传输时延。
  10. 根据权利要求6所述的装置，其中，所述第一节点的组件类型包括：计算型和存储型。
PCT/CN2016/073955 2015-09-15 2016-02-17 Hadoop集群组件指标的采集方法及装置 WO2016165471A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510585652.0A CN106533730B (zh) 2015-09-15 2015-09-15 Hadoop集群组件指标的采集方法及装置
CN201510585652.0 2015-09-15

Publications (1)

Publication Number Publication Date
WO2016165471A1 true WO2016165471A1 (zh) 2016-10-20

Family

ID=57127106

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/073955 WO2016165471A1 (zh) 2015-09-15 2016-02-17 Hadoop集群组件指标的采集方法及装置

Country Status (2)

Country Link
CN (1) CN106533730B (zh)
WO (1) WO2016165471A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220100769A1 (en) * 2020-09-29 2022-03-31 Cerner Innovation, Inc. System and method for improved state identification and prediction in computerized queries

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112425122A (zh) * 2018-07-17 2021-02-26 脸谱公司 检测警报的通信网络洞察力

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102923538A (zh) * 2012-07-06 2013-02-13 天津大学 基于物联网的电梯健康管理与维护系统及采集和评估方法
CN103618644A (zh) * 2013-11-26 2014-03-05 曙光信息产业股份有限公司 一种基于hadoop集群的分布式监控系统及其方法
US20140064066A1 (en) * 2012-08-29 2014-03-06 Nec Laboratories America, Inc. Data Processing
CN104037943A (zh) * 2014-06-18 2014-09-10 东北大学 一种提高电网电压质量的电压监测方法及系统

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101841565B (zh) * 2010-04-20 2013-07-31 中国科学院软件研究所 数据库集群系统负载均衡方法和数据库集群系统
CN103178990A (zh) * 2011-12-20 2013-06-26 中国移动通信集团青海有限公司 一种网络设备性能监控方法及网络管理系统
CN103605418B (zh) * 2013-10-23 2017-01-04 曙光信息产业(北京)有限公司 集群服务器的功耗调节方法和装置
CN103970587B (zh) * 2014-04-21 2018-02-13 华为技术有限公司 一种资源调度的方法、设备和系统
CN104750780B (zh) * 2015-03-04 2017-10-13 北京航空航天大学 一种基于统计分析的Hadoop配置参数优化方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102923538A (zh) * 2012-07-06 2013-02-13 天津大学 基于物联网的电梯健康管理与维护系统及采集和评估方法
US20140064066A1 (en) * 2012-08-29 2014-03-06 Nec Laboratories America, Inc. Data Processing
CN103618644A (zh) * 2013-11-26 2014-03-05 曙光信息产业股份有限公司 一种基于hadoop集群的分布式监控系统及其方法
CN104037943A (zh) * 2014-06-18 2014-09-10 东北大学 一种提高电网电压质量的电压监测方法及系统

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220100769A1 (en) * 2020-09-29 2022-03-31 Cerner Innovation, Inc. System and method for improved state identification and prediction in computerized queries
US11893030B2 (en) * 2020-09-29 2024-02-06 Cerner Innovation, Inc. System and method for improved state identification and prediction in computerized queries

Also Published As

Publication number Publication date
CN106533730B (zh) 2020-07-31
CN106533730A (zh) 2017-03-22

Similar Documents

Publication Publication Date Title
CN112988398B (zh) 一种微服务动态伸缩及迁移方法和装置
US20190391897A1 (en) Adaptive thresholds for containers
US20230224752A1 (en) Communication method, apparatus, and system
CN109120463B (zh) 流量预测方法及装置
CN112749056A (zh) 应用服务指标监控方法、装置、计算机设备和存储介质
CN115982265A (zh) 数据趋势检测方法、电子设备和存储介质
CN111277640B (zh) 用户请求处理方法、装置、系统、计算机设备和存储介质
CN106330558A (zh) 应用于软件定义网络的控制器负载预测系统及方法
CN105491079B (zh) 云计算环境中调整应用所需资源的方法及装置
US8180716B2 (en) Method and device for forecasting computational needs of an application
CN116643844B (zh) 面向电力超算云资源自动扩展的智能化管理系统及方法
WO2016165471A1 (zh) Hadoop集群组件指标的采集方法及装置
US11295224B1 (en) Metrics prediction using dynamic confidence coefficients
CN110191015B (zh) 基于cpi指标的云服务性能智能预测方法和装置
US11146447B2 (en) Method and apparatus of establishing computer network monitoring criteria
EP3025452B1 (en) Monitoring network use of resources
CN109560978B (zh) 网络流量检测方法、装置及系统和计算机可读存储介质
CN112073329B (zh) 分布式限流方法、装置、电子设备和存储介质
US20070288475A1 (en) Method for processing multiple continuous top-K queries
CN116546029A (zh) 负载均衡方法、装置、存储介质及计算机设备
US10089149B2 (en) Method for scheduling multiple periodic requests and scheduling device
CN115185683A (zh) 一种基于动态优化模型的云平台流处理资源分配方法
US10091068B1 (en) System for modeling distributed systems
CN114896296A (zh) 云服务资源配置方法、装置、电子设备及计算机可读介质
CN118113443B (zh) 任务调度方法、系统、程序产品、设备和介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16779446

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16779446

Country of ref document: EP

Kind code of ref document: A1