CN108712465A - Big data platform monitoring method - Google Patents
Big data platform monitoring method Download PDFInfo
- Publication number
- CN108712465A CN108712465A CN201810332243.3A CN201810332243A CN108712465A CN 108712465 A CN108712465 A CN 108712465A CN 201810332243 A CN201810332243 A CN 201810332243A CN 108712465 A CN108712465 A CN 108712465A
- Authority
- CN
- China
- Prior art keywords
- data
- flow
- big data
- data platform
- acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3055—Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
Abstract
Big data platform monitoring method, is related to computer technology.The present invention includes the following steps：1) the acquisition state data from big data platform, the status data include component states data；2) calculation processing is carried out according to set rule to the result of acquisition, and stores the result of calculating；3) if result of calculation meets alert if, then pass through api interface alert, which is characterized in that in the step 1), status data further includes flow system flow data, the flow system flow data include each flow of big data platform, component flow system flow.The invention has the advantages that can comprehensively, real time monitoring cluster, the operating condition that operation is calculated on node and node, note abnormalities and alert in time, ensure smoothly completing for scheduler task, there is higher sensitivity and reliability.
Description
Technical field
The present invention relates to computer technologies.
Background technology
For current magnanimity and fast changing big data, storage has not been final target, how from number
According to bonus of the middle acquisition including commercial value, it is only where real significance.By establishing big data platform, higher is obtained
Data value, this is only meaning of the data for enterprise.Although the final task of big data platform is not storage, data
Storage is basis.For the particularity of business, often can all Extract (transmission)-be carried out to data before data storage
Transform (conversion)-Load (load), this process are referred to as ETL process process.Due to the diversity of business, data are final
Before storage, other flow chart of data processing, such as data cleansing are often further comprised.
Since constantly having enterprise's framework big data platform, in order to improve working platform efficiency and fault discovery it is timely
Property, have become a bright spot of a big data platform for the monitoring alarm of health status.
Many enterprises do not carry out actual development for the monitoring alarm of big data platform at present, are often based on big
The application of data platform there is a problem, just manually be excluded.For the monitoring alarm work for carrying out big data platform
, common design method is that acquisition-storage-displaying-alarm, i.e., (system itself runs shape to the monitoring data acquired first
State, such as CPU, memory, disk, Internet Use etc.；The operating condition of various applications, such as database, container calculate operation
Deng).Storage is to preserve collection result by storage scheme.Displaying is to open up these data in web interface
Show, the situation of change of monitor control index is visualized.Alarm is to send out warning information in the form of mail, short message, wechat etc.
It goes.Such design (calculates operation, number just for system (CPU, memory, disk, big data platform component etc.), function
According to ETL etc.) component monitored, lack because data traffic lose or data traffic it is overstocked situations such as lead to big number
The monitoring alerted according to plateform system flow processing.
Invention content
The technical problem to be solved by the invention is to provide a kind of big data platform monitoring methods, are carried to big data platform
For more comprehensive monitoring.
The present invention solve the technical problem the technical solution adopted is that, big data platform monitoring method, including following steps
Suddenly：
1) the acquisition state data from big data platform, the status data include component states data；
2) calculation processing is carried out according to set rule to the result of acquisition, and stores the result of calculating；
If 3) result of calculation meets alert if, pass through api interface alert.
In the step 1), status data further includes flow system flow data, and the flow system flow data include that big data is flat
The flow system flow of each flow of platform, component.
The invention has the advantages that being capable of operation that is comprehensive, monitoring calculating operation on cluster, node and node in real time
Situation notes abnormalities and alerts in time, ensures smoothly completing for scheduler task, has higher sensitivity and reliability.
Description of the drawings
Fig. 1 is the system architecture diagram of the present invention.
Fig. 2 is flow system flow acquisition schematic diagram.
Specific implementation mode
Referring to Fig. 1~2.
The so-called flow system flow of the present invention refers to by some flow, component, calculating centre by under certain time frequency
The total amount of data of part etc., the referred to as flow system flow in the unit interval, abbreviation flow system flow.Such as in the Extract mistakes of ETL process
Journey is used as temporal frequency according to per minute, is M by the total amount of data of Extract processes, obtained flow system flow is every point
Clock M can be calculated as M/m.Often according to the size of data volume or the particularity of business, the temporal frequency of setting is also inconsistent.
Big data platform：The big data platform monitored is needed, includes elementary stream journey, finally stores and arrive to data
Big data platform, and the data based on storage are operated, analysis, excavate etc..
Acquisition：Acquisition component includes component states acquisition and flow system flow acquisition.Component states acquisition is main to be obtained greatly
The acquisition of the state of data platform system itself and the operating condition of various applications；Flow system flow acquisition is main to obtain big data
The flow system flow of each flow of platform, component.
Storage：By acquisition as a result, or the result that the result of acquisition has been carried out after logical calculated store.
API Sever：Information to meeting alarm regulation is sent to by various api interfaces in corresponding application, such as postal
Part, short message, wechat etc..
The big data platform monitoring framework of the present invention not only contains traditional monitor component, additionally provides based on system stream
The monitoring of amount.The framework of whole system such as Fig. 1.
The key of the present invention, that is, flow system flow acquisition, such as Fig. 2, traditional big data platform monitoring alarm are substantially all presence
The monitoring of component states.The present invention only it should be understood that big data platform flow, or get can with the point of data intercept,
It can carry out the acquisition of flow system flow.Figure below illustrates the point that can carry out flow system flow acquisition in detail according to data flow.
In practical application, collection point is reasonably configured in conjunction with scheduled business rule.
The temporal frequency of acquisition can reasonably be configured according to the rule of the data volume of business or business, still
The temporal frequency of each collection point must be consistent.Such as using minute as the frequency acquisition of data, then the acquisition frequency of all nodes
Rate is necessary for minute.
According to business rule, flow difference offset is pre-set, the data traffic of time frequency collection is compared,
If the value compared has been higher than flow difference offset, which is stored, is convenient for alarm display.To adopting
The data traffic of collection is compared the flow information acquired to all nodes under not only each temporal frequency and does to be compared two-by-two, also right
The data traffic that the flow of the same collection point carries out twice recently is judged, the accuracy of judgement is improved.
Claims (1)
1. big data platform monitoring method, includes the following steps：
1) the acquisition state data from big data platform, the status data include component states data；
2) calculation processing is carried out according to set rule to the result of acquisition, and stores the result of calculating；
If 3) result of calculation meets alert if, by api interface alert,
It is characterized in that,
In the step 1), status data further includes flow system flow data, and the flow system flow data include that big data platform is each
The flow system flow of a flow, component.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810332243.3A CN108712465A (en) | 2018-04-13 | 2018-04-13 | Big data platform monitoring method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810332243.3A CN108712465A (en) | 2018-04-13 | 2018-04-13 | Big data platform monitoring method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108712465A true CN108712465A (en) | 2018-10-26 |
Family
ID=63867275
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810332243.3A Pending CN108712465A (en) | 2018-04-13 | 2018-04-13 | Big data platform monitoring method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108712465A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112445674A (en) * | 2019-08-30 | 2021-03-05 | 中国石油化工股份有限公司 | Data processing method and storage medium of computer cluster |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN201937609U (en) * | 2011-01-20 | 2011-08-17 | 广东商学院 | Integrated warning platform for monitoring system |
| CN102355381A (en) * | 2011-08-18 | 2012-02-15 | 网宿科技股份有限公司 | Method and system for predicting flow of self-adaptive differential auto-regression moving average model |
| US20140226975A1 (en) * | 2013-02-13 | 2014-08-14 | Sodero Networks, Inc. | Method and apparatus for boosting data intensive processing through optical circuit switching |
| CN104615526A (en) * | 2014-12-05 | 2015-05-13 | 北京航空航天大学 | Monitoring system of large data platform |
| CN105183470A (en) * | 2015-09-06 | 2015-12-23 | 东南大学 | Natural language processing systematic service platform |
| CN105897498A (en) * | 2015-08-04 | 2016-08-24 | 乐视致新电子科技（天津）有限公司 | Business monitoring method and device |
| CN107193266A (en) * | 2017-07-11 | 2017-09-22 | 王焱华 | A kind of platform monitoring system of big data |
-
2018
- 2018-04-13 CN CN201810332243.3A patent/CN108712465A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN201937609U (en) * | 2011-01-20 | 2011-08-17 | 广东商学院 | Integrated warning platform for monitoring system |
| CN102355381A (en) * | 2011-08-18 | 2012-02-15 | 网宿科技股份有限公司 | Method and system for predicting flow of self-adaptive differential auto-regression moving average model |
| US20140226975A1 (en) * | 2013-02-13 | 2014-08-14 | Sodero Networks, Inc. | Method and apparatus for boosting data intensive processing through optical circuit switching |
| CN104615526A (en) * | 2014-12-05 | 2015-05-13 | 北京航空航天大学 | Monitoring system of large data platform |
| CN105897498A (en) * | 2015-08-04 | 2016-08-24 | 乐视致新电子科技（天津）有限公司 | Business monitoring method and device |
| CN105183470A (en) * | 2015-09-06 | 2015-12-23 | 东南大学 | Natural language processing systematic service platform |
| CN107193266A (en) * | 2017-07-11 | 2017-09-22 | 王焱华 | A kind of platform monitoring system of big data |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112445674A (en) * | 2019-08-30 | 2021-03-05 | 中国石油化工股份有限公司 | Data processing method and storage medium of computer cluster |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020259421A1 (en) | Method and apparatus for monitoring service system | |
| CN101876938B (en) | Message queue-based application software response time measuring method | |
| US11277483B2 (en) | Assessing user activity using dynamic windowed forecasting on historical usage | |
| CN105718351A (en) | Hadoop cluster-oriented distributed monitoring and management system | |
| CN106685703A (en) | Intelligent data collection and visual monitoring system | |
| US11012289B2 (en) | Reinforced machine learning tool for anomaly detection | |
| CN111339175B (en) | Data processing method, device, electronic equipment and readable storage medium | |
| CN101997709A (en) | Root alarm data analysis method and system | |
| CN110162445A (en) | The host health assessment method and device of Intrusion Detection based on host log and performance indicator | |
| CN109088775A (en) | abnormality monitoring method, device and server | |
| CN105871581A (en) | Method and device for processing of alarm information in cloud calculation | |
| CN114143220A (en) | Real-time data visualization platform | |
| CN110210980A (en) | A kind of driving behavior appraisal procedure, device and storage medium | |
| Mart et al. | Observability in kubernetes cluster: Automatic anomalies detection using prometheus | |
| CN108712465A (en) | Big data platform monitoring method | |
| CN110083529A (en) | Automated testing method, device, medium and electronic equipment | |
| US11556120B2 (en) | Systems and methods for monitoring performance of a building management system via log streams | |
| CN114756301B (en) | Log processing method, device and system | |
| WO2023273461A1 (en) | Robot operating state monitoring system, and method | |
| CN116300564A (en) | Automatic monitoring operation and maintenance platform supporting cross-region and cross-cluster mixed infrastructure | |
| CN110647070A (en) | Power environment monitoring system for super-large-scale data center | |
| CN109634808A (en) | A kind of chain type monitor event root cause analysis method based on association analysis | |
| CN115080363A (en) | System capacity evaluation method and device based on service log | |
| CN108241567A (en) | A kind of cloud system server state management map method | |
| CN114327392A (en) | Data processing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181026 |