CN109905267A - A kind of method and apparatus for big data system status monitoring - Google Patents
A kind of method and apparatus for big data system status monitoring Download PDFInfo
- Publication number
- CN109905267A CN109905267A CN201711310901.0A CN201711310901A CN109905267A CN 109905267 A CN109905267 A CN 109905267A CN 201711310901 A CN201711310901 A CN 201711310901A CN 109905267 A CN109905267 A CN 109905267A
- Authority
- CN
- China
- Prior art keywords
- big data
- data system
- information
- system information
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Landscapes
- Alarm Systems (AREA)
- Telephonic Communication Services (AREA)
Abstract
The present invention relates to field of computer technology more particularly to a kind of method and apparatus for big data system status monitoring.This method includes that computer equipment is called to acquire big data system information, and judge whether big data system mode exception occurs according to collected big data system information, when judging that big data system mode occurs abnormal according to collected big data system information, warning message is sent from trend user.Monitoring and alarm procedure to big data system mode are thus realized by computer equipment, improve the accuracy of monitoring and alarm efficiency and alarm.
Description
Technical field
The present invention relates to field of computer technology more particularly to a kind of methods and dress for big data system status monitoring
It sets.
Background technique
The server cluster of big data system (hadoop ecology) on statisticalling analyze software and hardware information related to summarizing because
Server is more, deployment software type is more and relevant information index is excessively complicated, is monitored to the state of entire cluster,
And the alert operation amount when the state of cluster occurs abnormal is very heavy, monitoring and alarm inefficiency.
For the above inefficiency, majority settling modes are still artificial monitoring cluster at present, by associated monitoring software with
And related command etc. is checked cluster information and is judged, manually alarms when cluster occurs abnormal.This mode step is numerous
Trivial, consuming working hour, inefficiency, in addition, phenomena such as often generating wrong report due to artificial carelessness, failing to report.
Summary of the invention
The present invention provides a kind of method and apparatus for big data system status monitoring, by calling computer equipment
It is automatic when acquiring big data system information, and judging that big data system occurs abnormal according to collected big data system information
Warning message is sent to user, improves the efficiency to big data system status monitoring and alert operation.
In a first aspect, the present invention provides a kind of method for big data system status monitoring, including call computer
Equipment executes:
Acquire big data system information；
Judge whether big data system mode exception occurs according to collected big data system information；
When big data system mode occurs abnormal, warning message is issued the user with.
It is further, described to issue the user with warning message, comprising:
Alarm mail is sent to user, or calls third party's interface, is dialed the police emergency number automatically to user.
Further, the acquisition big data system information, comprising:
Acquire a plurality of types of big data system informations；
It is described to judge whether big data system mode exception occurs according to collected big data system information, comprising:
For the big data system information of each type, the corresponding abnormal judgement rule of the big data system information are determined
Then, and according to the exception judgment rule judge whether the big data system information of the type is abnormal.
Further, described to judge whether big data system mode exception occurs according to collected big data system information
Include:
Collected big data system information is handled as preset format；
Judge whether big data system mode exception occurs according to the big data system information that processing is preset format.
Further, the acquisition big data system information includes: one or more of following parameter of acquisition:
HDFS space hold information, HDFSBLOCK block count information, HDFSBLOCK distributed intelligence, the space HDFS increase letter
Breath, HBASE merge queuing message, HBASE refresh queuing message, HBASEmemstore size information, flume stacking pressure information,
Flume rate information, kafka stacking pressure information, each progress information of cluster and cluster machine loading information.
Further, the method, further includes:
History warning message is obtained, and is referred to according to the stability index of history warning message analysis big data system, storage
One of number, performance index and loophole index or several generations alarm chart；
Alarm chart is sent to user.
Further, the method, further includes:
According to history warning message, to the stability index in big data system future, storage index, performance index and loophole
Chart is predicted in one of index or several generations；
Prediction chart is sent to user.
Second aspect, the present invention provides a kind of devices for big data system status monitoring, comprising:
Acquisition module, for calling computer equipment to acquire big data system information；
Judgment module, for judging big data system mode with the presence or absence of different according to collected big data system information
Often；
Alarm module, for issuing the user with warning message when big data system mode occurs abnormal.
Further, the alarm module is specifically used for that computer equipment is called to send alarm mail to client, or adjusts
With third party's interface, dial the police emergency number to user.
Further, the acquisition module is specifically used for that computer equipment is called to acquire a plurality of types of big data systems
Information；
The judgment module is specifically used for the big data system information for calling computer equipment to be directed to each type, determines
The corresponding abnormal judgment rule of the big data system information, and judge according to the exception judgment rule big data system of the type
Whether information is abnormal.
Further, the judgment module is also used to that computer equipment is called to handle collected big data system information
For preset format；Judge whether big data system mode exception occurs according to the big data system information that processing is preset format.
Provided by the present invention for the method and apparatus of big data system status monitoring, by calling computer equipment acquisition
Big data system information, and judge whether big data system mode exception occurs according to collected big data system information, when
When judging that big data system mode occurs abnormal according to collected big data system information, alarm signal is sent from trend user
Breath.Monitoring and alarm procedure to big data system mode are thus realized by computer equipment, improve monitoring and report
Alert efficiency and the accuracy of alarm.
Detailed description of the invention
Fig. 1 is the flow diagram provided by the present invention for the method for big data system status monitoring；
Fig. 2 is the structural schematic diagram provided by the present invention for the device of big data system status monitoring.
Specific embodiment
With reference to the accompanying drawings and examples, further description of the specific embodiments of the present invention.Following embodiment is only
For clearly illustrating technical solution of the present invention, and not intended to limit the protection scope of the present invention.
In a first aspect, the present invention provides a kind of methods for big data system status monitoring, referring to Fig. 1, this method
Including calling computer equipment to execute following process:
Step S1 acquires big data system information；
Step S2 judges whether big data system mode exception occurs according to collected big data system information；
Step S3 issues the user with warning message when big data system mode occurs abnormal.
By calling computer equipment to acquire big data system information, judge when according to collected big data system information
When big data system mode occurs abnormal out, warning message is sent from trend user.Thus realized by computer equipment
Monitoring and alarm to big data system mode improve the accuracy of monitoring and alarm efficiency and alarm.
In the specific implementation, warning message can be sent to user in several ways in above-mentioned step S3.Specifically
For, use is sent in the form of mail by there is abnormal related abnormal big data system information with big data system mode
Family, such user are not necessarily at corresponding computer equipment, can know that exception occurs in big data system.In addition, ought count greatly
Third party's interface can also be called by computer when occurring abnormal according to system mode, dialed the police emergency number automatically to user, this
Sample user can timely learning big data system mode there is exception, to take corresponding measure in time, avoid the occurrence of
Exception big data system is damaged.
In the specific implementation, above-mentioned step S1 can be specifically included:
Constantly or according to predetermined period acquire a variety of big data system informations；
For the big data system information of each type, the corresponding abnormal judgement rule of the big data system information are determined
Then, and according to the exception judgment rule judge whether the big data system information of the type is abnormal.
The fault-tolerance to big data system status monitoring so can be improved.
In the specific implementation, above-mentioned step S2 can be specifically included: be by the processing of collected big data system information
Preset format；
Judge whether big data system mode exception occurs according to the big data system information that processing is preset format.
The advantage of doing so is that can allow for being communicated between computer equipment and a variety of different types of equipment, than
If interface here can be hadoopjmx and the relevant interface, such as linuxshell of other assemblies etc., can adopt at this time
Data are parsed with json and generate the big data system information of preset format.
In the specific implementation, above-mentioned step S1 can be specifically included: acquire one in following parameter according to predetermined period
Kind is a variety of:
HDFS space hold information, HDFSBLOCK block count information, HDFSBLOCK distributed intelligence, the space HDFS increase letter
Breath, HBASE merge queuing message, HBASE refresh queuing message, HBASEmemstore size information, flume stacking pressure information,
Flume rate information, kafka stacking pressure information, each progress information of cluster and cluster machine loading information.
In the specific implementation, the method can also include:
History warning message is obtained, and is referred to according to the stability index of history warning message analysis big data system, storage
One of number, performance index and loophole index or several generations alarm chart；
Alarm chart is sent to user.
Since disk storage space alarm will affect storage index and stability index；The alarm of kafka stacking pressure will affect
Performance index；Process alarm will affect stability index and loophole index.Therefore, user can know big number according to alarm chart
There is exception in state according to which part of system, so as to take phase in time when big data system mode occurs abnormal
The processing treatment measures answered.
In the specific implementation, the method can also include:
According to history warning message, to the stability index in big data system future, storage index, performance index and loophole
Chart is predicted in one of index or several generations；
Prediction chart is sent to user.
Being designed in this way is advantageous in that, can be for user in predicting big data system it is possible that abnormal part, uses
Family properly protects preparation early, avoids causing damages to user.
In the specific implementation, all warning messages can be stored, corresponding report is periodically read by predetermined period
Alert information analyzes stability index, storage index, performance index and the loophole index of big data system according to warning message, and
Every index is predicted.
In the specific implementation, the detailed of the time of fire alarming of big data system all parts, alert frequency and alarm can be passed through
Every index of big data system is predicted in thin content analysis.
Alarming value P can be calculated as follows according to the alarm times t of every kind of index in the specific implementation:
P=(100-t)/100
In the specific implementation, if alarming value P is smaller and smaller over time, it is different to show that the state of the index occurs
Often.
For example, if the root warning message of disk storage space continuously occurs, and alarming value P gradually successively decreases, then can be with
Daily mean difference is obtained by analyzing data, and provide prediction result: the root of the disk may will thoroughly take in a few days.
Such user in time can clear up disk according to warning message.
Based on identical design, second aspect, the present invention provides a kind of device for big data system status monitoring,
Referring to fig. 2, the apparatus may include:
Acquisition module 201, for calling computer equipment to acquire big data system information；
Judgment module 202, for calling computer equipment to judge big data system according to collected big data system information
System state is with the presence or absence of abnormal；
Alarm module 203 is used for when big data system mode occurs abnormal, for calling computer equipment to send out to user
Warning message out.
In the specific implementation, above-mentioned alarm module 203 can be specifically used for sending alarm mail to user, or call
Third party's interface, dials the police emergency number to user.
In the specific implementation, the acquisition module 201 is specifically used for that computer equipment is called to acquire a plurality of types of big numbers
According to system information；The judgment module 202 is specifically used for that computer equipment is called to believe for the big data system of each type
Breath determines the corresponding abnormal judgment rule of the big data system information, and judges the big of the type according to the exception judgment rule
Whether data system information is abnormal.
The above is only a preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art
For member, without departing from the technical principles of the invention, several improvements and modifications can also be made, these improvements and modifications
Also it should be regarded as protection scope of the present invention.
Claims (10)
1. a kind of method for big data system status monitoring, which is characterized in that including calling computer equipment to execute:
Acquire big data system information；
Judge whether big data system mode exception occurs according to collected big data system information；
When big data system mode occurs abnormal, warning message is issued the user with.
2. the method as described in claim 1, which is characterized in that described to issue the user with warning message, comprising:
Alarm mail is sent to user, or calls third party's interface, is dialed the police emergency number automatically to user.
3. the method as described in claim 1, which is characterized in that the acquisition big data system information, comprising:
Acquire a plurality of types of big data system informations；
It is described to judge whether big data system mode exception occurs according to collected big data system information, comprising:
For the big data system information of each type, the corresponding abnormal judgment rule of the big data system information is determined, and
Judge whether the big data system information of the type is abnormal according to the exception judgment rule.
4. the method as described in claim 1, which is characterized in that described to judge big number according to collected big data system information
Include: according to whether system mode exception occurs
Collected big data system information is handled as preset format；
Judge whether big data system mode exception occurs according to the big data system information that processing is preset format.
5. the method as described in claim 1, which is characterized in that the acquisition big data system information includes: the following ginseng of acquisition
One or more of number:
HDFS space hold information, HDFSBLOCK block count information, HDFSBLOCK distributed intelligence, the space HDFS increase information,
HBASE merges queuing message, HBASE refreshes queuing message, HBASEmemstore size information, flume stacking pressure information, flume
Rate information, kafka stacking pressure information, each progress information of cluster and cluster machine loading information.
6. method as claimed in claim 5, which is characterized in that further include:
History warning message is obtained, and analyzes stability index, the storage index, property of big data system according to history warning message
It can one of index and loophole index or several generations alarm chart；
Alarm chart is sent to user.
7. method as claimed in claim 6, which is characterized in that further include:
According to history warning message, to the stability index in big data system future, storage index, performance index and loophole index
One of or several generations predict chart；
Prediction chart is sent to user.
8. a kind of device for big data system status monitoring characterized by comprising
Acquisition module, for calling computer equipment to acquire big data system information；
Judgment module, for calling computer equipment to judge that big data system mode is according to collected big data system information
It is no to there is exception；
Alarm module, for calling computer equipment to issue the user with warning message when big data system mode occurs abnormal.
9. device as claimed in claim 8, which is characterized in that the alarm module is specifically used for calling computer equipment to visitor
Family sends alarm mail, or calls third party's interface, dials the police emergency number to user.
10. device as claimed in claim 8, which is characterized in that
The acquisition module is specifically used for that computer equipment is called to acquire a plurality of types of big data system informations；
The judgment module is specifically used for the big data system information for calling computer equipment to be directed to each type, determines that this is big
The corresponding abnormal judgment rule of data system information, and judge according to the exception judgment rule big data system information of the type
It is whether abnormal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711310901.0A CN109905267A (en) | 2017-12-11 | 2017-12-11 | A kind of method and apparatus for big data system status monitoring |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711310901.0A CN109905267A (en) | 2017-12-11 | 2017-12-11 | A kind of method and apparatus for big data system status monitoring |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109905267A true CN109905267A (en) | 2019-06-18 |
Family
ID=66942617
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711310901.0A Withdrawn CN109905267A (en) | 2017-12-11 | 2017-12-11 | A kind of method and apparatus for big data system status monitoring |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109905267A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111274088A (en) * | 2020-01-15 | 2020-06-12 | 平安科技（深圳）有限公司 | Real-time monitoring method, device, medium and electronic equipment for big data platform |
-
2017
- 2017-12-11 CN CN201711310901.0A patent/CN109905267A/en not_active Withdrawn
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111274088A (en) * | 2020-01-15 | 2020-06-12 | 平安科技（深圳）有限公司 | Real-time monitoring method, device, medium and electronic equipment for big data platform |
| WO2021143024A1 (en) * | 2020-01-15 | 2021-07-22 | 平安科技（深圳）有限公司 | Method and apparatus for real-time monitoring of big data platform, medium, and electronic device |
| CN111274088B (en) * | 2020-01-15 | 2021-08-24 | 平安科技（深圳）有限公司 | Real-time monitoring method, device, medium and electronic equipment for big data platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110661659B (en) | Alarm method, device and system and electronic equipment | |
| CN111143102B (en) | Abnormal data detection method and device, storage medium and electronic equipment | |
| CN105681128A (en) | Method and device for monitoring big data system state | |
| US8862119B2 (en) | Method and apparatus for telecommunications network performance anomaly events detection and notification | |
| CN105095056A (en) | Method for monitoring data in data warehouse | |
| CN103490917B (en) | The detection method of troubleshooting situation and device | |
| CN108599977B (en) | System and method for monitoring system availability based on statistical method | |
| CN109992473A (en) | Monitoring method, device, equipment and the storage medium of application system | |
| CN111163073A (en) | Flow data processing method and device | |
| CN108986418A (en) | intelligent alarm method, device, equipment and storage medium | |
| CN108039971A (en) | A kind of alarm method and device | |
| CN110012000B (en) | Command detection method and device, computer equipment and storage medium | |
| CN107465652B (en) | Operation behavior detection method, server and system | |
| CN109905267A (en) | A kind of method and apparatus for big data system status monitoring | |
| CN106982141A (en) | Weblogic examples monitoring method and device | |
| CN113123955B (en) | Plunger pump abnormity detection method and device, storage medium and electronic equipment | |
| CN110633165B (en) | Fault processing method, device, system server and computer readable storage medium | |
| CN113760634A (en) | Data processing method and device | |
| CN114610560B (en) | System abnormality monitoring method, device and storage medium | |
| CN116108394A (en) | Industrial control system flow abnormality detection method, device and medium | |
| KR101973728B1 (en) | Integration security anomaly symptom monitoring system | |
| CN112581715B (en) | Battery high-temperature alarm method, device and system | |
| CN115118614A (en) | Operation abnormality detection method, operation abnormality detection device, electronic device, and storage medium | |
| CN109508356B (en) | Data abnormality early warning method, device, computer equipment and storage medium | |
| CN113297039A (en) | Data monitoring method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20190618 |