CN109962792A - A kind of full link monitoring system based on big data - Google Patents
A kind of full link monitoring system based on big data Download PDFInfo
- Publication number
- CN109962792A CN109962792A CN201711399297.3A CN201711399297A CN109962792A CN 109962792 A CN109962792 A CN 109962792A CN 201711399297 A CN201711399297 A CN 201711399297A CN 109962792 A CN109962792 A CN 109962792A
- Authority
- CN
- China
- Prior art keywords
- monitoring
- log
- unit
- information
- big data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0604—Management of faults, events, alarms or notifications using filtering, e.g. reduction of information by using priority, element types, position or time
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
- H04L43/045—Processing captured monitoring data, e.g. for logfile generation for graphical visualisation of monitoring data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
- H04L43/0817—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking functioning
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Environmental & Geological Engineering (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention relates to telecom operators' electronic system O&M fields, more particularly to a kind of full link monitoring system based on big data, including monitoring objective, data pick-up unit, log transmission unit, storage unit, monitoring show unit and task schedule service framework.Monitoring objective includes the operating condition of Resource Middleware server etc..The status information that each monitoring objective is run is recorded and keeps in into journal file by data pick-up unit.Log transmission unit, which carries out the log recording that all monitoring objectives generate to summarize cleaning, filters out problematic log recording, and is carried out structure conversion, extracts necessary information.Storage unit safeguards information for storing effective monitoring log.Task schedule service framework carries out analysis for the log information of storage and depth is excavated；Monitoring shows unit, and graphically by current monitor, target tightening is presented to operation maintenance personnel, consequently facilitating tracking and Resolving probiems of the operation maintenance personnel to system.
Description
Technical field
The present invention relates to telecom operators' electronic system O&M fields, and in particular to a kind of full link prison based on big data
Control system.
Background technique
As the type service service scale of mobile operating service business is growing, bring the technology of mobile operation with
The development of business.With the development of technology and business, the framework of service system is made to become increasingly complex, system architecture is from original list
For one framework to the framework or now fluent micro services framework of basic SOA, framework brings many benefits, but also gives simultaneously
Operation maintenance personnel brings many problems, for example multiple applications while reporting an error, it is not known which appears in, when the needs that largely report an error spend very much
Between could position, there is a pile and is indifferent at all etc. extremely in project.Then we can uniformly collect log specified to one
Platform platform.But they are generally there are the call relation between tracking business of having no idea, and have no idea quickly to position application
Performance bottleneck the disadvantages of.Based on above status, it would be desirable to make the monitoring system of a full link.
Summary of the invention
1, technical problem to be solved:
The present invention provides a kind of full link monitoring system based on big data.It is the monitoring log by system in this system, point
The platform environment for analysing and showing the various states of current system, by the log information obtained from the operation node of system, and to it
After being collected cleaning, summarizing and convert, these information are stored in distributed search engine environment, then by this
A little information are analyzed, and are finally showed in a manner of chart etc. or are notified to O&M and system developer, thus to promote system
The indexs such as stability, high availability of uniting provide effective reference frame.
2, technical solution:
A kind of full link monitoring system based on big data, it is characterised in that: including monitoring objective, data pick-up unit, log
Transmission unit, storage unit, monitoring show unit and task schedule service framework.
The monitoring objective includes resource allocation conditions, the operating condition of middleware server, the state of application service, week
The interface of edge system calls and the operating condition of App client.
The status information that the data pick-up unit runs each monitoring objective, it is respectively specific square by monitoring objective
Formula and format, to monitor log and interface mode is recorded and keeps in into journal file.
The log transmission unit summarizes the log recording that all monitoring objectives generate, and carries out to these logs
Cleaning, filters out problematic or incomplete log recording, and carried out structure conversion, extracts necessary information, and
Time, location information are added, consequently facilitating analysis and storage.
Storage unit stores effective monitoring log information, and safeguards to these information, consequently facilitating to these
Information is inquired and is analyzed.
Task schedule service framework carries out analysis for the log information of storage and depth is excavated, thus the fortune of positioning system
Row situation and various performance issues；Task schedule service framework also carries out early warning analysis and notice to a monitoring objective simultaneously.
Monitoring shows unit for graphically concentrating each performance and state index of current monitor target
It is presented to operation maintenance personnel, consequently facilitating tracking and Resolving probiems of the operation maintenance personnel to system.
In above-mentioned scheme, the resource allocation conditions that the monitoring objective includes are database, Nginx server, delay
It deposits, the distribution condition of CPU and memory, disk, Internet resources.
Further, the log transmission unit further includes cache module, the good log of the log transmission cell processing
Cache module is transmitted to be stored.
3, the utility model has the advantages that
(1) comprehensively, not only the resource allocation conditions of monitor supervision platform, also monitor the operation feelings of intermediate server to the monitoring of this monitoring system
The interface of condition and perimeter systems calls situation.
(2) this monitoring system can quickly position the performance bottleneck of the positioning application of application.The present invention greatly facilitates fortune
The operation management to software for seeking quotient's operation maintenance personnel works, and can automatically generate corresponding chart, reduces document processing jobs, and
The performance bottleneck that application can timely be grasped, effectively increases working efficiency.
Detailed description of the invention
Fig. 1 is general frame figure of the invention.
Specific embodiment
The present invention is further detailed with reference to the accompanying drawing.
A kind of full link monitoring system based on big data, including monitoring objective, data pick-up unit, log transmission list
Member, storage unit, monitoring show unit and task schedule service framework.
The monitoring objective includes resource allocation conditions, the operating condition of middleware server, the state of application service, week
The interface of edge system calls and the operating condition of App client.
The status information that the data pick-up unit runs each monitoring objective, it is respectively specific square by monitoring objective
Formula and format, to monitor log and interface mode is recorded and keeps in into journal file.
The log transmission unit summarizes the log recording that all monitoring objectives generate, and carries out to these logs
Cleaning, filters out problematic or incomplete log recording, and carried out structure conversion, extracts necessary information, and
Time, location information are added, consequently facilitating analysis and storage.
Storage unit safeguards these information for storing effective monitoring log information, consequently facilitating to this
A little information are inquired and are analyzed.
Task schedule service framework is used to carry out analysis for the log information of storage and depth is excavated, thus positioning system
Operation conditions and various performance issues；Task schedule service framework also carries out early warning analysis to a monitoring objective and leads to simultaneously
Know.
Monitoring shows unit for graphically concentrating each performance and state index of current monitor target
It is presented to operation maintenance personnel, consequently facilitating tracking and Resolving probiems of the operation maintenance personnel to system.
In above-mentioned scheme, the resource allocation conditions that the monitoring objective includes are database, Nginx server, delay
It deposits, the distribution condition of CPU and memory, disk, Internet resources.
Further, the log transmission unit further includes cache module, the good log of the log transmission cell processing
Cache module is transmitted to be stored.
Although the present invention has been described by way of example and in terms of the preferred embodiments, they be not it is for the purpose of limiting the invention, it is any ripe
This those skilled in the art is practised, without departing from the spirit and scope of the invention, can make various changes or retouch from working as, therefore guarantor of the invention
Shield range should be subject to what claims hereof protection scope was defined.
Claims (3)
1. a kind of full link monitoring system based on big data, it is characterised in that: including monitoring objective, data pick-up unit, day
Will transmission unit, storage unit, monitoring show unit and task schedule service framework；
The monitoring objective includes resource allocation conditions, the operating condition of middleware server, the state of application service, periphery system
The interface of system calls and the operating condition of App client；
The status information that the data pick-up unit runs each monitoring objective, by monitoring objective respectively specific mode and
Format, to monitor log and interface mode is recorded and keeps in into journal file；
The log transmission unit summarizes the log recording that all monitoring objectives generate, and carries out to these logs clear
It washes, filters out problematic or incomplete log recording, and carried out structure conversion, extract necessary information, Yi Jitian
Between added-time, location information, consequently facilitating analysis and storage；
Storage unit stores effective monitoring log information, and safeguards to these information, consequently facilitating to these information
It is inquired and is analyzed；
Task schedule service framework carries out analysis for the log information of storage and depth is excavated, thus the operation shape of positioning system
Condition and various performance issues；Task schedule service framework also carries out early warning analysis and notice to a monitoring objective simultaneously；
Monitoring shows unit and presents for graphically concentrating each performance and state index of current monitor target
To operation maintenance personnel, consequently facilitating tracking and Resolving probiems of the operation maintenance personnel to system.
2. a kind of electric business channel monitoring system based on big data according to claim 1, it is characterised in that: the monitoring
The resource allocation conditions that target includes are the distribution of database, Nginx server, caching, CPU and memory, disk, Internet resources
Situation.
3. a kind of full link monitoring system based on big data according to claim 1, it is characterised in that: the log passes
Defeated unit further includes cache module, and good log transmission to the cache module of the log transmission cell processing is stored.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711399297.3A CN109962792A (en) | 2017-12-22 | 2017-12-22 | A kind of full link monitoring system based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711399297.3A CN109962792A (en) | 2017-12-22 | 2017-12-22 | A kind of full link monitoring system based on big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109962792A true CN109962792A (en) | 2019-07-02 |
Family
ID=67018738
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711399297.3A Pending CN109962792A (en) | 2017-12-22 | 2017-12-22 | A kind of full link monitoring system based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109962792A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110837459A (en) * | 2019-11-07 | 2020-02-25 | 广东省科技基础条件平台中心 | Big data-based operation performance analysis method and system |
| CN111092645A (en) * | 2019-11-18 | 2020-05-01 | 航天恒星科技有限公司 | Real-time monitoring and processing system of satellite communication system |
| CN111104287A (en) * | 2019-12-20 | 2020-05-05 | 南京亚派科技股份有限公司 | Data acquisition method of dual-power automatic transfer switch online management system |
| CN111125152A (en) * | 2019-12-26 | 2020-05-08 | 积成电子股份有限公司 | Full link data control method based on data processing process model |
| CN112559280A (en) * | 2020-12-04 | 2021-03-26 | 国网安徽省电力有限公司信息通信分公司 | Data full link monitoring method based on data center station |
| CN113114483A (en) * | 2021-03-17 | 2021-07-13 | 上海浦东发展银行股份有限公司 | Service log full-link tracking method and system |
| CN113537824A (en) * | 2021-08-03 | 2021-10-22 | 南方电网数字电网研究院有限公司 | Power grid data full link monitoring system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105760281A (en) * | 2016-03-01 | 2016-07-13 | 浪潮软件集团有限公司 | Method for monitoring and analyzing performance data of Linux system server |
| EP3151465A1 (en) * | 2015-09-29 | 2017-04-05 | Toshiba TEC Kabushiki Kaisha | Transmission of log information for device maintenance through a network |
| CN107046481A (en) * | 2017-04-18 | 2017-08-15 | 国网福建省电力有限公司 | A kind of information system integrated network management system comprehensive analysis platform |
| CN107493208A (en) * | 2017-08-30 | 2017-12-19 | 苏州朗动网络科技有限公司 | Full the link performance monitoring method and system of a kind of medium-and-large-sized operation system |
-
2017
- 2017-12-22 CN CN201711399297.3A patent/CN109962792A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3151465A1 (en) * | 2015-09-29 | 2017-04-05 | Toshiba TEC Kabushiki Kaisha | Transmission of log information for device maintenance through a network |
| CN105760281A (en) * | 2016-03-01 | 2016-07-13 | 浪潮软件集团有限公司 | Method for monitoring and analyzing performance data of Linux system server |
| CN107046481A (en) * | 2017-04-18 | 2017-08-15 | 国网福建省电力有限公司 | A kind of information system integrated network management system comprehensive analysis platform |
| CN107493208A (en) * | 2017-08-30 | 2017-12-19 | 苏州朗动网络科技有限公司 | Full the link performance monitoring method and system of a kind of medium-and-large-sized operation system |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110837459A (en) * | 2019-11-07 | 2020-02-25 | 广东省科技基础条件平台中心 | Big data-based operation performance analysis method and system |
| CN111092645A (en) * | 2019-11-18 | 2020-05-01 | 航天恒星科技有限公司 | Real-time monitoring and processing system of satellite communication system |
| CN111092645B (en) * | 2019-11-18 | 2022-04-19 | 航天恒星科技有限公司 | Real-time monitoring and processing system of satellite communication system |
| CN111104287A (en) * | 2019-12-20 | 2020-05-05 | 南京亚派科技股份有限公司 | Data acquisition method of dual-power automatic transfer switch online management system |
| CN111125152A (en) * | 2019-12-26 | 2020-05-08 | 积成电子股份有限公司 | Full link data control method based on data processing process model |
| CN111125152B (en) * | 2019-12-26 | 2023-10-13 | 积成电子股份有限公司 | Full-link data management and control method based on data processing process model |
| CN112559280A (en) * | 2020-12-04 | 2021-03-26 | 国网安徽省电力有限公司信息通信分公司 | Data full link monitoring method based on data center station |
| CN112559280B (en) * | 2020-12-04 | 2023-08-22 | 国网安徽省电力有限公司信息通信分公司 | Data full-link monitoring method based on data center station |
| CN113114483A (en) * | 2021-03-17 | 2021-07-13 | 上海浦东发展银行股份有限公司 | Service log full-link tracking method and system |
| CN113537824A (en) * | 2021-08-03 | 2021-10-22 | 南方电网数字电网研究院有限公司 | Power grid data full link monitoring system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109962792A (en) | A kind of full link monitoring system based on big data | |
| CN107943668B (en) | Computer server cluster log monitoring method and monitor supervision platform | |
| JP2022153557A (en) | Data analysis service for distributed type industrial performance monitoring | |
| JP2023116784A (en) | Distributed industrial performance monitoring and analysis | |
| EP2487860B1 (en) | Method and system for improving security threats detection in communication networks | |
| US10152361B2 (en) | Event stream processing cluster manager | |
| CN111124819B (en) | Method and device for full link monitoring | |
| US20150046512A1 (en) | Dynamic collection analysis and reporting of telemetry data | |
| US10983855B2 (en) | Interface for fault prediction and detection using time-based distributed data | |
| US20230115255A1 (en) | Systems and methods for predictive assurance | |
| US10567557B2 (en) | Automatically adjusting timestamps from remote systems based on time zone differences | |
| US11449798B2 (en) | Automated problem detection for machine learning models | |
| US11212173B2 (en) | Model-driven technique for virtual network function rehoming for service chains | |
| US11188409B2 (en) | Data lifecycle management | |
| US10372572B1 (en) | Prediction model testing framework | |
| CN107463490B (en) | Cluster log centralized collection method applied to platform development | |
| CN111046007A (en) | Method, apparatus and computer program product for managing a storage system | |
| CN111448551B (en) | Method and system for tracking application activity data from a remote device and generating a corrective action data structure for the remote device | |
| CN115514618A (en) | Alarm event processing method and device, electronic equipment and medium | |
| CN115202973A (en) | Application running state determining method and device, electronic equipment and medium | |
| CN113094243A (en) | Node performance detection method and device | |
| CN116094905B (en) | Full-link monitoring system | |
| CN116401138B (en) | Operating system running state detection method and device, electronic equipment and medium | |
| CN110633259A (en) | HDFS-based fault pushing system | |
| CN118158065A (en) | Cloud component security baseline dynamic restoration system and method based on tenant policy |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190702 |