CN110557291A - Network service monitoring system - Google Patents
Network service monitoring system Download PDFInfo
- Publication number
- CN110557291A CN110557291A CN201910881353.XA CN201910881353A CN110557291A CN 110557291 A CN110557291 A CN 110557291A CN 201910881353 A CN201910881353 A CN 201910881353A CN 110557291 A CN110557291 A CN 110557291A
- Authority
- CN
- China
- Prior art keywords
- data
- monitoring data
- network service
- monitoring
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/12—Discovery or management of network topologies
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L67/025—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/133—Protocols for remote procedure calls [RPC]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/51—Discovery or management thereof, e.g. service location protocol [SLP] or web services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The network service monitoring system provided by the embodiment of the invention relates to the technical field of networks, has diversified monitoring indexes, supports the monitoring of heterogeneous network services, can adapt to complex and diversified network service environments, acquires monitoring data in real time, generates corresponding alarm information according to the monitoring data, and improves the compatibility, the real-time property and the system throughput of the monitoring system.
Description
Technical Field
The invention relates to the technical field of networks, in particular to a network service monitoring system.
Background
With the continuous development of information technology, various networks have become indispensable tools in life and work. The situation of network bandwidth is complex and variable, and how to effectively and reasonably monitor various complex network services is always a problem.
Most of the current network service monitoring systems have the following defects:
(1) the monitoring index is single and cannot reflect the real state of the current network service;
(2) The monitoring network service type is single, and the monitoring network service type cannot adapt to complex and diverse network service environments;
(3) The data acquisition is delayed, and the monitoring data cannot be acquired in real time.
Disclosure of Invention
In order to solve the defects in the prior art, an embodiment of the present invention provides a network service monitoring system, including:
The system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for judging the type of network service through data characteristics, carrying out classification and point burying on the network service according to the type, acquiring different monitoring data, converting the monitoring data into data in a json format and sending the monitoring data to a kafka message queue;
The service module is used for receiving the monitoring data from the kafka message queue by utilizing a micro-service architecture and a restful interface, judging the type of the network service according to the monitoring data, writing the type of the network service into the monitoring data, and storing the data into the storage module;
And the analysis module is used for reading the monitoring data from the storage module in real time, analyzing the monitoring data in real time by using a spark streaming big data real-time analysis platform, generating corresponding analysis data and generating alarm information in real time according to a preset alarm rule.
Preferably, the acquisition module is further configured to acquire monitoring data and perform reverse analysis on the acquired data by using a section-oriented programming technology.
Preferably, the service module is further configured to discard the monitoring data that fails to be converted, and write the receiving time, the importance degree, and the shortest storage time of the monitoring data into the monitoring data.
Preferably, the analysis data includes status of network service, time consumption, link call relation, call topology, log and system throughput.
The network service monitoring system provided by the embodiment of the invention has the following beneficial effects:
(1) Zero intrusion
All links are zero-invasive to developers, the developers do not need to care about monitoring implementation details, and any code logic related to monitoring does not need to be introduced into code development.
(2) Supporting heterogeneous network services
The acquisition module divides the services into a single application, an RPC service, a Restful service, a message queue service, a database service and a cache service according to the type of the acquired monitoring data, and supports data acquisition in a mixed mode.
(3) Real-time property
all links of the system consider real-time performance, and after service calling is realized, information such as the state, time consumption, link calling relation, topology, logs, system throughput and the like of network service can be acquired in real time, and alarm information is generated in real time according to preset alarm rules.
(4) High concurrency
All links of the system consider performance bottlenecks possibly generated in practical application, and technologies such as kafka message queues, micro-services and the like are adopted to carry out peak clipping and elastic capacity expansion on a large amount of data generated by large concurrency, so that the system throughput of hundred thousand-level concurrency scale is realized.
Drawings
Fig. 1 is a schematic structural diagram of a network service monitoring system according to an embodiment of the present invention.
Detailed Description
the invention is described in detail below with reference to the figures and the embodiments.
referring to fig. 1, a network service monitoring system provided in an embodiment of the present invention includes:
And the acquisition module is used for judging the type of the network service according to the data characteristics, performing classification and point burying on the network service according to the type, acquiring different monitoring data, converting the monitoring data into data in a json format and sending the monitoring data to the kafka message queue.
the data characteristics comprise data types, and the network service types comprise single application, RPC service, Restful service, message queue service, database service and cache service.
And the service module is used for receiving the monitoring data from the kafka message queue by using the micro-service architecture and the restful interface, judging the type of the network service according to the monitoring data, writing the type of the network service into the monitoring data, and storing the data into the storage module.
As a specific embodiment, when receiving data, the service module improves the efficiency of receiving data by registering a plurality of micro service instances, and alleviates the performance problem caused when receiving large concurrent monitoring data.
when the storage module stores data, the received data is preferentially stored in the memory, the storage request is quickly responded, in order to prevent the monitoring data from occupying too large storage capacity of the hard disk along with the passage of time, the storage module starts a multithreading example to asynchronously compress and persistently store the data in the hard disk, and the time efficiency and the storage efficiency can simultaneously achieve the best performance.
And the analysis module is used for reading the monitoring data from the storage module in real time, analyzing the monitoring data in real time by using a spark streaming big data real-time analysis platform, generating corresponding analysis data and generating alarm information in real time according to a preset alarm rule.
The alarm rules are configurable options, and different alarm rules are flexibly set according to state thresholds, time-consuming thresholds, calling link length thresholds, log keywords and system throughput thresholds of different network services.
Optionally, the collecting module is further configured to collect the monitoring data and perform reverse analysis on the obtained data by using a section-oriented programming technique.
Optionally, the service module is further configured to discard the monitoring data that fails to be converted, and write the receiving time, the importance degree, and the shortest storage time of the monitoring data into the monitoring data.
The acquisition module is used for zero intrusion to a service developer when acquiring monitoring data, the service developer does not need to care about a monitoring implementation mode, service data are acquired through a section-oriented programming technology, meanwhile, reverse analysis is carried out on the acquired data, and core codes are based on assembly programming and realize efficient judgment of network service types by using a jni technology.
Optionally, the analysis data includes status of network services, time consumption, link invocation relationship, invocation topology, log, and system throughput.
The network service monitoring system provided by the embodiment of the invention has diversified monitoring indexes, supports monitoring of heterogeneous network services, can adapt to complex and diversified network service environments, can acquire monitoring data in real time and generate alarm information according to the monitoring data, and improves the compatibility, the real-time property and the system throughput of the monitoring system.
in the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
It will be appreciated that the relevant features of the method and apparatus described above are referred to one another.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
the algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In addition, the memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
these computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
it should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
as will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (4)
1. A network service monitoring system, comprising:
The system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for judging the type of network service through data characteristics, carrying out classification and point burying on the network service according to the type, acquiring different monitoring data, converting the monitoring data into data in a json format and sending the monitoring data to a kafka message queue;
The service module is used for receiving the monitoring data from the kafka message queue by utilizing a micro-service architecture and a restful interface, judging the type of the network service according to the monitoring data, writing the type of the network service into the monitoring data, and storing the data into the storage module;
and the analysis module is used for reading the monitoring data from the storage module in real time, analyzing the monitoring data in real time by using a spark streaming real-time analysis platform, generating corresponding analysis data and generating alarm information in real time according to a preset alarm rule.
2. The network service monitoring system of claim 1,
The acquisition module is also used for acquiring monitoring data and reversely analyzing the acquired data by using a section-oriented programming technology.
3. The network service monitoring system of claim 1, wherein the service module is further configured to discard the monitoring data with failed conversion, and write the receiving time, the importance degree, and the shortest storage time of the monitoring data into the monitoring data.
4. The network service monitoring system of claim 1, wherein the analysis data includes status of network services, elapsed time, link invocation relationship, invocation topology, log, and system throughput.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910881353.XA CN110557291A (en) | 2019-09-18 | 2019-09-18 | Network service monitoring system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910881353.XA CN110557291A (en) | 2019-09-18 | 2019-09-18 | Network service monitoring system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110557291A true CN110557291A (en) | 2019-12-10 |
Family
ID=68740654
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910881353.XA Pending CN110557291A (en) | 2019-09-18 | 2019-09-18 | Network service monitoring system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110557291A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111597094A (en) * | 2020-05-20 | 2020-08-28 | 神州数码融信软件有限公司 | IT system monitoring method and device |
| CN112486708A (en) * | 2020-12-16 | 2021-03-12 | 中国联合网络通信集团有限公司 | Processing method and processing system of page operation data |
| CN112862428A (en) * | 2021-01-18 | 2021-05-28 | 北京神州数字科技有限公司 | Supervision reporting system based on micro-service |
| CN114422386A (en) * | 2022-01-20 | 2022-04-29 | 南方电网数字电网研究院有限公司 | Monitoring method and device for micro-service gateway |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100131928A1 (en) * | 2008-11-21 | 2010-05-27 | Sun Microsystems, Inc. | Automated testing and qualification of software-based, network service products |
| CN109714192A (en) * | 2018-11-29 | 2019-05-03 | 深圳供电局有限公司 | Monitoring method and system for monitoring cloud platform |
-
2019
- 2019-09-18 CN CN201910881353.XA patent/CN110557291A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100131928A1 (en) * | 2008-11-21 | 2010-05-27 | Sun Microsystems, Inc. | Automated testing and qualification of software-based, network service products |
| CN109714192A (en) * | 2018-11-29 | 2019-05-03 | 深圳供电局有限公司 | Monitoring method and system for monitoring cloud platform |
Non-Patent Citations (3)
| Title |
|---|
| 刘嘉裕: "基于分布式微服务全链路实时监控系统设计与实现", 《中国优秀硕士学位论文全文数据库（电子期刊）》 * |
| 谢山青: "电信BSS业务监控系统的设计与实现", 《中国优秀硕士学位论文全文数据库（电子期刊）》 * |
| 谭一鸣: "基于微服务架构的平台化服务框架的设计与实现", 《中国优秀硕士学位论文全文数据库（电子期刊）》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111597094A (en) * | 2020-05-20 | 2020-08-28 | 神州数码融信软件有限公司 | IT system monitoring method and device |
| CN112486708A (en) * | 2020-12-16 | 2021-03-12 | 中国联合网络通信集团有限公司 | Processing method and processing system of page operation data |
| CN112486708B (en) * | 2020-12-16 | 2023-11-07 | 中国联合网络通信集团有限公司 | Page operation data processing method and processing system |
| CN112862428A (en) * | 2021-01-18 | 2021-05-28 | 北京神州数字科技有限公司 | Supervision reporting system based on micro-service |
| CN112862428B (en) * | 2021-01-18 | 2023-10-13 | 北京神州数字科技有限公司 | Supervision reporting system based on micro-service |
| CN114422386A (en) * | 2022-01-20 | 2022-04-29 | 南方电网数字电网研究院有限公司 | Monitoring method and device for micro-service gateway |
| CN114422386B (en) * | 2022-01-20 | 2023-08-11 | 南方电网数字电网研究院有限公司 | Monitoring method and device for micro-service gateway |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110557291A (en) | Network service monitoring system | |
| US10831562B2 (en) | Method and system for operating a data center by reducing an amount of data to be processed | |
| US20180365085A1 (en) | Method and apparatus for monitoring client applications | |
| WO2020147336A1 (en) | Micro-service full-link monitoring system and method | |
| US8843632B2 (en) | Allocation of resources between web services in a composite service | |
| CN108829508B (en) | Task processing method and device | |
| CN106713396B (en) | Server scheduling method and system | |
| CN113596078B (en) | Service problem positioning method and device | |
| CN110046187B (en) | Data processing system, method and device | |
| CN111177237B (en) | Data processing system, method and device | |
| CN111966289A (en) | Partition optimization method and system based on Kafka cluster | |
| CN109951326B (en) | Data processing method, device, storage medium and processor | |
| CN110908870B (en) | Method and device for monitoring resources of mainframe, storage medium and equipment | |
| CN112860412B (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN112579552A (en) | Log storage and calling method, device and system | |
| CN113342806A (en) | Big data processing method and device, storage medium and processor | |
| CN111796993B (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN107704362A (en) | A kind of method and device based on Ambari monitoring big data components | |
| CN113626194A (en) | Report file generation method, device, equipment and readable storage medium | |
| CN116126621A (en) | Task monitoring method of big data cluster and related equipment | |
| CN112540842A (en) | Method and device for dynamically adjusting system resources | |
| CN108810299B (en) | Information analysis method, medium and equipment | |
| CN112596974A (en) | Full link monitoring method, device, equipment and storage medium | |
| CN111211938A (en) | Biological information software monitoring system and method | |
| CN117435367B (en) | User behavior processing method, device, equipment, storage medium and program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191210 |
|
| RJ01 | Rejection of invention patent application after publication |