CN115955388A - A Distributed Cloud Comprehensive Alarm System - Google Patents

Abstract

The invention relates to the technical field of cloud computing, in particular to a distributed cloud comprehensive alarm system, which consists of an alarm message access layer, a stream processing layer, a data layer and an application layer; the beneficial effects are that: the distributed cloud comprehensive alarm system provided by the invention realizes the centralized access, analysis and processing of the alarm of the nano-tube distributed cloud platform. Meanwhile, the method supports personnel in each professional operation and maintenance queue to quickly find, respond in time and accurately position faults, shortens the service interruption time and realizes key guarantee of important customers and important services. Meanwhile, the system technical architecture supports heterogeneous cloud and has the characteristics of high performance, high reliability, expandability and the like.

Description

A Distributed Cloud Comprehensive Alarm System

technical field

The invention relates to the technical field of cloud computing, in particular to a distributed cloud comprehensive alarm system.

Background technique

Distributed Cloud (Distributed Cloud) refers to the cloud service provider (Cloud Service Provider, CSP) distributes public cloud services to different physical locations, and the CSP is responsible for the operation, governance, update and evolution of cloud services. Distributed clouds generally adopt a centralized operation and maintenance mode. In this context, the operation and maintenance support system generally adopts a two-level architecture: distributed cloud platform, and center-side operation and maintenance support platform.

In the existing technology, the distributed cloud platform is deployed on the client side, and the monitoring data, alarm data, performance data, and capacity data are sent back to the center side through a dedicated network; the operation and maintenance support platform on the center side is based on the distributed cloud platform. Return data to realize the core functions of centralized operation and maintenance, including distributed cloud capacity monitoring, performance monitoring, alarm monitoring, etc. Among them, the alarm system is the most important subsystem in the entire operation and maintenance support system, and the architecture design is particularly important.

However, the traditional alarm system cannot realize the centralized access, analysis and processing of the alarms of the managed distributed cloud platform; at the same time, it is difficult to support the professional operation and maintenance queue personnel to quickly discover, respond in time, and accurately locate faults; it cannot shorten the duration of business interruption, Realize key guarantees for important customers and important businesses.

Contents of the invention

The purpose of the present invention is to provide a distributed cloud integrated alarm system to solve the problems raised in the above background technology.

In order to achieve the above object, the present invention provides the following technical solutions: a distributed cloud integrated alarm system, the distributed cloud integrated alarm system is composed of an alarm message access layer, a stream processing layer, a data layer and an application layer;

The alarm message access layer, based on SD-WAN, realizes the network link connection between the distributed cloud platform and the Center side;

The stream processing layer, based on the message flow drive of the message cluster, realizes the core function of alarm analysis and processing of the distributed cloud platform;

Data layer, using heterogeneous multi-database system architecture to construct the data layer;

The application layer implements core application scenarios based on the data layer.

Preferably, the alarm component in the distributed cloud platform, based on the Http request agent, is forwarded to the access agent on the central side through the SD-WAN network, and then forwarded to the message cluster through the access agent;

The SD-WAN network address is uniformly planned within the entire distributed cloud, and through policy control, one-way request access from the Http request agent on the distributed cloud platform to the Http access agent on the center side is realized;

The message cluster is built based on the open source message middleware kafka, and adopts a distributed cluster deployment architecture.

Preferably, the stream processing layer is built based on Flink distributed clusters, which supports high availability, high performance, and scalability; the stream processing layer includes Source nodes, standard stream mapping nodes, CMDB associated nodes, alarm convergence nodes, AI processing nodes, and automatic order dispatching nodes and persistent nodes.

Preferably, the Source node realizes the access of Kafka message flow based on the functions provided by the Flink framework;

The standard stream mapping node is mapped to a standard alarm message stream through the analysis of the string stream, and the subsequent processing nodes are all processed based on the standard alarm message stream;

The CMDB association node, based on the CMDB library, associates the configuration attributes of the alarm object for subsequent alarm analysis and processing, and updates the alarm status of the alarm object in the memory database;

The alarm convergence node is the core component of alarm analysis and processing;

AI processing node, based on external AI engine, encapsulates AI algorithm, realizes cluster analysis of unknown alarm messages; based on knowledge base, associates fault handling knowledge information;

The automatic dispatch node, based on the rule engine, will finally analyze and process the output alarm message to form work order information, match the fault processing queue and personnel, and push it to the peripheral work order system to realize automatic dispatch;

The persistent node writes the alarm message record output by the final analysis and processing into the relational database, and writes the original alarm message into the time series database.

Preferably, the alarm convergence node implements the following core functions:

Based on the real-time state of the alarm object in the memory database and the convergence rules in the rule engine, the alarms of different levels and the same alarm object can be combined to effectively avoid the peak processing pressure caused by the alarm storm on the subsequent processing nodes;

Alarm silencing based on rules.

Preferably, the data layer is composed of CMDB library, memory database, relational database and time series database.

Preferably, the CMDB library adopts an open source graph database ArangoDB to support information association matching of alarm objects;

The memory database adopts distributed Redis cluster to cache dictionary data and status data of alarm objects;

The relational database adopts the open source MariaDB high-availability cluster, which is used to store configuration data, persist alarm processing record data, and provide data query for upper-layer applications;

The time series database adopts InfluxDB distributed cluster to store original alarm records and provide data support for upper-layer application display.

Preferably, the application layer includes the following subsystems: alarm presentation, alarm analysis, rule configuration, and scene monitoring.

Preferably, the alarm display is oriented to various professional operation and maintenance personnel, and an alarm application display view is provided based on the alarm data;

Alarm analysis, for professional operation and maintenance personnel, provides fault location and root cause analysis functions;

Rule configuration, for configuration managers, provides the front-end configuration function of alarm rules;

Scenario monitoring, oriented to key security scenarios, realizes the alarm monitoring large screen, network topology, and application topology scenario monitoring functions of re-insurance objects.

Compared with prior art, the beneficial effect of the present invention is:

The distributed cloud comprehensive alarm system proposed by the present invention realizes the centralized access, analysis and processing of the alarms of the managed distributed cloud platform. At the same time, support the professional operation and maintenance queue personnel to quickly discover, respond in time, and accurately locate faults, shorten the duration of business interruption, and achieve key guarantees for important customers and important businesses. At the same time, the system technical architecture supports heterogeneous clouds, and has the characteristics of high performance, high reliability, and scalability.

Description of drawings

Fig. 1 is the architecture diagram of the distributed cloud comprehensive alarm system of the present invention;

Fig. 2 is a technical architecture diagram of the distributed cloud comprehensive alarm system of the present invention;

Fig. 3 is an alarm access architecture diagram of the distributed cloud comprehensive alarm system of the present invention;

Fig. 4 is a diagram of an alarm stream processing model of the distributed cloud integrated alarm system of the present invention.

Detailed ways

In order to clearly and completely describe the purpose, technical solution, and advantages of the present invention, the embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are part of the embodiments of the present invention, rather than all embodiments, and are only used to explain the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. All other embodiments obtained under the premise of creative work all belong to the protection scope of the present invention.

Embodiment one

Please refer to Fig. 1 to Fig. 4, the present invention provides a kind of technical solution: a kind of distributed cloud comprehensive warning system, described distributed cloud comprehensive warning system is made up of warning message access layer, stream processing layer, data layer and application layer ;

The alarm message access layer, based on SD-WAN, realizes the network link connection between the distributed cloud platform and the Center side; the alarm component in the distributed cloud platform, based on the Http request agent, forwards the access to the center side through the SD-WAN network Agent, and then forwarded to the message cluster through the access agent; the SD-WAN network address is uniformly planned within the entire distributed cloud, and through policy control, it is realized from the Http request agent on the distributed cloud platform to the Http access agent on the center side One-way request access; the message cluster is built based on the open source message middleware kafka, and adopts a distributed cluster deployment architecture;

Stream processing layer, driven by message stream based on message cluster, realizes the core functions of alarm analysis and processing of distributed cloud platform; stream processing layer is built based on Flink distributed cluster, supports high availability, high performance, and scalability; stream processing layer includes Source nodes , standard flow mapping node, CMDB associated node, alarm convergence node, AI processing node, automatic order dispatching node, and persistent node; the Source node realizes the access of Kafka message flow based on the functions provided by the Flink framework; the standard flow mapping node, through The parsing of the string stream is mapped to a standard alarm message flow, and the subsequent processing nodes are processed based on the standard alarm message flow; the CMDB association node is based on the CMDB library, and the configuration attributes of the associated alarm object are used for subsequent alarm analysis and processing, and Update the alarm status of the alarm object in the memory database; the alarm convergence node is the core component of alarm analysis and processing; the AI processing node, based on the external AI engine, encapsulates the AI algorithm, and realizes the cluster analysis of unknown alarm messages; based on the knowledge base, associated faults Processing knowledge information; automatic order dispatching nodes, based on the rule engine, will finally analyze and process the output alarm messages to form work order information, match fault processing queues and personnel, and push them to peripheral work order systems to realize automatic order dispatching; persistent nodes, Write the alarm message record output by the final analysis and processing into the relational database, and write the original alarm message into the time series database; the alarm convergence node realizes the following core functions: based on the real-time status of the alarm object in the memory database, the convergence rules in the rule engine, Realize the merging of alarms at different levels and the same alarm object, effectively avoiding the peak processing pressure caused by alarm storms on subsequent processing nodes; realize alarm silence based on rules;

The data layer adopts heterogeneous multi-database system architecture to construct the data layer; the data layer is composed of CMDB library, memory database, relational database and time series database; the CMDB library uses the open source graph database ArangoDB to support information association and matching of alarm objects; The database uses a distributed Redis cluster to cache dictionary data and status data of alarm objects; the relational database uses an open-source MariaDB high-availability cluster to store configuration data, persist alarm processing record data, and provide Data query; time series database adopts InfluxDB distributed cluster to store original alarm records and provide data support for upper-layer application display;

The application layer implements core application scenarios based on the data layer; the application layer includes the following subsystems: alarm display, alarm analysis, rule configuration, and scene monitoring; alarm display, for professional operation and maintenance personnel, provides alarm application display views based on alarm data; alarm Analysis, for professional operation and maintenance personnel, provides fault location and root cause analysis functions; rule configuration, for configuration management personnel, provides front-end configuration functions for alarm rules; scene monitoring, for key guarantee scenarios, realizes alarm monitoring of re-insurance objects Screen, network topology, and application topology scene monitoring functions.

Embodiment two

Architecture method of comprehensive alarm system for distributed cloud platform.

As shown in Figure 1, it is the architecture of the comprehensive alarm system implemented by the present invention. The system is divided into four layers as a whole: alarm message access, stream processing layer, data layer, and application layer, as follows.

1. Alarm message access layer

The distributed cloud platform is based on the SD-WAN network connection and the data return link on the center side. SD-WAN network is a software-controlled wide area network, technically based on VPN tunnel technology, which can realize safe and reliable data transmission.

The alarm component of the distributed cloud platform is responsible for the monitoring and alarm of platform operation, and sends the alarm information back to the comprehensive alarm system on the central side based on the message. In order to achieve concurrent access to distributed cloud messages, alarm messages are encapsulated based on internal protocols and adopt the JSON data format. The specific structure is described as follows:

1) The header part of the message, including: cloud platform code, number of alarm records, message length, trigger time and other metadata information;

2). The message data part contains a list of specific alarm message records.

The alarm messages of the distributed cloud platform are uniformly connected to the message cluster of the comprehensive alarm system on the central side, and drive the upper stream processing layer for analysis and processing.

The message cluster uses the open source message middleware kafka, and is based on a distributed cluster architecture.

2. Stream processing layer

As shown in Figure 2, it is the technical architecture of the integrated alarm system of the distributed cloud platform. Among them, the stream processing layer is the core subsystem of the comprehensive alarm, based on the message flow drive of the message cluster, to realize the core function of alarm analysis and processing of the distributed cloud platform.

The stream processing layer is built based on the Flink distributed cluster, which supports high availability, high performance, and scalability.

Each processing node (operator) module of the stream processing layer logically includes: access, standardization, object association, convergence, AI operator, automatic dispatch, notification, and persistence.

In actual operation, based on the framework and configuration, the concurrent processing of each processing node can be realized, and the concurrency of different nodes can be adjusted according to the actual load.

3. Data layer

The comprehensive alarm system of the distributed cloud adopts a high-performance architecture due to the complexity of multiple data sources and processing logic, and the data layer adopts a multi-database system architecture.

The CMDB library is built using the open source graph database ArangoDB to support the information association and matching of alarm objects.

The memory database uses a distributed Redis cluster to cache dictionary data and status data of alarm objects.

The relational database adopts the open source MariaDB high-availability cluster, which is used to store configuration data, persist alarm processing record data, and provide data query for upper-layer applications.

The time series database is used to store original alarm records and provide data support for upper-layer application display.

4. Application layer

The application layer implements core application scenarios based on the data layer, mainly including the following subsystems: alarm display, alarm analysis, rule configuration, and scene monitoring.

Alarm display, for professional operation and maintenance personnel, provides alarm analysis and display based on alarm data.

Alarm analysis, for professional operation and maintenance personnel, provides fault location and root cause analysis functions.

Rule configuration, for configuration managers, provides the front-end configuration function of alarm rules.

Scenario monitoring, oriented to key security scenarios, realizes scene monitoring functions such as alarm monitoring large screen, network topology, and application topology of re-insurance objects.

5. Center-side cluster deployment architecture

The center side system involved in the present invention is based on Kubernetes container cluster deployment, adopts high availability, load balancing architecture, specifically as follows:

1), the message middleware adopts the open source Kafka cluster architecture;

2), the database adopts MariaDB Galera cluster "one master and two slaves" architecture;

3), the time series database adopts InfluxDB distributed cluster architecture;

4), stream processing adopts Flink distributed cluster architecture, and is based on Kubernetes and Docker image deployment;

5) The core application logic components are deployed with multiple copies of Service to achieve high availability and load balancing.

Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications and substitutions can be made to these embodiments without departing from the principle and spirit of the present invention. and modifications, the scope of the invention is defined by the appended claims and their equivalents.

Claims (9)

1. A distributed cloud integrated alarm system, characterized in that: the distributed cloud integrated alarm system is composed of alarm message access layer, stream processing layer, data layer and application layer; The alarm message access layer, based on SD-WAN, realizes the network link connection between the distributed cloud platform and the Center side; The stream processing layer, based on the message flow drive of the message cluster, realizes the core function of alarm analysis and processing of the distributed cloud platform; Data layer, using heterogeneous multi-database system architecture to construct the data layer; The application layer implements core application scenarios based on the data layer. 2. A kind of distributed cloud comprehensive warning system according to claim 1, it is characterized in that: the warning component in the distributed cloud platform, based on Http request agent, is forwarded to the access agent of central side by SD-WAN network, Then forward to the message cluster through the access agent; The SD-WAN network address is uniformly planned within the entire distributed cloud, and through policy control, one-way request access from the Http request agent on the distributed cloud platform to the Http access agent on the center side is realized; The message cluster is built based on the open source message middleware kafka, and adopts a distributed cluster deployment architecture. 3. A kind of distributed cloud integrated alarm system according to claim 1, characterized in that: the stream processing layer is based on Flink distributed cluster construction, supports high availability, high performance, and scalability; the stream processing layer includes Source nodes, standard Flow mapping node, CMDB association node, alarm convergence node, AI processing node, automatic dispatch node and persistence node. 4. a kind of distributed cloud comprehensive warning system according to claim 3, is characterized in that: Source node realizes the access of Kafka message flow based on the function that Flink framework provides; The standard stream mapping node is mapped to a standard alarm message stream through the analysis of the string stream, and the subsequent processing nodes are all processed based on the standard alarm message stream; The CMDB association node, based on the CMDB library, associates the configuration attributes of the alarm object for subsequent alarm analysis and processing, and updates the alarm status of the alarm object in the memory database; The alarm convergence node is the core component of alarm analysis and processing; AI processing node, based on external AI engine, encapsulates AI algorithm, realizes cluster analysis of unknown alarm messages; based on knowledge base, associates fault handling knowledge information; The automatic dispatch node, based on the rule engine, will finally analyze and process the output alarm message to form work order information, match the fault processing queue and personnel, and push it to the peripheral work order system to realize automatic dispatch; The persistent node writes the alarm message record output by the final analysis and processing into the relational database, and writes the original alarm message into the time series database. 5. A kind of distributed cloud comprehensive warning system according to claim 4, is characterized in that: the warning convergence node realizes the following core functions: Based on the real-time state of the alarm object in the memory database and the convergence rules in the rule engine, the alarms of different levels and the same alarm object can be combined to effectively avoid the peak processing pressure caused by the alarm storm on the subsequent processing nodes; Alarm silencing based on rules. 6. A distributed cloud comprehensive alarm system according to claim 1, wherein the data layer is composed of a CMDB library, an internal memory database, a relational database and a time series database. 7. A kind of distributed cloud comprehensive warning system according to claim 6, is characterized in that: CMDB storehouse, adopts the open-source graph database ArangoDB, is used to support the information association matching of warning object; The memory database adopts distributed Redis cluster to cache dictionary data and status data of alarm objects; The relational database adopts the open source MariaDB high-availability cluster, which is used to store configuration data, persist alarm processing record data, and provide data query for upper-layer applications; The time series database adopts InfluxDB distributed cluster to store original alarm records and provide data support for upper-layer application display. 8. A distributed cloud integrated alarm system according to claim 1, wherein the application layer includes the following subsystems: alarm presentation, alarm analysis, rule configuration, and scene monitoring. 9. A distributed cloud comprehensive alarm system according to claim 8, characterized in that: alarm display, for professional operation and maintenance personnel, provides an alarm application display view based on alarm data; Alarm analysis, for professional operation and maintenance personnel, provides fault location and root cause analysis functions; Rule configuration, for configuration managers, provides the front-end configuration function of alarm rules; Scenario monitoring, oriented to key security scenarios, realizes the alarm monitoring large screen, network topology, and application topology scenario monitoring functions of re-insurance objects.

Images