Disclosure of Invention
In order to solve the problems, the invention provides an electric power data monitoring event analysis system and method, which realize distributed acquisition, edge calculation analysis and centralized report summarization of events, and improve the flexibility of event processing system deployment, the efficiency accuracy of event positioning and the overall robustness of the system.
According to some embodiments, the following technical scheme is adopted in the disclosure:
an electric power data monitoring event analysis system comprises an event acquisition module, an event service host and an event database, wherein:
the event acquisition modules comprise a plurality of modules and are configured to remotely acquire event information of the network device, the workstation and the isolation device;
the event service host is configured to acquire event information uploaded by the event acquisition modules through a message bus, realize automatic control over current limiting or fusing requested by each event acquisition module according to the quantity and flow of each event acquisition module, acquire the event information, analyze and extract event key information, and sequentially transmit corresponding event information according to a preset priority;
the event database is configured to store and classify the event information summarized by the event service host, and provide an access interface to receive the request of the event service host and provide the event information to be read.
As an optional implementation manner, the event collection module includes an SNMP microservice module, an SYSLOG microservice module and a microservice module, the SNMP microservice module is configured to perform event recording of network access and traffic overrun events for a router and a gateway-off device, and the SYSLOG microservice module performs event recording of events such as login and operation of a workstation in a station; the microserver module with the self-defined event recording interface records the events of operation, maintenance and operation of the microserver.
As an alternative embodiment, the event collection module has a standard SNMP protocol interface, a standard SYSLOG protocol interface, or/and a custom direct log call interface.
As an alternative implementation, the event collection module adopts a read mode, when an event occurs, the event can be responded quickly, the event is first recorded into the cache, the event is recorded into the cache in time sequence after being analyzed and processed, and the cached high-level event is sent to the event service host through the message bus with the cache.
In an alternative embodiment, the event service host includes at least two servers, one of the servers is used as a main server, the other server is used as a standby server, and hot switching is performed between the main server and the standby server.
As an alternative implementation, the primary server and the standby server perform timing or real-time synchronization according to the number of events, the time interval, and the states of the primary and standby machines, so as to ensure the integrity of the event records.
As an alternative embodiment, the event service host adopts a local RPC technology, adopts a load balancing mode, and realizes automatic control of current limiting or fusing of the event acquisition module request according to the number of links and the flow of the event acquisition module, thereby realizing high concurrency processing and high-efficiency summarizing functions of each event acquisition module.
As an alternative implementation, the event service host has a main stream SQL database access interface, and backups the event record to the database according to the actual requirement, so as to perform more generalized persistent record.
As an alternative embodiment, the event database includes a local KV database and an event SQL database, where the local KV database is used to store the aggregated event classification data, and the event SQL database is used to store the backup data.
The working method based on the system comprises the following steps:
each event acquisition module carries out initial configuration and remotely acquires event information of a network device, a workstation and an isolation device;
selecting a main server or a standby server in the event service host, monitoring the acquired event information, and replacing another event service host if the monitoring fails;
and acquiring uploaded event information, analyzing and screening the events, and if the events accord with the preset priority, sequentially uploading corresponding event information to wait for subsequent processing.
Compared with the prior art, the beneficial effect of this disclosure is:
the invention provides a terminal self-adaptive event monitoring and analyzing technology, develops an event monitoring and positioning analysis system, realizes distributed acquisition, edge calculation analysis and centralized report summarization of events, and improves the flexibility of event processing system deployment, the efficiency accuracy of event positioning and the overall robustness of the system.
The present disclosure provides a distributed high-concurrency data drainage technique, which implements hierarchical high-speed recording and serialization processing of events, and improves event recording efficiency and event recording integrity of a front end under a condition of big data concurrency.
The invention provides an event online configurable restart-free deployment technology, which realizes the consistent deployment of system functions and actual environment requirements, and improves the flexibility and adaptability of an event recording mode and the time integrity of events.
According to the distributed event processing method and system, the distributed arrangement of the event acquisition modules is adopted, the micro-service architecture is adopted, the distributed arrangement is supported, the expansion and the transplantation are easy, the shunting of event processing is realized to a certain extent, and the processing and recording pressure during a large amount of burst data is avoided; the hot standby function can be configured during distributed deployment, and the efficiency and timeliness of information transmission are improved.
The event acquisition module disclosed by the invention adopts a micro-service modular design, the interface and the functional module are easy to expand, and event recording can be carried out on other types of devices according to the requirements.
According to the method and the device, the corresponding event information is transmitted in sequence according to the preset priority, the requirements of rapid real-time performance and integrity of event records under the conditions of high speed, high concurrency and the like can be met, the event information with high priority is guaranteed to be uploaded and analyzed preferentially, a certain event is located rapidly, and the timeliness of event processing of a transformer substation is guaranteed.
The event recording grade, the event grade, the recording time interval, the recording capacity, the recording file format and the like can be flexibly configured according to actual requirements, can be automatically covered according to the configured time interval, and ensure that the storage capacity is not out of limit.
According to the method and the device, the event information is analyzed, screened and graded, and the event information with high priority is preferentially transmitted and displayed to remind relevant personnel to process, so that the events are timely solved, and the accurate positioning capability and the processing response capability of the events and accidents of the transformer substation are improved.
The specific implementation mode is as follows:
the present disclosure is further described with reference to the following drawings and examples.
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
As described in the background, event records of the current substation are classified in an event record classification manner; the first-level buffer is used, so that the loss is easy and the recording is not timely; the format of the recorded file content is fixed and can not be configured; the recording speed is slow, and the recording is unreliable when the large data volume is high and concurrent.
To solve this problem, the present embodiment provides an event analysis system deployed in a substation monitoring device, as shown in fig. 1, including a collection front end and a server end.
The acquisition front end comprises a plurality of event acquisition modules which are configured to remotely acquire event information of the network device, the workstation and the isolation device;
the server side comprises an event service host and an event database, wherein:
the event service host is configured to acquire the event information uploaded by the event acquisition modules through a message bus, realize automatic control on the current limiting or fusing requested by each event acquisition module according to the quantity and flow of each event acquisition module, acquire the event information, analyze and extract event key information, and sequentially transmit corresponding event information according to a preset priority;
and the event database is configured to store and classify the event information summarized by the event service host, and provides an access interface to receive the request of the event service host and provide the event information required to be read.
Specifically, the event collection module has a functional service interface: a standard SNMP protocol interface, a standard SYSLOG protocol interface and a self-defined direct log calling interface. The system can respectively carry out remote event acquisition on the in-station equipment with the standard specification, such as a network device, a workstation, an isolation device and the like, so that the security events in the station can be recorded by diaries; the provided log recording interface can record events such as man-machine operation, maintenance, operation and the like on the machine, and a switchable debugging interface is provided for debugging of initial operation and maintenance personnel; because of the adoption of the micro-service modular design, the interfaces and the functional modules are easy to expand, and the event recording can be carried out on other types of devices according to the needs.
The interfaces are provided in the form of a micro-service module, can be deployed at an acquisition device end or independently set a micro-service host to form an event acquisition front end, and can be used for acquiring related logs in a targeted manner; for example, the SNMP micro-service module records events such as network access, flow overrun and the like aiming at equipment such as a router, a network shutdown and the like, and the SYSLOG micro-service module records events such as login, operation and the like of a workstation in the station; the micro service module with a user-defined event recording interface records events such as operation, maintenance, operation and the like of the local computer;
by the deployment mode, distributed deployment of micro-services and acquisition of various types of events are realized, shunting of event processing is realized to a certain extent, and processing and pressure recording during a large amount of burst data are avoided; the acquisition front end adopts a READTOR mode, the event can respond quickly when occurring, the event is firstly recorded into a cache, and is recorded into a KV database according to the time sequence after being analyzed and processed (the database has small volume, low requirements on software and hardware of the acquisition front end and high read-write speed), and meanwhile, the cached high-level event is sent to an event service host through a message bus with the cache. The mode of caching + KV library ensures the high speed and high efficiency of event record, and the message bus transmission technology with cache ensures the integrity of event transmission;
configuration items such as acquisition strategies, priorities, transmission directions, event types and the like are configured in advance, and uploading and filtering are carried out on event information through the configuration items, so that the priority uploading and transmission bandwidth of events can be ensured.
Events recorded by the event acquisition front end are collected to the event service host through the message bus to form a summary of all relevant events in the station. The event service host adopts a local RPC technology, if the event receiving adopts a thread pool + load balancing mode, the management of the function module adopts a service discovery technology and an RPC service management technology, and the automatic control of current limiting, fusing and the like of the acquisition front-end request is realized according to the number and the flow of links of the acquisition front-end, so that the high concurrent processing and the efficient summarizing function of the events of each acquisition module are realized; and the collected events are classified and recorded into a local KV library of the host, so that a global data basis is provided for subsequent in-station event analysis.
As shown in fig. 2, for the event collection front end, first, the loading configuration of each event collection module is performed, so as to implement initialization, and then, main/standby selection is performed, if an event service host (host) is selected, the event service host is registered to the message bus, so as to implement monitoring, and if monitoring is successful, the event collection front end waits for the occurrence of an event, so as to collect the event. If the monitoring is unsuccessful, the main-standby switching is carried out, and the events are collected after the occurrence of the events.
Meanwhile, the event service host has a main stream SQL database access interface, and can backup event records into a database according to actual requirements to carry out more generalized persistent records.
The two event service hosts have a hot standby function, and can perform timing or real-time synchronization according to the number of events, time intervals and states of the main hosts and the standby hosts, so that the integrity of event records is ensured.
As shown in fig. 3, the event service host is loaded and configured, initialization is implemented, whether the host is used or not is confirmed, if yes, the host is registered to the message bus, if successful, subsequent processing is continued, if unsuccessful, the host and standby are switched to the standby, and subsequent processing is performed. The subsequent processing comprises load balancing, corresponding loading and starting of the functional module are found, events are waited for, if the events occur, whether the events are high concurrency is determined, if the events occur, dynamic capacity expansion is carried out, if the events do not occur, screening, classifying and grading processing of the events are carried out, and information corresponding to preset conditions is transmitted to enter a database for backup.
If the time information which does not meet the preset conditions exists, instance information is created, the instance information is added into service management, load balancing and event processing are carried out, and the information in the database is combined according to the time sequence.
In conclusion, the system adopts a micro-service architecture, supports distributed deployment and is easy to expand and transplant; the hot standby function can be configured during distributed deployment.
The event record, search and processing performance is high by adopting a record mode of a cache, a second-level cache technology and a local database and adopting a high-speed search engine technology and a buffer type message bus technology, the requirements on the quick real-time performance and the integrity of the event record under the conditions of high speed, high concurrency and the like can be met, the event record is permanently stored, the event record can be automatically covered according to the configured time interval, and the storage capacity is not out of limit.
Meanwhile, the event recording parameters can be configured, and are specifically represented as follows:
the monitoring device and the device type can be configured, and certain types of events of the device are recorded;
the uploading platform and the uploading strategy are configurable, so that the source address and the destination address of an event can be recorded, the source direction and the destination direction can be recorded, and the recording integrity of the flow direction of the event is ensured;
the event recording grade, the event grade, the recording time interval, the recording capacity, the recording file format and the like can be flexibly configured according to actual requirements, and the log has types of auditing, operating, events and the like.
The system reserves a log event uploading interface, supports a standardized log uploading protocol (syslog and the like), is configurable, supports log import and export, can export the csv file and the report form of the common text file, and supports remote uploading of the log export file.
The system has a basic authority identification function, can configure whether a remote user has user authority for browsing, downloading, modifying and deleting, analyzes and processes after event collection, can analyze and process according to categories such as equipment type, log type, event type and event grade, generates an event report, and is convenient for quickly positioning a certain event.
For large-scale application occasions with high recording efficiency requirements, each service module can be respectively deployed on different hosts to customize configuration files, so that the function customization of the hosts is realized, and events with different grades and different types are respectively recorded. And remote data transmission is avoided when the system is completely deployed on a single host, so that higher recording and analysis efficiency is achieved.
Each service module supports the distributed cluster and the hot standby function through a message bus, and ensures that each host stores complete event records.
By configuring the address of the background host, the user authority and the like, the local event record can be posted to the background host according to the filtering condition, so that the remote event browsing is realized, and the event record file can also be uploaded to the background host.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure, and various modifications and changes may be made to the present disclosure by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present disclosure should be included in the protection scope of the present disclosure.
Although the present disclosure has been described with reference to specific embodiments, it should be understood that the scope of the present disclosure is not limited thereto, and those skilled in the art will appreciate that various modifications and changes can be made without departing from the spirit and scope of the present disclosure.