CN113487170A - Full link monitoring system with layered technical architecture - Google Patents

Full link monitoring system with layered technical architecture Download PDF

Info

Publication number
CN113487170A
CN113487170A CN202110749348.0A CN202110749348A CN113487170A CN 113487170 A CN113487170 A CN 113487170A CN 202110749348 A CN202110749348 A CN 202110749348A CN 113487170 A CN113487170 A CN 113487170A
Authority
CN
China
Prior art keywords
data
layer
monitoring
monitoring data
service function
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110749348.0A
Other languages
Chinese (zh)
Inventor
高伟
李劲松
庞日海
丘志新
张崇哲
陈韩霏
兰天
康凯
周秀沅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangzhou Xin'an Data Co ltd
Original Assignee
Guangzhou Xin'an Data Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangzhou Xin'an Data Co ltd filed Critical Guangzhou Xin'an Data Co ltd
Priority to CN202110749348.0A priority Critical patent/CN113487170A/en
Publication of CN113487170A publication Critical patent/CN113487170A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • G06Q10/06Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063Operations research, analysis or management
    • G06Q10/0631Resource planning, allocation, distributing or scheduling for enterprises or organisations
    • G06Q10/06315Needs-based resource requirements planning or analysis
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/06Energy or water supply
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y04INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
    • Y04SSYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
    • Y04S10/00Systems supporting electrical power generation, transmission or distribution
    • Y04S10/50Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications

Landscapes

  • Business, Economics & Management (AREA)
  • Human Resources & Organizations (AREA)
  • Engineering & Computer Science (AREA)
  • Economics (AREA)
  • Strategic Management (AREA)
  • Theoretical Computer Science (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Health & Medical Sciences (AREA)
  • Marketing (AREA)
  • General Physics & Mathematics (AREA)
  • General Business, Economics & Management (AREA)
  • Tourism & Hospitality (AREA)
  • Physics & Mathematics (AREA)
  • Public Health (AREA)
  • Primary Health Care (AREA)
  • Water Supply & Treatment (AREA)
  • Development Economics (AREA)
  • Educational Administration (AREA)
  • General Health & Medical Sciences (AREA)
  • Game Theory and Decision Science (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Testing And Monitoring For Control Systems (AREA)

Abstract

The invention provides a full link monitoring system with a layered technical architecture, which comprises the following architecture layers: the data source layer is in butt joint with various data platforms and is used for collecting and managing the monitoring data of the various data platforms; the data acquisition layer comprises a plurality of data acquisition devices, the plurality of data acquisition devices are used for acquiring monitoring data of a plurality of data platforms from the data source layer and then sending the acquired monitoring data to the data transmission bus; the persistent layer acquires the monitoring data from the data transmission bus and performs storage management on the acquired monitoring data; the business logic layer analyzes and summarizes corresponding monitoring data in the persistent layer according to business function requirements from preset application; and the application layer acquires the service function requirements from the preset application through the application server, sends the service function requirements to the service logic layer, and then integrates and displays the monitoring data analysis and summary results obtained by the service logic layer according to the service function requirements.

Description

Full link monitoring system with layered technical architecture
Technical Field
The invention relates to the technical field of data processing, in particular to a full link monitoring system with a layered technical architecture.
Background
At present, the overall power supply in China is in a tension state, the power shortage in partial areas and partial time periods is serious, and with the continuous expansion of the scale of a power grid and the continuous increase of the complexity of the power grid in China, the power consumption analysis of power grid data of each power utilization area, power utilization peak periods and the like is needed, so that a reasonable power supply scheme is obtained and implemented to relieve the tension of the power supply. For this reason, the grid system usually collects grid data generated during operation from the grids in each power consumption area, transmits the collected grid data to the data center, and then performs comprehensive power consumption analysis on the grid data in the data center by using the data analysis system.
However, because the number of service systems of the power grid is large at present, and the amount of power grid data to be collected by the power grid system is very large, so that the problems of incomplete power grid data collection, low power grid data collection quality and the like often occur in the process of collecting power grid data, at present, data platforms such as a big data analysis platform, a real-time data service platform, an unstructured data service platform, a data asset management platform, a data quality management platform and the like are generally adopted to monitor the power grid data in a full link from production to application to decommissioning. However, because the amount of the power grid data is very large, the corresponding amount of the monitoring data is also very large, and when an operation and maintenance person of the power grid system needs a certain item of the monitoring data, the operation and maintenance person of the power grid system is difficult to accurately acquire the needed monitoring data in time due to the large amount of the monitoring data.
Disclosure of Invention
The invention aims to solve the technical problem of how to enable operation and maintenance personnel of a power grid system to timely and accurately acquire required monitoring data.
To solve the above technical problem, the present invention provides a full link monitoring system with a layered technical architecture, which comprises the following architecture layers:
the data source layer is in butt joint with various data platforms and is used for collecting and managing the monitoring data of the various data platforms;
the data acquisition layer comprises a plurality of data acquisition devices, the plurality of data acquisition devices are used for acquiring monitoring data of a plurality of data platforms from the data source layer and then sending the acquired monitoring data to the data transmission bus;
the persistent layer acquires the monitoring data from the data transmission bus and performs storage management on the acquired monitoring data;
the business logic layer analyzes and summarizes corresponding monitoring data in the persistent layer according to business function requirements from preset application;
and the application layer acquires a service function requirement from a preset application through the application server, sends the service function requirement to the service logic layer, and then integrates and displays a monitoring data analysis and summarization result obtained by the service logic layer according to the service function requirement.
Preferably, the plurality of data platforms include a big data analysis platform, a real-time data service platform, an unstructured data service platform, a data asset management platform and/or a data quality management platform.
Preferably, the plurality of data collectors include an index data collector, a log data collector, a metadata collector, a task data collector, an application data collector and/or a cluster monitoring data collector.
Preferably, the data transmission bus is a Kafka cluster.
Preferably, after the persistent layer acquires the monitoring data, the persistent layer filters, cleans and converts the monitoring data, and then loads the filtered, cleaned and converted monitoring data into a storage container for storage management.
Preferably, the storage container includes a MySQL database adopting a dual-server hot-standby configuration mode.
Preferably, after the service logic layer obtains a service function requirement from a preset application, a data processing means of data integration, data processing and data analysis is adopted to perform service logic analysis processing on corresponding monitoring data in the persistent layer, so that the corresponding monitoring data is summarized according to the service function requirement.
Preferably, the application layer is provided with a data caching module, and the data caching module performs caching processing on the monitoring data analysis summary result obtained corresponding to the service function requirement of which the acquisition frequency exceeds a preset threshold value.
The invention has the following beneficial effects: in the full-link monitoring system provided by the invention, the application layer acquires the service function requirement from the preset application through the application server, and then sends the service function requirement to the service logic layer, the service logic layer analyzes and summarizes the corresponding monitoring data in the persistent layer according to the service function requirement to obtain the monitoring data analysis and summarization result, and then the application layer integrates and displays the monitoring data analysis and summarization result, so that the operation and maintenance personnel of the power grid system only need to input the service function requirement on the preset application, for example, the monitoring data required by the operation and maintenance personnel of the power grid system, and the application layer can display the corresponding monitoring data analysis and summarization result, thereby enabling the operation and maintenance personnel of the power grid system to timely and accurately acquire the required monitoring data.
Drawings
Fig. 1 is a block diagram of a full link monitoring system with a hierarchical technology architecture.
Detailed Description
The invention is described in further detail below with reference to specific embodiments.
The present embodiment provides a full link monitoring system with a layered technical architecture, as shown in fig. 1, the system includes five architecture layers, which are a data source layer, a data acquisition layer, a persistent layer, a business logic layer and an application layer, respectively, where the data source layer is located at the bottommost layer of the full link monitoring system, the data acquisition layer is located between the data source layer and the persistent layer, the persistent layer is located between the data acquisition layer and the business logic layer, the business logic layer is located between the persistent layer and the application layer, and the application layer is located at the topmost layer of the full link monitoring system.
The data source layer is in butt joint with various data platforms and is used for collecting and managing the monitoring data of the various data platforms, and the data source of the monitoring data can be divided into a relational database, a non-relational database, a text, an API (application programming interface) and the like according to type division. The multiple data platforms comprise a big data analysis platform, a real-time data service platform, an unstructured data service platform, a data asset management platform and/or a data quality management platform. In this embodiment, the data source layer interfaces with a plurality of data platforms and different components, different storage devices and different work tools therein, for example: the system comprises an MPP mass data real-time analysis architecture, an Oracle database, a MySQL database, a Kafka data storage tool, a Hadoop distributed system infrastructure, a basic detail layer in a GP cluster, an OGG data operation tool, a June operation tool, a Hive data warehouse tool and the like.
The data acquisition layer comprises a plurality of data acquisition devices, the plurality of data acquisition devices are used for acquiring monitoring data of a plurality of data platforms from the data source layer, ETL processing is carried out on the acquired monitoring data, and then the acquired monitoring data are sent to the data transmission bus. In this implementation, the plurality of data collectors include an index data collector, a log data collector, a metadata collector, a task data collector, an application data collector and/or a cluster monitoring data collector; the data acquisition mode comprises full acquisition, incremental acquisition, stream acquisition and the like. ETL processing includes data extraction (Extract), data transformation (Transform), and data loading (Load) in order to integrate scattered, messy, and non-uniform data together. The data transmission bus is a Kafka cluster, the data acquisition unit sends the acquired monitoring data to the Kafka cluster, and the safety of the monitoring data is guaranteed without loss by utilizing the distributed, high-availability and high-throughput characteristics of the Kafka cluster.
The persistent layer acquires monitoring data from the data transmission bus, and after the monitoring data is acquired, the monitoring data is filtered, cleaned and converted, and then the filtered, cleaned and converted monitoring data is loaded into a storage container for storage management. Wherein the filtering step is used for filtering invalid collected data, such as repeated dirty data; the cleaning step is to inspect and check the monitoring data, check the consistency of the data, process invalid values and missing values in the data, and the like; and the conversion step is used for converting data codes, data units and data granularity in the monitoring data according to the design of the data model. The storage container comprises a MySQL database and a Kafka container of a Kafka cluster, wherein: the MySQL database adopts a dual-computer hot standby configuration mode and is used for loading metadata and index data and ensuring the redundant backup of the data and the high availability of service; the Kafka container is used to load log pipeline data to facilitate subsequent flow computations.
In this embodiment, the persistent layer provides a data persistent interface to the outside, persistent data storage is implemented through technologies such as mybatis and hibernate, the data acquisition layer pushes data to the data persistent interface, and the data persistent layer performs automatic persistent update according to a storage mechanism after receiving the data, so that persistence of data management is implemented.
The service logic layer acquires a service function requirement from a preset application, analyzes and summarizes corresponding monitoring data in the persistent layer according to the service function requirement, and specifically, after acquiring the service function requirement from the preset application, the service logic layer performs service logic analysis processing on the corresponding monitoring data in the persistent layer by using data processing means of data integration, data processing and data analysis, so that the corresponding monitoring data is summarized according to the service function requirement.
The business logic layer has three analysis and summary modes for data, including data summary analysis, real-time flow calculation and relation analysis: the data summarization analysis is to perform batch processing calculation on data, such as summarization of resource use conditions of tenants every day and summarization of daily execution conditions of scheduling tasks, and the batch processing adopts Hive storage process, Spark and MySQL storage process to realize analysis and summarization of index data; the real-time Stream calculation specifically adopts a Kafka Stream and Spark Stream framework, loads data from a Kafka cluster in real time, and realizes logic processing by writing KSQL or Spark SQL; the relationship analysis is specifically to monitoring dependence of applications, indexes and scheduling tasks and blood relationship analysis, the relationships between the applications, the applications and the indexes and between the applications and the scheduling tasks are stored through a graph database, a blood relationship network is constructed, the blood relationship is subjected to reasoning analysis, and a blood relationship dependence graph and an influence analysis graph are obtained.
The application layer acquires the service function requirements from the preset application through the application server, then sends the service function requirements to the service logic layer, and then integrates and displays the monitoring data analysis and summary results obtained by the service logic layer according to the service function requirements, so that the functions of real-time monitoring of data, intelligent alarm, intelligent relationship discovery, abnormal processing scheme recommendation, blood relationship analysis, key application influence analysis, log analysis and the like are realized. The application layer is provided with a data caching module, and the data caching module is used for caching the monitoring data analysis and summary result obtained by the service function requirement with the acquisition times exceeding a preset threshold (for example, ten times).
In summary, in the full-link monitoring system provided in this embodiment, the application layer obtains a service function requirement from a preset application through the application server, and then sends the service function requirement to the service logic layer, the service logic layer analyzes and summarizes corresponding monitoring data in the persistent layer according to the service function requirement to obtain a monitoring data analysis and summarization result, and then the application layer integrates and displays the monitoring data analysis and summarization result.
The above description is only the embodiments of the present invention, and the scope of protection is not limited thereto. The insubstantial changes or substitutions will now be made by those skilled in the art based on the teachings of the present invention, which fall within the scope of the claims.

Claims (8)

1. A full link monitoring system with a layered technical architecture is characterized by comprising the following architecture layers:
the data source layer is in butt joint with various data platforms and is used for collecting and managing the monitoring data of the various data platforms;
the data acquisition layer comprises a plurality of data acquisition devices, the plurality of data acquisition devices are used for acquiring monitoring data of a plurality of data platforms from the data source layer and then sending the acquired monitoring data to the data transmission bus;
the persistent layer acquires the monitoring data from the data transmission bus and performs storage management on the acquired monitoring data;
the service logic layer acquires service function requirements from preset application, and analyzes and summarizes corresponding monitoring data in the persistence layer according to the service function requirements;
and the application layer acquires a service function requirement from a preset application through the application server, sends the service function requirement to the service logic layer, and then integrates and displays a monitoring data analysis and summarization result obtained by the service logic layer according to the service function requirement.
2. The full link monitoring system with a hierarchical technical architecture as set forth in claim 1 wherein the plurality of data platforms include a big data analytics platform, a real-time data services platform, an unstructured data services platform, a data asset management platform, and/or a data quality management platform.
3. The system according to claim 1, wherein the plurality of data collectors include an index data collector, a log data collector, a metadata collector, a task data collector, an application data collector, and/or a cluster monitoring data collector.
4. The system of claim 1, wherein the data transfer bus is a Kafka cluster.
5. The system according to claim 1, wherein the persistent layer filters, cleans and converts the monitoring data after acquiring the monitoring data, and loads the filtered, cleaned and converted monitoring data into a storage container for storage management.
6. The system according to claim 5, wherein the storage container comprises a MySQL database in a dual-server hot-standby configuration mode.
7. The system according to claim 1, wherein after the service logic layer obtains the service function requirement from the preset application, the service logic layer performs service logic analysis processing on the corresponding monitoring data in the persistent layer by using data processing means of data integration, data processing and data analysis, so that the corresponding monitoring data are summarized according to the service function requirement.
8. The system according to claim 1, wherein the application layer is provided with a data caching module, and the data caching module performs caching processing on the analysis and summary results of the monitoring data obtained corresponding to the service function requirements whose acquisition times exceed a preset threshold.
CN202110749348.0A 2021-07-01 2021-07-01 Full link monitoring system with layered technical architecture Pending CN113487170A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110749348.0A CN113487170A (en) 2021-07-01 2021-07-01 Full link monitoring system with layered technical architecture

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110749348.0A CN113487170A (en) 2021-07-01 2021-07-01 Full link monitoring system with layered technical architecture

Publications (1)

Publication Number Publication Date
CN113487170A true CN113487170A (en) 2021-10-08

Family

ID=77940250

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110749348.0A Pending CN113487170A (en) 2021-07-01 2021-07-01 Full link monitoring system with layered technical architecture

Country Status (1)

Country Link
CN (1) CN113487170A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114143177A (en) * 2021-12-01 2022-03-04 云赛智联股份有限公司 Business service monitoring system and monitoring method based on data blood margin
CN116757680A (en) * 2023-08-14 2023-09-15 深圳联友科技有限公司 Integration method of multiple monitoring platforms

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106339509A (en) * 2016-10-26 2017-01-18 国网山东省电力公司临沂供电公司 Power grid operation data sharing system based on large data technology
CN106997400A (en) * 2017-05-25 2017-08-01 南京多伦科技股份有限公司 A kind of monitoring of transit equipment O&M and data analysis system based on cloud service
CN108769207A (en) * 2018-05-30 2018-11-06 郑州云海信息技术有限公司 A kind of cloud platform resource monitoring method and system
CN112199430A (en) * 2020-10-15 2021-01-08 苏州龙盈软件开发有限公司 Business data processing system and method based on data middling station
CN112632025A (en) * 2020-08-25 2021-04-09 南方电网科学研究院有限责任公司 Power grid enterprise management decision support application system based on PAAS platform

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106339509A (en) * 2016-10-26 2017-01-18 国网山东省电力公司临沂供电公司 Power grid operation data sharing system based on large data technology
CN106997400A (en) * 2017-05-25 2017-08-01 南京多伦科技股份有限公司 A kind of monitoring of transit equipment O&M and data analysis system based on cloud service
CN108769207A (en) * 2018-05-30 2018-11-06 郑州云海信息技术有限公司 A kind of cloud platform resource monitoring method and system
CN112632025A (en) * 2020-08-25 2021-04-09 南方电网科学研究院有限责任公司 Power grid enterprise management decision support application system based on PAAS platform
CN112199430A (en) * 2020-10-15 2021-01-08 苏州龙盈软件开发有限公司 Business data processing system and method based on data middling station

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
周文琼;: "大数据环境下的电力客户服务数据分析系统", 计算机系统应用, no. 04 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114143177A (en) * 2021-12-01 2022-03-04 云赛智联股份有限公司 Business service monitoring system and monitoring method based on data blood margin
CN116757680A (en) * 2023-08-14 2023-09-15 深圳联友科技有限公司 Integration method of multiple monitoring platforms
CN116757680B (en) * 2023-08-14 2024-01-19 深圳联友科技有限公司 Integration method of multiple monitoring platforms

Similar Documents

Publication Publication Date Title
CN110022226B (en) Object-oriented data acquisition system and acquisition method
CN107256219B (en) Big data fusion analysis method applied to mass logs of automatic train control system
CN106651633B (en) Power utilization information acquisition system based on big data technology and acquisition method thereof
CN109669406A (en) A kind of remote online monitoring system and its workflow of industrial equipment
CN111077870A (en) Intelligent OPC data real-time acquisition and monitoring system and method based on stream calculation
CN113487170A (en) Full link monitoring system with layered technical architecture
CN108846076A (en) The massive multi-source ETL process method and system of supporting interface adaptation
CN102521781B (en) Safe region-crossing equipment uniform monitoring method based on independent monitoring services, and monitoring system for the same
CN112016828B (en) Industrial equipment health management cloud platform architecture based on streaming big data
CN108964269A (en) Power distribution network O&M and total management system
Liu et al. A big data framework for electric power data quality assessment
CN105303316A (en) National power grid distribution network fault processing system
CN112559634A (en) Big data management system based on computer cloud computing
CN115391444A (en) Heterogeneous data acquisition and interaction method, device, equipment and storage medium
CN114238388A (en) Heterogeneous data collection and retrieval system based on multiple protocols
CN112883001A (en) Data processing method, device and medium based on marketing and distribution through data visualization platform
CN110555583A (en) method for uniformly processing wide-area operation data of intelligent power grid dispatching control system
CN104601374B (en) Network failure processing method and device for Digit Control Machine Tool
CN110750596A (en) Process design method for realizing information sharing of medical institution
CN109561155B (en) Remote centralized monitoring and operation and maintenance method for substation equipment
CN107015540A (en) A kind of creation data real-time acquisition system based on DCS
CN111258758A (en) Streaming data processing system
CN116149849A (en) Edge computing method for intelligent water affair complex time scale data fusion
CN202331187U (en) Remote PLC (programmable logic controller) state monitoring and fault alarm system supporting WEB view
CN111427930A (en) Low-voltage photovoltaic energy storage microgrid device monitoring management system, method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination