CN110633259A - HDFS-based fault pushing system - Google Patents

HDFS-based fault pushing system Download PDF

Info

Publication number
CN110633259A
CN110633259A CN201910721100.6A CN201910721100A CN110633259A CN 110633259 A CN110633259 A CN 110633259A CN 201910721100 A CN201910721100 A CN 201910721100A CN 110633259 A CN110633259 A CN 110633259A
Authority
CN
China
Prior art keywords
data
hdfs
fault
module
threshold
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910721100.6A
Other languages
Chinese (zh)
Inventor
周洁
胡牧
杨志
刘铭
张琦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nari Information and Communication Technology Co
Original Assignee
Nari Information and Communication Technology Co
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nari Information and Communication Technology Co filed Critical Nari Information and Communication Technology Co
Priority to CN201910721100.6A priority Critical patent/CN110633259A/en
Publication of CN110633259A publication Critical patent/CN110633259A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/182Distributed file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • G06Q10/06Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063Operations research, analysis or management
    • G06Q10/0631Resource planning, allocation, distributing or scheduling for enterprises or organisations
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • G06Q10/20Administration of product repair or maintenance
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/06Energy or water supply
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y04INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
    • Y04SSYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
    • Y04S10/00Systems supporting electrical power generation, transmission or distribution
    • Y04S10/50Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Human Resources & Organizations (AREA)
  • Theoretical Computer Science (AREA)
  • Economics (AREA)
  • Strategic Management (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Tourism & Hospitality (AREA)
  • Marketing (AREA)
  • Entrepreneurship & Innovation (AREA)
  • General Business, Economics & Management (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Development Economics (AREA)
  • Educational Administration (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Game Theory and Decision Science (AREA)
  • Public Health (AREA)
  • Water Supply & Treatment (AREA)
  • General Health & Medical Sciences (AREA)
  • Primary Health Care (AREA)
  • Remote Monitoring And Control Of Power-Distribution Networks (AREA)

Abstract

The invention provides a fault pushing system based on an HDFS (Hadoop distributed File System), which is applied to power transmission and transformation equipment and comprises the following components: the data information acquisition module is used for acquiring power transmission and transformation equipment, meteorological information, traffic road conditions and maintenance personnel data in real time; the storage module divides the data acquired by the data information acquisition module into structured data and unstructured data and correspondingly stores the structured data and the unstructured data in the HDFS; the partitioning module is used for creating a new data block on the HDFS storage node and sending data to the corresponding HDFS storage node for storage according to the obtained information of the HDFS storage node; the scheduling module is used for realizing the scheduling and circulation of information among the modules by the coordination of the system scheduling module; and the analysis module sets a safety threshold and a warning threshold for the stored data, and generates early warning, tracking and processing mechanisms of faults with different dimensions according to the difference between the safety threshold and the warning threshold. The invention can quickly establish a fault data processing mechanism and better realize fault processing.

Description

HDFS-based fault pushing system
Technical Field
The invention relates to a fault processing mode, in particular to a fault pushing system based on an HDFS.
Background
The load data of the power transmission and transformation equipment and the power grid resource data are respectively stored in different intranet systems, so that when the load data of the power transmission and transformation equipment is collected and stored in a public database of the power grid system, the load data of the power transmission and transformation equipment needs to be analyzed according to a power standard specification, and the purpose of use can be achieved.
The load data of the power transmission and transformation equipment mainly comprises various real-time information (including frequency, generator power, line power, bus voltage and the like) of the power grid power transmission and transformation equipment, is mainly stored in a power integrated automation system, mainly provides various real-time information of a power grid for power grid dispatching management personnel, and carries out dispatching decision management and control on the power grid, thereby ensuring the safe operation of the power grid, improving the quality of the power grid and improving the economical efficiency of the operation of the power grid. The data generation period is short, and the period of changing some data is 1 minute, such as the power of a generator, the power of a line and the like;
in the prior art, data access is only single access and storage, and the judgment and record of possible faults in the access process are lack of accurate classification, and are not suitable for ensuring the integrity of data.
Disclosure of Invention
The invention aims to provide a fault pushing System based on an HDFS (Hadoop Distributed File System), wherein the HDFS is short for the Hadoop Distributed File System and is an implementation of the Hadoop abstract File System. The Hadoop abstract file system may be integrated with local systems, Amazon S3, etc., or even may operate via the Web protocol (webhsfs). The files of the HDFS are distributed on cluster machines, and meanwhile, the copies are provided for fault tolerance and reliability guarantee. For example, direct operations of writing and reading files by clients are distributed on all machines in the cluster, and no single point of performance pressure exists.
HDFS provides high throughput data access and is well suited for application on large-scale data sets. HDFS relaxes a portion of the POSIX constraints to achieve the goal of streaming file system data.
Once the data has a fault, the fault information needs to be pushed in the first time, so that a user can timely solve the problem behind the fault, and the requirement on the timeliness of data transmission is high.
In order to achieve the above object, the present invention provides a HDFS-based fault push system, which is applied to power transmission and transformation equipment, and includes:
the data information acquisition module is used for acquiring power transmission and transformation equipment, meteorological information, traffic road conditions and maintenance personnel data in real time;
the storage module divides the data acquired by the data information acquisition module into structured data and unstructured data and correspondingly stores the structured data and the unstructured data in the HDFS;
the partitioning module is used for creating a new data block on the HDFS storage node and sending data to the corresponding HDFS storage node for storage according to the obtained information of the HDFS storage node;
the scheduling module is used for realizing the scheduling and circulation of information among the modules by the coordination of the system scheduling module;
and the analysis module sets a safety threshold and a warning threshold for the stored data, and generates early warning, tracking and processing mechanisms of faults with different dimensions according to the difference between the safety threshold and the warning threshold.
Different from the prior art, the data information acquisition module only acquires the data of the power transmission and transformation equipment, and then analyzes the fault location according to the data change; in the scheme, external influence data including but not limited to meteorological information are collected together, the meteorological data has a very important reference function corresponding to the processing flow reference once a fault occurs, for example, the fault is in a micro state, and the fault in the micro state can be processed temporarily under the condition of clear weather;
although the fault is in a micro state, the current weather is in a severe weather condition, the fault also needs to be immediately processed, and under the severe weather condition, the fault in the micro state may also be evolved into a large fault, which may affect the normal operation of the power transmission and transformation equipment.
As for the traffic road condition, in order to better process the path for reference at the planning position, similarly to which maintenance personnel are respectively dispatched according to the fault types and which traffic road is taken, the fault can be eliminated more quickly; because the maintenance personnel with different experiences are selected according to different fault types, the fault clearing time is greatly promoted.
In a preferred embodiment of the present invention, after the partitioning module obtains the location and the name of the file to be processed, the partitioning module includes a first partitioning cluster and a second partitioning cluster, stores the basic data of the structured data in the first partitioning cluster, and stores the unstructured loose data in the second partitioning cluster.
In a preferred embodiment of the present invention, if the stored data exceeds the safety threshold, sending abnormal data to the background server for warning if the stored data exceeds the safety threshold; meanwhile, the data information acquisition module can call current meteorological information and traffic road conditions, and meanwhile, the data information acquisition module is matched with maintenance personnel to be ready and can process the data according to the data progress.
In a preferred embodiment of the present invention, if the stored data exceeds the warning threshold, a processing flow is sent to the backend server, and the processing flow and the current state preferably select an optimal processing flow path.
In a preferred embodiment of the present invention, if the stored data exceeds the safety threshold and the warning threshold, a monitoring data list is automatically generated, and when it is determined that the storage load of the server storing the monitoring data list exceeds a preset load threshold, a server for storing the monitoring data list is newly added.
In a preferred embodiment of the present invention, the monitoring system further includes an early warning module, which performs analysis according to the monitoring data detail table, and performs fault diagnosis, fault analysis, and fault prediction of the electric transmission and transformation equipment, so as to perform early prediction of a fault.
In a preferred embodiment of the present invention, the early warning module classifies the classification model of the failure mode to obtain a decision network of the failure, and obtains a failure type according to the decision network, and automatically generates a processing flow.
Compared with the prior art, the invention has the beneficial effects that:
the invention integrates various acquired data, divides the data into structured data and unstructured data, uniformly stores the data in the HDFS, establishes a set of warning and processing flow of fault data inside, ensures the timeliness and accuracy of fault processing, and can predict the fault in advance according to long-term use.
Drawings
Fig. 1 is a functional block diagram of embodiment 1 of the present invention.
Fig. 2 is a functional block diagram of embodiment 2 of the present invention.
Detailed Description
The present invention is described in detail with reference to the embodiments shown in the drawings, but it should be understood that these embodiments are not intended to limit the present invention, and those skilled in the art should understand that functional, methodological, or structural equivalents or substitutions made by these embodiments are within the scope of the present invention.
Referring to fig. 1, fig. 1 is a block diagram illustrating the working principle of the present invention.
Example 1:
in this embodiment, the HDFS-based fault push system applied to power transmission and transformation equipment includes:
the data information acquisition module is used for acquiring power transmission and transformation equipment, meteorological information, traffic road conditions and maintenance personnel data in real time;
different from the prior art, the data information acquisition module only acquires the data of the power transmission and transformation equipment, and then the fault location is analyzed according to the data change; in the scheme, external influence data including but not limited to meteorological information are collected together, the meteorological data has a very important reference function corresponding to the processing flow reference once a fault occurs, for example, the fault is in a micro state, and the fault in the micro state can be processed temporarily under the condition of clear weather;
although the fault is in a micro state, the current weather is in a severe weather condition, the fault also needs to be immediately processed, and under the severe weather condition, the fault in the micro state may also be evolved into a large fault, which may affect the normal operation of the power transmission and transformation equipment.
As for the traffic road condition, in order to better process the path for reference at the planning position, similarly to which maintenance personnel are respectively dispatched according to the fault types and which traffic road is taken, the fault can be eliminated more quickly; because the maintenance personnel with different experiences are selected according to different fault types, the fault clearing time is greatly promoted.
The storage module divides the data acquired by the data information acquisition module into structured data and unstructured data and correspondingly stores the structured data and the unstructured data in the HDFS;
the partitioning module is used for creating a new data block on the HDFS storage node and sending data to the corresponding HDFS storage node for storage according to the obtained information of the HDFS storage node;
according to the scheme, different databases are correspondingly generated and stored according to different structured data and unstructured data, so that the data can be split better, and fault data can be positioned more quickly; for example, real-time calculations of device data are at a priority level and calculations of non-device facilities are at a sub-priority level.
The scheduling module is used for realizing the scheduling and circulation of information among the modules by the coordination of the system scheduling module;
and the analysis module sets a safety threshold and a warning threshold for the stored data, and generates early warning, tracking and processing mechanisms of faults with different dimensions according to the difference between the safety threshold and the warning threshold.
Furthermore, after the partition module obtains the position and the name of the file to be processed, the partition module comprises a first partition cluster and a second partition cluster, the basic data of the structured data is stored in the first partition cluster, and the unstructured loose data is stored in the second partition cluster.
The data storage and analysis module analyzes the data more quickly by dividing the data into different partition clusters.
Example 2:
compared with embodiment 1, the present solution provides a processing relationship between the stored data and the current threshold, specifically as follows:
if the stored data exceeds the safety threshold, sending abnormal data to a background server for warning if the stored data exceeds the safety threshold; meanwhile, the data information acquisition module can call current meteorological information and traffic road conditions, and meanwhile, the data information acquisition module is matched with maintenance personnel to be ready and can process the data according to the data progress.
And if the stored data exceeds the warning threshold value, sending a processing flow to the background server, wherein the processing flow and the current state preferably select an optimal processing flow path.
More importantly, if the stored data exceeds a safety threshold and a warning threshold, a monitoring data detail table is automatically generated, and when the fact that the storage load of the server storing the monitoring data detail table exceeds a preset load threshold is determined, a server used for storing the monitoring data detail table is newly added.
Example 3:
referring to fig. 2, in addition to the technical solutions described in the foregoing embodiment 1, embodiment 2, and embodiment 3, the present solution further includes an early warning module, which performs analysis according to the monitoring data detail table, and performs fault diagnosis, fault analysis, and fault prediction on the electric transmission and transformation equipment, so as to implement early prediction of a fault.
In addition, the early warning module classifies the classification model of the fault mode to obtain a fault decision network, obtains a fault type according to the decision network, and automatically generates a processing flow.
The invention integrates various acquired data, divides the data into structured data and unstructured data, uniformly stores the data in the HDFS, establishes a set of warning and processing flow of fault data inside, ensures the timeliness and accuracy of fault processing, and can predict the fault in advance according to long-term use.
The above-listed detailed description is only a specific description of a possible embodiment of the present invention, and they are not intended to limit the scope of the present invention, and equivalent embodiments or modifications made without departing from the technical spirit of the present invention should be included in the scope of the present invention.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Furthermore, it should be understood that although the present description refers to embodiments, not every embodiment may contain only a single embodiment, and such description is for clarity only, and those skilled in the art should integrate the description, and the embodiments may be combined as appropriate to form other embodiments understood by those skilled in the art.

Claims (7)

1. HDFS based fault push system is applied to power transmission and transformation equipment, and its characterized in that includes:
the data information acquisition module is used for acquiring power transmission and transformation equipment, meteorological information, traffic road conditions and maintenance personnel data in real time;
the storage module divides the data acquired by the data information acquisition module into structured data and unstructured data and correspondingly stores the structured data and the unstructured data in the HDFS;
the partitioning module is used for creating a new data block on the HDFS storage node and sending data to the corresponding HDFS storage node for storage according to the obtained information of the HDFS storage node;
the scheduling module is used for realizing the scheduling and circulation of information among the modules by the coordination of the system scheduling module;
and the analysis module sets a safety threshold and a warning threshold for the stored data, and generates early warning, tracking and processing mechanisms of faults with different dimensions according to the difference between the safety threshold and the warning threshold.
2. The HDFS-based failure pushing system according to claim 1, wherein after the partition module obtains the location and name of the file to be processed, the partition module includes a first partition cluster and a second partition cluster, stores the basic data of the structured data in the first partition cluster, and stores the unstructured loose data in the second partition cluster.
3. The HDFS-based failure pushing system according to claim 1, wherein if the stored data exceeds a safety threshold, sending abnormal data to a background server alert if the stored data exceeds the safety threshold; meanwhile, the data information acquisition module can call current meteorological information and traffic road conditions, and meanwhile, the data information acquisition module is matched with maintenance personnel to be ready and can process the data according to the data progress.
4. The HDFS-based failure push system according to claim 1, wherein if the stored data exceeds a warning threshold, a process flow is sent to the backend server, which process flow and the current state prefer the best process flow path.
5. The HDFS-based failure pushing system according to any one of claims 1 to 4, wherein a monitoring data list is automatically generated if the stored data exceeds a safety threshold and a warning threshold, and a server for storing the monitoring data list is newly added when it is determined that a storage load of a server storing the monitoring data list exceeds a preset load threshold.
6. The HDFS-based fault pushing system according to claim 5, further comprising an early warning module, wherein the early warning module is used for analyzing according to the monitoring data detail table, and performing fault diagnosis, fault analysis and fault prediction on the power transmission and transformation equipment, so that the fault can be predicted in advance.
7. The HDFS-based fault pushing system according to claim 6, wherein the early warning module classifies the classification models of the fault modes to obtain a fault decision network, and obtains fault types according to the decision network, and automatically generates a processing flow.
CN201910721100.6A 2019-08-06 2019-08-06 HDFS-based fault pushing system Pending CN110633259A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910721100.6A CN110633259A (en) 2019-08-06 2019-08-06 HDFS-based fault pushing system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910721100.6A CN110633259A (en) 2019-08-06 2019-08-06 HDFS-based fault pushing system

Publications (1)

Publication Number Publication Date
CN110633259A true CN110633259A (en) 2019-12-31

Family

ID=68969264

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910721100.6A Pending CN110633259A (en) 2019-08-06 2019-08-06 HDFS-based fault pushing system

Country Status (1)

Country Link
CN (1) CN110633259A (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105590160A (en) * 2015-10-30 2016-05-18 国网山东省电力公司青岛供电公司 Three-dimensional emergency command method and system on the basis of GIS
CN107146018A (en) * 2017-05-04 2017-09-08 北京许继电气有限公司 Data Management Analysis method and system based on electric network state monitoring system

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105590160A (en) * 2015-10-30 2016-05-18 国网山东省电力公司青岛供电公司 Three-dimensional emergency command method and system on the basis of GIS
CN107146018A (en) * 2017-05-04 2017-09-08 北京许继电气有限公司 Data Management Analysis method and system based on electric network state monitoring system

Similar Documents

Publication Publication Date Title
CN107943668B (en) Computer server cluster log monitoring method and monitor supervision platform
EP3798846B1 (en) Operation and maintenance system and method
CN108039959B (en) Data situation perception method, system and related device
CN111260504B (en) Intelligent power grid monitoring method and system and intelligent power grid controller
CN104616205A (en) Distributed log analysis based operation state monitoring method of power system
CN106210124B (en) Unified cloud data center monitoring system
CN113179173B (en) Operation and maintenance monitoring system for expressway system
CN104092575A (en) Resource monitoring method and system
CN112134357A (en) Power distribution automation master station system based on cloud service
CN111858251B (en) Data security audit method and system based on big data computing technology
CN111259073A (en) Intelligent business system running state studying and judging system based on logs, flow and business access
CN112865311B (en) Method and device for monitoring message bus of power system
CN109962792A (en) A kind of full link monitoring system based on big data
CN115280741A (en) System and method for autonomous monitoring and recovery in hybrid energy management
CN110659180A (en) Data center infrastructure management system based on cluster technology
CN109361576A (en) A kind of PIM monitoring data processing method and system
CN113675944A (en) Intelligent analysis decision-making system and method for photovoltaic power station
CN110334900B (en) Data processing method
CN116594840A (en) Log fault acquisition and analysis method, system, equipment and medium based on ELK
CN106649034A (en) Visual intelligent operation and maintenance method and platform
CN110633259A (en) HDFS-based fault pushing system
CN110647070A (en) Power environment monitoring system for super-large-scale data center
Wang et al. Fault diagnosis and trace method of power system based on big data platform
CN115766768A (en) Method and device for designing sensing center in computational power network operating system
EP4071670A1 (en) Technical system for a centralized generation of a plurality of trained, retrained and/or monitored machine learning models, wherein the generated machine learning models are executed decentral

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
AD01 Patent right deemed abandoned
AD01 Patent right deemed abandoned

Effective date of abandoning: 20230228