CN112035208A - Multi-form power equipment of transformer substation and Internet of things collection and display big data access method - Google Patents
Multi-form power equipment of transformer substation and Internet of things collection and display big data access method Download PDFInfo
- Publication number
- CN112035208A CN112035208A CN202011012820.4A CN202011012820A CN112035208A CN 112035208 A CN112035208 A CN 112035208A CN 202011012820 A CN202011012820 A CN 202011012820A CN 112035208 A CN112035208 A CN 112035208A
- Authority
- CN
- China
- Prior art keywords
- data
- power equipment
- internet
- substation
- big data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000004364 calculation method Methods 0.000 claims abstract description 23
- 238000007726 management method Methods 0.000 claims abstract description 14
- 238000004458 analytical method Methods 0.000 claims abstract description 13
- 230000000007 visual effect Effects 0.000 claims abstract description 8
- 238000012544 monitoring process Methods 0.000 claims description 30
- 238000012545 processing Methods 0.000 claims description 13
- 238000007689 inspection Methods 0.000 claims description 11
- 238000001514 detection method Methods 0.000 claims description 9
- 230000007547 defect Effects 0.000 claims description 8
- 238000003745 diagnosis Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 8
- 238000003860 storage Methods 0.000 claims description 8
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 230000008030 elimination Effects 0.000 claims description 4
- 238000003379 elimination reaction Methods 0.000 claims description 4
- 230000006870 function Effects 0.000 claims description 4
- 238000012423 maintenance Methods 0.000 claims description 4
- 238000004519 manufacturing process Methods 0.000 claims description 4
- 238000003672 processing method Methods 0.000 claims description 4
- 230000003068 static effect Effects 0.000 claims description 4
- 230000004931 aggregating effect Effects 0.000 claims description 3
- 238000004422 calculation algorithm Methods 0.000 claims description 3
- 230000008859 change Effects 0.000 claims description 3
- 238000005111 flow chemistry technique Methods 0.000 claims description 3
- 238000010801 machine learning Methods 0.000 claims description 3
- 238000009826 distribution Methods 0.000 claims description 2
- 238000007418 data mining Methods 0.000 abstract description 4
- 230000005540 biological transmission Effects 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 3
- 238000005094 computer simulation Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000004907 flux Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000004171 remote diagnosis Methods 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
- G06F16/244—Grouping and aggregation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24554—Unary operations; Data partitioning operations
- G06F16/24556—Aggregation; Duplicate elimination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/06—Electricity, gas or water supply
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
- Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
- Y04S10/00—Systems supporting electrical power generation, transmission or distribution
- Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
Abstract
The access method for the polymorphic power equipment of the transformer substation and the big data collected and displayed by the Internet of things collects the polymorphic power equipment of the transformer substation and the big data of the Internet of things; dividing collected multi-form power equipment of the transformer substation and big data of the Internet of things into structured data, semi-structured data and unstructured data according to data types; according to different obtained data types, introducing the structured data into the HDFS of Hadoop by Sqoop, the semi-structured data by Flume and the unstructured data by Kafka; the Yarn resource management system allocates resources for different computing tasks and selects a computing mode according to computing requirements; oozie carries out task scheduling, and Zookeeper carries out configuration and scheduling of a frame on a platform; and carrying out visual analysis on the processed big data of the substation power equipment. According to the invention, different calculation modes are adopted for polymorphic big data of power equipment and the Internet of things, so that the memory utilization rate is improved, the delay is reduced, useful data are extracted, analysis such as data mining is carried out, and the guarantee is provided for safe, stable and economic operation of a transformer substation.
Description
Technical Field
The invention belongs to the technical field of power equipment of transformer substations, and relates to a polymorphic power equipment of a transformer substation and an Internet of things acquisition and display big data access method.
Background
The intelligent power grid technology is continuously developed, on the basis of high integration of the traditional power technology, technologies such as manufacturing, information, control, internet and automation are combined, a large amount of data information is collected in each link in the whole power process, information is analyzed, mined and extended, decision is optimized according to the information, and a good technical basis is provided for safe and efficient operation of power equipment.
Since the appearance of big data concepts, the big data concepts have profoundly influenced the world, and particularly, the big data concepts are widely applied in various fields related to consumption. Big data in the power industry also becomes one of main support systems of new-period power grid functions, and mining of related data in numerous power works becomes key content of power main work through data acquisition, transmission, storage and processing of power equipment.
The prior art document 1 (Jianglidan; Wangyu; Zhongxinyu; Liutian; Korea; Xuswords; Ligang; Zhang xi Zheng; transformation equipment intelligent monitoring analysis system [ P ] based on big data, Chinese patent No. CN110932405A,2020-03-27) includes a central monitoring system and a terminal system, but the patent has a low utilization rate of monitoring signals and cannot realize a visual effect.
In prior art document 2 (li yu xuan, fault diagnosis system and method [ P ] based on big data analysis, chinese patent: CN111124735A,2020-05-08), the system includes a computer, and a general software analysis module, a hardware data acquisition and monitoring module, a fuzzy diagnosis initial unit, an expert remote diagnosis data terminal, a fault preprocessing module, and an alarm module are disposed in the computer, but the patent is not matched to the special application of the substation, and the data configuration and scheduling cannot meet the requirement of monitoring the state of the substation equipment.
Disclosure of Invention
In order to solve the problems in the prior art, the invention aims to provide a multi-form power equipment of a transformer substation and an access method for internet of things collection and display big data.
The invention adopts the following technical scheme:
the access method for the polymorphic power equipment of the transformer substation and the collected and displayed big data of the Internet of things comprises the following steps:
step 1, collecting polymorphic power equipment of a transformer substation and Internet of things big data;
step 2, dividing the multi-form power equipment of the transformer substation and the big data of the Internet of things collected in the step 1 into structured data, semi-structured data and unstructured data according to data types;
step 3, transmitting the structured data, the semi-structured data and the unstructured data obtained in the step 2 to the HDFS by utilizing Sqoop, Flume and Kafka respectively,
the HDFS is a Hadoop distributed file system, Sqoop is an open source tool, flash is a log collecting, aggregating and transmitting system, and Kafka is an open source stream processing platform;
step 4, the HDFS and the Yarn resource management system in the step 3 are utilized to distribute resources for different computing tasks, and a computing mode is selected according to computing requirements;
the adopted calculation modes comprise Mapreduce off-line calculation, Spark calculation and Flink real-time calculation;
step 5, Oozie schedules the computing task of the computing mode selected in the step 4, Zookeeper configures and schedules the open source flow processing platform Kafka,
wherein, Oozie is a workflow scheduling program system, and Zookeeper is a distributed application program coordination service of an open source code;
and 6, carrying out visual analysis on the substation power equipment big data processed in the steps 4 and 5.
In the step 1, the substation power equipment big data comprises video data generated by video monitoring equipment, picture data generated by unmanned aerial vehicle inspection, equipment fault information text, equipment inspection record, maintenance record, fault trip record, fault typical case library, defect and defect elimination record, online monitoring and live detection data.
In the step 2, the structured data includes a transformer vibration signal, the transformer vibration signal belongs to online monitoring and live detection data, the semi-structured data includes logs, and the unstructured data includes video data generated by video monitoring equipment.
In the step 3, the HDFS comprises an Hbase and a Hive, the Hbase is a non-relational database and is used for storing unstructured data, the unstructured data comprises video data generated by video monitoring equipment, picture data generated by unmanned aerial vehicle inspection and electronic files in the circulation process of an office system, and a structured storage cluster is built on a PC Server; the Hive static table batch processing method maps structured data files including online monitoring and charged detection data into a database table, provides an SQL query function, converts SQL sentences into MapReduce tasks for execution,
wherein SQL is a structured query language.
In the step 4, the Yarn resource management system provides uniform resource management and scheduling for the upper application.
In the step 4, the Mapreduce offline calculation is to decompose a large data processing task into single tasks which are executed in parallel in the server cluster according to the codes of the power equipment, and the calculation results of the tasks are combined together to calculate a final result.
In the step 4, the Spark calculation includes four modules, Spark sql, Spark MLlib, Spark streaming and Spark graph,
the Spark sql is used for data query of the power equipment, the Spark MLlib introduces a machine learning algorithm for optimizing production operation, evaluating equipment state, analyzing major accidents after events and counting year, month and day of the power equipment, Spark streaming processing is used for online monitoring abnormal data and real-time fault diagnosis of the power equipment, and Spark graph is used for visual analysis of complex data of the power equipment.
In the step 4, the Flink calculates and processes the borderless and borderless data sets in real time, and is used for online monitoring of abnormal data and real-time fault diagnosis of the power equipment.
In the step 5, the task scheduling method of the Oozie frame includes: time-based scheduling is supported as well as scheduling based on data availability.
In the step 5, the configuration and scheduling method of the Zookeeper frame includes: and coordinating and solving the balanced distribution of an application system in the distributed cluster, maintaining and monitoring the state change of the stored data, and performing cluster management based on the data.
Compared with the prior art, the invention has the beneficial effects that:
according to the invention, different calculation modes are adopted for big data of the power equipment, so that the memory utilization rate is improved, the delay is reduced, useful data are extracted, analysis such as data mining is carried out, and the guarantee is provided for safe, stable and economic operation of a transformer substation.
Drawings
Fig. 1 is a flow chart of a method for accessing polymorphic power equipment of a transformer substation and collected and displayed big data of the internet of things.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. The embodiments described herein are only some embodiments of the invention, and not all embodiments. All other embodiments obtained by a person skilled in the art without making any inventive step on the basis of the spirit of the present invention are within the scope of protection of the present invention.
The invention discloses a substation polymorphic power equipment and an Internet of things acquisition and display big data access method, which comprises a structured data transmission method based on a Sqoop frame; file log (semi-structured data) transmission method based on the flash framework; an unstructured data transmission and storage method based on Kafka; an HDFS-based file repository; a non-relational database based on Hbase; hive-based static batch processing method; based on the Yarn resource management system; a Mapreduce-based computational model; a Spark framework based data mining analysis method; a real-time data calculation method based on a Flink framework; a task scheduling method based on an Oozie framework; a configuration and scheduling method based on a Zookeeper frame;
as shown in fig. 1, a method for accessing polymorphic power equipment of a transformer substation and collected and displayed big data of the internet of things comprises the following steps:
step 1, collecting polymorphic power equipment of a transformer substation and Internet of things big data;
step 2, dividing the multi-form power equipment of the transformer substation and the big data of the Internet of things collected in the step 1 into structured data, semi-structured data and unstructured data according to data types;
step 3, transmitting the structured data, the semi-structured data and the unstructured data obtained in the step 2 to the HDFS by utilizing Sqoop, Flume and Kafka respectively,
the HDFS is a Hadoop distributed file system, Sqoop is an open source tool, flash is a log collecting, aggregating and transmitting system, and Kafka is an open source stream processing platform;
step 4, the HDFS and the Yarn resource management system in the step 3 are utilized to distribute resources for different computing tasks, and a computing mode is selected according to computing requirements;
the adopted calculation modes comprise Mapreduce off-line calculation, Spark calculation and Flink real-time calculation;
step 5, Oozie schedules the computing task of the computing mode selected in the step 4, Zookeeper configures and schedules the open source flow processing platform Kafka,
wherein, Oozie is a workflow scheduling program system, and Zookeeper is a distributed application program coordination service of an open source code;
and 6, carrying out visual analysis on the substation power equipment big data processed in the steps 4 and 5.
Polymorphic power equipment of transformer substation and thing networking data include: video data generated by a large amount of video monitoring equipment, picture data generated by unmanned aerial vehicle routing inspection, various types of electronic files in the circulation process of an office system, equipment fault information texts of the same factory, the same type and the same period, equipment inspection records, maintenance records, fault trip records, fault typical case libraries, defect and defect elimination records, online monitoring and live detection data, bad working condition information and the like.
The structured data transmission method of the Sqoop frame comprises the following steps: and transmitting the structured data including data of on-line monitoring, charged detection and the like into an HDFS storage layer of Hadoop, and vice versa. The data are mutually converted, and a framework is provided for mass data transmission.
File log (semi-structured data) transmission method of the flash framework: and the semi-structured data including equipment inspection records, maintenance records, fault trip records, fault typical case base, defect and defect elimination records and the like are transmitted to the HDFS through a flux component, and the method is based on a streaming framework and is simpler and more flexible.
The unstructured data transmission and storage method of Kafka comprises the following steps: the high-throughput-based publish-subscribe pattern message queue system processes unstructured data, including video data generated by a large number of video monitoring devices, picture data generated by unmanned aerial vehicle inspection, various electronic files in the circulation process of an office system and the like. Has four main characteristics: a, high throughput, low latency, processing hundreds of thousands of messages per second; b, persistence, reliability, the message is permanently stored in a local disk, and data backup is supported to prevent data loss; c, fault tolerance, allowing cluster failure; and d, high concurrency, and capacity of simultaneously reading and writing by a plurality of users.
Hbase non-relational database: similar to a relational database, the system is suitable for unstructured data storage, comprises video data generated by a large amount of video monitoring equipment, picture data generated by unmanned aerial vehicle inspection, various electronic files in the circulation process of an office system and the like, and can build a large-scale structured storage cluster on a low-cost PC Server to increase the calculation and storage capacity.
Hive static batch processing method: and mapping the structured data file including data such as online monitoring, charged detection and the like into a database table, providing an SQL query function, and converting the SQL statement into a MapReduce task for execution.
Wherein SQL is a structured query language.
The Yarn resource management system: unified resource management and scheduling is provided for upper layer applications.
Mapreduce's computational model: the large data processing task is decomposed into a plurality of single tasks which can be executed in parallel in the server cluster, and the calculation results of the tasks are combined together to calculate the final result.
The data mining analysis method of the Spark framework comprises the following steps: the core is mainly divided into four modules: spark sql, Spark MLlib, Spark streaming, Spark graph. Spark sql is used for the purposes of fast querying data of the power equipment and the like. The Spark MLlib introduces a machine learning algorithm for production operation optimization, equipment state evaluation, major accident post analysis, year, month and day statistics and other purposes of the power equipment. Spark streaming processing is used for on-line monitoring of abnormal data and real-time fault diagnosis of power equipment. The Spark graph is used for visual analysis of complex data of the power equipment and the like.
The method for calculating the real-time data of the Flink framework comprises the following steps: the method has the advantages of low delay, high throughput rate, item-by-item processing, and good suitability for processing borderless and borderless data sets, and is used for the purposes of online monitoring of abnormal data and real-time fault diagnosis of electric equipment and the like.
A task scheduling method of an Oozie framework; time-based scheduling is supported as well as scheduling based on data availability.
The configuration and scheduling method of the Zookeeper frame comprises the following steps: the method solves the consistency problem of the application system in the distributed cluster, maintains and monitors the state change of the stored data, and performs cluster management based on the data.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting the same, and although the present invention is described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications and equivalents may be made to the embodiments of the invention without departing from the spirit and scope of the invention, which is to be covered by the claims.
Claims (10)
1. The access method for the polymorphic power equipment of the transformer substation and the big data collected and displayed by the Internet of things is characterized by comprising the following steps: the method comprises the following steps:
step 1, collecting polymorphic power equipment of a transformer substation and Internet of things big data;
step 2, dividing the multi-form power equipment of the transformer substation and the big data of the Internet of things collected in the step 1 into structured data, semi-structured data and unstructured data according to data types;
step 3, transmitting the structured data, the semi-structured data and the unstructured data obtained in the step 2 to the HDFS by utilizing Sqoop, Flume and Kafka respectively,
the HDFS is a Hadoop distributed file system, Sqoop is an open source tool, flash is a log collecting, aggregating and transmitting system, and Kafka is an open source stream processing platform;
step 4, the HDFS and the Yarn resource management system in the step 3 are utilized to distribute resources for different computing tasks, and a computing mode is selected according to computing requirements;
the adopted calculation modes comprise Mapreduce off-line calculation, Spark calculation and Flink real-time calculation;
step 5, Oozie schedules the computing task of the computing mode selected in the step 4, Zookeeper configures and schedules the open source flow processing platform Kafka,
wherein, Oozie is a workflow scheduling program system, and Zookeeper is a distributed application program coordination service of an open source code;
and 6, carrying out visual analysis on the substation power equipment big data processed in the steps 4 and 5.
2. The substation polymorphic power equipment and internet of things collection and display big data access method according to claim 1, characterized in that:
in the step 1, the substation power equipment big data comprises video data generated by video monitoring equipment, picture data generated by unmanned aerial vehicle inspection, equipment fault information text, equipment inspection record, maintenance record, fault trip record, fault typical case library, defect and defect elimination record, online monitoring and live detection data.
3. The substation polymorphic power equipment and internet of things collection and display big data access method according to claim 1, characterized in that:
in the step 2, the structured data includes a transformer vibration signal, the transformer vibration signal belongs to online monitoring and live detection data, the semi-structured data includes logs, and the unstructured data includes video data generated by video monitoring equipment.
4. The substation polymorphic power equipment and internet of things collection and display big data access method according to claim 1, characterized in that:
in the step 3, the HDFS comprises an Hbase and a Hive, the Hbase is a non-relational database and is used for storing unstructured data, the unstructured data comprises video data generated by video monitoring equipment, picture data generated by unmanned aerial vehicle inspection and electronic files in the circulation process of an office system, and a structured storage cluster is built on a PC Server; the Hive static table batch processing method maps structured data files including online monitoring and charged detection data into a database table, provides an SQL query function, converts SQL sentences into MapReduce tasks for execution,
wherein SQL is a structured query language.
5. The substation polymorphic power equipment and internet of things collection and display big data access method according to claim 1, characterized in that:
in the step 4, the Yarn resource management system provides uniform resource management and scheduling for the upper application.
6. The substation polymorphic power equipment and internet of things collection and display big data access method according to claim 1, characterized in that:
in the step 4, the Mapreduce offline calculation is to decompose a large data processing task into single tasks which are executed in parallel in the server cluster according to the codes of the power equipment, and the calculation results of the tasks are combined together to calculate a final result.
7. The substation polymorphic power equipment and internet of things collection and display big data access method according to claim 1, characterized in that:
in the step 4, the Spark calculation includes four modules, Spark sql, Spark MLlib, Spark streaming and Spark graph,
the Spark sql is used for data query of the power equipment, the Spark MLlib introduces a machine learning algorithm for optimizing production operation, evaluating equipment state, analyzing major accidents after events and counting year, month and day of the power equipment, Spark streaming processing is used for online monitoring abnormal data and real-time fault diagnosis of the power equipment, and Spark graph is used for visual analysis of complex data of the power equipment.
8. The substation polymorphic power equipment and internet of things collection and display big data access method according to claim 1, characterized in that:
in the step 4, the Flink calculates and processes the borderless and borderless data sets in real time, and is used for online monitoring of abnormal data and real-time fault diagnosis of the power equipment.
9. The substation polymorphic power equipment and internet of things collection and display big data access method according to claim 1, characterized in that:
in the step 5, the task scheduling method of the Oozie frame includes: time-based scheduling is supported as well as scheduling based on data availability.
10. The substation polymorphic power equipment and internet of things collection and display big data access method according to claim 1, characterized in that:
in the step 5, the configuration and scheduling method of the Zookeeper frame includes: and coordinating and solving the balanced distribution of an application system in the distributed cluster, maintaining and monitoring the state change of the stored data, and performing cluster management based on the data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011012820.4A CN112035208A (en) | 2020-09-23 | 2020-09-23 | Multi-form power equipment of transformer substation and Internet of things collection and display big data access method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011012820.4A CN112035208A (en) | 2020-09-23 | 2020-09-23 | Multi-form power equipment of transformer substation and Internet of things collection and display big data access method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112035208A true CN112035208A (en) | 2020-12-04 |
Family
ID=73575074
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011012820.4A Pending CN112035208A (en) | 2020-09-23 | 2020-09-23 | Multi-form power equipment of transformer substation and Internet of things collection and display big data access method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112035208A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105139281A (en) * | 2015-08-20 | 2015-12-09 | 北京中电普华信息技术有限公司 | Method and system for processing big data of electric power marketing |
| CN108335075A (en) * | 2018-03-02 | 2018-07-27 | 华南理工大学 | A kind of processing system and method for Logistics Oriented big data |
| CN110865997A (en) * | 2019-11-08 | 2020-03-06 | 国网四川省电力公司电力科学研究院 | Online identification method for hidden danger of power system equipment and application platform thereof |
-
2020
- 2020-09-23 CN CN202011012820.4A patent/CN112035208A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105139281A (en) * | 2015-08-20 | 2015-12-09 | 北京中电普华信息技术有限公司 | Method and system for processing big data of electric power marketing |
| CN108335075A (en) * | 2018-03-02 | 2018-07-27 | 华南理工大学 | A kind of processing system and method for Logistics Oriented big data |
| CN110865997A (en) * | 2019-11-08 | 2020-03-06 | 国网四川省电力公司电力科学研究院 | Online identification method for hidden danger of power system equipment and application platform thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107256219B (en) | Big data fusion analysis method applied to mass logs of automatic train control system | |
| CN111077870A (en) | Intelligent OPC data real-time acquisition and monitoring system and method based on stream calculation | |
| CN108039959B (en) | Data situation perception method, system and related device | |
| CN107330056B (en) | Wind power plant SCADA system based on big data cloud computing platform and operation method thereof | |
| CN106651633A (en) | Power utilization information acquisition system and method based on big data technology | |
| CN108718345A (en) | A kind of digitlization workshop industrial data Network Transmitting system | |
| Liu et al. | Real-time complex event processing and analytics for smart grid | |
| CN114416855A (en) | Visualization platform and method based on electric power big data | |
| CN111178587A (en) | Spark framework-based short-term power load rapid prediction method | |
| CN111259073A (en) | Intelligent business system running state studying and judging system based on logs, flow and business access | |
| CN112559634A (en) | Big data management system based on computer cloud computing | |
| CN112884452A (en) | Intelligent operation and maintenance multi-source data acquisition visualization analysis system | |
| CN113176948A (en) | Edge gateway, edge computing system and configuration method thereof | |
| CN111428895A (en) | Intelligent ammeter fault diagnosis support center | |
| CN115564071A (en) | Method and system for generating data labels of power Internet of things equipment | |
| Wu et al. | An auxiliary decision-making system for electric power intelligent customer service based on Hadoop | |
| CN111414355A (en) | Offshore wind farm data monitoring and storing system, method and device | |
| CN111078731A (en) | Hbase-based power grid operation data collaborative query method and device and storage medium | |
| CN112035208A (en) | Multi-form power equipment of transformer substation and Internet of things collection and display big data access method | |
| Guilan et al. | Real-time big data technologies of energy internet platform | |
| Chen et al. | Research on equipment situation display based on multi-source data fusion | |
| CN112784129A (en) | Pump station equipment operation and maintenance data supervision platform | |
| Yu et al. | Research on integrated operation and maintenance acquisition and monitoring technology for new generation dispatching control system | |
| Cao et al. | Cloud-based Approach for Prevention Maintenance Management Platform of Printing Press equipment | |
| CN112508276B (en) | Power grid rapid diagnosis and optimization system and optimization method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201204 |
|
| RJ01 | Rejection of invention patent application after publication |