CN114020763A - Kafka-based real-time power generation data acquisition and transmission and data monitoring method - Google Patents
Kafka-based real-time power generation data acquisition and transmission and data monitoring method Download PDFInfo
- Publication number
- CN114020763A CN114020763A CN202111336012.8A CN202111336012A CN114020763A CN 114020763 A CN114020763 A CN 114020763A CN 202111336012 A CN202111336012 A CN 202111336012A CN 114020763 A CN114020763 A CN 114020763A
- Authority
- CN
- China
- Prior art keywords
- data
- power generation
- calculation
- engine
- collection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/256—Integrating or interfacing systems involving database management systems in federated or virtual databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Computer Security & Cryptography (AREA)
- Supply And Distribution Of Alternating Current (AREA)
Abstract
The invention relates to the technical field of data transmission, and discloses a kafka-based real-time power generation data acquisition transmission and data monitoring method, which is applied to the collection and calculation of power generation data of a multi-source heterogeneous power station and the transmission of a multi-source storage medium. According to the method for real-time power generation data acquisition, transmission and data monitoring based on kafka, by setting an intelligent scheduling task of a core engine, dynamically updating scheduling configuration and dynamic synchronous data collection or calculation configuration, intelligently collecting and analyzing power generation data by a data collection engine, and intelligently calculating multi-dimension power generation data to a multi-source storage medium by a data elimination calculation engine in batches to perform data compensation in the core engine, the problems that the power generation data of a multi-source heterogeneous power station are various, the data cannot be integrally managed, the analysis and power generation data calculation and solidification are optimized, and the power generation data of the multi-source power station are heterogeneous are solved, and log monitoring is provided so as to monitor the execution condition of the task in the execution process.
Description
Technical Field
The invention relates to the technical field of data transmission, in particular to a kafka-based real-time power generation data acquisition transmission and data monitoring method.
Background
Solar energy will become one of the main global energy sources in the 21 st century, and is the most original energy source, almost all other energy sources on the earth are directly or indirectly from solar energy, and photovoltaic power generation is the most direct embodiment of using solar energy. An inverter and a collector are basically needed for photovoltaic power generation. Due to diversification of inverter manufacturers, photovoltaic power station enterprises can use inverters and collectors of different brands at different time stages, and therefore power station data are distributed in data services of different manufacturers. Moreover, the dimensions of the power generation data stored by different inverter manufacturers are different at present, and the data structures are variable, so that the data cannot be analyzed and shared on the same platform. In real life, a photovoltaic power station builder can purchase equipment of different photovoltaic equipment manufacturers according to real-time market quotations in different time stages and different geographic areas. For a power station construction enterprise, the power generation condition of a power station owned by the enterprise cannot be monitored on one platform, and the data dimensions provided by different equipment providers are different, so that unified data planning is lacked, which leads to the problem that the enterprise cannot perfectly plan the data of the power station and carry out integration analysis on the data.
The prior art has the following defects and shortcomings:
how to share power generation data of multi-source heterogeneous power stations in the same type of data service platform and get through internal data relation to provide data sharing, application and analysis for production guidance of enterprises becomes a problem to be solved urgently by technical staff in the field at present.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a method for acquiring, transmitting and monitoring the real-time power generation data based on kafka, which can solve the problem of the existing method for acquiring, transmitting and monitoring the real-time power generation data based on kafka; the method comprises the steps of setting the collection calculation and the multi-source storage medium transmission of the power generation data of the multi-source heterogeneous power station, wherein the collection calculation and the multi-source storage medium transmission of the power generation data of the multi-source heterogeneous power station comprise the steps of monitoring synchronous power station data collection or calculation scheduling configuration by a core engine in real time, scheduling the data collection engine and a data calculation engine by the core engine, collecting multi-source heterogeneous data by the data collection engine for preliminary data arrangement, calculating the power generation data in batches by the data calculation engine, and persistently scheduling compensation tasks for abnormally collected data by a target data source and the core engine, so that the problems of realizing the collection calculation and the multi-source storage medium transmission of the power generation data of the multi-source heterogeneous power station and providing task monitoring are effectively solved.
In order to achieve the purpose of the kafka-based real-time power generation data acquisition transmission and data monitoring method, the invention provides the following technical scheme: the method comprises the steps of collecting and calculating power generation data of a multi-source heterogeneous power station and transmitting multi-source storage media, wherein the collecting and calculating of the power generation data of the multi-source heterogeneous power station and the transmitting of the multi-source storage media comprise the steps of monitoring synchronous power station data collection or calculation scheduling configuration by a core engine in real time, scheduling the data collection engine and a data calculation engine by the core engine, collecting multi-source heterogeneous data by the data collection engine for preliminary data arrangement, calculating power generation data by the data calculation engine in batches, and performing compensation task scheduling on abnormally collected data by a target data source and the core engine in a lasting mode.
Preferably, the core engine real-time monitoring synchronous power station data collection or calculation scheduling configuration comprises a core control engine and a task scheduling center, the core control engine stores power station data collection or calculation configuration, when the core control engine detects that configuration data changes, the synchronous power station data collection or calculation configuration is configured to the data collection or calculation engine, the core control engine stores task scheduling configuration, when the core control engine detects that the scheduling configuration changes, the synchronous latest task scheduling configuration is configured to the task scheduling center, and the core control engine monitors kafka in real time and performs maintenance management on topic.
Preferably, the monitoring of kafka and the maintenance and management of topic by the core control engine in real time includes monitoring of data collection or calculation configuration change by the core control engine, if a new power generation data collection configuration exists, adding a corresponding kafka topic, if the number of kafka topic partitions needs to be updated, updating the number of kafka topic partitions, and if an existing collection configuration is deleted, deleting the corresponding kafka topic.
Preferably, the scheduling of the data collection engine and the data calculation engine by the core engine includes that the core engine schedules other execution engines through a task scheduling center, the task scheduling center stores the latest task scheduling rule configuration and monitors tasks meeting the scheduling in real time, when the data collection tasks meeting the scheduling exist, the task scheduling center schedules the data collection engine and stores a scheduling log, when the data calculation tasks meeting the scheduling exist, the task scheduling center schedules the data collection engine and stores the scheduling log, the task scheduling center determines which tasks need compensation scheduling according to the scheduling log and the scheduling configuration, and if the tasks needing compensation scheduling are found, compensation task scheduling is performed.
Preferably, the data collection engine collects the multi-source heterogeneous data for preliminary data arrangement comprises the steps that the data collection engine adopts a corresponding data collection processor according to scheduling task information, if the data collection processor is in a webpage crawler mode, the webpage data collection processor is selected, if the data collection processor is in an api-json interface mode, the api-json collection processor is selected, if the data collection processor is in an api-xml interface mode, the api-xml collection processor is selected, if the data collection processor is in an excel file mode, the excel file collection processor is selected, and after collection and processing, the data are pushed to a corresponding kafka topoc.
Preferably, the data calculation engine calculates the power generation data in batch, persists to the target data source and comprises a data calculation engine, a corresponding data calculation processor is adopted according to scheduling task information, if the data calculation processor is a power aggregation power generation processor, the power generation tools are aggregated into corresponding time power generation, if the data calculation processor is an hour-level power generation processor, the power generation is aggregated into an hour level, if the data calculation processor is a day-level power generation processor, the power generation is aggregated into a day-time level, if the data calculation processor is a day-level power generation curing processor, the power generation data on the corresponding date is cured into a corresponding target storage medium, the power generation data is analyzed according to the data source configuration thread to be cured into a time sequence database Influxdb if the data processor comprises an infiluxdb, the power generation data is cured into a cache redis if the data processor comprises a Myql, the power generation data is cured into a relational database Mysql, if the MyOSS comprises the MyOSS, the data is cured into a corresponding catalog in the OSS according to the data relationship, if kafka is included, the calculated target data is pushed to the new kafka topic.
Preferably, the method is applied to collection calculation and multi-source storage medium transmission of power generation data of a multi-source heterogeneous power station, and the power generation data transmission calculation program of the power station realizes the steps of the data scheduling, calculation, transmission and monitoring method in any one of claims 1 to 6.
Preferably, the kafka-based real-time power generation data acquisition, transmission and data monitoring method comprises the following steps:
(1) configuring data source information in a core control module in a core engine, wherein the data source information comprises data source codes, data source names, data types, access connections, access permit information, data dimensions, data topic and data mapping relations, and is used as data source collection configuration information;
(2) configuring data calculation information in a core control module in a core engine, wherein the data calculation information comprises calculation rules, data source codes, data dimensions, topic, target dimensions and a target data source, and the data calculation configuration information is used as the data source calculation configuration information;
(3) monitoring the state of the collection/calculation configuration in real time by the core control engine, and carrying out corresponding synchronous operation if the state changes;
(4) if the data source collection configuration changes, updating the collection configuration in the data search engine and the corresponding information of kafka topic, and reloading the corresponding collection processor;
(5) if the data calculation configuration is changed, updating the calculation rules in the data calculation engine and reloading the calculation processor;
(6) the task scheduling center monitors the configured scheduling configuration information in real time, and if the scheduling conditions are met, the task meeting the conditions is performed correspondingly;
(7) the data collection engine collects multi-source isomerism to perform preliminary data arrangement, and pushes processed data to corresponding topic in corresponding kafka;
(8) the data calculation engine calculates the power generation data in batch, and the power generation data are persisted to a target data source or pushed to kafka again to wait for the next round of calculation;
(9) the task scheduling center continuously monitors the tasks which meet the requirements and need to be scheduled;
(10) the core control engine continues to monitor for changes in configuration.
Compared with the prior art, the invention provides a kafka-based real-time power generation data acquisition transmission and data monitoring method, which has the following beneficial effects:
1. the method for collecting, transmitting and monitoring the real-time power generation data based on the kafka can solve the problems that the power generation data of the multi-source heterogeneous power station is various in types, the data cannot be managed in an overall mode, the analysis power generation data calculation and solidification cannot be optimized, and the power generation data of the multi-source power station is heterogeneous, and provides log monitoring to monitor the execution condition of the task in the execution process by setting an intelligent scheduling task of a core engine, dynamically updating scheduling configuration and dynamic synchronous data collection or calculation configuration, intelligently collecting and analyzing the power generation data by a data collection engine, and intelligently calculating the multi-dimensional power generation data to the multi-source storage medium and intelligently compensating the data by a data elimination calculation engine.
Drawings
FIG. 1 is a schematic diagram of the principles of the present invention;
FIG. 2 is a diagram of the actual interaction of various services of the present invention;
FIG. 3 is a schematic diagram of a process for analyzing raw power generation data by the receipt collection engine according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1-3, a kafka-based real-time power generation data acquisition, transmission and data monitoring method includes collecting and calculating power generation data of a multi-source heterogeneous power station and transmitting multi-source storage media, wherein the collecting and calculating power generation data of the multi-source heterogeneous power station and the transmitting of the multi-source storage media include core engine real-time monitoring synchronous power station data collection or calculation scheduling configuration, a core engine schedules a data collection engine and a data calculation engine, the data collection engine collects multi-source heterogeneous data for preliminary data arrangement, the data calculation engine calculates power generation data in batches, and the data collection data and the core engine persistently perform compensation task scheduling on abnormally collected data.
In summary, the core engine real-time monitoring synchronous power station data collection or calculation scheduling configuration comprises a core control engine and a task scheduling center, the core control engine stores power station data collection or calculation configuration, when the core control engine detects that configuration data changes, the synchronous power station data collection or calculation configuration is configured to the data collection or calculation engine, the core control engine stores task scheduling configuration, when the core control engine detects that the scheduling configuration changes, the synchronous latest task scheduling is configured to the task scheduling center, and the core control engine monitors kafka in real time and performs maintenance management on topic. The core control engine monitors kafka in real time and carries out maintenance management on the topic, wherein the maintenance management comprises the core control engine monitoring data collection or calculation configuration change, if new power generation data collection configuration exists, the corresponding kafka topic is newly added, if the number of kafka topic partitions needs to be updated, the number of the kafka topic partitions is updated, and if the existing collection configuration is deleted, the corresponding kafka topic is deleted. The core engine schedules the data collection engine and the data calculation engine, and comprises the steps that the core engine schedules other execution engines through a task scheduling center, the task scheduling center stores latest task scheduling rule configuration and monitors tasks meeting scheduling in real time, when the data collection tasks meeting scheduling exist, the task scheduling center schedules the data collection engine and stores scheduling logs, when the data calculation tasks meeting scheduling exist, the task scheduling center schedules the data collection engine and stores the scheduling logs, the task scheduling center determines which tasks need compensation scheduling according to the scheduling logs and the scheduling configuration, and if the tasks needing compensation scheduling are found, compensation task scheduling is carried out. The data collection engine collects multi-source isomerism and carries out preliminary data arrangement, and the data collection engine adopts a corresponding data collection processor according to scheduling task information, selects a webpage data collection processor if the mode is a webpage crawler mode, selects an api-json collection processor if the mode is an api-json interface mode, selects the api-xml collection processor if the mode is the api-xml interface mode, selects an excel file collection processor if the mode is an excel file mode, and pushes data to a corresponding kafka topic after collection and processing. The data calculation engine calculates the power generation data in batch, the data calculation engine persists to a target data source and adopts a corresponding data calculation processor according to scheduling task information, if the data calculation processor is a power aggregation power generation processor, the power generation tools are aggregated into corresponding time power generation, if the data calculation processor is an hour-level power generation processor, the power generation is aggregated into an hour level, if the data calculation processor is a day-level power generation processor, the power generation is aggregated into a day-time level, if the data calculation processor is a day-level power generation curing processor, the power generation data corresponding to the date is cured into a corresponding target storage medium, the power generation data is analyzed according to a data source configuration thread and goes to the direction, if the data calculation processor contains inflixdb, the data is cured into a time sequence database inflixdb, if the data contains redis, the data is cured into a cache redis, if the data contains Myql, the data is cured into a relational database Mysql, if the data contains OSS, the data calculation processor is cured into a corresponding catalog in the OSS according to the data relationship, if kafka is included, the calculated target data is pushed to the new kafka topic. The method is applied to collection calculation and multi-source storage medium transmission of power generation data of a multi-source heterogeneous power station, and a power generation data transmission calculation program of the power station realizes the steps of the data scheduling, calculation, transmission and monitoring method in any one of claims 1 to 6.
The work use flow and the installation method of the invention are as follows: configuring data source information including data source codes, data source names, data types, access connections, access permit information, data dimensions, data topic and data mapping relations in a core control module in a core engine, and taking the data source information as data source collection configuration information; configuring data calculation information including a calculation rule, a data source code, a data dimension, a topic, a target dimension and a target data source in a core control module in a core engine, wherein the data calculation information is used as data source calculation configuration information; the core control engine monitors the state of the collection/calculation configuration in real time, and if the state changes, corresponding synchronous operation is carried out; if the data source collection configuration changes, updating the collection configuration in the data search engine and the corresponding information of kafka topic, and reloading the corresponding collection processor; if the data calculation configuration changes, updating the calculation rules in the data calculation engine and reloading the calculation processor; the task scheduling center monitors the configured scheduling configuration information in real time, and if the scheduling conditions are met, tasks meeting the conditions correspondingly are carried out; the data collection engine collects multi-source isomerism to perform preliminary data arrangement, and pushes processed data to corresponding topic in corresponding kafka; the data calculation engine calculates the power generation data in batch, and the power generation data are persisted to a target data source or pushed to kafka again to wait for the next round of calculation; the task scheduling center continuously monitors the tasks which meet the requirements and need to be scheduled; the core control engine continues to monitor for changes in configuration.
The method is developed based on a distributed message queue Kafka tool when being used, is particularly used for collecting and monitoring the power generation data of the multi-source heterogeneous power station to a uniform target source in a timing, large batch and multi-thread mode, and comprises the steps of monitoring the real-time of a core engine and synchronizing the collection configuration of the power generation data source of the power station and the data calculation dimension rule; the method comprises the steps that a core engine scheduling data acquisition engine acquires power generation data from a power generation data source of a heterogeneous power station; the data acquisition engine sends the multi-source heterogeneous power generation data to a kafka message queue; the core engine scheduling data calculation engine performs aggregation calculation on the power generation data; the data calculation engine solidifies the calculated data into the corresponding target data source according to the data source configuration; the core engine is provided with a scheduling compensation mechanism and can perform task scheduling compensation according to scheduling conditions, the problem that the multi-source heterogeneous power station power generation data are various and the data cannot be managed comprehensively is solved, a safe, quick, efficient and large-batch calculation method for synchronizing the multi-source heterogeneous power station power generation data collection and calculation to unified target source data is disclosed, the problems of calculation solidification of the analysis power generation data and heterogeneous multi-source power station power generation data are optimized, log monitoring is provided, and the execution condition of the tasks in the execution process is monitored.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (8)
1. A method for real-time power generation data acquisition transmission and data monitoring based on kafka is characterized in that: the method is applied to collection calculation and multi-source storage medium transmission of power generation data of the multi-source heterogeneous power station, and the collection calculation and the multi-source storage medium transmission of the power generation data of the multi-source heterogeneous power station comprise the steps of monitoring synchronous power station data collection or calculation scheduling configuration by a core engine in real time, scheduling the data collection engine and a data calculation engine by the core engine, collecting multi-source heterogeneous data by the data collection engine for preliminary data arrangement, calculating power generation data in batches by the data calculation engine, and persistently scheduling compensation tasks for abnormally collected data by a target data source and the core engine.
2. The kafka-based real-time power generation data acquisition, transmission and data monitoring method according to claim 1, wherein: the core engine real-time monitoring synchronous power station data collection or calculation scheduling configuration comprises a core control engine and a task scheduling center, the core control engine stores power station data collection or calculation configuration, when the core control engine detects that configuration data changes, the synchronous power station data collection or calculation configuration is configured to the data collection or calculation engine, the core control engine stores task scheduling configuration, when the core control engine detects that the scheduling configuration changes, the latest task scheduling configuration is synchronously configured to the task scheduling center, and the core control engine monitors kafka in real time and maintains and manages topic.
3. The kafka-based real-time power generation data acquisition, transmission and data monitoring method according to claim 2, wherein: the core control engine monitors kafka in real time and carries out maintenance management on the topic, wherein the maintenance management comprises the core control engine monitoring data collection or calculation configuration change, if a new power generation data collection configuration exists, the corresponding kafka topic is newly added, if the number of kafka topic partitions needs to be updated, the number of the kafka topic partitions is updated, and if the existing collection configuration is deleted, the corresponding kafka topic is deleted.
4. The kafka-based real-time power generation data acquisition, transmission and data monitoring method according to claim 1, wherein: the core engine schedules the data collection engine and the data calculation engine, and comprises the steps that the core engine schedules other execution engines through a task scheduling center, the task scheduling center stores latest task scheduling rule configuration and monitors tasks meeting scheduling in real time, when the data collection tasks meeting scheduling exist, the task scheduling center schedules the data collection engine and stores scheduling logs, when the data calculation tasks meeting scheduling exist, the task scheduling center schedules the data collection engine and stores the scheduling logs, the task scheduling center determines which tasks need to be compensated and scheduled according to the scheduling logs and the scheduling configuration, and if the tasks needing to be compensated and scheduled are found, compensation task scheduling is carried out.
5. The kafka-based real-time power generation data acquisition, transmission and data monitoring method according to claim 1, wherein: the data collection engine collects multi-source isomerism and carries out preliminary data arrangement, the data collection engine adopts a corresponding data collection processor according to scheduling task information, if the data collection processor is in a webpage crawler mode, the webpage data collection processor is selected, if the data collection processor is in an api-json interface mode, the api-json collection processor is selected, if the data collection processor is in an api-xml interface mode, the api-xml collection processor is selected, if the data collection processor is in an excel file mode, the excel file collection processor is selected, and after collection and processing, the data are pushed to a corresponding kafka topic.
6. The kafka-based real-time power generation data acquisition, transmission and data monitoring method according to claim 1, wherein: the data calculation engine calculates the power generation data in batch, the data calculation engine persists to a target data source and comprises a data calculation processor, a power aggregation power generation processor, a small-level power generation processor, a day-time level power generation curing processor, a target storage medium, a relational database Mysql, an OSS (open service system) and a data relation database OSS (open service system), wherein the data calculation processor adopts corresponding data according to scheduling task information, aggregates the power generation tools into corresponding time power generation, aggregates the power generation into an hour level according to the small-level power generation processor, aggregates the power generation into a day-time level according to the day-level power generation curing processor, the data of the corresponding date are cured into the corresponding target storage medium according to the day-level power generation curing processor, the data of the corresponding date are analyzed according to a data source configuration thread, the data direction of the power generation data are cured into the time-sequence database Influxdb if the data contain redis, the Myql, the data are cured into the relational database Mysql if the OSS is contained, if kafka is included, the calculated target data is pushed to the new kafka topic.
7. The kafka-based real-time power generation data acquisition, transmission and data monitoring method according to claim 1, wherein: the method is applied to collection calculation and multi-source storage medium transmission of power generation data of a multi-source heterogeneous power station, and the power generation data transmission calculation program of the power station realizes the steps of the data scheduling, calculation, transmission and monitoring method in any one of claims 1 to 6.
8. The kafka-based real-time power generation data acquisition, transmission and data monitoring method according to claim 1, comprising the steps of:
(1) configuring data source information in a core control module in a core engine, wherein the data source information comprises data source codes, data source names, data types, access connections, access permit information, data dimensions, data topic and data mapping relations, and is used as data source collection configuration information;
(2) configuring data calculation information in a core control module in a core engine, wherein the data calculation information comprises calculation rules, data source codes, data dimensions, topic, target dimensions and a target data source, and the data calculation configuration information is used as the data source calculation configuration information;
(3) monitoring the state of the collection/calculation configuration in real time by the core control engine, and carrying out corresponding synchronous operation if the state changes;
(4) if the data source collection configuration changes, updating the collection configuration in the data search engine and the corresponding information of kafka topic, and reloading the corresponding collection processor;
(5) if the data calculation configuration is changed, updating the calculation rules in the data calculation engine and reloading the calculation processor;
(6) the task scheduling center monitors the configured scheduling configuration information in real time, and if the scheduling conditions are met, the task meeting the conditions is performed correspondingly;
(7) the data collection engine collects multi-source isomerism to perform preliminary data arrangement, and pushes processed data to corresponding topic in corresponding kafka;
(8) the data calculation engine calculates the power generation data in batch, and the power generation data are persisted to a target data source or pushed to kafka again to wait for the next round of calculation;
(9) the task scheduling center continuously monitors the tasks which meet the requirements and need to be scheduled;
(10) the core control engine continues to monitor for changes in configuration.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111336012.8A CN114020763A (en) | 2021-11-12 | 2021-11-12 | Kafka-based real-time power generation data acquisition and transmission and data monitoring method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111336012.8A CN114020763A (en) | 2021-11-12 | 2021-11-12 | Kafka-based real-time power generation data acquisition and transmission and data monitoring method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114020763A true CN114020763A (en) | 2022-02-08 |
Family
ID=80063544
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111336012.8A Pending CN114020763A (en) | 2021-11-12 | 2021-11-12 | Kafka-based real-time power generation data acquisition and transmission and data monitoring method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114020763A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114185914A (en) * | 2022-02-16 | 2022-03-15 | 西安热工研究院有限公司 | Complementary calculation method, system, equipment and storage medium for calculating label data |
-
2021
- 2021-11-12 CN CN202111336012.8A patent/CN114020763A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114185914A (en) * | 2022-02-16 | 2022-03-15 | 西安热工研究院有限公司 | Complementary calculation method, system, equipment and storage medium for calculating label data |
| CN114185914B (en) * | 2022-02-16 | 2022-04-29 | 西安热工研究院有限公司 | Complementary calculation method, system, equipment and storage medium for calculating label data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110022226B (en) | Object-oriented data acquisition system and acquisition method | |
| CN112565011B (en) | Novel power consumption information acquisition system based on cloud platform | |
| CN110047014B (en) | User electric quantity data restoration method based on load curve and historical electric quantity | |
| CN112178756B (en) | Intelligent heat supply management system and working method thereof | |
| CN111142414A (en) | Intelligent power plant management and control system based on edge cloud cooperation | |
| CN108183869B (en) | Electric quantity data acquisition system based on distributed message queue | |
| CN112886922B (en) | Calculation method, device and storage medium for loss electric quantity of photovoltaic power station | |
| CN114020763A (en) | Kafka-based real-time power generation data acquisition and transmission and data monitoring method | |
| CN114745409A (en) | Remote control method for intelligent Internet of things equipment | |
| CN110555583A (en) | method for uniformly processing wide-area operation data of intelligent power grid dispatching control system | |
| CN111798655B (en) | Operation data minute-level acquisition method suitable for power Internet of things platform area | |
| CN112052965A (en) | Visual electric power equipment online management system and management method | |
| CN111915048A (en) | Novel power failure management method based on intelligent analysis decision theory technology | |
| CN115543345A (en) | Distributed computing system for power time sequence data and implementation method thereof | |
| CN111090698B (en) | Alarm synchronization method and device for centralized monitoring of distribution network 1+ N system | |
| CN110796567B (en) | Production early report application framework based on regulation and control cloud | |
| CN112261509A (en) | Meter reading scheduling method for electricity consumption information acquisition terminal | |
| CN112329408A (en) | Intelligent report reporting system of power grid regulation and control office system | |
| CN111401577A (en) | Device management method, device and storage medium | |
| Tian et al. | RETRACTED: Research on Big Data Analysis Platform of Power Grid Enterprise Accounting Based on Cloud Computing | |
| CN116361389B (en) | Data synchronization link method and system based on national network marketing acquisition system | |
| CN117374978B (en) | Grid-connected scheduling management method and system constructed by combining knowledge graph | |
| CN107908476B (en) | Data processing method and device based on distributed cluster | |
| Sun et al. | Mechanism design for unified management of power grid planning data | |
| CN114371952A (en) | Process automation system based on ITSM system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |