CN111737325A - Power data analysis method and device based on big data technology - Google Patents
Power data analysis method and device based on big data technology Download PDFInfo
- Publication number
- CN111737325A CN111737325A CN202010448021.5A CN202010448021A CN111737325A CN 111737325 A CN111737325 A CN 111737325A CN 202010448021 A CN202010448021 A CN 202010448021A CN 111737325 A CN111737325 A CN 111737325A
- Authority
- CN
- China
- Prior art keywords
- data
- index
- electric power
- power
- big
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000005516 engineering process Methods 0.000 title claims abstract description 24
- 238000007405 data analysis Methods 0.000 title claims description 20
- 238000003860 storage Methods 0.000 claims abstract description 24
- 238000007781 pre-processing Methods 0.000 claims abstract description 11
- 238000013500 data storage Methods 0.000 claims description 11
- 230000014509 gene expression Effects 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 8
- 238000007670 refining Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 abstract description 3
- 230000015556 catabolic process Effects 0.000 abstract 1
- 239000003245 coal Substances 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000004519 manufacturing process Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 239000000446 fuel Substances 0.000 description 5
- 238000007726 management method Methods 0.000 description 5
- 238000010248 power generation Methods 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000003066 decision tree Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 229910052799 carbon Inorganic materials 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000013439 planning Methods 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/06—Electricity, gas or water supply
Abstract
The invention discloses a method and a device for analyzing electric power data based on big data technology, which are characterized in that electric power data in an area are collected by a multi-path complex link method, and electric power original data are obtained after preprocessing and stored; classifying the original power data according to data types, defining the classified data to form an index data set, and performing differentiated classified storage; and releasing the index data, and inquiring and displaying the index data from the database according to the display requirement. The invention is based on big data technology to classify, identify, store and display the electric power data, the data acquisition is accurate, the display can be switched rapidly according to the scene, a differential storage method is adopted, and the data to be stored is stored in a plurality of cache servers in a full amount, so that the condition that the calculation is influenced by the breakdown of a single server is avoided.
Description
Technical Field
The invention relates to the technical field of data analysis, in particular to a power data analysis method and device based on a big data technology.
Background
The regional strategic operation cockpit reflects the whole operation state of an enterprise in real time, provides real-time, reliable and accurate data analysis for the enterprise, provides powerful support for operation management and scientific decision, can further accurately and timely grasp and adjust the development direction of the enterprise, promotes the high-efficiency development of the enterprise, and improves the economic benefit and competitiveness of the enterprise. The cockpit is constructed by aiming at reasonably developing and utilizing networks and information resources, organically integrating related information resources of a group company and existing application systems of the local part of a regional company, increasing functions required by safe production, strategic planning, financial assets, fuel materials, marketing and the like of the company, constructing a unified information platform covering all services of the regional company, realizing service data sharing among the systems of the regional company, providing powerful information support for efficient management and scientific decision of the regional company and affiliated units, and promoting the optimization of system management processes of the regional company and the improvement of management efficiency.
As the number of company information systems in each area increases, the amount of data increases. Due to the limitation of data acquisition, the accuracy and consistency of data cannot be guaranteed, so that the analysis of cockpit data is inaccurate, and accurate decision support cannot be provided for enterprise managers.
In order to control the business operation and operation conditions of an enterprise in time, an enterprise manager often needs to comprehensively analyze different business data to form a corresponding business scene, but the business data often has strong speciality. The traditional technology is limited in service scene processing capacity, the scene switching speed is slow, and the requirements of enterprise managers cannot be quickly responded.
Disclosure of Invention
In order to overcome the defects in the prior art, the invention provides a power data analysis method and device based on a big data technology, and solves the problems of inaccurate and inconsistent data acquisition and incapability of providing rapid scene switching.
In order to achieve the above purpose, the invention adopts the following technical scheme: a power data analysis method based on big data technology,
acquiring power data in the region by a multi-path complex link method, preprocessing the power data to obtain power original data, and storing the power original data;
classifying the original power data according to data types, defining the classified data to form an index data set, and performing differentiated classified storage;
and releasing the index data, and inquiring and displaying the index data from the database according to the display requirement.
Furthermore, the power data source is power data issued by a regional internal power system and a central dispatching website.
Further, the acquiring of the power data in the area by the multi-path complex link method includes:
setting data acquisition configuration, wherein the content of the data acquisition configuration comprises a data type, a data request address, a protocol type of the data request address and a data address loading sequence request frequency expression;
registering the data acquisition configuration as acquisition configuration information into a Zookeeper cluster of a distributed coordinator;
acquiring acquisition configuration information from a Zookeeper cluster, establishing different acquisition channels according to protocol types of data request addresses in the acquisition configuration information, creating acquisition tasks according to the data request addresses, request frequency expressions and data address loading sequences in the configuration, distributing the acquisition tasks of the same protocol type to the same acquisition channels for operation, and returning through the acquisition channels after the acquisition tasks operate and acquire data.
Further, the preprocessed power raw data is stored, and the storage method comprises the following steps:
the real-time data storage adopts a Redis database, the non-real-time data is stored by adopting a big data Hadoop and Hbase system, and the data is indexed into an Elastic Search during the big data storage;
and setting the validity period of the data for the real-time data stored in the Redis database, automatically clearing the data after the expiration, and automatically prolonging the validity period of the data if the data in the Redis is accessed.
Further, the defining the classified data to form an index data set includes:
refining data characteristics according to the classified data;
removing the duplication and combining the same meaning index data in different classifications;
the data pointer name is encoded.
Further, the data characteristics include a time period of the data, main data to which the data belongs, a unit of the data, a value type of the data, and whether the data is calculated.
Further, the method for encoding the data index name includes:
adding a random code to the Chinese initial of the data index;
or a unique character string consisting of three parts of data type, period and serial number.
Further, the automatic classification storage process is as follows:
after the classified data are defined to form an index data set, a data set with the access times higher than a set threshold and the query efficiency lower than the set threshold in a big data storage Hbase database is placed into a Redis cluster and is also placed into an ElasticSearch cluster, the Redis cluster and the ElasticSearch cluster are respectively arranged in different cache servers, and the databases are mutually backed up.
Further, the display is inquired from the database according to the display requirement, including,
when certain index data needs to be accessed according to the display requirement, reading from Redis, if the index data cannot be read from Redis, reading from ElasticSearch, returning to display when any one of the two-level caches reads the cache data, and if the two-level caches do not read the cache data, directly reading from the Hbase database.
A big data technology-based power data analysis device comprises:
the method comprises the steps that a storage module is obtained, electric power data in an area are collected through a multi-path complex link method, and electric power original data are obtained after preprocessing and stored;
the index data set forming module is used for classifying the original power data according to the data types, defining the classified data to form an index data set and storing the index data set in a differentiated classification mode;
and the query display module is used for issuing the index data, querying the index data from the database according to the display requirement and displaying the index data.
The invention achieves the following beneficial effects: the method comprises the steps of acquiring original power data in an area by a multi-path complex link method, preprocessing the original power data and storing the preprocessed original power data; the method comprises the steps of automatically classifying and identifying the original power data according to data types, defining the classified data to form an index data set, and automatically classifying and storing the index data set; and issuing the index data, and performing diversified display through a visual component. The electric power data are classified, identified, stored and displayed based on a big data technology, the data acquisition is accurate, and the display can be rapidly switched according to scenes.
The differential storage method is adopted, and the total data to be stored is stored in a plurality of cache servers, so that the condition that the calculation is influenced by the downtime of a single server is avoided.
Drawings
FIG. 1 is a flow chart of a method in an embodiment of the invention;
fig. 2 is a schematic illustration of a method in an embodiment of the invention.
Detailed Description
The invention is further described below with reference to the accompanying drawings. The following examples are only for illustrating the technical solutions of the present invention more clearly, and the protection scope of the present invention is not limited thereby.
As shown in fig. 1 to 2, a power data analysis method based on big data technology includes the steps of:
acquiring original power data in an area by a multi-path complex link method, preprocessing and storing the original power data;
the power data sources generally comprise data issued by a regional internal system and a central dispatching network station, the data sources are complex, and the data have the characteristics of uneven distribution, large difference of distribution environments, large quantity of network stations (systems), large data quantity and the like.
Internal system data generally divide into a plurality of aspects such as environmental protection, fuel, marketing, daily profit, comprehensive statistics, people's resources, carbon emission, and internal system data includes: coal supply, coal consumption, coal storage, coal heat value of coal entering a factory, coal heat value of coal entering a furnace, coal storage and consumption of a coal yard, annual accumulation of power consumption of a power plant and annual accumulation of equivalent operation.
The data issued by the medium-voltage regulating network station generally comprises nine major categories of electric quantity conditions, power receiving conditions, load conditions, power utilization conditions, peak load limit conditions, electric energy quality, hydropower running conditions, thermal power running conditions and scheduling running conditions. The specific data issued by the central adjusting network station comprises: the power generation capacity of the whole network, the power supply capacity of the whole province, the power supply capacity of the periphery, the power transmission and reception situation, the load situation (the highest power generation load of the overall regulation, the lowest power generation load of the overall regulation, the peak-valley difference of the overall regulation load), the power consumption situation (the region), and the peak-shifting and power limiting situation (the region).
The method for acquiring the power data in the area through the multi-path complex link comprises the following steps:
1) and setting data acquisition configuration, wherein the content of the data acquisition configuration comprises data types, data request addresses, protocol types of the data request addresses and data address loading sequence request frequency expressions.
Data categories such as marketing class data, production class data, fuel class data, and the like;
a data category can configure a plurality of data request addresses;
the request frequency expression represents a cron expression for setting request requirements, such as the expression: "5923? "indicates that 23 points 59 request data once; ? "indicates that data is requested every five minutes;
the protocol type of the data request address comprises webservice, restful, mq and websocket; and the data request addresses of different protocols are called by adopting different calling methods.
2) Registering the data acquisition configuration as acquisition configuration information into a Zookeeper cluster of a distributed coordinator, wherein each Zookeeper cluster server node has complete acquisition configuration information;
3) acquiring acquisition configuration information from the Zookeeper, establishing different acquisition channels according to the protocol type of a data request address in the configuration information, creating acquisition tasks according to the data request address, a request frequency expression and a data address loading sequence in the configuration, and distributing the acquisition tasks of the same protocol type to the same acquisition channels for operation. And after the collection task runs and acquires data, returning through the collection channel.
ZooKeeper is a highly available framework for distributed data management and system coordination. A typical application scenario is a publishing and subscribing model, that is, a so-called configuration center, where a publisher publishes (registers) data to a ZooKeeper node, so that the subscriber dynamically acquires the data, and centralized management and dynamic update of configuration information are realized. The maintenance of the data request address list is realized by relying on the open source technology.
Preprocessing the acquired data, and then performing differential storage: the real-time data is stored in a memory mode, so that rapid data reading and writing are realized; for non-real-time data (including historical archive data), high-availability, high-throughput large data storage based on a distributed file system is employed. Meanwhile, the data query engine is provided to complete the rapid data query function.
The real-time data storage is carried out in a Redis database mode, the validity period of data is required to be set for the real-time data stored in the Redis database, the data is automatically cleared after the expiration, and if the data in the Redis is accessed, the validity period of the data can be automatically prolonged, so that the real-time data of the hot spot are always in the Redis database, and the data reading speed is greatly improved.
The non-real-time data are stored by adopting a big data Hadoop and Hbase system, and the data are indexed into the Elastic Search when the big data are stored, so that the non-real-time data and the historical data can be efficiently retrieved.
Step two, automatically classifying the stored original data according to the data types, defining the classified data to form an index data set, and performing differentiated classified storage;
step 1, performing data classification integration on the original data in the data storage according to data types, namely marking different service labels on the data, such as marketing data, production data, fuel data and the like. The methods used here are both dynamic and static. The dynamic division method belongs to an automatic identification and classification mode, and is used for automatically classifying an original data set by using a decision tree algorithm in a data mining technology, and the static division method belongs to a manual intervention mode, and is used for manually performing intervention and processing when an abnormal alarm occurs in the dynamic division method. And (3) automatically classifying and identifying the data stored in the step one by using a decision tree algorithm, wherein the classified and identified data are generally classified into A type, B type, C type, D type and E type, wherein the A type represents marketing type data, the B type represents fuel type data, the C type represents environment-friendly type data, the D type represents financial type data, and the E type represents production type data.
Step 2, data definition is to comb and summarize the classified data to form an index set;
the data definition process is as follows:
1) and refining data characteristics according to the classified data, wherein the data characteristics comprise the time period of the data, the main data to which the data belongs, the unit of the data, the value type of the data and whether the data is calculated. Wherein the time period means the statistical time dimension of the data, such as the generation of electricity is divided by the degree of the day, the degree of the month and the degree of the year, the degree of the day is obtained by filling or from a third-party system, and the degree of the month and the degree of the year can be accumulated by the degree of the day; the main data of the data comprises the attribute of a power plant, the attribute of a unit or the attribute of a power station; whether the data is computed indicates whether the data was obtained directly from the source or is counted against existing data.
2) And removing the duplicate, and combining the data of the indexes with the same meaning in different classifications, wherein the production classification and the marketing classification have the data of the power generation amount, the meaning and the data are the same, so that the indexes are only required to be classified into one in the definition, and the indexes can be respectively quoted when a specific service system is used.
3) And coding the data index name, wherein the coding needs to ensure global uniqueness, and the index name is the Chinese meaning of the data index. There are two general coding rules: firstly, adding a random code to the Chinese initial of a data index, for example, the code of the generated energy daily index is FDLRD 001; secondly, the unique character string is composed of three parts of first-level classification, period and serial number, and the rule is as follows:
| rank order number | Level 0 (2 position) | Level 1 (2 position) | Grade 2 (4 position) |
| Hierarchical names | First-level classification coding | Periodic encoding | Concrete data coding (running water) |
A set of differential storage method structures is built by using Redis (distributed cache), Hadoop, Hbase (big data storage) and ElasticSerach components, and a redundant and full-scale differential storage method process is provided.
The differential classification storage process comprises the following steps:
after the classified data is defined to form an index data set, a data set which is frequently accessed in a large data storage Hbase and has low query efficiency needs to be placed into a Redis cluster, and meanwhile, the data set is also stored into an ElasticSearch cluster. The Redis cluster and the ElasticSearch cluster are respectively arranged in different cache servers, and databases are backed up with each other to prevent data loss caused by damage of a certain database node.
The difference between the differentiated cache and the traditional cache is mainly two points: firstly, an elastic search is added to serve as a secondary cache, reading is carried out from a primary cache Redis when the cache is read, and if the primary cache cannot be read, reading is continued from the secondary cache Elastic search; secondly, the cache data of each Redis node in the Redis and ElasticSearch clusters is full, that is, any node in the Redis cluster cannot access, the cache reading from other Redis nodes is not affected, the cache loss does not exist, and the ElasticSearch is similar.
The differentiated storage provides three methods to push data in the big data Hbase into the cache. The first is to automatically load the data in Hbase at program start and buffer the part of the data into a multi-way buffer (i.e. Redis and elastic search); the second method provides the operation of manually triggering the load cache; the third is passive loading, that is, the buffer is read first, and if the buffer is not in the buffer, the Hbase database is read and the data read from the database is loaded into the multi-way buffer.
And step three, issuing the index data, inquiring from the database according to the display requirement, and performing diversified display through the visual component.
Step 1, issuing the classified data through a service interface, wherein the service interface mainly has two forms: restful and websocket. Real-time data are issued through a websocket protocol interface, and non-real-time data and historical data are issued through a restful protocol interface.
And step 2, after the data are published, performing data through a visualization component such as echarts and the like, and performing diversified display such as tables, graphs and the like in a rendering mode according to system requirements.
In the process of data presentation, how to quickly query and extract a large amount of data is one of the important factors influencing the presentation speed and effect.
When certain index data is to be accessed, reading is firstly carried out from Redis, if the index data cannot be read from Redis, reading is carried out from elastic search, and any one of the two-level caches can be read into cache data to be returned for showing. And if no cache data is read from the two-level cache, directly reading from the Hbase database.
The method comprises the steps of acquiring original power data in an area by a multi-path complex link method, preprocessing the original power data and storing the preprocessed original power data; the method comprises the steps of automatically classifying and identifying the original power data according to data types, defining the classified data to form an index data set, and automatically classifying and storing the index data set; and issuing the index data, and performing diversified display through a visual component. The electric power data are classified, identified, stored and displayed based on a big data technology, the data acquisition is accurate, and the display can be rapidly switched according to scenes.
The differential storage method is adopted, and the total data to be stored is stored in a plurality of cache servers, so that the condition that the calculation is influenced by the downtime of a single server is avoided.
A big data technology-based power data analysis device comprises:
the method comprises the steps that a storage module is obtained, electric power data in an area are collected through a multi-path complex link method, and electric power original data are obtained after preprocessing and stored;
the index data set forming module is used for classifying the original power data according to the data types, defining the classified data to form an index data set and storing the index data set in a differentiated classification mode;
and the query display module is used for issuing the index data, querying the index data from the database according to the display requirement and displaying the index data.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A power data analysis method based on big data technology is characterized in that:
acquiring power data in the region by a multi-path complex link method, preprocessing the power data to obtain power original data, and storing the power original data;
classifying the original power data according to data types, defining the classified data to form an index data set, and performing differentiated classified storage;
and releasing the index data, and inquiring and displaying the index data from the database according to the display requirement.
2. The electric power data analysis method based on big data technology as claimed in claim 1, wherein: the power data source is power data issued by a regional internal power system and a central dispatching website.
3. The electric power data analysis method based on big data technology as claimed in claim 1, wherein: the method for collecting the power data in the area through the multi-path complex link comprises the following steps:
setting data acquisition configuration, wherein the content of the data acquisition configuration comprises a data type, a data request address, a protocol type of the data request address and a data address loading sequence request frequency expression;
registering the data acquisition configuration as acquisition configuration information into a Zookeeper cluster of a distributed coordinator;
acquiring acquisition configuration information from a Zookeeper cluster, establishing different acquisition channels according to protocol types of data request addresses in the acquisition configuration information, creating acquisition tasks according to the data request addresses, request frequency expressions and data address loading sequences in the configuration, distributing the acquisition tasks of the same protocol type to the same acquisition channels for operation, and returning through the acquisition channels after the acquisition tasks operate and acquire data.
4. The electric power data analysis method based on big data technology as claimed in claim 1, wherein: the electric power original data obtained after the preprocessing is stored, and the storage method comprises the following steps:
the real-time data storage adopts a Redis database, the non-real-time data is stored by adopting a big data Hadoop and Hbase system, and the data is indexed into an Elastic Search during the big data storage;
and setting the validity period of the data for the real-time data stored in the Redis database, automatically clearing the data after the expiration, and automatically prolonging the validity period of the data if the data in the Redis is accessed.
5. The electric power data analysis method based on big data technology as claimed in claim 1, wherein: the defining the classified data to form an index data set includes:
refining data characteristics according to the classified data;
removing the duplication and combining the same meaning index data in different classifications;
the data pointer name is encoded.
6. The electric power data analysis method based on big data technology as claimed in claim 5, wherein: the data characteristics comprise the time period of the data, the main data to which the data belongs, the unit of the data, the value type of the data and whether the data is calculated.
7. The electric power data analysis method based on big data technology as claimed in claim 5, wherein: the method for coding the data index name comprises the following steps:
adding a random code to the Chinese initial of the data index;
or a unique character string consisting of three parts of data type, period and serial number.
8. The electric power data analysis method based on big data technology according to claim 1, characterized in that: the automatic classification and storage process comprises the following steps:
after the classified data are defined to form an index data set, a data set with the access times higher than a set threshold and the query efficiency lower than the set threshold in a big data storage Hbase database is placed into a Redis cluster and is also placed into an ElasticSearch cluster, the Redis cluster and the ElasticSearch cluster are respectively arranged in different cache servers, and the databases are mutually backed up.
9. The electric power data analysis method based on big data technology according to claim 8, characterized in that: and inquiring the display data from the database according to the display requirement, including,
when certain index data needs to be accessed according to the display requirement, reading from Redis, if the index data cannot be read from Redis, reading from ElasticSearch, returning to display when any one of the two-level caches reads the cache data, and if the two-level caches do not read the cache data, directly reading from the Hbase database.
10. The utility model provides an electric power data analysis device based on big data technique which characterized in that: the method comprises the following steps:
the method comprises the steps that a storage module is obtained, electric power data in an area are collected through a multi-path complex link method, and electric power original data are obtained after preprocessing and stored;
the index data set forming module is used for classifying the original power data according to the data types, defining the classified data to form an index data set and storing the index data set in a differentiated classification mode;
and the query display module is used for issuing the index data, querying the index data from the database according to the display requirement and displaying the index data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010448021.5A CN111737325A (en) | 2020-05-25 | 2020-05-25 | Power data analysis method and device based on big data technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010448021.5A CN111737325A (en) | 2020-05-25 | 2020-05-25 | Power data analysis method and device based on big data technology |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111737325A true CN111737325A (en) | 2020-10-02 |
Family
ID=72647652
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010448021.5A Pending CN111737325A (en) | 2020-05-25 | 2020-05-25 | Power data analysis method and device based on big data technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111737325A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112269830A (en) * | 2020-10-20 | 2021-01-26 | 苏州莱锦机电自动化有限公司 | Big data analysis method, system, computer equipment and storage medium thereof |
| CN113342804A (en) * | 2021-03-06 | 2021-09-03 | 广东信通通信有限公司 | Big data-based dissociative data tagged reutilization method |
| CN113656370A (en) * | 2021-08-16 | 2021-11-16 | 南方电网数字电网研究院有限公司 | Data processing method and device for power measurement system and computer equipment |
| CN115174585A (en) * | 2022-08-30 | 2022-10-11 | 平安银行股份有限公司 | Message generation method based on Elasticissearch, redis data management system and master control equipment |
| CN113656370B (en) * | 2021-08-16 | 2024-04-30 | 南方电网数字电网集团有限公司 | Data processing method and device for electric power measurement system and computer equipment |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104317800A (en) * | 2014-09-19 | 2015-01-28 | 山东大学 | Hybrid storage system and method for mass intelligent power utilization data |
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN106878092A (en) * | 2017-03-28 | 2017-06-20 | 上海以弈信息技术有限公司 | A kind of network O＆M monitor in real time of multi-source heterogeneous data fusion is presented platform with analysis |
| CN106899638A (en) * | 2015-12-21 | 2017-06-27 | 中国科学院信息工程研究所 | A kind of fusions networking relationships system for solving Heterogeneous data and method |
| CN108763436A (en) * | 2018-05-25 | 2018-11-06 | 福州大学 | A kind of distributed data-storage system based on ElasticSearch and HBase |
| CN110414802A (en) * | 2019-07-08 | 2019-11-05 | 南京华盾电力信息安全测评有限公司 | Conglomerate Analysis of Policy Making flight deck system |
| CN111031040A (en) * | 2019-12-12 | 2020-04-17 | 山东有人信息技术有限公司 | Cloud service method and system supporting multiple protocol network access modes |
| CN111177191A (en) * | 2019-11-28 | 2020-05-19 | 南京华盾电力信息安全测评有限公司 | Key index calculation analysis method based on carbon emission transaction |
-
2020
- 2020-05-25 CN CN202010448021.5A patent/CN111737325A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104317800A (en) * | 2014-09-19 | 2015-01-28 | 山东大学 | Hybrid storage system and method for mass intelligent power utilization data |
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN106899638A (en) * | 2015-12-21 | 2017-06-27 | 中国科学院信息工程研究所 | A kind of fusions networking relationships system for solving Heterogeneous data and method |
| CN106878092A (en) * | 2017-03-28 | 2017-06-20 | 上海以弈信息技术有限公司 | A kind of network O＆M monitor in real time of multi-source heterogeneous data fusion is presented platform with analysis |
| CN108763436A (en) * | 2018-05-25 | 2018-11-06 | 福州大学 | A kind of distributed data-storage system based on ElasticSearch and HBase |
| CN110414802A (en) * | 2019-07-08 | 2019-11-05 | 南京华盾电力信息安全测评有限公司 | Conglomerate Analysis of Policy Making flight deck system |
| CN111177191A (en) * | 2019-11-28 | 2020-05-19 | 南京华盾电力信息安全测评有限公司 | Key index calculation analysis method based on carbon emission transaction |
| CN111031040A (en) * | 2019-12-12 | 2020-04-17 | 山东有人信息技术有限公司 | Cloud service method and system supporting multiple protocol network access modes |
Non-Patent Citations (2)
| Title |
|---|
| 冷程浩: "企业战略经营驾驶舱的研究", 电脑编程技巧与维护, pages 14 - 15 * |
| 赵弘俊: "供电企业信息驾驶舱研究", 中国优秀硕士学位论文全文数据库, pages 150 - 438 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112269830A (en) * | 2020-10-20 | 2021-01-26 | 苏州莱锦机电自动化有限公司 | Big data analysis method, system, computer equipment and storage medium thereof |
| WO2022082892A1 (en) * | 2020-10-20 | 2022-04-28 | 苏州莱锦机电自动化有限公司 | Big data analysis method and system, and computer device and storage medium thereof |
| CN113342804A (en) * | 2021-03-06 | 2021-09-03 | 广东信通通信有限公司 | Big data-based dissociative data tagged reutilization method |
| CN113656370A (en) * | 2021-08-16 | 2021-11-16 | 南方电网数字电网研究院有限公司 | Data processing method and device for power measurement system and computer equipment |
| CN113656370B (en) * | 2021-08-16 | 2024-04-30 | 南方电网数字电网集团有限公司 | Data processing method and device for electric power measurement system and computer equipment |
| CN115174585A (en) * | 2022-08-30 | 2022-10-11 | 平安银行股份有限公司 | Message generation method based on Elasticissearch, redis data management system and master control equipment |
| CN115174585B (en) * | 2022-08-30 | 2022-11-11 | 平安银行股份有限公司 | Message generation method based on Elasticissearch, redis data management system and master control equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107402976B (en) | Power grid multi-source data fusion method and system based on multi-element heterogeneous model | |
| CN111737325A (en) | Power data analysis method and device based on big data technology | |
| US9734228B2 (en) | Digital analytics system | |
| CN107256443A (en) | Line loss real-time computing technique based on business and data integration | |
| CN111104394A (en) | Energy data warehouse system construction method and device | |
| CN113064866B (en) | Power business data integration system | |
| CN101566981A (en) | Method for establishing dynamic virtual data base in analyzing and processing system | |
| CN111241129B (en) | Industrial production enterprise index data acquisition and calculation system | |
| CN106649687A (en) | Method and device for on-line analysis and processing of large data | |
| CN111080261A (en) | Visual data asset management system based on big data | |
| CN105335822A (en) | Smart power grid unified data model modeling method for big data analysis | |
| CN110555452A (en) | network problem processing method and device based on intelligent clustering | |
| CN111160865A (en) | Workflow management method and device | |
| CN111159154A (en) | Energy data warehouse system | |
| CN109165207B (en) | Drinking water mass data storage management method and system based on Hadoop | |
| CN116993227B (en) | Heat supply analysis and evaluation method, system and storage medium based on artificial intelligence | |
| CN109768878B (en) | Network work order calculation method and device based on big data | |
| Klisarova-Belcheva et al. | Business intelligence and analytics–contemporary system model | |
| CN115470279A (en) | Data source conversion method, device, equipment and medium based on enterprise data | |
| CN114416785A (en) | Stream type enterprise big data processing method and storage medium | |
| CN115292754A (en) | Credit investigation platform credit data risk monitoring method | |
| CN114648163A (en) | Energy consumption data management system and method suitable for smart city | |
| CN109189818B (en) | Tobacco data granularity division method in value-added service environment | |
| Zhao et al. | Hadoop-Based Power Grid Data Quality Verification and Monitoring Method | |
| CN112380164B (en) | Electric power system scene data management method, device and system based on snapshot technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |