CN108804606B - Method and system for migrating power measurement data to HBase - Google Patents
Method and system for migrating power measurement data to HBase Download PDFInfo
- Publication number
- CN108804606B CN108804606B CN201810533864.8A CN201810533864A CN108804606B CN 108804606 B CN108804606 B CN 108804606B CN 201810533864 A CN201810533864 A CN 201810533864A CN 108804606 B CN108804606 B CN 108804606B
- Authority
- CN
- China
- Prior art keywords
- module
- log
- data
- hbase
- query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 23
- 238000005259 measurement Methods 0.000 title claims abstract description 19
- 230000005012 migration Effects 0.000 claims abstract description 35
- 238000013508 migration Methods 0.000 claims abstract description 35
- 238000006243 chemical reaction Methods 0.000 claims abstract description 8
- 230000001502 supplementing effect Effects 0.000 claims abstract description 5
- 230000007704 transition Effects 0.000 claims abstract description 3
- 230000002159 abnormal effect Effects 0.000 claims description 23
- 230000008569 process Effects 0.000 claims description 8
- 238000012546 transfer Methods 0.000 claims description 5
- 238000001514 detection method Methods 0.000 claims description 3
- 230000003203 everyday effect Effects 0.000 claims description 3
- 238000007689 inspection Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 239000013589 supplement Substances 0.000 claims description 3
- 230000005856 abnormality Effects 0.000 claims 1
- 230000005540 biological transmission Effects 0.000 abstract description 8
- 238000007418 data mining Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Business, Economics & Management (AREA)
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Economics (AREA)
- Public Health (AREA)
- Water Supply & Treatment (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a method and a system for migrating power measurement data to HBase, wherein the system comprises the following steps: the query module: querying data from a relational database through JDBC; a configuration module: the method is used for configuring connection, tables, query conditions, result output formats, writing modes and the like; a log module: recording the state of each table in each transition; a conversion module: converting the result data searched by the query module into a format required by the write-in module according to the corresponding HBase structure; a writing module: writing the data into HBase through API according to the write-in parameters of the configuration module; a management module: and starting a migration task at regular time and supplementing unsuccessfully migrated data according to the log information recorded by the log module. Compared with the prior art, the method has the advantages of automatic migration, increment delay synchronization support, automatic supplementary transmission, high efficiency and the like.
Description
Technical Field
The invention relates to the field of power data measurement, in particular to a method and a system for migrating power measurement data to HBase.
Background
In order to improve the operation level of enterprises and further explore the potential value of data, massive power measurement data need to be migrated from a traditional relational database to a big data platform, and high-level applications such as rapid query, load prediction, anomaly analysis and the like are realized through various data mining tools or query engines based on the big data platform.
At present, the relational database can be imported into the HBase through tools such as Sqoop, but the relational database can be used with other scheduling tools (such as Oozie) to realize a timing task, and an automatic retransmission function is not provided, so that automatic retransmission of migration failure data cannot be realized.
Disclosure of Invention
The present invention is directed to a method for migrating power measurement data to HBase, which overcomes the above-mentioned drawbacks of the prior art.
The purpose of the invention can be realized by the following technical scheme:
a method for migrating power measurement class data to HBase comprises the following steps:
1) the management module starts a migration task at regular time every day, retrieves a migration log recorded by the log module, judges whether abnormal state records or log loss exist in the historical migration date, if yes, performs the step 2), and if not, performs the step 3);
2) the management module generates a missing and abnormal log information list and executes the step 4);
3) the query module reads the parameters of the configuration module, queries the data of the previous day of the data table to be migrated in batches according to the parameters, and performs the step 5);
4) the query module reads the parameters of the configuration module, queries the earliest date data in the missing and abnormal log information list corresponding to the data table to be migrated in batches according to the parameters, deletes the record from the list, and performs step 5);
5) the conversion module converts the query result of the step 3) or the step 4) into a structure required by HBase according to the result output format parameter in the configuration module, and then the step 6) is carried out;
6) the write-in module writes HBase in a multithreading and parallel mode according to the write-in mode parameters of the configuration module through the API, and step 7) is carried out;
7) the log module records or updates the log, and the step 8) is carried out;
8) the management module judges whether a log record has a log of the previous day or not, if so, the step 11) is carried out, and if not, the step 9) is carried out;
9) the query module checks and generates a missing and abnormal log information list, if no information exists in the information list, the step 10 is carried out, and if information exists, the step 4 is returned;
10) the query module reads the parameters of the configuration module to query the data in the previous day of the data table to be migrated in batches according to the parameters, and returns to the step 5);
11) the management module ends the current migration task.
The step 10) further comprises the following steps:
and checking whether log records with abnormal times more than or equal to 2 exist in the log records and finishing the execution, if so, sending out a warning log to remind the manual inspection.
In the process of transferring the power measurement data to the HBase, if the log module and the management module are abnormal, log loss is caused, if other modules are abnormal, the log module records abnormal logs, the later-day transfer task is automatically supplemented according to the log records, and when the supplement transfer is invalid, the management module generates an alarm log to remind manual intervention detection processing.
The configuration module parameters include connection information, a table and query conditions.
A system for migrating power metering class data to HBase, the system comprising:
the query module: querying data from a relational database through JDBC;
a configuration module: the device is used for configuring connection, tables, query conditions, result output formats and writing modes;
a log module: recording the state of each information table in each transition;
a conversion module: converting the result data searched by the query module into a format required by the write-in module according to the corresponding HBase structure;
a writing module: writing the data into HBase through API according to the write-in parameters of the configuration module;
a management module: and starting a migration task at regular time and supplementing unsuccessfully migrated data according to the log information recorded by the log module.
Compared with the prior art, the invention has the following advantages:
1) the method can realize the automatic migration of the power measurement data from the traditional relational database to the HBase database.
2) The invention supports the incremental delay synchronization function.
3) The invention can automatically supplement and transmit the migration failure data.
4) The invention can lead the power measurement data into the HBase with higher efficiency, solves the problem of automatic supplementary transmission of missing data, reduces the execution difficulty of transferring the power measurement data to the HBase, and improves the flexibility, convenience and stability of data transfer.
Drawings
FIG. 1 is a flow chart of the method of the present invention.
Detailed Description
The invention is described in detail below with reference to the figures and specific embodiments.
Examples
The invention can set an optimal writing mode according to the hardware information of the host and the HBase cluster information, and consists of six functional modules, namely an independently researched and developed query module, a configuration module, a log module, a conversion module, a writing module and a management module.
The query module queries data from a relational database through JDBC;
the configuration module is used for configuring connection, tables, query conditions, result output formats, writing modes and the like;
the log module records the state of each table in each migration;
the conversion module converts the result data searched by the query module into a format required by the write-in module according to the corresponding HBase structure;
the writing module writes data into the HBase through the API according to the writing parameters of the configuration module;
and the management module starts a migration task at regular time and transmits unsuccessfully migrated data according to the log information recorded by the log module.
Therefore, automatic migration of the power measurement data from the traditional relational database to the HBase database is realized, the increment delay synchronization function is supported, and automatic supplementary transmission of migration failure data can be realized.
As shown in fig. 1, the main steps of the present invention are as follows:
beginning: the migration task is started by the management module periodically every day.
Step 1: the management module retrieves the migration log recorded by the log module and judges whether abnormal state record or log missing exists in the historical migration date. If yes, executing step 2, and if no, executing step 3.
Step 2: the management module generates a missing and abnormal log information list. Step 4 is performed.
And step 3: and the query module reads the parameters of the configuration module and queries the data in the previous day of the data table to be migrated in batches according to the parameters. Step 5 is performed.
And 4, step 4: and the query module reads the parameters of the configuration module, queries the earliest date data in the list of the generation missing and abnormal log information corresponding to the data table to be migrated in batches according to the parameters, and deletes the record from the list. Step 5 is performed.
And 5: and the conversion module converts the query result in the step 3 or the step 4 into a structure required by the HBase according to the parameters of the configuration module. Step 6 is performed.
Step 6: and the writing module writes HBase in a multithreading and parallel mode through the API according to the parameters of the configuration module. Step 7 is performed.
And 7: the log module records or updates the log. Step 8 is performed.
And 8: the management module judges whether the log record has a log of the previous day, if so, the step 11 is executed, and if not, the step 9 is executed.
And step 9: the query module checks and generates a missing and abnormal log information list, if no information exists, the step 10 is executed, and if information exists, the step 4 is executed.
Step 10: and the query module reads the parameters of the configuration module and queries the data in the previous day of the data table to be migrated in batches according to the parameters. Step 5 is performed.
Step 11: and checking whether the log records with the abnormal times more than or equal to 2 exist in the log records or not, and finishing the execution. If a warning log is sent, the manual inspection is reminded.
And (4) ending: and the management module finishes the current migration task.
In the whole migration process, if the log module and the management module are abnormal, log loss can be caused; and other modules record the abnormal logs into the log module when the abnormal events occur. And the migration tasks in the next day can be automatically and additionally transferred according to the log records. When the supplementary transmission is invalid, the management module generates an alarm log to remind manual intervention detection processing, so that the automation and reliability of the migration are ensured by the mechanism.
If a small amount of data needs to be additionally transmitted in the data migration process, the Sqoop retransmits all the data, all the transmitted data become historical versions, and the storage burden of the system is greatly increased. If the Sqoop needs to achieve the supplementary transmission effect of the method, the data condition in the HBase table needs to be manually inquired, and then the SQL statement in the Sqoop is changed, when the data quantity in the HBase table is extremely large, the section information of the failure data is difficult to find, and the operation feasibility is extremely low.
In the data migration process, the data migration is forcibly ended in a direct Kill process mode. The data which is migrated before Kill by the migration method process is recorded in the log, the Kill back process is stopped, and the log is lost. And restarting the migration task, wherein the management module automatically transmits the data of the date of the missing log in a supplementing manner according to the log, only transmits the missing data after Kill execution in a supplementing manner, and covers the previous data to make the previous data become a historical version. Therefore, the accuracy and convenience of the data supplementary transmission range are improved, and the requirement degree of manual participation is greatly reduced.
The method can realize the automatic migration of the power measurement data from the traditional relational database to the HBase database, support the increment delay synchronization function, realize the automatic supplementary transmission of the migration failure data, introduce the power measurement data into the HBase with higher efficiency, solve the problem of the automatic supplementary transmission of the missing data, reduce the execution difficulty of the migration of the power measurement data to the HBase, and improve the flexibility, convenience and stability of the data migration.
Claims (1)
1. A method for migrating power measurement data to HBase is characterized by comprising the following steps:
1) the management module starts a migration task at regular time every day, retrieves a migration log recorded by the log module, judges whether abnormal state records or log loss exist in the historical migration date, if yes, performs the step 2), and if not, performs the step 3);
2) the management module generates a missing and abnormal log information list and executes the step 4);
3) the query module reads parameters of the configuration module, queries data of a data table to be migrated in batches according to the parameters, and performs step 5), wherein the parameters of the configuration module comprise connection information, the table and query conditions;
4) the query module reads the parameters of the configuration module, queries the earliest date data in the missing and abnormal log information list corresponding to the data table to be migrated in batches according to the parameters, deletes the record from the list, and performs step 5);
5) the conversion module converts the query result of the step 3) or the step 4) into a structure required by HBase according to the result output format parameter in the configuration module, and then the step 6) is carried out;
6) the write-in module writes HBase in a multithreading and parallel mode according to the write-in mode parameters of the configuration module through the API, and step 7) is carried out;
7) the log module records or updates the log, and the step 8) is carried out;
8) the management module judges whether a log record has a log of the previous day or not, if so, the step 11) is carried out, and if not, the step 9) is carried out;
9) the query module checks and generates a missing and abnormal log information list, if no information exists in the information list, the step 10 is carried out, and if information exists, the step 4 is returned;
10) the query module reads the parameters of the configuration module and queries data of the previous day of the data table to be migrated in batches according to the parameters, and returns to the step 5), whether log records with abnormal times more than or equal to 2 exist in the log records or not is checked, and the execution is finished, if so, a warning log is sent out to remind manual inspection;
11) the management module finishes the current migration task;
in the process of transferring the power measurement data to HBase, if log loss is caused by abnormality of a log module and a management module, if other modules are abnormal, the log module records abnormal logs, the transfer task in the next day is automatically supplemented according to the log records, and when the supplement transfer is invalid, the management module generates an alarm log to remind manual intervention detection processing;
the system for realizing the method for migrating the power measurement data to the HBase comprises the following steps:
the query module: querying data from a relational database through JDBC;
a configuration module: the device is used for configuring connection, tables, query conditions, result output formats and writing modes;
a log module: recording the state of each information table in each transition;
a conversion module: converting the result data searched by the query module into a format required by the write-in module according to the corresponding HBase structure;
a writing module: writing the data into HBase through API according to the write-in parameters of the configuration module;
a management module: and starting a migration task at regular time and supplementing unsuccessfully migrated data according to the log information recorded by the log module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810533864.8A CN108804606B (en) | 2018-05-29 | 2018-05-29 | Method and system for migrating power measurement data to HBase |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810533864.8A CN108804606B (en) | 2018-05-29 | 2018-05-29 | Method and system for migrating power measurement data to HBase |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108804606A CN108804606A (en) | 2018-11-13 |
| CN108804606B true CN108804606B (en) | 2021-08-31 |
Family
ID=64089200
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810533864.8A Active CN108804606B (en) | 2018-05-29 | 2018-05-29 | Method and system for migrating power measurement data to HBase |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108804606B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112783856A (en) * | 2019-11-07 | 2021-05-11 | 北京沃东天骏信息技术有限公司 | Data migration method and device |
| CN117591497B (en) * | 2024-01-18 | 2024-05-03 | 中核武汉核电运行技术股份有限公司 | Nuclear power historical data cross-system migration method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202167018U (en) * | 2011-06-08 | 2012-03-14 | 上海市电力公司 | Power supply reliability data statistical treatment device suitable for power system |
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN105243155A (en) * | 2015-10-29 | 2016-01-13 | 贵州电网有限责任公司电力调度控制中心 | Big data extracting and exchanging system |
| CN106651633A (en) * | 2016-10-09 | 2017-05-10 | 国网浙江省电力公司信息通信分公司 | Power utilization information acquisition system and method based on big data technology |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102799555B (en) * | 2012-07-24 | 2014-03-12 | 中国电力科学研究院 | Design method capable of configuring data interaction tool in electric power information system and system |
| CN103631907B (en) * | 2013-11-26 | 2016-09-07 | 中国科学院信息工程研究所 | A kind of method and system that relational data is migrated to HBase |
| US20160055233A1 (en) * | 2014-08-25 | 2016-02-25 | Ca, Inc. | Pre-join tags for entity-relationship modeling of databases |
| CN104616205B (en) * | 2014-11-24 | 2019-10-25 | 北京科东电力控制系统有限责任公司 | A kind of operation states of electric power system monitoring method based on distributed information log analysis |
| US10108623B2 (en) * | 2014-12-12 | 2018-10-23 | International Business Machines Corporation | Merging database operations for serializable transaction execution |
| CN104598540A (en) * | 2014-12-31 | 2015-05-06 | 国家电网公司 | Timing data migration device and using method thereof |
| CN104750870B (en) * | 2015-04-20 | 2018-11-27 | 河海大学 | A kind of log storage system and implementation method based on HBase |
| CN107577588B (en) * | 2017-09-26 | 2021-04-09 | 北京中安智达科技有限公司 | Intelligent operation and maintenance system for mass log data |
-
2018
- 2018-05-29 CN CN201810533864.8A patent/CN108804606B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202167018U (en) * | 2011-06-08 | 2012-03-14 | 上海市电力公司 | Power supply reliability data statistical treatment device suitable for power system |
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN105243155A (en) * | 2015-10-29 | 2016-01-13 | 贵州电网有限责任公司电力调度控制中心 | Big data extracting and exchanging system |
| CN106651633A (en) * | 2016-10-09 | 2017-05-10 | 国网浙江省电力公司信息通信分公司 | Power utilization information acquisition system and method based on big data technology |
Non-Patent Citations (2)
| Title |
|---|
| Transforming relational database into HBase: A case study;Chongxin Li;《2010 IEEE International Conference on Software Engineering and Service Sciences》;20100819;第683-687页 * |
| 云平台下电力设备监测大数据存储优化与并行处理技术研究;宋亚奇;《中国博士学位论文全文数据库 工程科技Ⅱ辑》;20170215;第C042-216页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108804606A (en) | 2018-11-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200409977A1 (en) | Generic Multi-Source Heterogeneous Large-Scale Data Synchronization Client-Server Method | |
| CN101719149B (en) | Data synchronization method and device | |
| CN100487700C (en) | Data processing method and system of data library | |
| CN102637214B (en) | Method and system for synchronizing general data among database services | |
| CN102495906A (en) | Incremental data migration method capable of realizing breakpoint transmission | |
| CN108804606B (en) | Method and system for migrating power measurement data to HBase | |
| CN109241156A (en) | The method updated based on ETL tool from relevant database to non-relational database | |
| CN104317800A (en) | Hybrid storage system and method for mass intelligent power utilization data | |
| CN104850635B (en) | Shortage of data automaticly inspects and centralized monitoring method under a kind of multi-data source environment | |
| CN103631870A (en) | System and method used for large-scale distributed data processing | |
| CN107423452A (en) | A kind of power network heterogeneous database synchronously replicates moving method | |
| CN102790786A (en) | Environment monitoring platform based on mobile internet and monitoring method | |
| CN110688399A (en) | Stream type calculation real-time report system and method | |
| CN111460019A (en) | Data conversion method and middleware of heterogeneous data source | |
| CN114218218A (en) | Data processing method, device and equipment based on data warehouse and storage medium | |
| CN112035563A (en) | Real-time database system based on shared storage | |
| CN115309761A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN110515955B (en) | Data storage and query method and system, electronic equipment and storage medium | |
| CN109800425B (en) | Semantic miss automatic error correction processing method and system for cross-platform information exchange | |
| CN118170736A (en) | Data synchronization method and system for power system | |
| CN110738586A (en) | weather integrated service system based on CIMISS and comprehensive database data | |
| CN116361389B (en) | Data synchronization link method and system based on national network marketing acquisition system | |
| CN111753010B (en) | Data acquisition network architecture of railway contact network and implementation method | |
| CN116302178B (en) | Acceleration processing method and device for column storage data | |
| Lin et al. | Research on Heterogeneous Data Transfer Method for Power Energy Management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |