CN108804606A - A kind of electric power measures class Data Migration to the method and system of HBase - Google Patents
A kind of electric power measures class Data Migration to the method and system of HBase Download PDFInfo
- Publication number
- CN108804606A CN108804606A CN201810533864.8A CN201810533864A CN108804606A CN 108804606 A CN108804606 A CN 108804606A CN 201810533864 A CN201810533864 A CN 201810533864A CN 108804606 A CN108804606 A CN 108804606A
- Authority
- CN
- China
- Prior art keywords
- module
- data
- hbase
- record
- parameter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000005012 migration Effects 0.000 title claims abstract description 35
- 238000013508 migration Methods 0.000 title claims abstract description 35
- 238000000034 method Methods 0.000 title claims abstract description 25
- 230000008901 benefit Effects 0.000 claims abstract description 10
- 238000006243 chemical reaction Methods 0.000 claims abstract description 8
- 238000007726 management method Methods 0.000 claims description 24
- 230000002159 abnormal effect Effects 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 9
- 238000007689 inspection Methods 0.000 claims description 6
- 210000004209 hair Anatomy 0.000 claims description 4
- 238000012550 audit Methods 0.000 claims description 3
- 238000001514 detection method Methods 0.000 claims description 3
- 230000005856 abnormality Effects 0.000 claims description 2
- 230000005611 electricity Effects 0.000 claims 1
- 238000005259 measurement Methods 0.000 abstract description 3
- 230000002567 autonomic effect Effects 0.000 abstract description 2
- 230000001360 synchronised effect Effects 0.000 abstract 1
- 230000008676 import Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Business, Economics & Management (AREA)
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Economics (AREA)
- Public Health (AREA)
- Water Supply & Treatment (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to a kind of method and system of electric power measurement class Data Migration to HBase, which includes：Enquiry module：By JDBC from relational data library inquiry data；Configuration module：For configuring connection, table, querying condition, result output format, writing mode etc.；Journal module：Record the state that every table migrates every time；Conversion module：The result data that enquiry module is found is converted to the format of writing module needs according to corresponding HBase structures；Writing module：It is write data into HBase by API according to configuration module write parameters；Management module：Start by set date migrates task and mends the data for passing and migrating not successfully according to the log information of journal module record.Compared with prior art, the present invention has many advantages, such as Autonomic Migration Framework, supports that incremental time delay is synchronous, mends biography automatically, is efficient.
Description
Technical field
The present invention relates to electric power data fields of measurement, and class Data Migration is measured to HBase's more particularly, to a kind of electric power
Method and system.
Background technology
It is horizontal to improve enterprise operation, further excavate data potential value, need by the electric power metric data of magnanimity from
Traditional Relational DataBase moves to big data platform, is drawn by the various Data Mining Tools based on big data platform or inquiry
It holds up, realizes the advanced applications such as quick search, load prediction, anomaly analysis.
Currently, can by the tools such as Sqoop realize by relevant database import HBase in, but its need with it is other
Scheduling tool (such as Oozie) cannot achieve migration and lose with the use of timed task, and no biography function of benefit automatically just can be achieved
The automatic benefit for losing data passes.
Invention content
It is an object of the present invention to overcome the above-mentioned drawbacks of the prior art and provide a kind of electric power to measure class number
According to the method for moving to HBase.
The purpose of the present invention can be achieved through the following technical solutions：
A kind of electric power measures class Data Migration to the method for HBase, includes the following steps：
1) the daily start by set date of management module migrates task, and retrieves the migration daily record of journal module record, judges history
It migrates with the presence or absence of abnormality record or daily record missing in the date, if so, step 2) is carried out, if it is not, then carrying out step 3)；
2) management module generates missing, abnormal log information list, and executes step 4)；
3) enquiry module reads configuration module parameter, and inquires the number for needing migrating data table proxima luce (prox. luc) in batches by parameter
According to progress step 5)；
4) enquiry module reads configuration module parameter, and being inquired in batches by parameter needs corresponding generate of migrating data table to lack
The earliest date data become estranged in abnormal log information list, and it is removed from the list this record, carry out step 5)；
5) conversion module is turned the query result of step 3) or step 4) by the result output format parameter in configuration module
It is changed to the structure of HBase needs, carries out step 6)；
6) by API HBase is written according to configuration module writing mode parameter multi-threaded parallel, and carry out in writing module
Step 7)；
7) journal module record or Update log, carry out step 8)；
8) management module judges whether have proxima luce (prox. luc) daily record in log recording, if so, step 11) is carried out, if it is not, then
Carry out step 9)；
9) enquiry module inspection generates missing and abnormal log information list, if being carried out without information in information list
Step 10, if there is information, return to step 4)；
10) enquiry module reads configuration module parameter and inquires the tables of data proxima luce (prox. luc) data that need to be migrated in batches by parameter, and
Return to step 5)；
11) management module terminates current migration task.
It is further comprising the steps of in the step 10)：
Log recording and execution in audit log record with the presence or absence of frequency of abnormity more than or equal to 2 terminate, if so, hair
Go out and alert daily record, reminds hand inspection.
During electric power measures class Data Migration to HBase, if journal module, management module are abnormal, cause
Daily record lacks, if other modules are abnormal, journal module recording exceptional daily record migrates task latter day according to log recording
Automatic mend passes, and when generation, which is mended, passes invalid situation, management module generates alarm log, reminds manual intervention detection process.
The configuration module parameter includes link information, table and querying condition.
A kind of electric power measures class Data Migration to the system of HBase, which includes：
Enquiry module：By JDBC from relational data library inquiry data；
Configuration module：For configuring connection, table, querying condition, result output format and writing mode；
Journal module：Record the state that every information table migrates every time；
Conversion module：The result data that enquiry module is found is converted into writing module needs according to corresponding HBase structures
Format；
Writing module：It is write data into HBase by API according to configuration module write parameters；
Management module：Start by set date migrates task and mends what biography migrated not successfully according to the log information of journal module record
Data.
Compared with prior art, the present invention has the following advantages：
1) this method may be implemented electric power and measure class data from traditional Relational DataBase to HBase databases from moving
It moves.
2) present invention supports incremental time delay synchronizing function.
3) present invention can accomplish that the automatic benefit for migrating miss data passes.
4) electric power can be measured class data with greater efficiency and import HBase by the present invention, and it is automatic to solve missing data
Mend pass the problem of, reduce electric power measure class Data Migration to HBase difficulty of implementation, improve Data Migration flexibility,
Convenience and stability.
Description of the drawings
Fig. 1 is flow chart of the method for the present invention.
Specific implementation mode
The present invention is described in detail with specific embodiment below in conjunction with the accompanying drawings.
Embodiment
The present invention can set best writing mode according to host hardware information, HBase cluster informations, by independent research
Six enquiry module, configuration module, journal module, conversion module, writing module, management module function module compositions.
Enquiry module is by JDBC from relational data library inquiry data；
Configuration module is for configuring connection, table, querying condition, result output format, writing mode etc.；
Journal module records the state that every table migrates every time；
The result data that enquiry module is found is converted to writing module needs by conversion module according to corresponding HBase structures
Format；
Writing module is write data by API in HBase according to configuration module write parameters；
Management module start by set date migrates task and mends what biography migrated not successfully according to the log information of journal module record
Data.
To realize that electric power measures class data from traditional Relational DataBase to the Autonomic Migration Framework of HBase databases, support
Incremental time delay synchronizing function can accomplish that the automatic benefit for migrating miss data passes.
As shown in Figure 1, the key step flow of the present invention is：
Start：Task is migrated by management module start by set date daily.
Step 1：Management module retrieves the migration daily record of journal module record, judges in the history migration date with the presence or absence of different
Normal state recording has daily record missing.Step 2 is executed if having, if without step 3 is executed.
Step 2：Management module generates missing, abnormal log information list.Execute step 4.
Step 3：Enquiry module reads configuration module parameter, inquires the previous number of days of tables of data that need to be migrated in batches by parameter
According to.Execute step 5.
Step 4：Enquiry module reads configuration module parameter, inquires the corresponding generation of tables of data that need to be migrated in batches by parameter
The earliest date data in missing, abnormal log information list, and it is removed from the list this record.Execute step 5.
Step 5：Step 3 or step 4 query result are converted to the knot of HBase needs by configuration module parameter by conversion module
Structure.Execute step 6.
Step 6：HBase is written by API, according to configuration module parameter multi-threaded parallel in writing module.Execute step 7.
Step 7：Journal module records or Update log.Execute step 8.
Step 8：Management module judges whether have proxima luce (prox. luc) daily record in log recording, and step 11 is executed if having, if without holding
Row step 9.
Step 9：Enquiry module inspection generates missing, abnormal log information list, if step 10 is executed without information, if having
Information executes step 4.
Step 10：Enquiry module reads configuration module parameter and inquires the previous number of days of tables of data that need to be migrated in batches by parameter
According to.Execute step 5.
Step 11：Log recording and execution in audit log record with the presence or absence of frequency of abnormity more than or equal to 2 terminate.If
Daily record is given a warning, hand inspection is reminded.
Terminate：Management module terminates when time migration task.
In entire transition process, if journal module, management module are abnormal, daily record can be caused to lack；Other module hairs
In raw abnormal meeting recording exceptional daily record to journal module.Latter day migration task can mend biography automatically according to log recording.Work as hair
When raw benefit passes invalid situation, management module generates alarm log, reminds manual intervention detection process, ensures migration with this mechanism
Automation and reliability.
If need to mend biography low volume data in data migration process, Sqoop can retransmit total data, all to have passed data
Become old version, system storage burden has been significantly greatly increased.If the benefit that Sqoop will reach this method passes effect, need artificially to look into
Table data cases in HBase are ask, then change the SQL statement in Sqoop, when data volume is very big in HBase table, it is difficult to find mistake
The section information of data is lost, operation possibility is extremely low.
The present invention forces to terminate Data Migration by direct Kill processes mode in data migration process.Moving method
Process is completed its date of the data of migration before by Kill and has been recorded in daily record, and process stops after Kill, thereafter day
Will lacks.Restart migration task, the data that management module will lack the date of daily record according to daily record automatically mend biography, only mend and pass Kill
Missing data after execution covers previous data and becomes old version.To improve the accuracy that data mend biography range
With convenience, the desirability artificially participated in is greatly reduced.
This method may be implemented electric power and measure class data from traditional Relational DataBase to HBase databases from moving
It moves, supports incremental time delay synchronizing function, can accomplish the automatic benefit biography for migrating miss data, it can be with greater efficiency by amount of power
It surveys class data and imports HBase, and solve the problems, such as that missing data mends biography automatically, reduce electric power measurement class Data Migration and arrive
The difficulty of implementation of HBase improves flexibility, convenience and the stability of Data Migration.
Claims (5)
1. a kind of electric power measures class Data Migration to the method for HBase, which is characterized in that include the following steps：
1) the daily start by set date of management module migrates task, and retrieves the migration daily record of journal module record, judges that history migrates
With the presence or absence of abnormality record or daily record missing in date, if so, step 2) is carried out, if it is not, then carrying out step 3)；
2) management module generates missing, abnormal log information list, and executes step 4)；
3) enquiry module reads configuration module parameter, and inquires the data for needing migrating data table proxima luce (prox. luc) in batches by parameter, into
Row step 5)；
4) enquiry module reads configuration module parameter, and inquire in batches by parameter need corresponding generations of migrating data table lack with
The earliest date data in abnormal log information list, and it is removed from the list this record, carry out step 5)；
5) conversion module is converted to the query result of step 3) or step 4) by the result output format parameter in configuration module
The structure that HBase needs carries out step 6)；
6) by API HBase is written according to configuration module writing mode parameter multi-threaded parallel, and carry out step in writing module
7)；
7) journal module record or Update log, carry out step 8)；
8) management module judges whether have proxima luce (prox. luc) daily record in log recording, if so, step 11) is carried out, if it is not, then carrying out
Step 9)；
9) enquiry module inspection generates missing and abnormal log information list, if carrying out step without information in information list
10, if there is information, return to step 4)；
10) enquiry module reads configuration module parameter and inquires the tables of data proxima luce (prox. luc) data that need to be migrated in batches by parameter, and returns
Step 5)；
11) management module terminates current migration task.
2. a kind of electric power according to claim 1 measures class Data Migration to the method for HBase, which is characterized in that described
Step 10) in, it is further comprising the steps of：
Log recording and execution in audit log record with the presence or absence of frequency of abnormity more than or equal to 2 terminate, if so, sending out police
Daily record is accused, hand inspection is reminded.
3. a kind of electric power according to claim 1 measures class Data Migration to the method for HBase, which is characterized in that in electricity
During strength surveys class Data Migration to HBase, if journal module, management module are abnormal, daily record is caused to lack, if its
Its module is abnormal, then the daily record of journal module recording exceptional, and latter day migration task mends biography according to log recording automatically, works as hair
When raw benefit passes invalid situation, management module generates alarm log, reminds manual intervention detection process.
4. a kind of electric power according to claim 1 measures class Data Migration to the method for HBase, which is characterized in that described
Configuration module parameter include link information, table and querying condition.
5. a kind of system realized electric power according to any one of claims 1-4 and measure method of the class Data Migration to HBase,
It is characterized in that, the system includes：
Enquiry module：By JDBC from relational data library inquiry data；
Configuration module：For configuring connection, table, querying condition, result output format and writing mode；
Journal module：Record the state that every information table migrates every time；
Conversion module：The result data that enquiry module is found is converted to the lattice of writing module needs according to corresponding HBase structures
Formula；
Writing module：It is write data into HBase by API according to configuration module write parameters；
Management module：Start by set date migrates task and mends the number for passing and migrating not successfully according to the log information of journal module record
According to.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810533864.8A CN108804606B (en) | 2018-05-29 | 2018-05-29 | Method and system for migrating power measurement data to HBase |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810533864.8A CN108804606B (en) | 2018-05-29 | 2018-05-29 | Method and system for migrating power measurement data to HBase |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108804606A true CN108804606A (en) | 2018-11-13 |
| CN108804606B CN108804606B (en) | 2021-08-31 |
Family
ID=64089200
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810533864.8A Active CN108804606B (en) | 2018-05-29 | 2018-05-29 | Method and system for migrating power measurement data to HBase |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108804606B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112783856A (en) * | 2019-11-07 | 2021-05-11 | 北京沃东天骏信息技术有限公司 | Data migration method and device |
| CN117591497A (en) * | 2024-01-18 | 2024-02-23 | 中核武汉核电运行技术股份有限公司 | Nuclear power historical data cross-system migration method |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202167018U (en) * | 2011-06-08 | 2012-03-14 | 上海市电力公司 | Power supply reliability data statistical treatment device suitable for power system |
| CN102799555A (en) * | 2012-07-24 | 2012-11-28 | 中国电力科学研究院 | Design method capable of configuring data interaction tool in electric power information system and system |
| CN103631907A (en) * | 2013-11-26 | 2014-03-12 | 中国科学院信息工程研究所 | Method and system for migrating relational data to HBbase |
| CN104598540A (en) * | 2014-12-31 | 2015-05-06 | 国家电网公司 | Timing data migration device and using method thereof |
| CN104616205A (en) * | 2014-11-24 | 2015-05-13 | 北京科东电力控制系统有限责任公司 | Distributed log analysis based operation state monitoring method of power system |
| CN104750870A (en) * | 2015-04-20 | 2015-07-01 | 河海大学 | Log storage system based on HBase and implementation method |
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN105243155A (en) * | 2015-10-29 | 2016-01-13 | 贵州电网有限责任公司电力调度控制中心 | Big data extracting and exchanging system |
| US20160055233A1 (en) * | 2014-08-25 | 2016-02-25 | Ca, Inc. | Pre-join tags for entity-relationship modeling of databases |
| US20160171002A1 (en) * | 2014-12-12 | 2016-06-16 | International Business Machines Corporation | Merging database operations for serializable transaction execution |
| CN106651633A (en) * | 2016-10-09 | 2017-05-10 | 国网浙江省电力公司信息通信分公司 | Power utilization information acquisition system and method based on big data technology |
| CN107577588A (en) * | 2017-09-26 | 2018-01-12 | 北京中安智达科技有限公司 | A kind of massive logs data intelligence operational system |
-
2018
- 2018-05-29 CN CN201810533864.8A patent/CN108804606B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202167018U (en) * | 2011-06-08 | 2012-03-14 | 上海市电力公司 | Power supply reliability data statistical treatment device suitable for power system |
| CN102799555A (en) * | 2012-07-24 | 2012-11-28 | 中国电力科学研究院 | Design method capable of configuring data interaction tool in electric power information system and system |
| CN103631907A (en) * | 2013-11-26 | 2014-03-12 | 中国科学院信息工程研究所 | Method and system for migrating relational data to HBbase |
| US20160055233A1 (en) * | 2014-08-25 | 2016-02-25 | Ca, Inc. | Pre-join tags for entity-relationship modeling of databases |
| CN104616205A (en) * | 2014-11-24 | 2015-05-13 | 北京科东电力控制系统有限责任公司 | Distributed log analysis based operation state monitoring method of power system |
| US20160171002A1 (en) * | 2014-12-12 | 2016-06-16 | International Business Machines Corporation | Merging database operations for serializable transaction execution |
| CN104598540A (en) * | 2014-12-31 | 2015-05-06 | 国家电网公司 | Timing data migration device and using method thereof |
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN104750870A (en) * | 2015-04-20 | 2015-07-01 | 河海大学 | Log storage system based on HBase and implementation method |
| CN105243155A (en) * | 2015-10-29 | 2016-01-13 | 贵州电网有限责任公司电力调度控制中心 | Big data extracting and exchanging system |
| CN106651633A (en) * | 2016-10-09 | 2017-05-10 | 国网浙江省电力公司信息通信分公司 | Power utilization information acquisition system and method based on big data technology |
| CN107577588A (en) * | 2017-09-26 | 2018-01-12 | 北京中安智达科技有限公司 | A kind of massive logs data intelligence operational system |
Non-Patent Citations (2)
| Title |
|---|
| CHONGXIN LI: "Transforming relational database into HBase: A case study", 《2010 IEEE INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEERING AND SERVICE SCIENCES》 * |
| 宋亚奇: "云平台下电力设备监测大数据存储优化与并行处理技术研究", 《中国博士学位论文全文数据库 工程科技Ⅱ辑》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112783856A (en) * | 2019-11-07 | 2021-05-11 | 北京沃东天骏信息技术有限公司 | Data migration method and device |
| CN117591497A (en) * | 2024-01-18 | 2024-02-23 | 中核武汉核电运行技术股份有限公司 | Nuclear power historical data cross-system migration method |
| CN117591497B (en) * | 2024-01-18 | 2024-05-03 | 中核武汉核电运行技术股份有限公司 | Nuclear power historical data cross-system migration method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108804606B (en) | 2021-08-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11409764B2 (en) | System for data management in a large scale data repository | |
| EP3513314B1 (en) | System for analysing data relationships to support query execution | |
| EP2929467B1 (en) | Integrating event processing with map-reduce | |
| EP3513313A1 (en) | System for importing data into a data repository | |
| CN105183860B (en) | Method of data synchronization and system | |
| US20130066882A1 (en) | Data storage method and system | |
| CN104008201A (en) | Numerical control machine tool production information obtaining method | |
| CN104461671A (en) | Method and system for periodically managing code modification report | |
| CN104850635B (en) | Shortage of data automaticly inspects and centralized monitoring method under a kind of multi-data source environment | |
| CN104866619A (en) | Data monitoring method and system for data warehouse | |
| CN112347071B (en) | Power distribution network cloud platform data fusion method and power distribution network cloud platform | |
| US20200250188A1 (en) | Systems, methods and data structures for efficient indexing and retrieval of temporal data, including temporal data representing a computing infrastructure | |
| CN108804606A (en) | A kind of electric power measures class Data Migration to the method and system of HBase | |
| CN110442651A (en) | A method of it is uploaded automatically based on kettle realization excel data and triggers scheduling | |
| CN112817958A (en) | Electric power planning data acquisition method and device and intelligent terminal | |
| CN106250429A (en) | A kind of data pick-up method based on sqoop | |
| CN117290180A (en) | Monitoring method, equipment and medium based on time sequence data analysis equipment operation state | |
| EP2568399A2 (en) | Data storage method and system | |
| CN107633056A (en) | Data managing method for power information acquisition terminal | |
| CN112448840B (en) | Communication data quality monitoring method, device, server and storage medium | |
| CN112559641A (en) | Processing method and device of pull chain table, readable storage medium and electronic equipment | |
| CN111198893A (en) | Data updating method and device, readable medium and electronic equipment | |
| CN112148459B (en) | Processing method, device, readable medium and equipment for node association data | |
| CN113467407B (en) | Fault information collection method, system and equipment for distributed control system | |
| CN113553320B (en) | Data quality monitoring method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |