CN104318481A - Power-grid-operation-oriented holographic time scale measurement data extraction conversion method - Google Patents
Power-grid-operation-oriented holographic time scale measurement data extraction conversion method Download PDFInfo
- Publication number
- CN104318481A CN104318481A CN201410181171.9A CN201410181171A CN104318481A CN 104318481 A CN104318481 A CN 104318481A CN 201410181171 A CN201410181171 A CN 201410181171A CN 104318481 A CN104318481 A CN 104318481A
- Authority
- CN
- China
- Prior art keywords
- data
- extraction
- time scale
- scale measurement
- measurement data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000005259 measurement Methods 0.000 title claims abstract description 51
- 238000006243 chemical reaction Methods 0.000 title claims abstract description 46
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000013075 data extraction Methods 0.000 title claims abstract description 25
- 238000000605 extraction Methods 0.000 claims abstract description 47
- 238000011068 loading method Methods 0.000 claims abstract description 13
- 238000012545 processing Methods 0.000 claims abstract description 10
- 239000000284 extract Substances 0.000 claims description 28
- 230000008569 process Effects 0.000 claims description 25
- 230000008859 change Effects 0.000 claims description 12
- 230000005540 biological transmission Effects 0.000 claims description 8
- 238000013499 data model Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 4
- 238000007596 consolidation process Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 2
- 230000002123 temporal effect Effects 0.000 claims description 2
- 230000006870 function Effects 0.000 abstract description 6
- 238000004140 cleaning Methods 0.000 abstract description 5
- 206010063385 Intellectualisation Diseases 0.000 abstract 1
- 238000004458 analytical method Methods 0.000 description 9
- 238000007726 management method Methods 0.000 description 7
- 230000008676 import Effects 0.000 description 4
- 238000013500 data storage Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000009412 basement excavation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000006116 polymerization reaction Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Water Supply & Treatment (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a power-grid-operation-oriented holographic time scale measurement data extraction conversion method, and belongs to the technical field of a database. According to the method, firstly, the holographic time scale measurement data extraction is carried out; then, the holographic time scale measurement data conversion is carried out; finally, the holographic time scale measurement data loading is carried out; and the holographic time scale measurement data is integrated with power grid models stored in a relational database and various other kinds of service data. The method has the advantages that data extraction conversion can be carried out in the field of holographic time scale measurement data processing; the effective extraction, cleaning and conversion are realized; the cost and the complexity degree of subsequent application development are greatly reduced; the total size of the data can be reduced; the quality of the data can be improved; function modules of the conventional system are effectively extracted and utilized; efficient up-application is realized on the premise of not influencing the functions, safety and stability of the conventional system; the value of the conventional system is sufficiently found; automation and intellectualization of the power grid operation processing are realized at a higher level; and the safe and stable operation of a power grid is guaranteed.
Description
Technical field
The invention belongs to database technical field, the present invention relates to a kind of holographic time scale measurement data extraction conversion method towards operation of power networks more precisely.
Background technology
Along with the continuous expansion of power grid construction scale, deepening continuously of the research application such as intelligent grid, intelligent substation, the collection point faced by dispatch automated system gets more and more.With regional dispatch automation system in the past significantly unlike, the data acquisition scale that large-scale area power dispatching automation system faces sharply rises, and indivedual area will reach the scale of collection point up to a million, and data storage size turns to TB level by from current GB level.In addition, along with improving constantly of dispatching automation level, each operation system is had higher requirement to dispatch automated system, low frequency had been difficult to according to the history data store of minute level the requirement meeting electrical network fine-grained management in the past, and the holographic all details of operation of power networks of preserving have become trend of the times, original periodicity storage of history data P mode can not meet the demands, should store according to the real time Sequentially continuous of change, to meet more application demand, this also will cause the growth of data storage size decades of times.Meanwhile, the storage organization strategy of historical data and query and search strategy also will become quite complicated.Traditional relational database all will be difficult to the demand meeting application on response speed, storage size, search efficiency and change memory mechanism.In recent years, dynamic information database (also known as the time series databases) Integrated predict model in electrical network make sampling with high precision and in holographic recording operation of power networks process with time target data become possibility.
But holographic time scale measurement data exist that total size is huge, quality is uneven, data itself lack problems such as the descriptions of model.Therefore, need a kind of data extracting process and instrument badly, can the holographic time scale measurement data of magnanimity be extracted and be cleaned, reduce DATA POPULATION scale, promote the quality of data, simultaneously and the electric network model be stored in relevant database and other kinds business datum carry out integrated, forming surface is to the Data Mart of types of applications, and for follow-uply comprising data mining, aid decision making, multi-service multiple senior application that is integrated, visual presentation lay the foundation.
Summary of the invention
The object of the invention is: make up the deficiencies in the prior art, a kind of holographic time scale measurement data extraction conversion method towards operation of power networks is provided, ensure that data correctly can carry out extracting, change and being loaded in dynamic information database according to the mode of expection, carry out integrated with the electric network model be stored in relevant database and other kinds business datum simultaneously.
Specifically, the present invention adopts following technical scheme to realize, and comprises the following steps:
1) extraction of holographic time scale measurement data: read the metadata in source data, access data and extracted data from multi-data source;
2) conversion of holographic time scale measurement data: according to constraint database and service logic, changing the data harmonization extracted in step 1) by data cleansing, data is consolidation form, with the needs of the data model and the quality of data that meet dynamic information database;
3) loading of holographic time scale measurement data: the holographic time scale measurement data converted are loaded in dynamic information database according to the data structure that target data model defines, to the different loading cycle of data acquisition of different business systems, adopt multi-threading parallel process mode to load data to improve program operational efficiency simultaneously;
4) by the holographic time scale measurement data be loaded in dynamic information database be stored in the integrated of electric network model in relevant database and other kinds business datum, form data warehouse, application server provides data supporting by the data in acquisition data warehouse for upper layer application.
Technique scheme is further characterized in that, described step 1) specifically comprises following process:
1-1) read the holographic time scale measurement data meta-model in each data source resource layer;
1-2) extract meta-model by far-end and extraction process is carried out to data, described far-end extracts meta-model and forms by extracting core, log pattern and extraction configure metadata, extract core and read the optimum configurations extracting configure metadata, target data is extracted according in relative set distally data source table, extraction mode is divided into full dose to extract and increment extraction two kinds, extract configure metadata to be made up of the table name needing to extract, field, type, database linkage information, extracted data temporal information, daily record model is used for noting down the daily record extracting process;
Described full dose extracts and refers to that the total data of the specific data field of data source all extracts by the rule according to the field extracted and extraction; Only extract in follow-up extraction process after the full dose that refers to described increment extraction has extracted and extracted data that are newly-increased in the table of source or that be modified from last time;
1-3) by transmission unit model, the data of extraction are distally transferred to center-side;
1-4) will the data buffer storage of center-side be transferred to isomery scratchpad area (SPA).
Technique scheme is further characterized in that, described increment extraction realizes based on the timestamp of dispatching services system, decides to extract which data by the value comparing the timestamp field of specifying extraction time and extraction source to show.
Technique scheme is further characterized in that, described step 2) in data conversion comprise inconsistent data conversion, with reference to conversion, string processing, directly mapping, date conversion, date computing, null value judgement, aggregate operation and/or set value.
Technique scheme is further characterized in that, in described step 3), when the loading of holographic time scale measurement data, the new real time data change produced is caught in real time, and carry out normal device image data, the location of out-of-limit data and seizure according to metric data Quality Codes, the data variation that the artificial set of filtering produces.
Technique scheme is further characterized in that, the real time data in described data warehouse and historical data are separated and stored, and carry out unified Modeling to real time data and historical data, externally provide unified access view.
Beneficial effect of the present invention is as follows: the present invention can carry out data extraction conversion in holographic time scale measurement data processing field, reaches effective extraction of 99%, cleaning and conversion, greatly reduces the cost carried out of subsequent applications and complexity.Simultaneously, the holographic time scale measurement data that the present invention utilizes dynamic information database to store, DATA POPULATION scale can be reduced, promote the quality of data, to seek to become more meticulous in dispatch automated system the breakthrough point of statistical study application, effective extraction, utilize the functional module of existing system, efficient upper layer application is realized under the prerequisite not affecting existing system function and security and stability, the value of abundant excavation existing system, higher level realizes the robotization of operation of power networks process, intellectuality, ensures the safe and stable operation of electrical network.
Accompanying drawing explanation
Fig. 1 is holographic time scale measurement data extraction conversion general frame schematic diagram of the present invention.
Fig. 2 is that meta-model schematic diagram is extracted in holographic time scale measurement data extraction of the present invention.
Fig. 3 is data warehouse architecture of the present invention.
Embodiment
With reference to the accompanying drawings and in conjunction with example, the present invention is described in further detail.
As shown in Figure 1, holographic time scale measurement data extraction conversion general frame of the present invention is divided into three levels, is respectively resource layer, analysis layer, administration and supervision authorities from lower to upper.Every one deck forms by some pieces, and each block represents a meta-model.
Wherein, resource layer describes the model of the metadata of various different types of data resource, and metadata refers to the abstracted information to data, mainly refers to holographic time scale measurement data here.The further describing and various rule of model in analysis layer definition resource layer is the part of most critical in data extraction frame model, includes extractions, conversion and loading meta-model.Extract and load meta-model and include metadata in data source, extraction and loading rule definition, conversion meta-model mainly completes data integration and gathering work.Data integration refers to and is reconstructed integrated to multiple data source, and cleaning and conversion, be converted to the definition meeting target data source model.Comprising the Heterogeneity solving data source, be divided into four levels: system, grammer, structure and semanteme.As data will carry out the conversion of time format data from data source to data warehouse, in the record in data source, the semanteme of attribute is normally different, as needed to solve null value, repetition values, different measurement unit problems.Assemble and refer to data are gathered and comprehensively, namely strengthen data granularity.The metadata schema of the scheduling of administration and supervision authorities defined analysis layer metadata and execution aspect, comprises the models such as job scheduling management, operation monitoring, recovery management, exception management, log management.
The process that holographic time scale measurement data are finally loaded into dynamic information database from heterogeneous data source by data extraction process is as follows:
(1) data first in heterogeneous data source are by holographic this data-interface of time scale measurement data meta-model in resource layer, by the extraction meta-model in analysis layer, these data are extracted, this meta-model defines decimation rule, and namely which data will extract and how to extract.
(2) for the conversion process of the data extracted through conversion meta-model, wherein conversion meta-model defines the metadata about transformation in source data and target data storage.These metadata about transformation contain transformation rule metadata, are determined by the business rule in scheduling field and the data memory format of dynamic information database.
(3) data after conversion meta-model, carry out Data import work for the treatment of by the loading meta-model saving loading rule.
(4) be loaded in dynamic information database finally by the metadata definition in resource layer.
Below said process is specifically described:
1, the extraction of holographic time scale measurement data: read the metadata in source data, access data and extracted data from multi-data source.
As shown in Figure 2, holographic time scale measurement data extraction of the present invention is extracted meta-model and is comprised 4 parts, be respectively far-end and extract-transmission-buffering-merging, namely the extraction technique using far-end to extract-transmit-cushion-merge solves the problem that multiple strange lands data source carries out data pick-up, far-end extraction is carried out respectively in each front end, then file transfer success extracted is to the data buffer of center-side, again these data are merged, efficiently solve consistance and the integrity issue of the data source extracted data different in strange land like this.Because first check extracted file status of processes before being transmitted, this guarantees the correct of data extraction process, check transmission running status simultaneously, if unsuccessful transmission unit model is by autonomous retransmission, and note down running log.
The detailed step of holographic time scale measurement data pick-up is as follows:
(1) the holographic time scale measurement data meta-model in each data source resource layer is read;
(2) extract meta-model through far-end and carry out extraction process to data, far-end extracts meta-model and forms by extracting core, log pattern and extraction configure metadata.Extract core and read the optimum configurations extracting configure metadata, extract target data according in relative set distally data source table, the extraction mode extracting core is divided into full dose and increment extraction.When integrated end carries out the initialization of data, be that full dose extracts for the first time, defined by business personnel and extract strategy, after selecting the rule of field and the extraction extracted, designer's executive routine, all extracts the total data of the specific data field of data source, puts into data scratchpad area (SPA).Can select the field needed, and define new field name for the field name of source database, data value is constant, also can show the old field of data through mathematical operation by source, and the data value that must make new advances is loaded in target database.After full dose has extracted, follow-up extraction process only needs extraction to extract data that are newly-increased in the table of source or that be modified from last time, i.e. increment extraction.Routine matter is all increment extraction.Realize increment extraction, need the change of table data in source in acquisition database exactly, the present invention adopts timestamp mode, and the timestamp based on dispatching services system realizes.Which decide to extract data by the value comparing the timestamp field of specifying extraction time and extraction source to show, namely each extract before first judge the up-to-date timestamp that records in dynamic information database, then remove operation system to get to be greater than all records of this timestamp according to this timestamp.This mode needs to increase a timestamp field on the table of source, in system when renewal or amendment source table data, and the value of modification time stamp field simultaneously.The timestamp of data inserting is specified by system time.Some database timestamp support upgrades automatically, and when the data of other field namely shown change, the value of timestamp field can be automatically updated into the moment of record change.In this case, only timestamp field need be added at source table when carrying out data extraction.Automatically the database upgraded for not supporting timestamp, then need operation system when upgrading business datum, stabs field manual update time by the mode of programming.Extract core and depend on extraction configure metadata, extract configure metadata and form by needing the information such as table name, field, type, database linkage information, extracted data time extracted.Daily record model is used for noting down the daily record extracting process;
(3) by transmission unit model, the data of extraction are distally transferred to center-side;
(4) data buffer storage of center-side will be transferred to isomery scratchpad area (SPA), this is because normally there is multiple data source to need to extract, the process extracted is normally asynchronous, therefore need next this process synchronous of a data buffering, simultaneously also in order to date restoring, the data of all front ends will be caused again to extract because of the failure of one end data pick-up like this.Finally the data extracted in these each data sources are merged, form a unified extracted file.
2, the conversion of holographic time scale measurement data: according to constraint database and service logic, be consolidation form by data cleansing, data conversion (merge, change and polymerization etc.) by data harmonization, with the needs of the data model and the quality of data that meet dynamic information database.
Due to data from many different systems, the situation that data redundancy even conflicts therefore may be there is.In fact the task of data cleansing is exactly filter undesirable data, gives competent business department by the result of filtration, extracts after being confirmed filter out or revise by service unit again.So both can improve the quality of the data be drawn into a certain extent, also obviously can reduce the burden of follow-up data extraction step, greatly enhance the efficiency of data extraction.Undesirable data mainly contain: data formatting error, as missing data, data value go beyond the scope or data layout illegal etc.; Data are imperfect, mainly refer to the disappearance that should have information; Data are inconsistent or have repeating data.
Owing to often there is inconsistent problem between data source, therefore data conversion must accomplish the unification of data name and form, non-existent data may need create new mathematical logic view and change accordingly in source database simultaneously, need to be handled as follows:
(1) inconsistent data conversion: this process is a process integrated, the data of the identical type of different business systems are unified, such as same producer is A001 and encodes in another system to be B001 at the coding of a system, needs unification to convert a coding to after extracting;
(2) with reference to conversion: usually use one or more fields of data source as Key in the transfer, go to search for particular value in an Associate array, and should unique value be obtained.This Associate array uses Hash algorithm realization, and before whole data extraction process starts, it is with regard to graftabl, and the help improved performance is very large;
(3) string processing: often can obtain customizing messages from certain String field of data source, have type conversion, character string intercepting etc. to the operation of character string, add abnormality processing simultaneously;
(4) directly map: data source field is identical with aiming field length or precision, without the need to doing any process;
(5) date conversion: because the date type form in dynamic information database is unified, adopts " YYYY-MM-DD hh:mm:ss " to represent the date.And in different data sources, different date formats can be adopted, so corresponding conversion is needed to the date format of data source field;
(6) date computing: based on the date, can calculate diurnal inequality, monthly error, duration etc. usually.The date operating function that general database provides all based on date type, and needs oneself date operating function collection a set of in dynamic information database
(7) null value judges: for the NULL value in data source field, can go wrong, therefore must judge null value, and convert specific value to when dynamic information database carries out analyzing and processing;
(8) aggregate operation: operation system generally stores very detailed data, and Data Warehouse is used to analysis, does not need very detailed data, operation system data need be polymerized according to data warehouse granularity.For some metric field in dynamic information database fact table, usually need to use aggregate function to get by the one or more field of data source, such as sum, avg, min, max, count, therefore need to do corresponding conversion;
(9) set value: this rule gets a value that the is fixing or system of dependence for aiming field, and does not rely on data source field.
3, the loading of holographic time scale measurement data: the data structure that the holographic time scale measurement data converted define according to target data model is loaded in dynamic information database.To the different loading cycle of data acquisition of different business systems, adopt multi-threading parallel process mode to load data simultaneously, improve program operational efficiency.
In traditional data warehouse system, by origin system by the load time of making an appointment and data layout, regularly need extract data be put in the interface of making an appointment, then by data extractor tool this part Data import to data warehouse.But, for holographic time scale measurement data, just must be loaded in data warehouse immediately, to support the needs of real-time tactical analysis once be produced by origin system.Therefore, data extractor tool also needs to catch in real time new real time data change (insertion, renewal etc.) produced, data variation is selectively located and catches, normal device image data, the location of out-of-limit data and seizure is carried out according to metric data Quality Codes, the data variation that the artificial set of filtering produces, meets the requirement of zero-lag, minimizes the Invasive degree to origin system, reduce the load of origin system, guarantee that origin system performance does not decline, improper machine.
The each holographic time scale measurement data variation captured is distributed in form of a message, and comprise multiple data variation in same affairs, also just contain multiple messages, these message carry out individual transmission in a network.Data extractor tool adopts efficient Data Dissemination, after making each data variation captured put into message queue, completed the distribution of data by message queue, ensure the consistance of transmission of messages and integrality, simultaneously the transaction dependency of service data and time dependence effectively.
Be undressed data in the message received, if carry out intricately cleaning and conversion operations to these data, the requirement of external inquiry to real-time property cannot be met; Otherwise the dirty data comprised can have a strong impact on the quality of data.Need under the prerequisite ensureing the quality of data, realize real-time, efficient Data import, rationally effective organizing is carried out to the inside subring joint that cleaning and the transfer process of data comprise, thus the speed of raising data processing and concurrency.Simultaneously according to the different demands of user to the quality of data, the instant data loaded are treated with a certain discrimination, reasonable distribution system resource, improve Data import performance.
4, integrated by holographic time scale measurement data and the electric network model be stored in relevant database and other kinds business datum, being about to the data be loaded in dynamic information database carries out integrated with the electric network model be stored in relevant database and other kinds business datum, forms data warehouse.The data warehouse architecture formed as shown in Figure 3.Application server passes through to obtain the data in data warehouse, for the upper layer application such as Real-time Alarm, immediate analysis, report customization provide data supporting.
In order to farthest reduce to inquire about the negative effect that brings to system of conflicting, ensure that data warehouse runs normally and efficiently, real time data and historical data are usually separated and are stored.In order to minimize the impact on query facility, do not need query facility to understand the method obtaining different types of data, but once propose inquiry request, just can obtain the data after " Seamless integration-".
In order to provide the effective Organization And Management strategy of real time data and historical data, make it to be operated in efficiently in a kind of operating load environment of mixing, data warehouse carries out unified Modeling to real time data and historical data, unified access view is externally provided, solve " inquiry conflict " and " inquiry inconsistency " problem that real time data inquiry is produced, ensure the non-blocking of query processing process and the consistance of Query Result, the timely information of real time data and historical data is merged, the query manipulation submitted to is provided to the Integrated service of " transparent ", simultaneously, strengthen the management to load, make integrated after data warehouse run efficiently.
Data warehouse can automatic analysis query statement, thus determines demand data, and from the data needed for different extracting section, for query facility after merging.Meanwhile, also can the ratio of real-time partial and history portion in automatic analysis desired data, thus select the migration strategy of data better, reduce data transmission, improve service performance.
Although the present invention with preferred embodiment openly as above, embodiment is not of the present invention for limiting.Without departing from the spirit and scope of the invention, any equivalence change done or retouching, belong to the protection domain of the present invention equally.Therefore the content that protection scope of the present invention should define with the claim of the application is standard.
Claims (6)
1., towards a holographic time scale measurement data extraction conversion method for operation of power networks, it is characterized in that, comprise the steps:
1) extraction of holographic time scale measurement data: read the metadata in source data, access data and extracted data from multi-data source;
2) conversion of holographic time scale measurement data: according to constraint database and service logic, changing the data harmonization extracted in step 1) by data cleansing, data is consolidation form, with the needs of the data model and the quality of data that meet dynamic information database;
3) loading of holographic time scale measurement data: the holographic time scale measurement data converted are loaded in dynamic information database according to the data structure that target data model defines, to the different loading cycle of data acquisition of different business systems, adopt multi-threading parallel process mode to load data to improve program operational efficiency simultaneously;
4) by the holographic time scale measurement data be loaded in dynamic information database be stored in the integrated of electric network model in relevant database and other kinds business datum, form data warehouse, application server provides data supporting by the data in acquisition data warehouse for upper layer application.
2. the holographic time scale measurement data extraction conversion method towards operation of power networks according to claim 1, it is characterized in that, described step 1) specifically comprises following process:
1-1) read the holographic time scale measurement data meta-model in each data source resource layer;
1-2) extract meta-model through far-end and extraction process is carried out to data, described far-end extracts meta-model and forms by extracting core, log pattern and extraction configure metadata, extract core and read the optimum configurations extracting configure metadata, target data is extracted according in relative set distally data source table, extraction mode is divided into full dose to extract and increment extraction two kinds, extract configure metadata to be made up of the table name needing to extract, field, type, database linkage information, extracted data temporal information, daily record model is used for noting down the daily record extracting process;
Described full dose extracts and refers to that the total data of the specific data field of data source all extracts by the rule according to the field extracted and extraction; Only extract in follow-up extraction process after the full dose that refers to described increment extraction has extracted and extracted data that are newly-increased in the table of source or that be modified from last time;
1-3) by transmission unit model, the data of extraction are distally transferred to center-side;
1-4) will the data buffer storage of center-side be transferred to isomery scratchpad area (SPA).
3. the holographic time scale measurement data extraction conversion method towards operation of power networks according to claim 2, it is characterized in that, described increment extraction realizes based on the timestamp of dispatching services system, decides to extract which data by the value comparing the timestamp field of specifying extraction time and extraction source to show.
4. the holographic time scale measurement data extraction conversion method towards operation of power networks according to claim 1, it is characterized in that, described step 2) in data conversion comprise inconsistent data conversion, with reference to conversion, string processing, directly mapping, date conversion, date computing, null value judgement, aggregate operation and/or set value.
5. the holographic time scale measurement data extraction conversion method towards operation of power networks according to claim 1, it is characterized in that, in described step 3), when the loading of holographic time scale measurement data, the new real time data change produced is caught in real time, and carry out normal device image data, the location of out-of-limit data and seizure according to metric data Quality Codes, the data variation that the artificial set of filtering produces.
6. the holographic time scale measurement data extraction conversion method towards operation of power networks according to claim 1, it is characterized in that, real time data in described data warehouse and historical data are separated and are stored, and carry out unified Modeling to real time data and historical data, externally provide unified access view.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410181171.9A CN104318481A (en) | 2014-05-04 | 2014-05-04 | Power-grid-operation-oriented holographic time scale measurement data extraction conversion method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410181171.9A CN104318481A (en) | 2014-05-04 | 2014-05-04 | Power-grid-operation-oriented holographic time scale measurement data extraction conversion method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104318481A true CN104318481A (en) | 2015-01-28 |
Family
ID=52373707
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410181171.9A Pending CN104318481A (en) | 2014-05-04 | 2014-05-04 | Power-grid-operation-oriented holographic time scale measurement data extraction conversion method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104318481A (en) |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104659915A (en) * | 2015-02-02 | 2015-05-27 | 国家电网公司 | ETL (Extract-Transform-Load) terminal monitoring device for smart power grid |
| CN105279234A (en) * | 2015-09-24 | 2016-01-27 | 国网山西省电力公司大同供电公司 | Cross-platform data acquisition and conversion method of state overhaul data of electric transmission and transformation equipment |
| CN106777933A (en) * | 2016-12-02 | 2017-05-31 | 郑州云海信息技术有限公司 | A kind of collecting method, apparatus and system |
| CN107315767A (en) * | 2017-05-17 | 2017-11-03 | 中国科学院计算机网络信息中心 | A kind of convergence method for reconstructing of flux data |
| CN107451206A (en) * | 2017-07-11 | 2017-12-08 | 成都优易数据有限公司 | A kind of method using data structure control data set-up procedure |
| CN107506448A (en) * | 2017-08-25 | 2017-12-22 | 国网辽宁省电力有限公司 | A kind of method and system of intelligent grid time scale measurement data interchange format |
| CN108052637A (en) * | 2017-12-20 | 2018-05-18 | 江苏瑞中数据股份有限公司 | The real-time picture and text monitoring method of power grid time scale measurement data access Life cycle |
| CN108052557A (en) * | 2017-11-30 | 2018-05-18 | 山东鲁能软件技术有限公司 | The method and system of GIS grid graph datas access full-service data center |
| CN108572973A (en) * | 2017-03-09 | 2018-09-25 | 派衍信息科技(苏州)有限公司 | A kind of real time data and historical data logical separation store processing system |
| CN108615142A (en) * | 2018-05-16 | 2018-10-02 | 北京化工大学 | A kind of quick campus card data analysing method that knowledge based excavates |
| CN108829830A (en) * | 2018-06-15 | 2018-11-16 | 四川众之金科技有限公司 | Data processing method and device |
| CN105205117B (en) * | 2015-09-09 | 2019-03-05 | 郑州悉知信息科技股份有限公司 | A kind of tables of data moving method and device |
| CN109491992A (en) * | 2018-11-14 | 2019-03-19 | 上海工程技术大学 | A kind of campus card data reading system |
| CN109656916A (en) * | 2018-12-14 | 2019-04-19 | 北京无线电测量研究所 | A kind of acquisition methods of radar target data |
| CN110602049A (en) * | 2019-08-14 | 2019-12-20 | 中国平安人寿保险股份有限公司 | Data transmission method, server and storage medium |
| CN110727725A (en) * | 2019-09-26 | 2020-01-24 | 广西电网有限责任公司电力科学研究院 | Data interface based on distribution network operating efficiency monitoring and analysis |
| CN112100227A (en) * | 2020-09-22 | 2020-12-18 | 国网辽宁省电力有限公司电力科学研究院 | Big data processing method based on multilevel heterogeneous data storage |
| CN112186901A (en) * | 2020-09-30 | 2021-01-05 | 国网智能科技股份有限公司 | Panoramic sensing monitoring method and system for transformer substation |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102880709A (en) * | 2012-09-28 | 2013-01-16 | 用友软件股份有限公司 | Data warehouse management system and data warehouse management method |
| CN103023970A (en) * | 2012-11-15 | 2013-04-03 | 中国科学院计算机网络信息中心 | Method and system for storing mass data of Internet of Things (IoT) |
| CN103473260A (en) * | 2013-06-25 | 2013-12-25 | 北京控制工程研究所 | Concurrency OLAP (On-Line Analytical Processing)-oriented test data hierarchy cluster query processing system and method |
-
2014
- 2014-05-04 CN CN201410181171.9A patent/CN104318481A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102880709A (en) * | 2012-09-28 | 2013-01-16 | 用友软件股份有限公司 | Data warehouse management system and data warehouse management method |
| CN103023970A (en) * | 2012-11-15 | 2013-04-03 | 中国科学院计算机网络信息中心 | Method and system for storing mass data of Internet of Things (IoT) |
| CN103473260A (en) * | 2013-06-25 | 2013-12-25 | 北京控制工程研究所 | Concurrency OLAP (On-Line Analytical Processing)-oriented test data hierarchy cluster query processing system and method |
Non-Patent Citations (3)
| Title |
|---|
| 李娜: "生产过程数据仓库ETL模块的研究与开发", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 段成，王增平，吴克河: "一种轻量级电网实时数据ETL系统的涉及与实现", 《电力系统保护与控制》 * |
| 郭抒翔: "ETL关键技术在电网综合数据平台中的应用", 《中国优秀硕士学位论文全文数据库 工程科技Ⅱ辑》 * |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104659915A (en) * | 2015-02-02 | 2015-05-27 | 国家电网公司 | ETL (Extract-Transform-Load) terminal monitoring device for smart power grid |
| CN105205117B (en) * | 2015-09-09 | 2019-03-05 | 郑州悉知信息科技股份有限公司 | A kind of tables of data moving method and device |
| CN105279234A (en) * | 2015-09-24 | 2016-01-27 | 国网山西省电力公司大同供电公司 | Cross-platform data acquisition and conversion method of state overhaul data of electric transmission and transformation equipment |
| CN106777933A (en) * | 2016-12-02 | 2017-05-31 | 郑州云海信息技术有限公司 | A kind of collecting method, apparatus and system |
| CN108572973A (en) * | 2017-03-09 | 2018-09-25 | 派衍信息科技(苏州)有限公司 | A kind of real time data and historical data logical separation store processing system |
| CN107315767A (en) * | 2017-05-17 | 2017-11-03 | 中国科学院计算机网络信息中心 | A kind of convergence method for reconstructing of flux data |
| CN107315767B (en) * | 2017-05-17 | 2020-08-04 | 中国科学院计算机网络信息中心 | Convergent reconstruction method of flux data |
| CN107451206A (en) * | 2017-07-11 | 2017-12-08 | 成都优易数据有限公司 | A kind of method using data structure control data set-up procedure |
| CN107506448A (en) * | 2017-08-25 | 2017-12-22 | 国网辽宁省电力有限公司 | A kind of method and system of intelligent grid time scale measurement data interchange format |
| CN108052557A (en) * | 2017-11-30 | 2018-05-18 | 山东鲁能软件技术有限公司 | The method and system of GIS grid graph datas access full-service data center |
| CN108052637A (en) * | 2017-12-20 | 2018-05-18 | 江苏瑞中数据股份有限公司 | The real-time picture and text monitoring method of power grid time scale measurement data access Life cycle |
| CN108052637B (en) * | 2017-12-20 | 2021-09-21 | 江苏瑞中数据股份有限公司 | Real-time image-text monitoring method for power grid time scale measurement data access full life cycle |
| CN108615142A (en) * | 2018-05-16 | 2018-10-02 | 北京化工大学 | A kind of quick campus card data analysing method that knowledge based excavates |
| CN108829830A (en) * | 2018-06-15 | 2018-11-16 | 四川众之金科技有限公司 | Data processing method and device |
| CN109491992A (en) * | 2018-11-14 | 2019-03-19 | 上海工程技术大学 | A kind of campus card data reading system |
| CN109656916A (en) * | 2018-12-14 | 2019-04-19 | 北京无线电测量研究所 | A kind of acquisition methods of radar target data |
| CN110602049A (en) * | 2019-08-14 | 2019-12-20 | 中国平安人寿保险股份有限公司 | Data transmission method, server and storage medium |
| CN110727725A (en) * | 2019-09-26 | 2020-01-24 | 广西电网有限责任公司电力科学研究院 | Data interface based on distribution network operating efficiency monitoring and analysis |
| CN110727725B (en) * | 2019-09-26 | 2022-06-24 | 广西电网有限责任公司电力科学研究院 | Data interface based on distribution network operating efficiency monitoring and analysis |
| CN112100227A (en) * | 2020-09-22 | 2020-12-18 | 国网辽宁省电力有限公司电力科学研究院 | Big data processing method based on multilevel heterogeneous data storage |
| CN112186901A (en) * | 2020-09-30 | 2021-01-05 | 国网智能科技股份有限公司 | Panoramic sensing monitoring method and system for transformer substation |
| CN112186901B (en) * | 2020-09-30 | 2022-07-15 | 国网智能科技股份有限公司 | Panoramic sensing monitoring method and system for transformer substation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104318481A (en) | Power-grid-operation-oriented holographic time scale measurement data extraction conversion method | |
| CN103593422B (en) | Virtual access management method of heterogeneous database | |
| CN102023983B (en) | Managing method of statistical space-time database | |
| CN110300963A (en) | Data management system in large-scale data repository | |
| Khattak et al. | Change management in evolving web ontologies | |
| CN103019728B (en) | Effective complex report parsing engine and parsing method thereof | |
| CN103425762A (en) | Telecom operator mass data processing method based on Hadoop platform | |
| CN106168965A (en) | Knowledge mapping constructing system | |
| CN106709035A (en) | Preprocessing system for electric power multi-dimensional panoramic data | |
| CN104598610A (en) | Step-by-step database data distribution uploading and synchronizing method | |
| CN106934014A (en) | A kind of network data excavation based on Hadoop and analysis platform and its method | |
| CN104573100A (en) | Step-by-step database synchronization method with autoincrement identifications | |
| CN109284346A (en) | A kind of distribution network planning method and device based on cloud computing | |
| CN112364046B (en) | Knowledge graph-based main data management method in heterogeneous environment | |
| CN109508355A (en) | A kind of data pick-up method, system and terminal device | |
| CN103186541A (en) | Generation method and device for mapping relationship | |
| CN107194533B (en) | Power distribution network full information model construction method and system | |
| CN113553313B (en) | Data migration method and system, storage medium and electronic equipment | |
| CN103150163A (en) | Map/Reduce mode-based parallel relating method | |
| CN105224527A (en) | Be applicable to the general ETL method of multiple object table update mode | |
| CN115858513A (en) | Data governance method, data governance device, computer equipment and storage medium | |
| CN112181960A (en) | Intelligent operation and maintenance framework system based on AIOps | |
| CN111913933B (en) | Power grid historical data management method and system based on unified support platform | |
| CN114218218A (en) | Data processing method, device and equipment based on data warehouse and storage medium | |
| CN106802905A (en) | A kind of synergistic data exchange method of isomorphism PLM system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| AD01 | Patent right deemed abandoned |
Effective date of abandoning: 20190521 |
|
| AD01 | Patent right deemed abandoned |