CN103605512A - System and method for data verification based on GTechnology platform - Google Patents
System and method for data verification based on GTechnology platform Download PDFInfo
- Publication number
- CN103605512A CN103605512A CN201310542927.3A CN201310542927A CN103605512A CN 103605512 A CN103605512 A CN 103605512A CN 201310542927 A CN201310542927 A CN 201310542927A CN 103605512 A CN103605512 A CN 103605512A
- Authority
- CN
- China
- Prior art keywords
- data
- verification
- check
- platform
- gtechnology
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention provides a system and a method for data verification based on a GTechnology platform. The system comprises a data access layer, a service logic layer and a user interface layer. The method includes that the system collects data from the GTechnology platform through a data access component to acquire to-be-verified data; a data verification module verifies the acquired data, acquires rule verification functions defined by various data through XML (extensible markup language) file analysis and executes the rule verification functions to complete data verification, wherein data against rules are record and stored in a problem list; based on the acquired problem list, a data processing module verifies the data according to data type, data values within a definition domain, null values and redundant data and coverts and cleans valid data. By the system and the method, junk data and redundant data can be processed effectively, and accordingly, data records in a database are ensured to be free of repetition and redundancy; problem data, incompatible data and the like in the GTechnology platform are cleaned, and accordingly, data accuracy and integrity are guaranteed.
Description
Technical field
The present invention relates to electric power system data processing technology field, relate in particular to a kind of based on GTechnology(geographical space Facilities Management Platform) data verification system and the data verification method of platform.
Background technology
Unified platform, around the first job requirement of wound, is joined in the unified battalion that builds, promotes, implemented based on GIS of the whole province by Guangdong Power Grid company.Wherein, the platform that GIS mainly applies is Gtechnology platform.At present, each power supply enterprise of the subordinate of Guangdong Power Grid company has carried out major network GIS and distribution GIS integrated, has built from 500kV equipment to the integrated full electric network model of 0.4kV user.In addition, built communication resource generalized information system, the vehicle management generalized information system etc. of establishing of part power supply enterprise.From online implementing operation so far, the about 900G of prefectures and cities of Guangdong Power Grid company office generalized information system power network resources data total amount, nearly 5000 users are online use simultaneously.Due to the system Construction initial stage, lack experience, lack unified codes and standards, prefectures and cities office has proposed different construction requirements in system Construction process, and the data of generalized information system be there are differences, and is unfavorable for that longitudinal data connects.Simultaneously, the existing data verification method of system also cannot system, the integrality that comprehensively guarantees system data, consistance, accuracy, uniqueness and timeliness, cause system data quality aspect to have more problem, comprise operation system data and GIS mathematical logic relation is numerous and diverse, redundant data is more, data content is inconsistent etc., have a strong impact on the practical level of operation system related application.
Summary of the invention
The object of the present invention is to provide a kind of based on check system and data verification method GTechnology platform, that adopt various dimensions checking data, this system and method can system, comprehensively guarantee the quality of data in network system.
Object of the present invention can realize by following technical measures:
A data verification system based on GTechnology platform, comprises data access layer, Business Logic and user interface layer; Described data access layer comprises MDAC, provide access system data interface, the interface that carries out exchanges data is provided, for carrying out data interaction with system, system data is inquired about; Described Business Logic comprises data check rule base, data check module and data processing module, for the logical process of Various types of data; Described user interface layer is the interface for man-machine interaction, intuitively shows the result of various data check Treatment Analysis.
Store and management is carried out in the mode of XML file in described data check rule base backstage.
Described data check module is write all kinds of methods of calibration according to the verification rule of different types of data, and forms data check function library for data are arranged and verification.
Described data processing module carries out data-switching and cleaning according to check results, for all kinds of problem datas are write to data processing method, forms pretreating scheme.
A data verification method based on GTechnology platform, concrete steps are as follows:
A, according to business demand, Various types of data and required verification thereof rule, verification function are configured, every class data can corresponding a plurality of verifications rules or are adopted XML file layout to be directly configured;
B, by MDAC, from GTechnology platform, carry out data acquisition, obtain needing the data of verification;
The data that C, data check module obtain step B are carried out verification, by resolving described XML file, obtain the defined regular verification function of Various types of data, executing rule verification function completes data check, and the data that break the rules are recorded and are kept in problem list;
Regular verification function described in step C is public function and the self-defining function in verification function library.Described public function is used for the data verification of typical data, common rule and checks, and for example arithmetic class function " double Sign (double x) " in C#, quotes " System.Math.dll " class libraries and can obtain; Self-defining function is used for the data verification of special object, particular service rule and checks, these functions are write voluntarily according to business rule, logic, system has been carried in internal memory while starting, under any circumstance can both use, and not need to be written into any class libraries or auxiliary function.
D, the problem list obtaining based on step C, from data type, data value within the scope of field of definition, null value, redundant data verified aspect several, to carrying out data-switching and cleaning by the data of checking.Described data-switching comprises conversion (as 1km transfers 1000m to), data type conversion (as character string type transfers date type to), data-mapping conversion (transferring mechanism coding 05 to as transferred Chinese description mapping to code ID， mechanism MingXX power supply administration), the replacement null value (as with 0) between data unit; Data scrubbing is mainly to delete redundant data.
Complete after a check sum data processing, checking treatment result will be saved in corresponding storehouse table, and system automatically generates abnormal data statistical report form and derives.
The present invention contrasts prior art, has the following advantages:
The present invention is based on GTechnology platform, from various dimensions, carry out data check, use this system and method effectively to process the junk data, the redundant data that exist, guarantee in database that data recording is without repetition, irredundant; Problem data in GTechnology platform, incompatible data etc. are cleared up, guarantee correctness and the integrality of data; Can effectively control the rational change range of different types of data in GTechnology platform, data error is controlled in set scope; Can support the operation of the system related services such as production, marketing, promote the level of IT application of power supply enterprise, be applicable to following GIS development need.
Accompanying drawing explanation
Fig. 1 is the data verification system configuration diagram that the present invention is based on GTechnology platform;
Fig. 2 is data check rule base, the data check function library model schematic diagram of data verification method;
Fig. 3 is the process flow diagram that the present invention is based on the data check of GTechnology platform.
Embodiment
Below the embodiment to the inventive method is by reference to the accompanying drawings elaborated.
As shown in Figure 1, a kind of data verification system based on GTechnology platform, comprises data access layer, Business Logic and user interface layer.Data access layer, for carrying out data interaction with system, is inquired about system data; Business Logic is for the logical process of Various types of data, comprise data check rule base, data check module and data processing module, data check module is for arranging and verification data, and data processing module carries out data-switching and cleaning according to check results; User interface layer is the interface for man-machine interaction, intuitively shows the result of various data check Treatment Analysis.
User interface layer is to using the operation/maintained/managed personnel of native system to provide a personal-machine mutual interface, system user of service can be carried out various data checks, data processing, analysis result statistics is derived by this interface, and data check rule base related various rules, parameter configuration are defined, managed.
Data check module in Business Logic, is to write all kinds of methods of calibration according to the verification rule of different types of data, and forms data check function library.Data processing module is to have write data processing method for all kinds of problem datas, forms pretreating scheme.Store and management is carried out in the mode of XML file in data check rule base backstage.
Data access layer is mainly MDAC, provide access system data interface, the interface that carries out exchanges data is provided.
A data verification method based on GTechnology platform, is characterized in that concrete steps are as follows:
A, as shown in Figure 2, the configuration of data check rule base, is configured Various types of data and required verification rule thereof, verification function according to business demand by user interface layer, and every class data can corresponding a plurality of verifications rules; Also can adopt XML file layout to be directly configured, form is as follows:
B, as shown in Figure 3, system is carried out data acquisition from GTechnology platform by MDAC, obtain needing the data of verification.
The data that C, data check module obtain step B are carried out verification, by analyzing XML file, obtain the defined regular verification function of Various types of data, and executing rule verification function completes data check, and the data that break the rules are recorded and are kept in problem list.
Regular verification function under in step C, i.e. public function in verification function library and self-defining function.Described public function is used for the data verification of typical data, common rule and checks, and is common data verification method, and for example arithmetic class function " double Sign (double x) " in C#, quotes " System.Math.dll " class libraries and can obtain; Self-defining function is used for the data verification of special object, particular service rule and checks, these functions are write voluntarily according to business rule, logic, system has been carried in internal memory while starting, under any circumstance can both use, and not need to be written into any class libraries or auxiliary function.
D, the problem list obtaining based on step C, data processing module from data type, data value within the scope of field of definition, null value, redundant data verified aspect several, to carrying out data-switching and cleaning by the data of checking.Data-switching comprises conversion (as 1km transfers 1000m to), data type conversion (as character string type transfers date type to), data-mapping conversion (transferring mechanism coding 05 to as transferred Chinese description mapping to code ID， mechanism MingXX power supply administration), the replacement null value (as with 0) between data unit; Data scrubbing is mainly to delete redundant data.
Complete after a check sum data processing, checking treatment result will be saved in corresponding storehouse table, and system automatically generates abnormal data statistical report form and derives.
Embodiments of the present invention are not limited to this; under the above-mentioned basic fundamental thought of the present invention prerequisite; modification, replacement or the change to other various ways that content of the present invention is made according to the ordinary skill knowledge of this area and customary means, within all dropping on rights protection scope of the present invention.
Claims (7)
1. the data verification system based on GTechnology platform, is characterized in that: comprise data access layer, Business Logic and user interface layer; Described data access layer comprises MDAC, provide access system data interface, the interface that carries out exchanges data is provided, for carrying out data interaction with system, system data is inquired about; Described Business Logic comprises data check rule base, data check module and data processing module, for the logical process of Various types of data; Described user interface layer is the interface for man-machine interaction, intuitively shows the result of various data check Treatment Analysis.
2. data verification system according to claim 1, is characterized in that: store and management is carried out in the mode of XML file in described data check rule base backstage.
3. data verification system according to claim 1, is characterized in that: described data check module is write all kinds of methods of calibration according to the verification rule of different types of data, and forms data check function library for data are arranged and verification.
4. data verification system according to claim 1, is characterized in that: described data processing module carries out data-switching and cleaning according to check results, for all kinds of problem datas are write to data processing method, forms pretreating scheme.
5. the data verification method based on GTechnology platform, is characterized in that comprising the steps:
A, according to business demand, Various types of data and required verification thereof rule, verification function are configured, every class data can corresponding a plurality of verifications rules or are adopted XML file layout to be directly configured;
B, by MDAC, from GTechnology platform, carry out data acquisition, obtain needing the data of verification;
The data that C, data check module obtain step B are carried out verification, by resolving described XML file, obtain the defined regular verification function of Various types of data, executing rule verification function completes data check, and the data that break the rules are recorded and are kept in problem list;
D, the problem list obtaining based on step C, from data type, data value within the scope of field of definition, null value, redundant data verified aspect several, to carrying out data-switching and cleaning by the data of checking;
F, complete after a check sum data processing, checking treatment result will be saved in corresponding storehouse table, and system automatically generates abnormal data statistical report form and derives.
6. data verification method according to claim 5, is characterized in that: the regular verification function described in step C is public function and the self-defining function in verification function library; Described public function is used for the data verification of typical data, common rule and checks; Described self-defining function is used for the data verification of special object, particular service rule and checks.
7. data verification method according to claim 5, is characterized in that: described data-switching comprises conversion, data type conversion, data-mapping conversion, the replacement null value between data unit; Described data scrubbing is for deleting redundant data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310542927.3A CN103605512A (en) | 2013-11-05 | 2013-11-05 | System and method for data verification based on GTechnology platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310542927.3A CN103605512A (en) | 2013-11-05 | 2013-11-05 | System and method for data verification based on GTechnology platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103605512A true CN103605512A (en) | 2014-02-26 |
Family
ID=50123742
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310542927.3A Pending CN103605512A (en) | 2013-11-05 | 2013-11-05 | System and method for data verification based on GTechnology platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103605512A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104239162A (en) * | 2014-09-09 | 2014-12-24 | 北京国双科技有限公司 | Data check method and data check device |
| CN105099813A (en) * | 2015-06-12 | 2015-11-25 | 北京信而泰科技股份有限公司 | Data verification method and device based on Ethernet tester |
| CN107220892A (en) * | 2017-05-27 | 2017-09-29 | 国家计算机网络与信息安全管理中心 | One kind melts data intelligence pretreating tool and method applied to magnanimity P2P net monetary allowances |
| CN107885609A (en) * | 2017-11-08 | 2018-04-06 | 泰康保险集团股份有限公司 | Service conflict processing method and processing device, storage medium and electronic equipment |
| CN108021468A (en) * | 2017-12-08 | 2018-05-11 | 泰康保险集团股份有限公司 | The method of calibration and device of business datum in medical treatment endowment cooperative system |
| CN109783417A (en) * | 2019-01-11 | 2019-05-21 | 四川长虹电器股份有限公司 | A kind of serial communication device and method |
| WO2020006929A1 (en) * | 2018-07-05 | 2020-01-09 | 平安科技（深圳）有限公司 | Data check method and device, and computer storage medium |
| CN110851486A (en) * | 2018-07-26 | 2020-02-28 | 珠海格力电器股份有限公司 | Data storage method and device |
| CN111382579A (en) * | 2020-01-13 | 2020-07-07 | 中船第九设计研究院工程有限公司 | Data preprocessing verification platform of ship pipeline manufacturing execution system |
| CN111737528A (en) * | 2020-06-23 | 2020-10-02 | Oppo（重庆）智能科技有限公司 | Data acquisition and verification method and device, electronic equipment and storage medium |
| CN112597645A (en) * | 2020-12-17 | 2021-04-02 | 武汉大学 | Power distribution network model checking method based on cloud platform |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101170378A (en) * | 2007-10-12 | 2008-04-30 | 上海亚太计算机信息系统有限公司 | Content-based electronic data calibration and conversion method compatible to multi-data specification |
| CN101788950A (en) * | 2010-01-27 | 2010-07-28 | 浪潮(山东)电子信息有限公司 | Data item calibration method based on JSP page |
-
2013
- 2013-11-05 CN CN201310542927.3A patent/CN103605512A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101170378A (en) * | 2007-10-12 | 2008-04-30 | 上海亚太计算机信息系统有限公司 | Content-based electronic data calibration and conversion method compatible to multi-data specification |
| CN101788950A (en) * | 2010-01-27 | 2010-07-28 | 浪潮(山东)电子信息有限公司 | Data item calibration method based on JSP page |
Non-Patent Citations (3)
| Title |
|---|
| 昌力: "基于规则的电力调度计划数据校验机制", 《电力系统自动化》 * |
| 肖峻: "基于规则的配电地理信息系统导出数据校验与修复", 《电力系统自动化》 * |
| 胡昊伟: "电力数据平台中数据校验与修复方法的研究", 《中国电子商情：通信市场》 * |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104239162A (en) * | 2014-09-09 | 2014-12-24 | 北京国双科技有限公司 | Data check method and data check device |
| CN105099813A (en) * | 2015-06-12 | 2015-11-25 | 北京信而泰科技股份有限公司 | Data verification method and device based on Ethernet tester |
| CN105099813B (en) * | 2015-06-12 | 2019-03-05 | 北京信而泰科技股份有限公司 | Data verification method and device based on ethernet test instrument |
| CN107220892B (en) * | 2017-05-27 | 2020-10-16 | 国家计算机网络与信息安全管理中心 | Intelligent preprocessing tool and method applied to massive P2P network loan financial data |
| CN107220892A (en) * | 2017-05-27 | 2017-09-29 | 国家计算机网络与信息安全管理中心 | One kind melts data intelligence pretreating tool and method applied to magnanimity P2P net monetary allowances |
| CN107885609A (en) * | 2017-11-08 | 2018-04-06 | 泰康保险集团股份有限公司 | Service conflict processing method and processing device, storage medium and electronic equipment |
| CN108021468A (en) * | 2017-12-08 | 2018-05-11 | 泰康保险集团股份有限公司 | The method of calibration and device of business datum in medical treatment endowment cooperative system |
| WO2020006929A1 (en) * | 2018-07-05 | 2020-01-09 | 平安科技（深圳）有限公司 | Data check method and device, and computer storage medium |
| CN110851486A (en) * | 2018-07-26 | 2020-02-28 | 珠海格力电器股份有限公司 | Data storage method and device |
| CN109783417A (en) * | 2019-01-11 | 2019-05-21 | 四川长虹电器股份有限公司 | A kind of serial communication device and method |
| CN109783417B (en) * | 2019-01-11 | 2022-03-15 | 四川长虹电器股份有限公司 | Serial port communication device and method |
| CN111382579A (en) * | 2020-01-13 | 2020-07-07 | 中船第九设计研究院工程有限公司 | Data preprocessing verification platform of ship pipeline manufacturing execution system |
| CN111737528A (en) * | 2020-06-23 | 2020-10-02 | Oppo（重庆）智能科技有限公司 | Data acquisition and verification method and device, electronic equipment and storage medium |
| CN112597645A (en) * | 2020-12-17 | 2021-04-02 | 武汉大学 | Power distribution network model checking method based on cloud platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103605512A (en) | System and method for data verification based on GTechnology platform | |
| US11451529B2 (en) | Security migration in a business intelligence environment | |
| CN102576432B (en) | Automated test execution plan generation | |
| CN104123227A (en) | Method for automatically generating testing cases | |
| CN103914564A (en) | Method and system for updating and warehousing urban pipelines on basis of ArcGIS (geographic information system) | |
| CN104598376A (en) | Data driving layered automation test system and method | |
| CN103957248A (en) | Public real-time data management cloud service platform based on Internet of Things | |
| CN103699693A (en) | Metadata-based data quality management method and system | |
| CN105487867A (en) | Lightweight visual satellite testing program design system and method | |
| CN105260841A (en) | Auditing system for power distribution network fault emergency repair receipt | |
| CN104915262A (en) | Calibration system and method based on EXCEL data structure | |
| CN104992022A (en) | Aeronautics electronic system interface management method based on models | |
| CN103577550A (en) | Grid equipment data verification method | |
| Zorrilla et al. | A reference framework for the implementation of data governance systems for industry 4.0 | |
| CN103606032B (en) | A kind of method in two dimension power grid GIS data set | |
| US20130268936A1 (en) | Workflow management system and method | |
| CN107526804B (en) | Railway investment project data statistics system and method | |
| CN116150408A (en) | Energy information management method, system, equipment and storage medium based on identification analysis | |
| CN104748757A (en) | Data updating method and device for navigation electronic map | |
| CN104239558A (en) | External Excel data uploading system and method based on SAP system | |
| Moloney et al. | Systems-of-systems methodology for strategic infrastructure decision making: Ireland as a case study | |
| CN107783870A (en) | A kind of server-compatible test result management method and system | |
| CN103617141A (en) | Method for electrical equipment field test and characteristic evaluation management | |
| CN100576806C (en) | Logical difference construction system data integration method in the peer-to-peer network | |
| CN110276225A (en) | A kind of water factory's equipment routing inspection data system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20140226 |