CN113723822A - Power supply service data management system - Google Patents
Power supply service data management system Download PDFInfo
- Publication number
- CN113723822A CN113723822A CN202111018032.0A CN202111018032A CN113723822A CN 113723822 A CN113723822 A CN 113723822A CN 202111018032 A CN202111018032 A CN 202111018032A CN 113723822 A CN113723822 A CN 113723822A
- Authority
- CN
- China
- Prior art keywords
- data
- layer
- power supply
- index
- management system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013523 data management Methods 0.000 title claims abstract description 26
- 238000012545 processing Methods 0.000 claims abstract description 23
- 238000003860 storage Methods 0.000 claims abstract description 22
- 238000007405 data analysis Methods 0.000 claims abstract description 21
- 238000010276 construction Methods 0.000 claims abstract description 14
- 238000013500 data storage Methods 0.000 claims abstract description 14
- 238000013480 data collection Methods 0.000 claims abstract description 12
- 238000004458 analytical method Methods 0.000 claims abstract description 5
- 230000006641 stabilisation Effects 0.000 claims abstract description 4
- 238000011105 stabilization Methods 0.000 claims abstract description 4
- 238000000034 method Methods 0.000 claims description 50
- 230000008569 process Effects 0.000 claims description 47
- 238000007726 management method Methods 0.000 claims description 34
- 238000009826 distribution Methods 0.000 claims description 28
- 238000013499 data model Methods 0.000 claims description 20
- 239000002131 composite material Substances 0.000 claims description 19
- 238000013439 planning Methods 0.000 claims description 10
- 238000013461 design Methods 0.000 claims description 9
- 238000004519 manufacturing process Methods 0.000 claims description 6
- 230000002093 peripheral effect Effects 0.000 claims description 6
- 239000000463 material Substances 0.000 claims description 5
- 230000006698 induction Effects 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 239000000284 extract Substances 0.000 abstract description 2
- 230000002708 enhancing effect Effects 0.000 abstract 1
- 239000010410 layer Substances 0.000 description 53
- 235000019580 granularity Nutrition 0.000 description 20
- 238000005259 measurement Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 4
- 238000000060 site-specific infrared dichroism spectroscopy Methods 0.000 description 4
- 230000010354 integration Effects 0.000 description 3
- 238000011144 upstream manufacturing Methods 0.000 description 3
- 239000000654 additive Substances 0.000 description 2
- 230000000996 additive effect Effects 0.000 description 2
- 238000012550 audit Methods 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000012384 transportation and delivery Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000013506 data mapping Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000003607 modifier Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000001376 precipitating effect Effects 0.000 description 1
- 238000001556 precipitation Methods 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 239000002994 raw material Substances 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000012732 spatial analysis Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
- 239000002344 surface layer Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
- G06Q10/103—Workflow collaboration or project management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Strategic Management (AREA)
- General Physics & Mathematics (AREA)
- Economics (AREA)
- Entrepreneurship & Innovation (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Marketing (AREA)
- Quality & Reliability (AREA)
- Operations Research (AREA)
- Probability & Statistics with Applications (AREA)
- Development Economics (AREA)
- Educational Administration (AREA)
- General Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Game Theory and Decision Science (AREA)
- Water Supply & Treatment (AREA)
- General Health & Medical Sciences (AREA)
- Primary Health Care (AREA)
- Public Health (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a power supply service data management system, which comprises a data access layer, a data gathering and storage layer, a data analysis layer and a data processing layer, wherein the data access layer is used for receiving data; the data access layer is used for realizing uniform data access and shielding the mutual influence of the transaction flow and the warehouse counting flow; the data collection and storage layer is used for establishing relatively complete public data with atomic granularity, and enhancing data sharing and model stabilization and reuse; the data analysis layer is used for carrying out moderate data summarization based on the data summarization and the storage layer or carrying out cross-theme and integrated pull-through data to provide preprocessed data for the data processing layer; and the data processing layer extracts related data from the data gathering and storage layer or the data analysis layer, carries out loading and individual demand processing, and quickly supports the power supply system to gather analysis application construction of each business department. The invention can make full use of the data resource of the power grid system, provide good data support for formulating the power supply scheme, and improve the quality and efficiency of formulating the power supply scheme.
Description
Technical Field
The invention relates to the technical field of power supply service, in particular to a power supply service data management system.
Background
In the process of power supply service, the work processes of making, repeating and executing a power supply scheme of marketing business often need a large amount of data resources, and the data relates to a plurality of business departments of a power supply company. In the power supply service system, the existing management systems are as follows: marketing MIS, GIS system, SCADA, distribution network automation, distribution network bar chart, distribution network resource management and the like accumulate a large amount of power network resource data, but because the systems are independent, the data resources of the systems can not be fully and effectively applied in the planning and repeating process of the service power supply scheme, almost all service schemes need to be audited by arranging special service conferences, so that the investigation, planning and repeating work of the service schemes are heavy and time-consuming, the existing power network resource information can not be fully utilized to improve the efficiency and quality of the planning and repeating of the service schemes, and the process of standardizing the repeating of the service schemes and implementing the management is a main problem of information management and construction of marketing services.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a power supply service data management system so as to fully and effectively utilize data resources of a power grid.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows.
The power supply service data management system comprises a data access layer, a data gathering and storage layer, a data analysis layer and a data processing layer;
the data access layer is respectively connected with the peripheral data resource subsystem through the access ports, so that unified data access is realized, and the mutual influence of the transaction flow and the warehouse counting flow is shielded;
the input end of the data gathering and storage layer is connected with the output end of the data access layer, public data which is relatively complete and has atomic granularity is established, and data sharing and model stabilization and reuse are enhanced;
the input end of the data analysis layer is connected with the output end of the data collection and storage layer, appropriate data collection is carried out on the basis of the data collection and storage layer, or cross-theme and integrated pull-through data is carried out, and preprocessing data are provided for the data processing layer;
and the input end of the data processing layer is connected with the output end of the data analysis layer, relevant data are extracted from the data gathering and storage layer or the data analysis layer, loading and individual demand processing are carried out, and the power supply system is quickly supported to gather analysis application construction of each business department.
In the power supply service data management system, the peripheral data resource subsystem comprises a regional power grid SCADA system, an electric energy acquisition system, a power distribution network automation system, a marketing MIS system, a power distribution network line loss system, a power distribution GIS system, a power distribution network bar graph system and a data interface server.
In the power supply service data management system, the data summarization and storage layer construction is driven by business and data, and the model data of the layer can completely and comprehensively reflect relevant business process event information; the attribute of the atomic index fact table needs to reserve the field with the service attribute in the source system source table.
In the power supply service data management system, the data collection and storage layer comprises a theme domain providing a high-order view of the model, and the theme domain mainly comprises marketing, safety production, planning and construction, material management, financial management, human resources, information management and comprehensive management.
According to the power supply service data management system, when the data analysis layer analyzes data, firstly index disassembly is carried out, and then when modeling is carried out based on the disassembled index disassembly process elements, theme modeling, logic modeling and physical modeling are mainly included.
According to the power supply service data management system, the index disassembly takes a report form or a business assessment index as input, decomposition is carried out according to an index specification definition framework, a composite index, a derivative index and an atomic index are identified, and a business process of the atomic index is determined.
In the power supply service data management system, the theme modeling refers to the business domain and the EA framework, data information is classified and described in the modes of induction, classification and the like of the business process, and a theme data model is formed in the mode of explaining the business domain, the sub-theme domain and the content of the sub-theme domain.
In the power supply service data management system, the logic data model obtained by the logic modeling is a refinement of the theme data model, and is composed of a group of entities and relations thereof, including the facts entities, the attributes in the entities, and the relations between the facts entities, the facts entities and the dimensions.
In the power supply service data management system, the physical data model obtained by physical modeling is a database architecture design by considering various specific technical implementation factors on the basis of the logical data model, so that the storage of data in the database is really realized.
In the power supply service data management system, the data analysis layer is driven by application requirements, determines the data use scene of a service, and builds public, common and stable derivative/composite index data.
Due to the adoption of the technical scheme, the technical progress of the invention is as follows.
The method has the advantages that the data resources of the power grid system can be fully utilized, good data support is provided for formulating the power supply scheme, the quality and the efficiency of formulating the power supply scheme are improved, and the like.
Drawings
FIG. 1 is a functional block diagram of the present invention;
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
The power supply service data management system has a structure shown in fig. 1, and includes a data access layer, a data collection and storage layer, a data analysis layer, and a data processing layer.
And the data access layer, abbreviated as ODS layer, is respectively connected with the peripheral data resource subsystem through the access ports, so that uniform data access is realized, the mutual influence of the transaction flow and the warehouse flow is shielded, and OLAP and OLTP resources are isolated.
The peripheral data resource subsystem comprises a regional power grid SCADA system, an electric energy acquisition system, a power distribution network automation system, a marketing MIS system, a power distribution network line loss system, a power distribution GIS system, a power distribution network bar graph system and a data interface server.
The regional power grid SCADA system mainly reads historical data of an SCADA operation historical library and analyzes an operation XML file, load information of a transformer substation and a circuit is mainly recorded in the historical library, one record is recorded every five minutes, an interface guarantees integrity and correctness of the data in a full-reading mode, and an SVG graphic function embedded into the SCADA system displays an image in the transformer substation and displays operation data.
The electric energy acquisition system mainly provides basic accounts and electric quantity data of a transformer substation and outgoing line intervals, a standard is required because the basic accounts of each power grid operation system are possibly inconsistent, and the electric quantity acquisition system acquires the electric quantity, so that the basic accounts are comprehensive.
The distribution network automation system is used for providing basic machine account information and operation information of distribution network switching stations, distribution transformers and ring main unit distribution network resources, traversing and analyzing dynamic data resource table names in a mode of reading an operation database, acquiring data, and embedding distribution network automation system functions and a mode of reading the database.
The marketing MIS system mainly provides the acceptance information of the service power supply scheme, and the basic information condition and the power utilization condition of the application user.
The distribution GIS system interface mainly provides a geographic information system of a power grid, and power grid resources in a geographic information range are inquired by accessing a GIS platform.
The distribution network bar chart system is a system for truly reflecting the distribution lines through bar charts, and the GIS-based distribution network bar chart management system fully exerts the spatial analysis capability and the database function of a geographic information system and realizes the automatic drawing of the distribution line bar chart.
The data interface server displays the power grid resource information of all the operating systems according to the types of a transformer substation, an outgoing line interval, a switching station, a distribution transformer, a ring main unit and the like in a hierarchical manner, and intelligently assists in manufacturing a power supply scheme by combining the use condition of the interval, the current condition, the load condition and the like, so that flexible and convenient system operation is provided for an application client, and basic information and flow information of the power supply scheme are stored in the data terminal.
The data access layer structure is attached to the source, the physical model is consistent with the business system model, and the business data is not subjected to data cleaning conversion and is kept to be original; the data integration system can integrate various source data and provide uniform data raw materials for other layers of the system.
The input end of the data gathering and storage layer, referred to as DWD layer for short, is connected with the output end of the data access layer, public data with relative integrity and atom granularity is established, data sharing and model stabilization and reuse are enhanced, and the data gathering and storage layer has good adaptability for future demands.
The DWD layer construction is driven by services and data, and model data of the layer can completely and comprehensively reflect relevant service process event information; the attribute of the atomic index fact table should reserve the field with the service attribute in the source table of the source system as much as possible. The DWD layer is built by modeling based on a Kimball dimension model as a theoretical basis, is subject-oriented, provides finest-granularity public atomic data with complete business attributes and traceable history, provides joint and non-summary detailed data (join) in a subject, models around the atomic indexes of a business process, reserves complete business attribute fields corresponding to the atomic indexes in the model, has the original finest granularity, and can provide base public consistency dimensions.
The data summarization and storage layer comprises a theme domain providing a high-level view of the model, and the theme domain mainly comprises marketing, safe production, planning and construction, material management, financial management, human resources, information management and comprehensive management.
And a marketing theme domain is established, unified marketing management application is established, all links of electric power marketing management are covered comprehensively, the standardization of marketing business is promoted, and the electric power marketing business level and the service quality are improved.
The safety production subject domain takes the equipment as a core and realizes equipment and safety information management of the equipment, such as basic information, equipment maintenance, equipment operation monitoring, production operation, technical supervision, disaster prevention emergency management and the like.
And planning and constructing a theme domain, wherein the theme domain comprises power grid planning, comprehensive planning and statistical energy-saving management application.
The material management subject field comprises information such as demand planning, purchasing, inventory and warehousing delivery of material management and corresponding supplier information.
The subject domain of financial management builds intensive and integrated financial management application, and realizes the centralized management of accounting, capital, fund and budget.
The human resource theme domain establishes an integrated and full-coverage human resource management application, provides informatization support for various services of human resource management, and improves the intensive and specialized levels of human resource management.
Information management subject domain, enterprise architecture and information service, precipitating knowledge maps of various power supply offices
The comprehensive management subject domain comprises comprehensive information management of enterprises such as administrative offices, party group construction, inspection and supervision, auditing, legal affairs and the like.
The input end of the data analysis layer, referred to as DWS layer for short, is connected with the output end of the data collection and storage layer, and proper data collection is carried out on the basis of the data collection and storage layer or cross-subject and integrated pull-through data is carried out, so that preprocessing data are provided for the data processing layer.
The data analysis layer is driven by application requirements, determines the data use scenes of the service, such as common dimensions, granularity and the like, and then processes the data. For public, common and stable derivative/composite index data, precipitation is built at this level. The method is characterized in that moderate summarization is carried out around the use mode and the common dimensions and granularity, and a cross-subject integrated pull-through broad table oriented to business objects is provided; for example, a client panoramic view, an equipment panoramic view and the like, the model design of the layer needs to consider more convenience and query performance of data consumption, the attributes of common redundancy and same granularity in the model strengthen the dimensionality degradation of the index, and more broad-tabulated means are adopted to construct the public index, so that the reusability of the public index is improved, and repeated processing is reduced.
When the data analysis layer analyzes the data, firstly index disassembly is carried out, and then when modeling is carried out based on the disassembled index disassembly process elements, theme modeling, logic modeling and physical modeling are mainly included.
The index disassembly takes a report form or a business assessment index as input, and is performed according to an index specification definition framework to carry out decomposition, identify a composite index, a derivative index and an atomic index, and determine the business process of the atomic index.
Wherein: the atomic index data needs to keep the finest granularity of the source data of the service system and keep complete service attributes as much as possible so as to adapt to future requirement expansion; the atomic indexes are uniformly defined, constructed, managed and operated, and different synonyms, synonyms and different sources of the same name are avoided; the atomic indexes need to be issued on a control platform after being reviewed by a data center team, are shared according to the application range and the requirement of data security level, and are not allowed to be generated by self definition.
The derivative indexes are uniformly defined, constructed, managed and operated, and different synonyms and synonyms of the same name are avoided; one derivative index can only be derived from one atom index data, and modifiers and dimensions are derived from attributes or measures in the atom index data; the derived indexes need to be released after the evaluation of a product team is passed, are shared according to the application range and the requirement of data security level, and are not allowed to be generated by self definition.
The composite indexes are uniformly defined, constructed, managed and operated, and different synonyms and synonyms of the same name are avoided; and the data center team is required to be issued on the control platform after passing the review, and the data center team is shared according to the application range and the requirement of data security level, and is not allowed to be generated by self definition. The composite index may be calculated from one or more composite/derivative/atomic indices by a superposition formula. According to the level of superposition calculation, the composite index can be divided into a first-level composite index and a multi-level composite index, wherein the first-level composite index is as follows: directly calculating and generating the atomic index or the derivative index through a superposition formula; multi-stage composite indexes: and continuously performing superposition calculation on the primary composite index to generate a secondary composite index, a tertiary composite index and other multi-level composite indexes.
The theme modeling refers to a business domain and an EA framework, classifies and describes data information in the modes of induction, classification and the like of a business process, and embodies and forms a theme data model in the mode of explaining the business domain, the sub-theme domain and the content of the sub-theme domain. And the index disassembly process elements and the data structure of the data middlebox have clear corresponding relation, so that the rationality and feasibility of driving the data modeling of the middlebox by taking index disassembly as an entry point are reflected.
The logic data model obtained by logic modeling is a refinement of the subject data model and is composed of a group of entities and relations thereof, wherein the relations comprise factual entities, attributes in the entities, relations among the factual entities, the factual entities and dimensions; by adopting the theory and the method of dimensional modeling facing to the business process, the logic model has 4 key activities, the modeling process must be clear, and the business process, the statement granularity, the confirmation dimension and the confirmation fact are respectively selected.
In the logic model, the fact table model design needs to clearly embody and explain the following elements of the model: primary key, business foreign key, dimension, metric, other attributes. The primary key is used for declaring fact table data granularity (DWD fact table keeps the most detailed data granularity), and comprises a physical primary key (self-generated serial number) and a service primary key (service number); the business foreign key is a key value of an upstream business process and a downstream business process, and a data flow and a relation of a business domain can be established by identifying and reserving the business foreign key in the logic model; the dimension refers to identifying and specifying dimension fields in the fact table, is specifically subdivided into a common dimension (enterprise unified main data) and a specific dimension (business domain specific main data information or reference data), and plans different management and control requirements. Further investigating the association matching condition of the dimension field and the dimension table to ensure the availability of the fact table; metrics are metric fields that determine fact tables, typically value type fields, such as amount, quantity, etc. And (4) performing certain algorithm summarization (such as COUNT, SUM and the like) by using the measurement field, and obtaining corresponding index data.
The overall process of logic model construction comprises three key processes of index disassembly, model design, ETL implementation and the like, wherein the model design is the core of the whole warehouse delivery process and is positioned in the middle of the index disassembly and implementation landing and plays a role in starting and stopping.
The logic model reflects the attributes of the factual entities and the attributes in the entities which are specifically divided, and the relationship between the factual entities and the dimensions. The logic model design follows the dimensional modeling specification process: selecting a business process, declaring granularity, validating dimensions, validating facts.
Selecting a service process: a business process is an operational type activity that an enterprise completes. In the working link, the atomic index and the business process are identified through index disassembling work, and logic model modeling work is carried out on the business process. Selecting a business process, wherein information such as links, upstream and downstream business operations and the like of the business process in a business domain module needs to be investigated; meanwhile, the system information items of the business process, including the source system, the source table, the upstream and downstream data tables, the system operation steps, etc., need to be investigated to understand the complete business process.
Statement granularity: data granularity means what each row defining a logical model fact table represents, reflecting the degree of detail of the fact table, and the same fact table cannot be mixed with a plurality of different fact granularities. The data granularity needed by different data levels is different, and in DWD detail data rubbing, the logic model design requires the atomic level granularity of a fact table to be reserved so as to support the maximum applicability of the model.
Confirming the dimension: the problem to be solved by a dimension is "how do business personnel describe data from business process metric events? "fact tables should be decorated with a robust set of dimensions that bear all possible single-valued descriptors in each measurement environment
Confirming the fact: the metrics fields and types used to analyze each of the fact tables formed are determined. Facts of different granularity must be put in different fact tables, and typical facts are additive values, such as the amount of electric charges paid, the number of devices, and the like.
The fact table model is modeled according to a business process, a single business process is designed into a transaction fact table, and a source system can be combined into the same fact table under the condition of normal splitting (such as the splitting according to the main table and the extended table with the same data granularity) so as to reduce the association; in principle, the transaction fact table does not independently set a self-growth ID primary key, and a source table ID is adopted as the primary key; for a fact table integrating multiple sources by multiple transactions, a primary key is combined to be an SSID (SSID source system code) and an ID (SSID source system code) of the source table, a new field is suggested to be built as the primary key, and the value of the new field is SSID I ID; for multi-source data integration, the measurement units of the same kind of factual measurement are consistent. If the sales volume is the same, the sales volume is changed into 'ten thousand stations' and 'ten stations'; a plurality of sales amount measures, including 'million yuan' and 'yuan', all using 'yuan'; in principle virtually all fields in the table have to be processed for their null values. A. The metrics field is empty, e.g., number type default set to "0". Date type, start Date, default "1900-01-01"; the end date, defaulted to "4712-12-31". (ii) a B. Other attribute fields, if null, default to "SNULL" (source is null); all facts in a DWD/DWS fact table need to be consistent with the granularity of the table definition, and there cannot be multiple different granularities of facts in the same fact table. Such as the fact that the head and the implementation do not lie in one; try to break the non-additivity metric down into the additive metrics. If the achievement rate is equal to the achievement amount/the target amount, the achievement amount and the target amount are used as measurement, and the achievement rate is calculated on a report or a BI tool; when the atomic data fact table of DWD is designed, except the public dimension, excessive special dimensions should not be designed; considering convenience and performance in use, if the specific dimension is too much, whether dimension integration or dimension degradation is needed to be considered in the fact table; in principle the fact table does not allow for long text, large field attributes such as fault description, feedback reason description, etc.; the DWD layer is required to contain all facts and attributes related to business processes as much as possible; in the DWD layer fact table, common dimensions and subject-specific dimensions (including miscellaneous dimensions), IDs and names appear in pairs at the same time, for example, the IDs corresponding to the source-specific dimensions should also be stored in the facts as other attribute fields. The miscellaneous item dimension plays a Key value in a fact table; the fact table needs to be added with a technical audit field.
The dimension table in the dimension model should save the codes and IDs used by the source service system and the description information as much as possible; the dimension table should identify the code which can be identified/has the service meaning by the source service user as the service key; public dimension: if any dimension is used by a plurality of fact tables, the dimension is designed as a dimension table; if one dimension table is used by a plurality of subject fact tables, the dimension table is designed as a public dimension table; the dimension attributes are preferentially obtained from the public dimension table, then obtained from the subject special dimension table and then obtained from other non-main dimension tables; merging attribute hierarchies of the dimensions into a single dimension by adopting a star model mode, and flattening the dimension hierarchy; when the dimension surface layer level is assigned, if the lower level is empty, the data of the previous level is used for supplementing; the dimension table is used by a single domain and can be used as a special dimension when not belonging to a public dimension and a miscellaneous dimension; miscellaneous dimension, which is longitudinally combined through class coding and dimension record, and uniquely distinguishes dimension attributes through fast code types and fast code service keys; miscellaneous dimension-boolean value, it is proposed to normalize whether a field is: 1, yes, 0, no; the miscellaneous dimensionality-Boolean value, if the result value contains a plurality of values, the ETL scheduling script is used for monitoring and alarming; the data updating mechanism of the dimension table applies update & insert, so that truncate can not be written in the whole quantity, and the strong dependence of the fact table processing and the dimension table processing is reduced; the Master data dimension Table creation Process creates 3 tables: current table, snapshot table, and pull-up table.
The physical data model obtained by physical modeling considers various specific technical implementation factors on the basis of the logical data model to design a database architecture, thereby really realizing the storage of data in a database.
The goal of the physical data model is to specify how the logical data model is implemented with the database schema, as well as actually storing the data. The contents of the physical data model include determining all tables and columns, defining primary keys (business primary keys or technology primary keys) identifying unique objects, and defining foreign keys for determining relationships between tables (primary foreign key relationships may not be implemented in the physical library based on performance, etc.). And meanwhile, the distribution of the database instances is defined based on non-functional factors such as performance, data volume and the like.
The logical model physics refers to adjusting and perfecting the designed logical model so that the logical model can be landed in a specific data storage environment (such as a database). Physical implementation requirements are embodied based on LDM, and the physical implementation requirements include consideration of data mapping, storage, performance, auditing and the like. LDM and PDM should in principle be consistent in data structure.
The physical model implementation process generally includes the following operations.
a) Physical and chemical naming standard
The naming bias of the logic model expresses the business meaning, the development and use environment of the technical level need to be considered when the table is physically established, the physical naming is carried out according to the specification, the field use habit is considered, and the pinyin initial letters are temporarily used as the naming rules of the table names and the column names.
b) Field data type unification
c) Introducing elements required for data storage
The physical model ultimately needs to be implemented on a data storage environment, and therefore data storage related elements, such as tablespaces, partition keys, etc. for relational databases, need to be added. The logical deletion identifier is added in the physical system, and in principle, the application program only performs logical deletion of data. Data storage physics is not limited to relational databases. And selecting an applicable relational storage mode and a non-relational storage mode (such as a graph database and the like) according to requirements of data application scenes, data access characteristics and the like.
d) Introducing elements required for performance optimization
And overlapping non-functional attributes of the database, and adding element expressions such as indexes, views, library and table splitting, data replication, read-write separation and the like.
e) Introducing elements required by data processing
The database table after landing needs to be used by the ETL process, so that attribute fields related to the ETL process need to be added, such as adding pure physical audit fields (DW _ ETL _ Dt, DW _ Ins _ Tm, DW _ Upd _ Tm), and recording the data change process.
The physical model drawing is designed by a PowerDesigner tool, the model versions are uniformly managed through SVN control, and the models are filed to an SVN server; the physical model is established and deployed, and a DDL script is required to be exported from a PowerDesigner file of the SVN server;
when the physical model is drawn, the principle of 'from left to right and from top to bottom' is observed as much as possible: the most core model in each sub-topic is placed in the most central position and displayed on one screen as much as possible (the distribution among the models should be compact but cannot be overlapped); the correlation lines between the models cannot in principle cross; if the structure is a father-son structure, the structure needs to be placed in an up-down mode; if the relationship is the upstream-downstream relationship between the two, the two are placed in a left-middle-right mode according to the sequence of the business process; if the historical data model exists, the historical data model is placed on the right side of the current model; the attributes of the same type are arranged close to each other (visit according to the classification sequence of the main key, the foreign key, the dimensionality, the measurement and the extended attribute).
And the input end of the data processing layer, called ADS layer for short, is connected with the output end of the data analysis layer, extracts related data from the data gathering and storage layer or the data analysis layer, carries out loading and individual demand processing, and quickly supports the power supply system to gather analysis application construction of each business department. The data processing layer is completely constructed facing application requirements, individualized and frequently-changed derivative/composite index data are subjected to multi-level summarization and processing calculation, application-based data assembly is carried out, such as wide-table marketing, transverse-table-to-longitudinal-table, trend index strings and the like, and applications such as front-end report inquiry, analysis charts, instrument panels and the like are met. The construction of a data processing layer needs to consider more tables to directly meet the use of application functions, calculation during data access is reduced as much as possible, query performance is improved, facts are widened, and common fields are redundant.
Claims (10)
1. Power supply service data management system, its characterized in that: the system comprises a data access layer, a data gathering and storage layer, a data analysis layer and a data processing layer;
the data access layer is respectively connected with the peripheral data resource subsystem through the access ports, so that unified data access is realized, and the mutual influence of the transaction flow and the warehouse counting flow is shielded;
the input end of the data gathering and storage layer is connected with the output end of the data access layer, public data which is relatively complete and has atomic granularity is established, and data sharing and model stabilization and reuse are enhanced;
the input end of the data analysis layer is connected with the output end of the data collection and storage layer, appropriate data collection is carried out on the basis of the data collection and storage layer, or cross-theme and integrated pull-through data is carried out, and preprocessing data are provided for the data processing layer;
and the input end of the data processing layer is connected with the output end of the data analysis layer, relevant data are extracted from the data gathering and storage layer or the data analysis layer, loading and individual demand processing are carried out, and the power supply system is quickly supported to gather analysis application construction of each business department.
2. The power supply service data management system according to claim 1, characterized in that: the peripheral data resource subsystem comprises a regional power grid SCADA system, an electric energy acquisition system, a power distribution network automation system, a marketing MIS system, a power distribution network line loss system, a power distribution GIS system, a power distribution network bar graph system and a data interface server.
3. The power supply service data management system according to claim 1, characterized in that: the data summarization and storage layer construction is driven by service and data, and the model data of the layer can completely and comprehensively reflect relevant service process event information; the attribute of the atomic index fact table needs to reserve the field with the service attribute in the source system source table.
4. The power supply service data management system according to claim 3, characterized in that: the data summarization and storage layer comprises a theme domain providing a high-order view of the model, and the theme domain mainly comprises marketing, safe production, planning and construction, material management, financial management, human resources, information management and comprehensive management.
5. The power supply service data management system according to claim 1, characterized in that: when the data analysis layer analyzes the data, firstly index disassembly is carried out, and then when modeling is carried out based on the disassembled index disassembly process elements, theme modeling, logic modeling and physical modeling are mainly carried out.
6. The power supply service data management system according to claim 5, characterized in that: the index disassembling takes a report form or a business assessment index as input, and is performed according to an index specification definition framework, so that a composite index, a derivative index and an atomic index are identified, and a business process of the atomic index is determined.
7. The power supply service data management system according to claim 5, characterized in that: the subject modeling refers to a business domain and an EA framework, classifies and describes data information in the modes of induction, classification and the like of a business process, and embodies and forms a subject data model in the mode of explaining the business domain, the sub-subject domain and the content thereof.
8. The power supply service data management system according to claim 7, characterized in that: the logic data model obtained by the logic modeling is a refinement of the theme data model and is composed of a group of entities and relations thereof, including factual entities, attributes in the entities, and relations between the factual entities, the factual entities and dimensions.
9. The power supply service data management system according to claim 8, characterized in that: the physical data model obtained by the physical modeling is based on the logical data model, various specific technical implementation factors are considered, the database architecture design is carried out, and the storage of data in the database is really realized.
10. The power supply service data management system according to claim 1, characterized in that: the data analysis layer is driven by application requirements, determines the data use scene of the service, and builds public, common and stable derivative/composite index data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111018032.0A CN113723822A (en) | 2021-09-01 | 2021-09-01 | Power supply service data management system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111018032.0A CN113723822A (en) | 2021-09-01 | 2021-09-01 | Power supply service data management system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113723822A true CN113723822A (en) | 2021-11-30 |
Family
ID=78680311
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111018032.0A Pending CN113723822A (en) | 2021-09-01 | 2021-09-01 | Power supply service data management system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113723822A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114706575A (en) * | 2022-06-07 | 2022-07-05 | 杭州比智科技有限公司 | Method and system for migrating and multiplexing data model |
| CN116431638A (en) * | 2023-04-12 | 2023-07-14 | 浪潮智慧科技有限公司 | Index processing method, equipment and medium for water conservancy industry |
| CN117350520A (en) * | 2023-12-04 | 2024-01-05 | 浙江大学高端装备研究院 | Automobile production optimization method and system |
-
2021
- 2021-09-01 CN CN202111018032.0A patent/CN113723822A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114706575A (en) * | 2022-06-07 | 2022-07-05 | 杭州比智科技有限公司 | Method and system for migrating and multiplexing data model |
| CN116431638A (en) * | 2023-04-12 | 2023-07-14 | 浪潮智慧科技有限公司 | Index processing method, equipment and medium for water conservancy industry |
| CN116431638B (en) * | 2023-04-12 | 2024-03-12 | 浪潮智慧科技有限公司 | Index processing method, equipment and medium for water conservancy industry |
| CN117350520A (en) * | 2023-12-04 | 2024-01-05 | 浙江大学高端装备研究院 | Automobile production optimization method and system |
| CN117350520B (en) * | 2023-12-04 | 2024-02-27 | 浙江大学高端装备研究院 | Automobile production optimization method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10853387B2 (en) | Data retrieval apparatus, program and recording medium | |
| CN107766579B (en) | Method for designing main data management system based on XBRL standard | |
| CN113723822A (en) | Power supply service data management system | |
| Paim et al. | DWARF: An approach for requirements definition and management of data warehouse systems | |
| CN112396404A (en) | Data center system | |
| CN101794226B (en) | Service software construction method and system adapting to multiple business abstraction levels | |
| US9158831B2 (en) | Business intelligent architecture system and method | |
| CN113392227B (en) | Metadata knowledge graph engine system oriented to rail transit field | |
| CN101506804A (en) | Methods and apparatus for maintaining consistency during analysis of large data sets | |
| Dolk | Integrated model management in the data warehouse era | |
| CN105868373A (en) | Method and device for processing key data of power service information system | |
| CN112527774A (en) | Data center building method and system and storage medium | |
| Rutz et al. | Practical implications of real time business intelligence | |
| Batini et al. | A Framework And A Methodology For Data Quality Assessment And Monitoring. | |
| CN113127538B (en) | High-precision spare part demand prediction method | |
| CN114358812A (en) | Multi-dimensional power marketing analysis method and system based on operation and maintenance big data | |
| CN115496337A (en) | Data system for supporting brain of enterprise | |
| CN115936296A (en) | Production and manufacturing data cockpit system of discrete manufacturing enterprise based on industrial internet big data lake | |
| Ramanujan et al. | VESPER: Visual exploration of similarity and performance metrics for computer-aided design repositories | |
| KR101903530B1 (en) | Optimization diagnostic system of business and IT system | |
| US11216486B2 (en) | Data retrieval apparatus, program and recording medium | |
| Stefanovic | Designing OLAP multidimensional systems for supply chain management | |
| CN101231717A (en) | Method and system of multidimensional dynamic parameter cost segregation | |
| Nimmagadda et al. | Applicability of data warehousing and data mining technologies in the Australian resources industry | |
| Li et al. | An Integrated Data Analysis and Application Service Platform for Power Grid Engineering |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication |