CN113064866A - Power business data integration system - Google Patents
Power business data integration system Download PDFInfo
- Publication number
- CN113064866A CN113064866A CN202110226550.5A CN202110226550A CN113064866A CN 113064866 A CN113064866 A CN 113064866A CN 202110226550 A CN202110226550 A CN 202110226550A CN 113064866 A CN113064866 A CN 113064866A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- common analysis
- information
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000010354 integration Effects 0.000 title claims abstract description 32
- 238000004458 analytical method Methods 0.000 claims abstract description 74
- 230000005611 electricity Effects 0.000 claims abstract description 54
- 230000002159 abnormal effect Effects 0.000 claims abstract description 16
- 238000013179 statistical model Methods 0.000 claims abstract description 13
- 238000007781 pre-processing Methods 0.000 claims abstract description 4
- 238000006243 chemical reaction Methods 0.000 claims description 26
- 238000004140 cleaning Methods 0.000 claims description 13
- 238000001514 detection method Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 6
- 230000008859 change Effects 0.000 claims description 6
- 238000003860 storage Methods 0.000 claims description 5
- 238000013499 data model Methods 0.000 claims description 3
- 238000004064 recycling Methods 0.000 claims description 2
- 238000013480 data collection Methods 0.000 claims 1
- 238000012545 processing Methods 0.000 abstract description 5
- 238000000034 method Methods 0.000 description 16
- 230000008569 process Effects 0.000 description 9
- 230000008901 benefit Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000005192 partition Methods 0.000 description 3
- 238000011084 recovery Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 241000533950 Leucojum Species 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000003908 quality control method Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/116—Details of conversion of file system types or formats
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Human Resources & Organizations (AREA)
- Economics (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Marketing (AREA)
- Health & Medical Sciences (AREA)
- Entrepreneurship & Innovation (AREA)
- Tourism & Hospitality (AREA)
- Computer Security & Cryptography (AREA)
- Development Economics (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Game Theory and Decision Science (AREA)
- Human Computer Interaction (AREA)
- Educational Administration (AREA)
- Public Health (AREA)
- Water Supply & Treatment (AREA)
- General Health & Medical Sciences (AREA)
- Primary Health Care (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides an electric power service data integration system, which comprises a data acquisition module, a data processing module and a data processing module, wherein the data acquisition module is used for acquiring service data of a user, daily meter reading data, meter reading abnormal data and customer service data; the data integration module is used for calling the common analysis objects, integrating the common analysis objects according to a preset integration rule, performing cross-domain integration on data of a bottom base table related to each common analysis object, and determining the association relation of the common analysis objects; the data summarizing module is used for carrying out multi-dimensional summarization according to the pre-stored preprocessing strategy and the incidence relation of each common analysis object to generate a multi-dimensional statistical model of the common analysis objects; and the client full-dimensional information module is used for determining the incidence relation of all relevant information of the electricity consumption client in the accounting period according to the common analysis object multi-dimensional statistical model and generating a client full-dimensional information model. The invention gets through all the islands related to the electric charge accounting service and realizes the full-dimensional and all-around display based on the electricity consumption customers.
Description
Technical Field
The invention relates to the technical field of electric power, in particular to an electric power service data integration system.
Background
At present, when an electricity charge accounting service person is doing accounting service, because a marketing system model is huge and complex, service data are dispersed in different fields or system functions, when all information of a customer needs to be acquired, a plurality of function pages need to be checked repeatedly, even different service systems (for example, the marketing system only has monthly electricity, and daily electricity data need to be checked from a metering automation system), the integration degree of the electricity service data is low, the data cannot be effectively counted integrally, and the abnormal location which cannot be accurately positioned when electricity charge checking is performed is caused, so that the work efficiency of electricity charge checking is greatly influenced. Therefore, it is necessary to adopt effective technical means to integrate and uniformly model data dispersed in different fields or modules to form a customer-based full-dimensional information model.
Disclosure of Invention
The invention aims to provide an electric power service data integration system, which solves the technical problems that the integration degree of electric power service data is low, the data cannot be effectively subjected to integral statistics, and the working efficiency of electric charge checking is influenced in the conventional method.
To achieve the above object, an embodiment of the present invention provides an electric power service data integration system, including:
the data acquisition module is used for acquiring business data, daily meter reading data, meter reading abnormal data and customer service data of a user from a plurality of data sources through a data interface;
the data integration module is used for calling common analysis objects from the business data, the daily meter reading data, the meter reading abnormal data and the customer service data, integrating the common analysis objects according to a preset integration rule, and performing cross-domain integration on data of a bottom base table related to each common analysis object to obtain a large-width table of each common analysis object; determining the incidence relation of the common analysis objects according to the broad table of the common analysis objects; wherein, the common analysis objects at least comprise electricity utilization customers, electricity utilization capacity, electricity selling quantity, electricity charge price, customer service work orders, customer service seats, telephone traffic records, business expansion work orders and business expansion matching items;
the data summarization module is used for performing multi-dimensional summarization according to the pre-stored preprocessing strategy and the incidence relation of each common analysis object and outputting basic statistical indexes and service themes; generating a common analysis object multi-dimensional statistical model according to the basic statistical indexes and the business theme;
and the client full-dimensional information module is used for determining the incidence relation of all relevant information of the electricity consumption client in the accounting period according to the common analysis object multi-dimensional statistical model and generating a client full-dimensional information model.
Preferably, the system further comprises a cache module, configured to store business data, daily meter reading data, abnormal meter reading data, and customer service data of the user collected from multiple data sources; storing a common analysis object multi-dimensional statistical model output by a data summarization module; and storing the client full-dimensional information model generated by the client full-dimensional information module.
Preferably, the service data of the user at least comprises: user files, business expansion work order basic information, meter change information, meter reading information, metering point relation, transformer information, calculation transformer change information, calculation transformer compensation capacity information, access electric quantity record, access electric quantity detail, metering point transformer relation, metering point electric quantity, quantity price detail and recheck work order information.
Preferably, the data acquisition module is further configured to analyze the service data, the daily meter reading data, the abnormal meter reading data, and the customer service data, and store the data according to a preset storage rule.
Preferably, the data summarization module further comprises:
the data cleaning module is used for identifying the same entity data record of each common analysis object from different data sources according to a preset target data conversion rule and carrying out consistency detection on the common analysis objects which have the same entity data but come from different data sources;
the code conversion module is used for detecting related data tables and field values related to code definition and code value in common analysis objects according to data dimensions and preset code value definition standards; determining the incidence relation between the record of the common analysis object and the code according to the detection result;
the multi-table merging module is used for correlating the plurality of service tables according to the correlation fields and realizing random exchange of rows and columns through merging of the plurality of service tables; and splitting the associated field to generate the stretching relation information of the wide table.
Preferably, the data cleaning module is further configured to generate difference information between the same common analysis object in different data sources according to the consistency detection result, and generate data quality information;
and according to a preset conversion rule, performing type conversion or value conversion on the common analysis object according to the data quality information to form intermediate data which can be spliced and automatically mapped.
Preferably, the code conversion module is further configured to detect whether the standard coded data value is within a preset reasonable value range, and when the standard coded data value is not within the preset reasonable value range, determine that an illegal coded value occurs, and generate a data problem list and a corresponding scheme; and identifying a newly added value in the dynamic code, and updating the corresponding code definition when the newly added dynamic code value is identified.
Preferably, the client full-dimension information module further comprises:
the CUB cube module is used for generating core data cube data and scheduling related computing resources to execute cube computation and summary statistics according to a preset core data cube and a core data model; storing the cube data in a temporary cube table, comparing the new cube data with the existing cube data, and generating cube updates according to the difference result;
the data loading module is used for loading the data output by the code conversion module, the data cleaning model and the CUB cube module into a temporary target data table, comparing the difference between the data in the temporary target data table and the data in the existing target data table, and generating updating information of the existing data table;
a parallel scheduling module: the parallel scheduling method is used for performing parallel scheduling on the code conversion module, the data cleaning model, the CUB cube model and the data loading module according to the task execution duration through a preset scheduling rule.
Preferably, the client full-dimensional information module is further configured to input a service or service application scenario required by the electricity consumer into the client full-dimensional information model, and output all relevant information in the process from electricity collection to electricity charge recycling for each bill of the electricity consumer.
In summary, the embodiment of the invention has the following beneficial effects:
the electric power service data integration system establishes a full data link from electric quantity acquisition to electric charge recovery for each bill of each electricity consumer, provides a clear view of the full life cycle of the electric charge bill for service personnel for checking and receiving, and simultaneously provides complete information support for handling electric quantity and electric charge customer complaints; the method takes a power consumption client as a center, and puts through all isolated islands related to power consumption accounting services around client archive information, monthly electric quantity and power consumption information, business expansion service information, customer service demand information, power consumption rechecking information, meter reading information, real-time power consumption load and other information, so that full-dimensional and all-dimensional display based on the power consumption client is realized.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is within the scope of the present invention for those skilled in the art to obtain other drawings based on the drawings without inventive exercise.
Fig. 1 is a schematic structural diagram of an electric power service data integration system according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating a client full-dimensional information model according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings.
Fig. 1 is a schematic diagram of an embodiment of a power service data integration system according to the present invention. In this embodiment, the method includes:
the data acquisition module is used for acquiring business data, daily meter reading data, meter reading abnormal data and customer service data of a user from a plurality of data sources through a data interface; it can be understood that the collection, analysis and storage of the text file are realized by using flash (log collection system) for real-time collection and transmission through Kfaka. Data is collected from a data source and the collected data is sent to a designated destination. In order to ensure that the transmission process is successful, the data is cached before being sent to the destination, and the cached data is deleted after the data really reaches the destination.
In a specific embodiment, data of each service system (or other related systems or databases) such as a marketing management system, a client comprehensive system, a metering automation system and the like is extracted in a batch synchronization, incremental updating and other manners, and all data are stamped. For example, the following business data can be obtained from the marketing management system; the method specifically comprises the following steps: user files, business expansion work order basic information, meter change information, meter reading information, metering point relation, transformer information, accounting transformer change information, accounting transformer compensation capacity information, access electric quantity records, access electric quantity details, metering point transformer relation, metering point electric quantity, quantity price details, recheck work order information and the like. Daily meter reading data and abnormal meter reading data of a public transformer user and a private transformer user can be obtained from a metering automation system and are used for assisting electric quantity abnormal analysis; the client appeal and client service data can be obtained from the client omnibearing system.
Specifically, after data acquisition, the service data, the daily meter reading data and the abnormal meter reading data need to be analyzed, and text files are acquired, analyzed and stored in a warehouse by transmitting through a Kfaka technology, specifically, a producer (data source) sends a message to a Kafka cluster, and classifies (Topic) the message, for example, two producers send a message classified as Topic1 for one of the producers and a message classified as Topic2 for the other producer, the Topic is a theme, the message can be classified by specifying the theme to the message, and a consumer can only pay attention to the message in the required Topic; the consumer continuously pulls messages from the cluster by establishing long connections with the kafka cluster, which can then be processed. The customer service data are stored according to a preset storage rule; when storing, when creating a topic, the number of partitions can be specified at the same time, the more the number of partitions is, the greater the throughput is, but the more resources kafka are needed, after receiving the message sent by the producer, the message can be stored in different partitions according to the balancing strategy. The method has the advantages that the centralization of internal data resources of enterprises such as marketing, customer omnibearing and metering is realized through batch offline data acquisition, batch real-time data acquisition and real-time flow data acquisition, and the preparation is made for the next step of data integration; and keeping consistent with the data of each source end service system, and not integrating and processing the data.
The data integration module is used for calling common analysis objects from business data, daily meter reading data, meter reading abnormal data and customer service data, integrating the common analysis objects according to preset integration rules, performing data cross-domain integration on a bottom base table related to each common analysis object to obtain a wide table of each common analysis object, wherein the wide table refers to a database table in which indexes, dimensions and attributes related to business topics are associated together, the efficiency problem in iterative computation in the data mining model training process can be greatly improved by placing related fields in the same table, it can be understood that data types contained in the common analysis objects are identified and stored in a classified manner through the data acquisition module, so that the indexes, dimensions and attributes related to each classified data can be clearly known, and a wide table for recording each data is obtained, the association relation of a certain item of data can be directly called, for example, information such as power consumption capacity, power selling amount, power charge and power price of a certain power consumer needs to be acquired, through a field corresponding to the power consumer, the information such as the power consumption capacity, the power selling amount, the power charge and power price can be acquired only by calling the data associated with the field, and then the corresponding information can be acquired by screening the dimensionality (for example, the data with the dimensionality being the power consumption capacity is called when the power consumption capacity is acquired); the incidence relation of the common analysis objects can be determined according to the broad table of the common analysis objects by the process; wherein, the common analysis objects at least comprise electricity utilization customers, electricity utilization capacity, electricity selling quantity, electricity charge price, customer service work orders, customer service seats, telephone traffic records, business expansion work orders and business expansion matching items; it can be understood that the method is mainly used for offline storage of mass historical data on the basis of data caching, and meanwhile, centralization of processes of analysis object integration, data standard unification, data quality control and the like is achieved. The wide table stores different contents in the same table, and the wide table is not in accordance with the model design specification of the three-norm form, so that the advantage is the improvement and convenience of the query performance.
In a specific embodiment, the integration of common analysis objects is performed according to the theme of the marketing and distribution data application. The data integration layer performs cross-domain integration on the bottom base table related to each analysis object according to the commonly used analysis objects to form a wide table of the analysis objects such as the electricity utilization customers, the electricity utilization capacity, the electricity selling amount and the like, and the wide table contains all commonly used analysis dimension information of the data analysis objects; and reconstructing data by using the subdivision relation and the association relation of the business main body, the business process and the business object.
The data summarization module is used for performing multi-dimensional summarization according to the pre-stored preprocessing strategy and the incidence relation of each common analysis object and outputting basic statistical indexes and service themes; generating a common analysis object multi-dimensional statistical model according to the basic statistical indexes and the business theme; it can be understood that, by obtaining the association relationship by the data integration module, the data multi-dimensional summarization and the centralization of the calculation process can be realized, for example, the electricity consumption users are summarized according to the dimension of the electricity charge and electricity price (for example, the summarization standard of the electricity charge and electricity price is 1, 2, 3 yuan, etc.), and the information of the electricity consumption users with the electricity price of 1, 2, 3 can be obtained through the association relationship, so as to obtain summarized data with the electricity charge and electricity price as the dimension; for another example, the quantity indexes of the customer service worksheets can be summarized in each level according to the time dimension to levels of 15 minutes, hours, days, weeks, ten days, months, seasons, half years, years and the like; similarly, summary data between any two parameters in each common analysis object can be obtained, such as electricity consumption client-electricity capacity, electricity consumption client-telephone traffic record, and the like. Forming a centralized basic statistical index and a theme system (any parameter is a theme, and the corresponding parameter can form the basic statistical index, and the statistical data corresponding to all the parameters form a data system); the statistical models of analysis objects such as electricity sales volume, electricity metering charge, customer service worksheets, electricity customers and the like are built in a modeling mode of a multidimensional model such as a star model and the like on the basis of data detail after data cleaning and conversion, for example, any topic (one parameter in a common analysis object) is selected by determining the statistical data, and other related common analysis objects form a statistical model (the star model is the most direct embodiment, the determined parameter is taken as the center, other parameters are scattered around the star model, and the related form can be expressed as other models such as a snowflake model, a constellation model and the like) similarly, so that the summary statistics of all common dimensions is carried out facing to the data analysis topic. When the work order is inquired or analyzed according to different time and units, the work order quantity index which corresponds to the time granularity and is summarized can be directly inquired, data re-filtering, screening and summarizing calculation are not needed, and response efficiency is improved.
Specifically, the data summarization module further comprises: the data cleaning module is used for identifying the same entity data record of each common analysis object from different data sources according to a preset target data conversion rule and carrying out consistency detection on the common analysis objects which have the same entity data but come from different data sources; the device is also used for generating difference information between the same common analysis object in different data sources according to the consistency detection result and generating data quality information; and according to a preset conversion rule, performing type conversion or value conversion on the common analysis object according to the data quality information to form intermediate data which can be spliced and automatically mapped.
The code conversion module is used for detecting related data tables and field values related to code definition and code value in common analysis objects according to data dimensions and preset code value definition standards; determining the incidence relation between the record of the common analysis object and the code according to the detection result; the device is also used for detecting whether the standard coded data value is in a preset reasonable value range, judging that an illegal coded value occurs when the standard coded data value is not in the preset reasonable value range, and generating a data problem list and a corresponding scheme; and identifying a newly added value in the dynamic code, and updating the corresponding code definition when the newly added dynamic code value is identified.
A multi-table merging module, configured to associate multiple service tables according to the association fields, and implement arbitrary exchange of rows and columns by merging the multiple service tables, for example, using a service as a driver to associate 2 or more tables according to the association fields in the forms of left connection, right connection, full connection, internal connection, and the like, so as to implement arbitrary exchange of rows and columns; and splitting the associated field to generate the stretching relation information of the wide table, thereby realizing the stretching function of the wide table.
And the client full-dimensional information module is used for determining the incidence relation of all relevant information of the electricity consumption client in the accounting period according to the common analysis object multi-dimensional statistical model and generating a client full-dimensional information model as shown in fig. 2. It can be understood that data fusion and sharing are performed facing to business and application scenes, value integration of data is achieved, and data service capability opening and business service capability opening of a marketing and distribution data mart are completed.
In a specific embodiment, the client full-dimensional information module is further used for inputting the required service or service application scene of the electricity utilization user into a client full-dimensional information model, and outputting all relevant information in the process from electricity collection to electricity charge recovery of each bill of the electricity utilization client; it can be understood that, from the analysis perspective, a user generally only needs to pay attention to the business domain topic analysis service provided by the data mart, and from the application perspective, the user can select different types of data products of the same topic domain according to a specific application scenario.
Specifically, the client full-dimension information module further includes: the CUB cube module is used for generating core data cube data and scheduling related computing resources to execute cube computation and summary statistics according to a preset core data cube and a core data model; storing the cube data in a temporary cube table, comparing the new cube data with the existing cube data, and generating cube updates according to the difference result; and generating a cube updating script according to the difference analysis result, and scheduling related computing resources to execute a cube updating task.
The data loading module is used for loading the data output by the code conversion module, the data cleaning model and the CUB cube module into a temporary target data table, comparing the difference between the data in the temporary target data table and the data in the existing target data table, and generating updating information of the existing data table; and taking the target model as a drive, loading the source data subjected to the dimension conversion and the data cleaning and the CUB cube into a temporary target data table, comparing the difference between the data in the temporary target data table and the data in the existing target data table, and automatically generating an execution script for updating the existing data table according to the difference comparison result. The system schedules related computing resources to distribute script execution tasks to different processing units for execution, records execution results, and processes various exceptions in the execution process.
A parallel scheduling module: the system comprises a code conversion module, a data cleaning module, a CUB cube module and a data loading module, wherein the code conversion module, the data cleaning module, the CUB cube module and the data loading module are used for carrying out parallel scheduling according to task execution duration through a preset scheduling rule; and optimizing according to the task execution duration, and scheduling in a parallel mode to shorten the whole operation period of the scheduling task.
The cache module is used for storing business data, daily meter reading data, meter reading abnormal data and customer service data of a user, which are collected from a plurality of data sources; storing a common analysis object multi-dimensional statistical model output by a data summarization module; and storing the client full-dimensional information model generated by the client full-dimensional information module.
In summary, the embodiment of the invention has the following beneficial effects:
the electric power service data integration system provided by the invention establishes a full data link from electric quantity acquisition to electric charge recovery aiming at each bill of each electricity consumer, provides a clear full life cycle view of the electric charge bill for service personnel for reading and checking and receiving, and simultaneously provides complete information support for processing electric quantity and electric charge customer complaints.
The method takes a power consumption client as a center, and puts through all isolated islands related to power consumption accounting services around client archive information, monthly electric quantity and power consumption information, business expansion service information, customer service demand information, power consumption rechecking information, meter reading information, real-time power consumption load and other information, so that full-dimensional and all-dimensional display based on the power consumption client is realized.
The above disclosure is only for the purpose of illustrating the preferred embodiments of the present invention, and it is therefore to be understood that the invention is not limited by the scope of the appended claims.
Claims (9)
1. An electric power service data integration system, comprising:
the data acquisition module is used for acquiring business data, daily meter reading data, meter reading abnormal data and customer service data of a user from a plurality of data sources through a data interface;
the data integration module is used for calling common analysis objects from the business data, the daily meter reading data, the meter reading abnormal data and the customer service data, integrating the common analysis objects according to a preset integration rule, and performing cross-domain integration on data of a bottom base table related to each common analysis object to obtain a large-width table of each common analysis object; determining the incidence relation of the common analysis objects according to the broad table of the common analysis objects; wherein, the common analysis objects at least comprise electricity utilization customers, electricity utilization capacity, electricity selling quantity, electricity charge price, customer service work orders, customer service seats, telephone traffic records, business expansion work orders and business expansion matching items;
the data summarization module is used for performing multi-dimensional summarization according to the pre-stored preprocessing strategy and the incidence relation of each common analysis object and outputting basic statistical indexes and service themes; generating a common analysis object multi-dimensional statistical model according to the basic statistical indexes and the business theme;
and the client full-dimensional information module is used for determining the incidence relation of all relevant information of the electricity consumption client in the accounting period according to the common analysis object multi-dimensional statistical model and generating a client full-dimensional information model.
2. The system of claim 1, further comprising a cache module for storing business data, daily meter reading data, abnormal meter reading data, and customer service data collected from a plurality of data sources; storing a common analysis object multi-dimensional statistical model output by a data summarization module; and storing the client full-dimensional information model generated by the client full-dimensional information module.
3. The system of claim 2, wherein the user's traffic data includes at least: user files, business expansion work order basic information, meter change information, meter reading information, metering point relation, transformer information, calculation transformer change information, calculation transformer compensation capacity information, access electric quantity record, access electric quantity detail, metering point transformer relation, metering point electric quantity, quantity price detail and recheck work order information.
4. The system of claim 3, wherein the data collection module is further configured to parse the service data, the daily meter reading data, the abnormal meter reading data, and the customer service data, and store the service data according to a preset storage rule.
5. The system of claim 4, wherein the data summarization module further comprises:
the data cleaning module is used for identifying the same entity data record of each common analysis object from different data sources according to a preset target data conversion rule and carrying out consistency detection on the common analysis objects which have the same entity data but come from different data sources;
the code conversion module is used for detecting related data tables and field values related to code definition and code value in common analysis objects according to data dimensions and preset code value definition standards; determining the incidence relation between the record of the common analysis object and the code according to the detection result;
the multi-table merging module is used for correlating the plurality of service tables according to the correlation fields and realizing random exchange of rows and columns through merging of the plurality of service tables; and splitting the associated field to generate the stretching relation information of the wide table.
6. The system of claim 5, wherein the data cleansing module is further configured to generate difference information between the same common analysis object in different data sources according to the consistency detection result, and generate data quality information;
and according to a preset conversion rule, performing type conversion or value conversion on the common analysis object according to the data quality information to form intermediate data which can be spliced and automatically mapped.
7. The system of claim 6, wherein the code conversion module is further configured to detect whether the standard coded data value is within a preset reasonable value range, and when detecting that the standard coded data value is not within the preset reasonable value range, determine that an illegal coded value occurs, and generate a data problem list and a corresponding scheme; and identifying a newly added value in the dynamic code, and updating the corresponding code definition when the newly added dynamic code value is identified.
8. The system of claim 7, wherein the customer full-dimension information module further comprises:
the CUB cube module is used for generating core data cube data and scheduling related computing resources to execute cube computation and summary statistics according to a preset core data cube and a core data model; storing the cube data in a temporary cube table, comparing the new cube data with the existing cube data, and generating cube updates according to the difference result;
the data loading module is used for loading the data output by the code conversion module, the data cleaning model and the CUB cube module into a temporary target data table, comparing the difference between the data in the temporary target data table and the data in the existing target data table, and generating updating information of the existing data table;
and the parallel scheduling module is used for performing parallel scheduling on the code conversion module, the data cleaning module, the CUB cube module and the data loading module according to the task execution duration through a preset scheduling rule.
9. The system of claim 8, wherein the customer full-dimensional information module is further configured to input a desired service or service application scenario of the electricity consumer into the customer full-dimensional information model, and output all relevant information from electricity collection to electricity charge recycling for each bill of the electricity consumer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110226550.5A CN113064866B (en) | 2021-03-02 | 2021-03-02 | Power business data integration system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110226550.5A CN113064866B (en) | 2021-03-02 | 2021-03-02 | Power business data integration system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113064866A true CN113064866A (en) | 2021-07-02 |
| CN113064866B CN113064866B (en) | 2024-03-01 |
Family
ID=76559464
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110226550.5A Active CN113064866B (en) | 2021-03-02 | 2021-03-02 | Power business data integration system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113064866B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113342798A (en) * | 2021-07-07 | 2021-09-03 | 广东电网有限责任公司 | Data management system based on data fusion |
| CN113627821A (en) * | 2021-08-25 | 2021-11-09 | 深圳供电局有限公司 | Method and system for identifying abnormal electricity utilization based on electricity utilization behavior characteristics |
| CN113743894A (en) * | 2021-08-30 | 2021-12-03 | 深圳供电局有限公司 | Method and system for establishing rechecking rule model for rechecking electric bill |
| CN113761036A (en) * | 2021-09-07 | 2021-12-07 | 国网福建省电力有限公司经济技术研究院 | Power grid statistics professional index multidimensional association table query display method |
| CN114267124A (en) * | 2021-12-22 | 2022-04-01 | 深圳供电局有限公司 | Charging information processing method and system |
| CN115858372A (en) * | 2022-12-19 | 2023-03-28 | 上海浦东发展银行股份有限公司 | OLAP system-based batch data construction and automatic verification method and system |
| CN116450717A (en) * | 2023-03-14 | 2023-07-18 | 深圳市英捷软件有限公司 | Data integration method and information management system for cross-service modules |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104915764A (en) * | 2015-05-29 | 2015-09-16 | 中国南方电网有限责任公司 | Electric power marketing management system based on client-user-account model and application thereof |
| CN106777244A (en) * | 2016-12-27 | 2017-05-31 | 国网浙江象山县供电公司 | A kind of power customer electricity consumption behavior analysis method and system |
| CN109669934A (en) * | 2018-12-11 | 2019-04-23 | 江苏瑞中数据股份有限公司 | A kind of data warehouse and its construction method suiting electric power customer service |
| CN110163621A (en) * | 2018-02-10 | 2019-08-23 | 广州供电局有限公司 | A kind of electric power customer service big data DSS |
| US20200311326A1 (en) * | 2019-03-29 | 2020-10-01 | Bert Labs Private Limited | Self-service artificial intelligence platform leveraging data-based and physics-based models for providing real-time controls and recommendations |
| CN112395361A (en) * | 2020-12-04 | 2021-02-23 | 华南理工大学 | Electric power engineering construction block chain management system |
-
2021
- 2021-03-02 CN CN202110226550.5A patent/CN113064866B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104915764A (en) * | 2015-05-29 | 2015-09-16 | 中国南方电网有限责任公司 | Electric power marketing management system based on client-user-account model and application thereof |
| CN106777244A (en) * | 2016-12-27 | 2017-05-31 | 国网浙江象山县供电公司 | A kind of power customer electricity consumption behavior analysis method and system |
| CN110163621A (en) * | 2018-02-10 | 2019-08-23 | 广州供电局有限公司 | A kind of electric power customer service big data DSS |
| CN109669934A (en) * | 2018-12-11 | 2019-04-23 | 江苏瑞中数据股份有限公司 | A kind of data warehouse and its construction method suiting electric power customer service |
| US20200311326A1 (en) * | 2019-03-29 | 2020-10-01 | Bert Labs Private Limited | Self-service artificial intelligence platform leveraging data-based and physics-based models for providing real-time controls and recommendations |
| CN112395361A (en) * | 2020-12-04 | 2021-02-23 | 华南理工大学 | Electric power engineering construction block chain management system |
Non-Patent Citations (1)
| Title |
|---|
| 周文琼: ""大数据环境下的电力客户服务数据分析系统"", 《计算机系统应用》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113342798A (en) * | 2021-07-07 | 2021-09-03 | 广东电网有限责任公司 | Data management system based on data fusion |
| CN113627821A (en) * | 2021-08-25 | 2021-11-09 | 深圳供电局有限公司 | Method and system for identifying abnormal electricity utilization based on electricity utilization behavior characteristics |
| CN113743894A (en) * | 2021-08-30 | 2021-12-03 | 深圳供电局有限公司 | Method and system for establishing rechecking rule model for rechecking electric bill |
| CN113761036A (en) * | 2021-09-07 | 2021-12-07 | 国网福建省电力有限公司经济技术研究院 | Power grid statistics professional index multidimensional association table query display method |
| CN114267124A (en) * | 2021-12-22 | 2022-04-01 | 深圳供电局有限公司 | Charging information processing method and system |
| CN115858372A (en) * | 2022-12-19 | 2023-03-28 | 上海浦东发展银行股份有限公司 | OLAP system-based batch data construction and automatic verification method and system |
| CN115858372B (en) * | 2022-12-19 | 2024-04-26 | 上海浦东发展银行股份有限公司 | Batch data construction and automatic verification method and system based on OLAP system |
| CN116450717A (en) * | 2023-03-14 | 2023-07-18 | 深圳市英捷软件有限公司 | Data integration method and information management system for cross-service modules |
| CN116450717B (en) * | 2023-03-14 | 2023-10-20 | 深圳市英捷软件有限公司 | Data integration method and information management system for cross-service modules |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113064866B (en) | 2024-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113064866B (en) | Power business data integration system | |
| CN109767255B (en) | Method for realizing intelligent operation and accurate marketing through big data modeling | |
| CN104820670B (en) | A kind of acquisition of power information big data and storage method | |
| CN105005570B (en) | Magnanimity intelligent power data digging method and device based on cloud computing | |
| CN109582667A (en) | A kind of multiple database mixing storage method and system based on power regulation big data | |
| CN110334274A (en) | Information-pushing method, device, computer equipment and storage medium | |
| CN108280541A (en) | Customer service strategies formulating method, device based on random forest and decision tree | |
| CN108388955A (en) | Customer service strategies formulating method, device based on random forest and logistic regression | |
| CN115934680B (en) | One-stop big data analysis processing system | |
| CN109582717A (en) | A kind of database unified platform and its read method towards electric power big data | |
| CN111026801A (en) | Method and system for assisting operation quick decision-making work of insurance type e-commerce | |
| CN112131203A (en) | Method and system for building data warehouse | |
| CN115423289A (en) | Intelligent plate processing workshop data processing method and terminal | |
| CN112632405A (en) | Recommendation method, device, equipment and storage medium | |
| CN111126852A (en) | BI application system based on big data modeling | |
| CN113642299A (en) | One-key generation method based on power grid statistical form | |
| CN111127099A (en) | E-commerce user analysis system based on big data and analysis method thereof | |
| Hu | E-commerce big data computing platform system based on distributed computing logistics information | |
| US20040139035A1 (en) | System and method for integration of value-added product costs | |
| CN117573646A (en) | Data management method and system based on dimension modeling | |
| CN111581302A (en) | Decision-making assisting system based on data warehouse | |
| CN116089490A (en) | Data analysis method, device, terminal and storage medium | |
| CN101140644A (en) | INTEDNET professional technology qualification reporting system | |
| CN115170196A (en) | Popularization method based on big data intelligent writing | |
| CN114218216A (en) | Resource management method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |