CN109542960A - A kind of data analysis domain system - Google Patents
A kind of data analysis domain system Download PDFInfo
- Publication number
- CN109542960A CN109542960A CN201811215551.4A CN201811215551A CN109542960A CN 109542960 A CN109542960 A CN 109542960A CN 201811215551 A CN201811215551 A CN 201811215551A CN 109542960 A CN109542960 A CN 109542960A
- Authority
- CN
- China
- Prior art keywords
- data
- layer
- module
- storage
- domain system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/06—Electricity, gas or water supply
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
- Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
- Y04S10/00—Systems supporting electrical power generation, transmission or distribution
- Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
Abstract
A kind of data analysis domain system characterized by comprising data active layer, data access layer, data store computation layer and united analysis service layer.
Description
Technical field
The present invention relates to a kind of data to analyze domain system.
Background technique
State's net covers eastern electric power and passes through SG186 engineering and SG-ERP System Construction, is completed integral information systems, big data
The construction of platform realizes unified data sharing, has accumulated magnanimity production run data.But with the continuous hair of technology
Exhibition gradually exposes the problems such as data sharing is insufficient, data extract repeatedly, data analysis support is inadequate.
Summary of the invention
In order to solve the above technical problems, the invention proposes a kind of data to analyze domain system characterized by comprising number
According to active layer, data access layer, data store computation layer and united analysis service layer.
Detailed description of the invention
Fig. 1 is the structure chart of data analysis domain system of the invention.
Fig. 2 is the flow chart of data integrating method of the invention.
Fig. 3 is the specific embodiment displaying interface that data analysis domain system of the invention is applied to business event application platform
Figure.
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, the present invention will be made further in conjunction with attached drawing
Detailed description.This description is to describe specific implementation consistent with the principles of the present invention by way of example, and not limitation
Mode, the description of these embodiments is detailed enough, so that those skilled in the art can practice the present invention, is not being taken off
Other embodiments can be used in the case where from scope and spirit of the present invention and can change and/or replace each element
Structure.Therefore, the following detailed description should not be understood from restrictive sense.
The present invention provides a kind of data to analyze domain system characterized by comprising data active layer, data access layer, number
According to storage computation layer and united analysis service layer, the data active layer is used to obtain data, including structural data obtains module,
Unstructured data obtains module and monitoring acquisition data acquisition module, and the data access layer is used for data from external data
Source imports data and analyzes domain system, including structural data AM access module, acquires measurement data AM access module and unstructured
Data access module, the data storage computation layer are used to provide the carrier that data are centrally stored and inquire, including data buffering
Module, data warehouse module, data package module and data mart module, the united analysis service layer is for providing unification
Data service and show service, including data-interface service module, data mining service module and self-service Analysis Service
Module.
According to the present invention, the data warehouse module includes detailed data layer and slightly summarizes layer, the detailed data layer
For storing the detailed data for forming the data of data active layer storage after coding unification, the data of same source, data normalization,
The layer that slightly summarizes summarizes the data of detailed data layer progress data light weight and/or light weight merging treatment for store
Data afterwards.Data in the data mart module are formed after processing conversion by the data of data warehouse, are used to support
The demand of united analysis service.The data package module includes the component calculated for data with data storage, the data
Computation module includes stream calculation component, memory computation module, batch computation module and inquiry computation module；The data storage group
Part includes hot spot data storage assembly, real-time data acquisition component and distributed file system component.
According to the present invention, the data access layer and data storage computation layer include: processor and memory, wherein are deposited
Reservoir is for storing computer program, and processor is used to execute the computer program of the memory storage, so that the data
Access layer and data storage computation layer cooperation execute following data integrated approach: as shown in Fig. 2, step A100, is taken out by data
It takes component or database synchronization copy mode to access, and stores to the data cache module；Step A200, by data buffer storage mould
Block number is drawn into data warehouse detailed data layer according to according to unified data model standard；Step A300, by data warehouse detail layer
Data pick-up slightly summarizes layer to data warehouse, is stored by the data model of data warehouse；Step A400, according to granularity need
It asks, data is polymerize according to the data model of Data Mart, store into Data Mart；Step A500, with Data Mart
In data based on, support united analysis service.
Further, the structural data obtains module for obtaining the relevant structural data of power grid, the structure
Changing data includes: grid operation data, production status data and producers' data.The unstructured data obtains module and uses
In obtaining the relevant unstructured data of power grid, the unstructured data includes audio-video, picture and file.The monitoring is adopted
For collection data acquisition module for obtaining the relevant monitoring acquisition data of power grid, the monitoring acquisition data include real-time voltage, real
When electric current and real time load.
Further, the united analysis service layer also provides for data routing and data gateway service.
Further, the data active layer is ephemeral data memory block, and data generally store 3 months, and overdue data will be into
The regular Gui Cang of row is simultaneously removed.
Further, the data of the data set urban district storage are mainly subject analysis data, the week of online data storage
Phase is generally longer, and data generally save 5 years.Data are after saving a period of time, it is contemplated that the bearing capacity of database, data
It should be filed.
In addition, system can also include:
It acquires metric data memory block: the data storage methods such as distributed column storage, distributed memory storage is provided, it is full
Sufficient service application is to the requirement such as high concurrent read-write, high efficiency storage, high scalability of mass data.Wherein, column storage is main
For providing the read and write access compared with low latency, the access request of high concurrent is born；It is real-time that memory storage is mainly used in high-performance
Query analysis scene.
Distributed file system: it is mainly used for storing unstructured data in analysis domain, using the data of distributed document
Storage architecture, PB grades of support or more scale unstructured data on-line storage, including unstructured data, semi-structured data,
Meet a large amount of, enriched data low cost storage demand.With the characteristic that write-once is repeatedly inquired, concurrency control can be reduced
System require, simplify data aggregate, support high-throughput access, should include file storage, file tree management, batch load,
The functions such as line monitoring.
As shown in figure 3, data analysis domain system of the invention to be applied to the specific embodiment of business event application platform
Show interface.
In addition, according to disclosed specification of the invention, other realizations of the invention are for those skilled in the art
Significantly.The various aspects of embodiment and/or embodiment can be used for system of the invention individually or with any combination
In method.Specification and example therein should be only be regarded solely as it is exemplary, the actual scope of the present invention and spirit by appended
Claims indicate.
Claims (9)
1. a kind of data analyze domain system characterized by comprising data active layer, data access layer, data store computation layer and
United analysis service layer, the data active layer are used to obtain data, including structural data obtains module, and unstructured data obtains
Modulus block and monitoring acquisition data acquisition module, the data access layer are used to data importing data analysis from external data source
Domain system, including structural data AM access module acquire measurement data AM access module and unstructured data AM access module, institute
It states data storage computation layer and is used to provide the carrier that data are centrally stored and inquire, including data buffering module, data warehouse mould
Block, data package module and data mart module, the united analysis service layer is for providing unified data service and exhibition
It now services, including data-interface service module, data mining service module and self-service Analysis Service module.
2. data according to claim 1 analyze domain system, which is characterized in that the structural data obtains module and is used for
The relevant structural data of power grid is obtained, the structural data includes: grid operation data, production status data and production people
Member's data.
3. data according to claim 1 analyze domain system, which is characterized in that the unstructured data obtains module and uses
In obtaining the relevant unstructured data of power grid, the unstructured data includes audio-video, picture and file.
4. data according to claim 1 analyze domain system, which is characterized in that the monitoring acquisition data acquisition module is used
Data are acquired in obtaining the relevant monitoring of power grid, the monitoring acquisition data include real-time voltage, real-time current and real time load.
5. data according to claim 1 analyze domain system, which is characterized in that the united analysis service layer is also used to mention
For data routing and data gateway service.
6. data according to claim 1 analyze domain system, which is characterized in that the data warehouse module includes detail number
It is used to store the data by data active layer storage by encoding unified, data with layer, the detailed data layer is slightly summarized according to layer
Detailed data that is homologous, being formed after data normalization, the layer that slightly summarizes is for storing the data of the detailed data layer
Carry out data light weight summarize and/or light weight merging treatment after data.
7. data according to claim 1 analyze domain system, which is characterized in that data in the data mart module by
The data of data warehouse are formed after processing conversion, are used to support the demand of united analysis service.
8. data according to claim 1 analyze domain system, which is characterized in that the data package module includes for counting
According to the component calculated and data store, the data computation module includes stream calculation component, memory computation module, batch calculating group
Part and inquiry computation module；The data storage component includes hot spot data storage assembly, real-time data acquisition component and point
Cloth file system component.
9. data according to claim 8 analyze domain system, which is characterized in that the data access layer and data storage meter
Calculating layer includes: processor and memory, wherein memory is for storing computer program, and processor is for executing the storage
The computer program of device storage, so that the data access layer and data storage computation layer cooperation execute the integrated side of following data
Method:
Step A100 is accessed by data pick-up component or database synchronization copy mode, and stored to the data buffer storage mould
Block；
Data cache module data are drawn into data warehouse detailed data layer according to unified data model standard by step A200；
Data warehouse detail layer data is drawn into data warehouse and slightly summarizes layer by step A300, by the data mould of data warehouse
Type is stored；
Data polymerize by step A400 according to Grained Requirements according to the data model of Data Mart, and Data Mart is arrived in storage
In；
Step A500 supports united analysis service based on the data in Data Mart.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811215551.4A CN109542960B (en) | 2018-10-18 | 2018-10-18 | Data analysis domain system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811215551.4A CN109542960B (en) | 2018-10-18 | 2018-10-18 | Data analysis domain system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109542960A true CN109542960A (en) | 2019-03-29 |
| CN109542960B CN109542960B (en) | 2023-03-14 |
Family
ID=65844493
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811215551.4A Active CN109542960B (en) | 2018-10-18 | 2018-10-18 | Data analysis domain system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109542960B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109976729A (en) * | 2019-05-05 | 2019-07-05 | 东北大学 | One kind depositing calculation and shows globally configurable Data Analysis Software architecture design method |
| CN112395281A (en) * | 2020-12-10 | 2021-02-23 | 太极计算机股份有限公司 | Heterogeneous multi-source data fusion system |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080065590A1 (en) * | 2006-09-07 | 2008-03-13 | Microsoft Corporation | Lightweight query processing over in-memory data structures |
| US20080270363A1 (en) * | 2007-01-26 | 2008-10-30 | Herbert Dennis Hunt | Cluster processing of a core information matrix |
| US20130074058A1 (en) * | 2012-09-18 | 2013-03-21 | Concurix Corporation | Memoization from Offline Analysis |
| CN104142980A (en) * | 2014-07-15 | 2014-11-12 | 中电科华云信息技术有限公司 | Big data-based metadata model management system and method |
| CN105184642A (en) * | 2015-09-02 | 2015-12-23 | 浪潮软件集团有限公司 | Comprehensive tax administration platform |
| CN105554070A (en) * | 2015-12-09 | 2016-05-04 | 北京中科云集科技有限公司 | Method based on police affair big data center service construction |
| US20160196304A1 (en) * | 2015-01-05 | 2016-07-07 | Saama Technologies Inc. | Abstractly implemented data analysis systems and methods therefor |
| CN106339509A (en) * | 2016-10-26 | 2017-01-18 | 国网山东省电力公司临沂供电公司 | Power grid operation data sharing system based on large data technology |
| CN107330080A (en) * | 2017-07-03 | 2017-11-07 | 北京希嘉创智教育科技有限公司 | A kind of data processing method, device and apply its computer equipment |
| CN108121778A (en) * | 2017-12-14 | 2018-06-05 | 浙江航天恒嘉数据科技有限公司 | A kind of heterogeneous database exchange and cleaning system and method |
| CN108390782A (en) * | 2018-02-12 | 2018-08-10 | 黄倚霄 | A kind of centralization application system performance question synthesis analysis method |
| CN108615143A (en) * | 2018-06-12 | 2018-10-02 | 湖南建工集团有限公司 | Device and method based on BIM models Yu O＆M information exchange in intelligent building management |
-
2018
- 2018-10-18 CN CN201811215551.4A patent/CN109542960B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080065590A1 (en) * | 2006-09-07 | 2008-03-13 | Microsoft Corporation | Lightweight query processing over in-memory data structures |
| US20080270363A1 (en) * | 2007-01-26 | 2008-10-30 | Herbert Dennis Hunt | Cluster processing of a core information matrix |
| US20130074058A1 (en) * | 2012-09-18 | 2013-03-21 | Concurix Corporation | Memoization from Offline Analysis |
| CN104142980A (en) * | 2014-07-15 | 2014-11-12 | 中电科华云信息技术有限公司 | Big data-based metadata model management system and method |
| US20160196304A1 (en) * | 2015-01-05 | 2016-07-07 | Saama Technologies Inc. | Abstractly implemented data analysis systems and methods therefor |
| CN105184642A (en) * | 2015-09-02 | 2015-12-23 | 浪潮软件集团有限公司 | Comprehensive tax administration platform |
| CN105554070A (en) * | 2015-12-09 | 2016-05-04 | 北京中科云集科技有限公司 | Method based on police affair big data center service construction |
| CN106339509A (en) * | 2016-10-26 | 2017-01-18 | 国网山东省电力公司临沂供电公司 | Power grid operation data sharing system based on large data technology |
| CN107330080A (en) * | 2017-07-03 | 2017-11-07 | 北京希嘉创智教育科技有限公司 | A kind of data processing method, device and apply its computer equipment |
| CN108121778A (en) * | 2017-12-14 | 2018-06-05 | 浙江航天恒嘉数据科技有限公司 | A kind of heterogeneous database exchange and cleaning system and method |
| CN108390782A (en) * | 2018-02-12 | 2018-08-10 | 黄倚霄 | A kind of centralization application system performance question synthesis analysis method |
| CN108615143A (en) * | 2018-06-12 | 2018-10-02 | 湖南建工集团有限公司 | Device and method based on BIM models Yu O＆M information exchange in intelligent building management |
Non-Patent Citations (3)
| Title |
|---|
| 孟维一: "基于数据仓库的保险商业智能系统设计与实现" * |
| 江樱;黄慧;卢文达;骆伟艺;: "基于大数据技术的电力全业务数据运营管理平台研究" * |
| 邱菊;王岩;黄佩卓;王洋;: "大型电力企业基于GBase分布式数据仓库建设初探" * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109976729A (en) * | 2019-05-05 | 2019-07-05 | 东北大学 | One kind depositing calculation and shows globally configurable Data Analysis Software architecture design method |
| CN109976729B (en) * | 2019-05-05 | 2021-10-22 | 东北大学 | Storage and computing display globally configurable data analysis software architecture design method |
| CN112395281A (en) * | 2020-12-10 | 2021-02-23 | 太极计算机股份有限公司 | Heterogeneous multi-source data fusion system |
| CN112395281B (en) * | 2020-12-10 | 2021-05-11 | 太极计算机股份有限公司 | Heterogeneous multi-source data fusion system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109542960B (en) | 2023-03-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| He et al. | QoE-driven big data architecture for smart city | |
| CN104933112B (en) | Distributed interconnection Transaction Information storage processing method | |
| CN104820670B (en) | A kind of acquisition of power information big data and storage method | |
| CN103390038B (en) | A kind of method of structure based on HBase and retrieval increment index | |
| CN109951463A (en) | A kind of Internet of Things big data analysis method stored based on stream calculation and novel column | |
| CN111400326B (en) | Smart city data management system and method thereof | |
| CN105139281A (en) | Method and system for processing big data of electric power marketing | |
| CN106326331A (en) | Intelligent power utilization data service system based on cloud computation | |
| CN103500173A (en) | Method for inquiring rail transit monitoring data | |
| CN103577602A (en) | Secondary clustering method and system | |
| CN102968422A (en) | System and method for controlling streaming data storage | |
| CN103365971A (en) | Mass data access processing system based on cloud computing | |
| CN109542960A (en) | A kind of data analysis domain system | |
| CN105354250A (en) | Data storage method and device for cloud storage | |
| CN106649687A (en) | Method and device for on-line analysis and processing of large data | |
| CN104113605A (en) | Enterprise cloud application development monitoring processing method | |
| CN108595605A (en) | A kind of construction method of car networking platform database | |
| CN111586091A (en) | Edge computing gateway system for realizing computing power assembly | |
| CN104717247A (en) | Method and system for dynamically scheduling storage resources in cloud storage system | |
| CN112632025A (en) | Power grid enterprise management decision support application system based on PAAS platform | |
| CN109669975A (en) | A kind of industry big data processing system and method | |
| CN102855391A (en) | Enhanced message service (EMS) multi-professional-data holographic storage and panoramic accident inversion method | |
| CN112328569A (en) | Construction method based on Flume distributed data collection architecture | |
| Dong et al. | Research on Architecture of Power Big Data High-Speed Storage System for Energy Interconnection | |
| CN110377757A (en) | A kind of real time knowledge map construction system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |