CN105138686A - Real-time application method for multi-level storage data - Google Patents
Real-time application method for multi-level storage data Download PDFInfo
- Publication number
- CN105138686A CN105138686A CN201510592013.7A CN201510592013A CN105138686A CN 105138686 A CN105138686 A CN 105138686A CN 201510592013 A CN201510592013 A CN 201510592013A CN 105138686 A CN105138686 A CN 105138686A
- Authority
- CN
- China
- Prior art keywords
- data
- request
- report engine
- calculating
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Abstract
The invention discloses a real-time application method for multi-level storage data. The method comprises the steps of 1, service object modeling; 2, distributed object caching; 3, distributed data demand activation; 4, dynamic routing; 5, agent-based computing; 6, summarizing and aggregating calculation. Compared with the prior art, the method has the advantages that extend property is high, the real-time performance of tasks is high, multi-dimension multi-angle data analysis and inquiry are flexible, real-time data extraction is provided for OMS system statement presentation, manual dispersed data reporting is avoided, statement dimensionality and inquiry conditions can be changed quickly, and drag-and-drop establishment of an inquiry statement can be achieved in a user-friendly mode.
Description
Technical field
The present invention relates to a kind of instant application process, be specifically related to a kind of instant application process for dynamic data attemper data.
Background technology
Along with progressively going deep into of electric power enterprise operating information system application, business datum amount will be huge gradually, business datum that each net is economized is all that dispersion is stored in local service system, cannot represent application at present by unified for the data of all dispersions, to user use and experience is made troubles.
Data warehouse and OLAP technology have become the focus of multidimensional analysis research, each manufacturer both provides oneself data warehouse solution and oneself decision analysis instrument, as AnalysisManager and OLAPService of Microsoft, the BusinessObjectExplorer of BO company and domestic some products etc.But these technology are only carried out analysis for data warehouse centralised storage and are represented.
At data multidimensional analysis field, common analytical approach utilizes data cube (Cube) to carry out aminated polyepichlorohydrin, but analyzing and processing can not disperse the data that are stored in each operation system storehouse.Field is calculated at internal memory, common analytical approach utilizes memory database, memory database passes through the all-in-one technology of internal memory computing technique and software and hardware combining, realizes high performance data query and analysis, meets the real-time demand of user to large data processing.But, internal memory computing platform based on brand-new framework, with the system software transformation of data with existing storehouse in move workload huge.
Summary of the invention
In order to solve the problems of the technologies described above, the invention provides a kind of instant application process for dynamic data attemper data.
In order to achieve the above object, the technical solution adopted in the present invention is:
For an instant application process for dynamic data attemper data, comprise the following steps,
Step one, business object modeling;
Create two kinds of object models according to service logic relation, a kind of is data object model for business datum, and a kind of is task object model for business task;
Step 2, distributed objects buffer memory;
Report Engine service starts initialization, loads object model, route and common dimension data;
Step 3, starts distributed data request;
When client initiates request report form showing, search according to the expired time range of request from historical snapshot, just directly return if existed, otherwise continue to Report Engine request, Report Engine generates computation requests object according to the index of request object, dimension and condition structure;
Step 4, dynamic routing;
Report Engine searches the request address in each area from routing table, by Hessain asynchronous transmission computation requests object;
Step 5, agency calculates;
Each calculating agent node is when initialization, by business datum prestrain, when receiving the computation requests of Report Engine, first judge whether institute's request resource is buffered, if be buffered, calculate agency and get resource data from buffer memory, by polymerization calculating, code conversion, date polymerization conversion, intermediate result collection will be calculated and send to main website message queue;
Step 6, gathers and is polymerized calculating;
Intermediate result collection in message queue gathers by Report Engine, the memory database being inserted into a temporary table calculates, Report Engine is that the computing unit lattice of each generation perform the calculating of internal memory expression formula, each cell generates a SQL statement by its constraint condition, the temporary table aggregate query value of memory database is filled into each computing unit lattice, and the set of form cell returns to the client of request the most at last.
By logical relation, object is divided into single object and object set, does not have logical relation between single object, subsistence logic relation between the object in object set.
Calculating agency adopts memory calculation data storehouse to calculate.
The beneficial effect that the present invention reaches: the present invention is compared to prior art, there is scalability good, task real-time performance is high, various dimensions data of multiple angles analysis and consult is flexible, for OMS report form showing provides real time data to extract, avoid separate data and manually make a report on workload, can change form dimension and querying condition fast, towed builds inquiry form more with open arms.
Accompanying drawing explanation
Fig. 1 is process flow diagram of the present invention.
Fig. 2 is system architecture diagram.
Embodiment
Below in conjunction with accompanying drawing, the invention will be further described.Following examples only for technical scheme of the present invention is clearly described, and can not limit the scope of the invention with this.
As shown in Figure 1, a kind of instant application process for dynamic data attemper data, comprises the following steps,
Step one, business object modeling.
Create two kinds of object models according to service logic relation, a kind of is data object model for business datum, and a kind of is task object model for business task.By logical relation, object is divided into single object and object set, and the single object degree of coupling is low, do not have logical relation between single object, can complete a certain generic task, the object set degree of coupling is high, subsistence logic relation between object in object set, multiple object just can complete a certain generic task together.
Step 2, distributed objects buffer memory.
Report Engine service starts initialization, loads object model, route and common dimension data.
Step 3, starts distributed data request.
When client initiates request report form showing, search according to the expired time range of request from historical snapshot, just directly return if existed, otherwise continue to Report Engine request, Report Engine generates computation requests object according to the index of request object, dimension and condition structure.
Step 4, dynamic routing.
Report Engine searches the request address in each area from routing table, by Hessain asynchronous transmission computation requests object.
Step 5, agency calculates.
Each calculating agent node is when initialization, by business datum prestrain, when receiving the computation requests of Report Engine, first judge whether institute's request resource is buffered, if be buffered, calculate agency and get resource data from buffer memory, by polymerization calculating, code conversion, date polymerization conversion, intermediate result collection will be calculated and send to main website message queue.
Calculating agency adopts memory calculation data storehouse to calculate, and it in the internal memory of server, directly can carry out analysis and consult operation at agent node to data deposit data, does not need the data of agent node to download, and decreases the time of data transmission.
Step 6, gathers and is polymerized calculating;
Intermediate result collection in message queue gathers by Report Engine, the memory database being inserted into a temporary table calculates, Report Engine is that the computing unit lattice of each generation perform the calculating of internal memory expression formula, each cell generates a SQL statement by its constraint condition, the temporary table aggregate query value of memory database is filled into each computing unit lattice, and the set of form cell returns to the client of request the most at last.
Said method is acted on behalf of three parts primarily of client browser, Report Engine and Distributed Calculation and has mutually been cooperated.
Client browser:
Browser major function has form issue, application management, unified storage and analytical model configuration and report form showing.Comprising report model modeling, create business object model, data source, dimension and index set; Dimension, index and form is generated based on business object; The difference of shielding bottom different pieces of information source storage mode, is completed the persistence of memory object data, is obtained the data of different regions operation system by a set of model.
Report Engine:
Report Engine is managed and is polymerized calculating formed by data, services, model analyzing, model buffer memory, task scheduling, Intelligent routing, dimension.Wherein model analyzing builds based on model Description standard and resolves, and is cached in H2 Memory management component, dimension management main point two kinds of privately owned dimensions and common dimension, and dimension values data are gathered by distributed way and obtain.Polymerization calculates the management achieved gathering table, mainly for the management of OLAP buffer memory, its buffer memory each calculate the Query Result of agency, i.e. cell data acquisition, if the data needed for computing module are not in the buffer, carry out inquiry from corresponding agent node and obtain data and buffer memory.
Distributed Calculation is acted on behalf of:
Distributed Calculation Agent components is made up of route monitoring, request analysis, data acquisition, polymerization computation module and computing unit lattice object, be responsible for the computation requests receiving main website transmission, precomputation is carried out by system business Data import to internal memory, comprise polymerization calculating, code conversion, date polymerization conversion, draw intermediate result collection, asynchronous transmission is to the message queue of main station system.
In calculating agency, adopt data syn-chronization and caching mechanism, the data that buffer memory is deposited are generally to the copy of the data being stored in database to its process data, just can data in read-write cache, lifting reading performance.When calculating the computation requests of agency's each answer main website, a lot of SQL statement to be performed to build multi-dimensional query result in underlying relational data storehouse, therefore should utilize the result set inquired as far as possible, save the access time of physical database.Because certain rule is followed in user's inquiry, inquiring about next time and often can use nearest Query Result, is carry high performance effective ways in the calculating agency therefore introduced by caching mechanism.
Said method, compared to prior art, has scalability good, and task real-time performance is high, and various dimensions data of multiple angles analysis and consult is flexible.
The above is only the preferred embodiment of the present invention; it should be pointed out that for those skilled in the art, under the prerequisite not departing from the technology of the present invention principle; can also make some improvement and distortion, these improve and distortion also should be considered as protection scope of the present invention.
Claims (3)
1., for an instant application process for dynamic data attemper data, it is characterized in that: comprise the following steps,
Step one, business object modeling;
Create two kinds of object models according to service logic relation, a kind of is data object model for business datum, and a kind of is task object model for business task;
Step 2, distributed objects buffer memory;
Report Engine service starts initialization, loads object model, route and common dimension data;
Step 3, starts distributed data request;
When client initiates request report form showing, search according to the expired time range of request from historical snapshot, just directly return if existed, otherwise continue to Report Engine request, Report Engine generates computation requests object according to the index of request object, dimension and condition structure;
Step 4, dynamic routing;
Report Engine searches the request address in each area from routing table, by Hessain asynchronous transmission computation requests object;
Step 5, agency calculates;
Each calculating agent node is when initialization, by business datum prestrain, when receiving the computation requests of Report Engine, first judge whether institute's request resource is buffered, if be buffered, calculate agency and get resource data from buffer memory, by polymerization calculating, code conversion, date polymerization conversion, intermediate result collection will be calculated and send to main website message queue;
Step 6, gathers and is polymerized calculating;
Intermediate result collection in message queue gathers by Report Engine, the memory database being inserted into a temporary table calculates, Report Engine is that the computing unit lattice of each generation perform the calculating of internal memory expression formula, each cell generates a SQL statement by its constraint condition, the temporary table aggregate query value of memory database is filled into each computing unit lattice, and the set of form cell returns to the client of request the most at last.
2. a kind of instant application process for dynamic data attemper data according to claim 1, it is characterized in that: by logical relation, object is divided into single object and object set, does not have logical relation between single object, subsistence logic relation between the object in object set.
3. a kind of instant application process for dynamic data attemper data according to claim 1, is characterized in that: calculate agency and adopt memory calculation data storehouse to calculate.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510592013.7A CN105138686B (en) | 2015-09-17 | 2015-09-17 | A kind of instant application process for multistage storage data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510592013.7A CN105138686B (en) | 2015-09-17 | 2015-09-17 | A kind of instant application process for multistage storage data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105138686A true CN105138686A (en) | 2015-12-09 |
| CN105138686B CN105138686B (en) | 2018-09-28 |
Family
ID=54724033
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510592013.7A Active CN105138686B (en) | 2015-09-17 | 2015-09-17 | A kind of instant application process for multistage storage data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105138686B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108268536A (en) * | 2016-12-30 | 2018-07-10 | 北京国双科技有限公司 | Database aggregation processing method and device |
| CN110276059A (en) * | 2019-06-24 | 2019-09-24 | 银联商务股份有限公司 | A kind for the treatment of method and apparatus of dynamic statement |
| CN110413610A (en) * | 2019-06-19 | 2019-11-05 | 中国平安财产保险股份有限公司 | Improve method and system, the database server of business datum report export efficiency |
| CN113486066A (en) * | 2021-07-15 | 2021-10-08 | 福建博思软件股份有限公司 | Method and terminal for hierarchically summarizing report forms |
| US11951530B2 (en) | 2020-04-29 | 2024-04-09 | Central Iron And Steel Research Institute | High-strength stainless steel rotor and method for preparing the same |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101882157A (en) * | 2010-06-21 | 2010-11-10 | 国家电网公司 | Aided analysis method and model |
| EP2660736A1 (en) * | 2012-04-30 | 2013-11-06 | Sap Ag | Partial merge in a multi-level storage architecture |
| CN103955502A (en) * | 2014-04-24 | 2014-07-30 | 科技谷（厦门）信息技术有限公司 | Visualized on-line analytical processing (OLAP) application realizing method and system |
| CN104376109A (en) * | 2014-11-28 | 2015-02-25 | 国家电网公司 | Multi-dimension data distribution method based on data distribution base |
-
2015
- 2015-09-17 CN CN201510592013.7A patent/CN105138686B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101882157A (en) * | 2010-06-21 | 2010-11-10 | 国家电网公司 | Aided analysis method and model |
| EP2660736A1 (en) * | 2012-04-30 | 2013-11-06 | Sap Ag | Partial merge in a multi-level storage architecture |
| CN103955502A (en) * | 2014-04-24 | 2014-07-30 | 科技谷（厦门）信息技术有限公司 | Visualized on-line analytical processing (OLAP) application realizing method and system |
| CN104376109A (en) * | 2014-11-28 | 2015-02-25 | 国家电网公司 | Multi-dimension data distribution method based on data distribution base |
Non-Patent Citations (1)
| Title |
|---|
| 李培军等: "电网业务中的海量数据存储技术", 《计算机系统应用》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108268536A (en) * | 2016-12-30 | 2018-07-10 | 北京国双科技有限公司 | Database aggregation processing method and device |
| CN110413610A (en) * | 2019-06-19 | 2019-11-05 | 中国平安财产保险股份有限公司 | Improve method and system, the database server of business datum report export efficiency |
| CN110413610B (en) * | 2019-06-19 | 2023-10-27 | 中国平安财产保险股份有限公司 | Method and system for improving export efficiency of business data report forms and database server |
| CN110276059A (en) * | 2019-06-24 | 2019-09-24 | 银联商务股份有限公司 | A kind for the treatment of method and apparatus of dynamic statement |
| CN110276059B (en) * | 2019-06-24 | 2023-10-27 | 银联商务股份有限公司 | Dynamic report processing method and device |
| US11951530B2 (en) | 2020-04-29 | 2024-04-09 | Central Iron And Steel Research Institute | High-strength stainless steel rotor and method for preparing the same |
| CN113486066A (en) * | 2021-07-15 | 2021-10-08 | 福建博思软件股份有限公司 | Method and terminal for hierarchically summarizing report forms |
| CN113486066B (en) * | 2021-07-15 | 2023-03-24 | 福建博思软件股份有限公司 | Method and terminal for hierarchically summarizing report forms |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105138686B (en) | 2018-09-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200301941A1 (en) | Large scale unstructured database systems | |
| US7917463B2 (en) | System and method for data warehousing and analytics on a distributed file system | |
| CN104903894B (en) | System and method for distributed networks database query engine | |
| US10585887B2 (en) | Multi-system query execution plan | |
| US10754877B2 (en) | System and method for providing big data analytics on dynamically-changing data models | |
| US7930277B2 (en) | Cost-based optimizer for an XML data repository within a database | |
| CN103106249B (en) | A kind of parallel data processing system based on Cassandra | |
| CN105138686A (en) | Real-time application method for multi-level storage data | |
| US8396828B2 (en) | Providing lightweight multidimensional online data storage for web service usage reporting | |
| CN109656958B (en) | Data query method and system | |
| CN103955502A (en) | Visualized on-line analytical processing (OLAP) application realizing method and system | |
| CN108664516A (en) | Enquiring and optimizing method and relevant apparatus | |
| CN103064933A (en) | Data query method and system | |
| CN106850258A (en) | A kind of Log Administration System, method and device | |
| CN104731969A (en) | Mass data join aggregation query method, device and system in distributed environment | |
| CN103970871A (en) | Method and system for inquiring file metadata in storage system based on provenance information | |
| CN103778251A (en) | SPARQL parallel query method facing large-scale RDF graph data | |
| WO2015041731A1 (en) | Interest-driven business intelligence systems including segment data | |
| CN103823846A (en) | Method for storing and querying big data on basis of graph theories | |
| CN105183809A (en) | Cloud platform data query method | |
| CN110309171A (en) | Data base query method, server and system | |
| CN106599190A (en) | Dynamic Skyline query method based on cloud computing | |
| CN106599189A (en) | Dynamic Skyline inquiry device based on cloud computing | |
| Kang et al. | Distributed graph cube generation using Spark framework | |
| CN208207819U (en) | A kind of big data analysis processing system based on extended node cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| CB03 | Change of inventor or designer information |
Inventor after: Xiang Yuliang Inventor after: Liu Gang Inventor after: Ge Weichun Inventor after: Ren Kaiyin Inventor after: Liu Shijin Inventor after: He Xiang Inventor after: Zhang Mingming Inventor after: Huang Gaopan Inventor after: Lei Zhenjiang Inventor after: Lv Xuming Inventor after: Li Zhao Inventor before: Xiang Yuliang Inventor before: Ren Kaiyin Inventor before: Liu Shijin Inventor before: He Xiang Inventor before: Zhang Mingming Inventor before: Huang Gaopan |
|
| COR | Change of bibliographic data | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20160206 Address after: 100031 Xicheng District West Chang'an Avenue, No. 86, Beijing Applicant after: State Grid Corporation of China Applicant after: Nanjing Nari Co., Ltd. Applicant after: NANJING NARI INFORMATION COMMUNICATION SCIENCE & TECHNOLOGY CO., LTD. Applicant after: Information & Telecommunication Branch of State Grid Jiangsu Electric Power Company Applicant after: State Grid Liaoning Electric Power Co., Ltd. information communication company Address before: 100031 Xicheng District West Chang'an Avenue, No. 86, Beijing Applicant before: State Grid Corporation of China Applicant before: Nanjing Nari Co., Ltd. Applicant before: NANJING NARI INFORMATION COMMUNICATION SCIENCE & TECHNOLOGY CO., LTD. Applicant before: Information & Telecommunication Branch of State Grid Jiangsu Electric Power Company |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 100031 Xicheng District West Chang'an Avenue, No. 86, Beijing Co-patentee after: NARI Group Corp. Patentee after: State Grid Corporation of China Co-patentee after: NARI INFORMATION AND COMMUNICATION TECHNOLOGY Co. Co-patentee after: INFORMATION & TELECOMMUNICATION BRANCH OF STATE GRID JIANGSU ELECTRIC POWER Co. Co-patentee after: State Grid Liaoning Electric Power Co.,Ltd. 's information communication common carrier Address before: 100031 Xicheng District West Chang'an Avenue, No. 86, Beijing Co-patentee before: NARI Group CORPORATION STATE GRID ELECTRIC POWER INSTITUTE Patentee before: State Grid Corporation of China Co-patentee before: NARI INFORMATION AND COMMUNICATION TECHNOLOGY Co. Co-patentee before: INFORMATION & TELECOMMUNICATION BRANCH OF STATE GRID JIANGSU ELECTRIC POWER Co. Co-patentee before: State Grid Liaoning Electric Power Co.,Ltd. 's information communication common carrier |
|
| CP01 | Change in the name or title of a patent holder |