CN105138686B - A kind of instant application process for multistage storage data - Google Patents
A kind of instant application process for multistage storage data Download PDFInfo
- Publication number
- CN105138686B CN105138686B CN201510592013.7A CN201510592013A CN105138686B CN 105138686 B CN105138686 B CN 105138686B CN 201510592013 A CN201510592013 A CN 201510592013A CN 105138686 B CN105138686 B CN 105138686B
- Authority
- CN
- China
- Prior art keywords
- data
- report
- request
- report engine
- instant application
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 13
- 230000008569 process Effects 0.000 title claims abstract description 10
- 238000006116 polymerization reaction Methods 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 8
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 230000005540 biological transmission Effects 0.000 claims description 5
- 230000036316 preload Effects 0.000 claims description 3
- 238000004458 analytical method Methods 0.000 abstract description 14
- 239000006185 dispersion Substances 0.000 abstract description 4
- 230000008859 change Effects 0.000 abstract description 2
- 238000005516 engineering process Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000013075 data extraction Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 229920002755 poly(epichlorohydrin) Polymers 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Abstract
The invention discloses a kind of instant application processes for multistage storage data, include the following steps, step 1, business object modeling；Step 2, distributed objects caching；Step 3 starts distributed data request；Step 4, dynamic routing；Step 5, agency calculate；Step 6 summarizes and polymerize calculating.The present invention is compared with the prior art, having the scalability can be good, task real-time performance is high, the analysis inquiry of various dimensions data of multiple angles is flexible, real time data is provided for OMS system report form showing to extract, it avoids dispersion data and manually makes a report on workload, can quickly change report dimension and querying condition, compare towed structure inquiry report with open arms.
Description
Technical field
The present invention relates to a kind of instant application processes, and in particular to a kind of instant application side for multistage storage data
Method.
Background technology
With gradually going deep into for electric power enterprise operating information system application, business datum amount will be gradually huge, and each net saves
Business datum all be dispersion be stored in local service system, the data of all dispersions can not uniformly be showed application at present, given
User use and experience is made troubles.
Data warehouse and OLAP technologies have become the hot spot of multidimensional analysis research, and each manufacturer both provides the number of oneself
According to warehouse solution and oneself Analysis of Policy Making tool, such as Analysis Manager and OLAP Service, the BO of Microsoft
The Business Object Explorer of company and some products etc. of the country.However these technologies are just for data warehouse
Centralised storage carries out analysis and shows.
In data multidimensional analysis field, common analysis method utilizes data cube（Cube）Aminated polyepichlorohydrin is carried out, but
It is unable to the data that analyzing processing dispersion is stored in each operation system library.In memory calculating field, common analysis method is
Using memory database, memory database realizes high property by memory computing technique and the all-in-one machine technology of software and hardware combining
The data query of energy and analysis, meet the real-time demand that user handles big data.However, memory computing platform is based on completely newly
It is huge to migrate workload in terms of with the system software transformation of data with existing library for framework.
Invention content
In order to solve the above technical problem, the present invention provides a kind of instant application processes for multistage storage data.
In order to achieve the above object, the technical solution adopted in the present invention is：
It is a kind of to include the following steps for the multistage instant application process for storing data,
Step 1, business object modeling；
Two kinds of object models, one is the data object model for business datum, one are created according to service logic relationship
Kind is the task object model for business task；
Step 2, distributed objects caching；
Report Engine service starts initialization, load object model, routing and common dimension data；
Step 3 starts distributed data request；
When client initiates request report form showing, searched according to the expired time range of request from historical snapshot, if
In the presence of just directly returning, otherwise continue to ask to Report Engine, Report Engine is according to index, dimension and the condition structure for asking object
It makes and generates computation requests object；
Step 4, dynamic routing；
Report Engine searches the request address in each area from routing table, passes through Hessain asynchronous transmission computation requests
Object；
Step 5, agency calculate；
Each agent node that calculates preloads business datum, in initialization when the computation requests for receiving Report Engine
When object, first judge whether asked resource has been buffered, if be buffered, calculates agency and take resource data from caching, lead to
Polymerization calculating, code conversion, date polymerization conversion are crossed, intermediate result set will be calculated and be sent to main website message queue；
Step 6 summarizes and polymerize calculating；
Report Engine summarizes the intermediate result set in main website message queue, is inserted into the memory database of an interim table
It calculates, Report Engine is that the computing unit lattice each generated execute memory expression formula and calculate, and each cell constrains item by it
Part generates a SQL statement, and the interim table aggregate query value of memory database is filled into each computing unit lattice, finally will report
Table unit lattice set returns to the client of request.
By logical relation, object is divided into single object and object set, does not have logical relation between single object, pair in object set
There are logical relations as between.
Agency is calculated to be calculated using memory calculation data library.
The advantageous effect that the present invention is reached：Compared with the prior art, having the scalability can good, task real-time by the present invention
Can be high, the analysis inquiry of various dimensions data of multiple angles is flexible, and real time data extraction is provided for OMS system report form showing, avoids point
Scattered data manually make a report on workload, can quickly change report dimension and querying condition, compare towed structure inquiry report with open arms
Table.
Description of the drawings
Fig. 1 is flow chart of the present invention.
Fig. 2 is system architecture diagram.
Specific implementation mode
The invention will be further described below in conjunction with the accompanying drawings.Following embodiment is only used for clearly illustrating the present invention
Technical solution, and not intended to limit the protection scope of the present invention.
As shown in Figure 1, a kind of instant application process for multistage storage data, includes the following steps,
Step 1, business object modeling.
Two kinds of object models, one is the data object model for business datum, one are created according to service logic relationship
Kind is the task object model for business task.By logical relation, object is divided into single object and object set, the single object degree of coupling
It is low, there is no logical relation between single object, can complete a certain generic task, the object set degree of coupling is high, the object in object set it
Between there are logical relations, multiple objects could complete a certain generic task together.
Step 2, distributed objects caching.
Report Engine service starts initialization, load object model, routing and common dimension data.
Step 3 starts distributed data request.
When client initiates request report form showing, searched according to the expired time range of request from historical snapshot, if
In the presence of just directly returning, otherwise continue to ask to Report Engine, Report Engine is according to index, dimension and the condition structure for asking object
It makes and generates computation requests object.
Step 4, dynamic routing.
Report Engine searches the request address in each area from routing table, passes through Hessain asynchronous transmission computation requests
Object.
Step 5, agency calculate.
Each agent node that calculates preloads business datum, in initialization when the computation requests for receiving Report Engine
When object, first judge whether asked resource has been buffered, if be buffered, calculates agency and take resource data from caching, lead to
Polymerization calculating, code conversion, date polymerization conversion are crossed, intermediate result set will be calculated and be sent to main website message queue.
It calculates agency to be calculated using memory calculation data library, it is stored in the memory of server data, can be with
Analysis inquiry operation directly is carried out to data in agent node, it is not necessary to the data of agent node be downloaded, number is reduced
According to the time of transmission.
Step 6 summarizes and polymerize calculating；
Report Engine summarizes the intermediate result set in main website message queue, is inserted into the memory database of an interim table
It calculates, Report Engine is that the computing unit lattice each generated execute memory expression formula and calculate, and each cell constrains item by it
Part generates a SQL statement, and the interim table aggregate query value of memory database is filled into each computing unit lattice, finally will report
Table unit lattice set returns to the client of request.
The above method is mainly acted on behalf of three parts by client browser, Report Engine and Distributed Calculation and has been cooperated
At.
Client browser：
Browser major function has report publication, application management, unified storage and analysis model configuration and report form showing.Its
Include report model modeling, creates business object model, data source, dimension and index set；Based on business object generate dimension,
Index and report；The difference of bottom different data sources storage mode is shielded, the persistence of memory object data is completed, by a set of
Model obtains the data of different regions operation system.
Report Engine：
Report Engine is by data service, model analyzing, model caching, task scheduling, Intelligent routing, dimension management and polymerization
Calculate composition.Wherein model analyzing is based on model Description standard structure and parses, and is cached in H2 Memory management components, dimension pipe
Reason is mainly divided to two kinds of privately owned dimensions and common dimension, dimension Value Data to be summarized by distributed way.Polymerization, which calculates, to be realized
To assembling the management of table, mainly for the management of OLAP cachings, each query result for calculating agency, i.e. unit have been cached
Lattice data acquisition system, if data needed for computing module not in the buffer, carry out inquiry from corresponding agent node and obtain data
And it caches.
Distributed Calculation is acted on behalf of：
Distributed Calculation Agent components are by routeing monitoring, request analysis, data acquisition, polymerization computation module and computing unit
Lattice object forms, and is responsible for receiving the computation requests object that main website is sent, system business data are loaded into memory and are expected
It calculates, including polymerization calculating, code conversion, date polymerization conversion, obtains intermediate result set, main website of the asynchronous transmission to main station system
Message queue.
Using data synchronization and caching mechanism in calculating agency, the data for caching storage are usually to being stored in database
Data handle it the copies of data, so that it may with the data in read-write cache, promote reading performance.Agency is calculated to answer every time
When the computation requests object of main website, many SQL statements are executed in underlying relational data library come build multi-dimensional query as a result, because
This should save the access time of physical database as far as possible using the result set inquired.Since user's inquiry follows
Certain rule, inquires and often uses nearest query result next time, therefore is during the calculating that caching mechanism is introduced into is acted on behalf of
Carry high performance effective ways.
Compared with the prior art, having the scalability can be good, and task real-time performance is high, various dimensions data of multiple angles for the above method
Analysis inquiry is flexible.
The above is only a preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art
For member, without departing from the technical principles of the invention, several improvement and deformations can also be made, these improvement and deformations
Also it should be regarded as protection scope of the present invention.
Claims (3)
1. a kind of instant application process for multistage storage data, it is characterised in that：Include the following steps,
Step 1, business object modeling；
Two kinds of object models are created according to service logic relationship, one is the data object model for business datum, one is
For the task object model of business task；
Step 2, distributed objects caching；
Report Engine service starts initialization, load object model, routing and common dimension data；
Step 3 starts distributed data request；
When client initiates request report form showing, searched according to the expired time range of request from historical snapshot, if there is
It just directly returns, otherwise continues to ask to Report Engine, Report Engine constructs life according to the index, dimension and condition of request object
At computation requests object；
Step 4, dynamic routing；
Report Engine searches the request address in each area from routing table, passes through Hessain asynchronous transmission computation requests objects；
Step 5, agency calculate；
Each agent node that calculates preloads business datum, in initialization when the computation requests object for receiving Report Engine
When, first judge whether asked resource has been buffered, if be buffered, calculates agency and take resource data from caching, by poly-
Total calculation, code conversion, date polymerization conversion, will be calculated intermediate result set and are sent to main website message queue；
Step 6 summarizes and polymerize calculating；
Report Engine summarizes the intermediate result set in main website message queue, is inserted into the memory database of an interim table to count
It calculates, Report Engine is that the computing unit lattice each generated execute the calculating of memory expression formula, and each cell is given birth to by its constraints
At a SQL statement, the interim table aggregate query value of memory database is filled into each computing unit lattice, finally by report list
First lattice set returns to the client of request.
2. a kind of instant application process for multistage storage data according to claim 1, it is characterised in that：By logic
Relationship, object are divided into single object and object set, do not have logical relation between single object, and there are logics between the object in object set
Relationship.
3. a kind of instant application process for multistage storage data according to claim 1, it is characterised in that：Calculate generation
Reason is calculated using memory calculation data library.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510592013.7A CN105138686B (en) | 2015-09-17 | 2015-09-17 | A kind of instant application process for multistage storage data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510592013.7A CN105138686B (en) | 2015-09-17 | 2015-09-17 | A kind of instant application process for multistage storage data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105138686A CN105138686A (en) | 2015-12-09 |
| CN105138686B true CN105138686B (en) | 2018-09-28 |
Family
ID=54724033
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510592013.7A Active CN105138686B (en) | 2015-09-17 | 2015-09-17 | A kind of instant application process for multistage storage data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105138686B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108268536A (en) * | 2016-12-30 | 2018-07-10 | 北京国双科技有限公司 | Database aggregation processing method and device |
| CN110413610B (en) * | 2019-06-19 | 2023-10-27 | 中国平安财产保险股份有限公司 | Method and system for improving export efficiency of business data report forms and database server |
| CN110276059B (en) * | 2019-06-24 | 2023-10-27 | 银联商务股份有限公司 | Dynamic report processing method and device |
| CN111471940B (en) | 2020-04-29 | 2021-09-10 | 钢铁研究总院 | High-strength stainless steel rotor and preparation method thereof |
| CN113486066B (en) * | 2021-07-15 | 2023-03-24 | 福建博思软件股份有限公司 | Method and terminal for hierarchically summarizing report forms |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101882157A (en) * | 2010-06-21 | 2010-11-10 | 国家电网公司 | Aided analysis method and model |
| EP2660736A1 (en) * | 2012-04-30 | 2013-11-06 | Sap Ag | Partial merge in a multi-level storage architecture |
| CN103955502A (en) * | 2014-04-24 | 2014-07-30 | 科技谷（厦门）信息技术有限公司 | Visualized on-line analytical processing (OLAP) application realizing method and system |
| CN104376109A (en) * | 2014-11-28 | 2015-02-25 | 国家电网公司 | Multi-dimension data distribution method based on data distribution base |
-
2015
- 2015-09-17 CN CN201510592013.7A patent/CN105138686B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101882157A (en) * | 2010-06-21 | 2010-11-10 | 国家电网公司 | Aided analysis method and model |
| EP2660736A1 (en) * | 2012-04-30 | 2013-11-06 | Sap Ag | Partial merge in a multi-level storage architecture |
| CN103955502A (en) * | 2014-04-24 | 2014-07-30 | 科技谷（厦门）信息技术有限公司 | Visualized on-line analytical processing (OLAP) application realizing method and system |
| CN104376109A (en) * | 2014-11-28 | 2015-02-25 | 国家电网公司 | Multi-dimension data distribution method based on data distribution base |
Non-Patent Citations (1)
| Title |
|---|
| 电网业务中的海量数据存储技术;李培军等;《计算机系统应用》;20140415;第23卷(第4期);正文第2.2.1节，图5 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105138686A (en) | 2015-12-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105138686B (en) | A kind of instant application process for multistage storage data | |
| CN104903894B (en) | System and method for distributed networks database query engine | |
| CN108363746B (en) | Unified SQL query system supporting multi-source heterogeneous data | |
| CN105824868B (en) | A kind of distributed data base data processing method and distributed data base system | |
| US20140074771A1 (en) | Query optimization | |
| US20140156636A1 (en) | Dynamic parallel aggregation with hybrid batch flushing | |
| US20160292224A1 (en) | Generating multiple query access plans for multiple computing environments | |
| CN108256087B (en) | Data importing, inquiring and processing method based on bitmap structure | |
| CN108664516A (en) | Enquiring and optimizing method and relevant apparatus | |
| US9930113B2 (en) | Data retrieval via a telecommunication network | |
| CN110134705A (en) | A kind of data query method, cache server and terminal | |
| US20200089678A1 (en) | Autonomic caching for in memory data grid query processing | |
| CN103778251A (en) | SPARQL parallel query method facing large-scale RDF graph data | |
| CN116108057B (en) | Distributed database access method, device, equipment and storage medium | |
| US11645281B1 (en) | Caching query plans in database systems | |
| CN112163001A (en) | High-concurrency query method, intelligent terminal and storage medium | |
| CN110096521A (en) | Log information processing method and device | |
| CN104199978A (en) | System and method for realizing metadata cache and analysis based on NoSQL and method | |
| CN100534084C (en) | Long-distance XML data updating method and system | |
| CN109299089A (en) | The calculating and storage method and calculating of a kind of label data of drawing a portrait and storage system | |
| CN111078494B (en) | Cache database monitoring method and server | |
| CN103218396B (en) | The management and running visual analysis method of static Web page is generated according to visitation frequency feature | |
| US20100332501A1 (en) | System and method for on-demand indexing | |
| CN101964816A (en) | Method and system for browsing data in browser/server (B/S) framework software system | |
| JP2011216029A (en) | Distributed memory database system, database server, data processing method, and program thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| CB03 | Change of inventor or designer information |
Inventor after: Xiang Yuliang Inventor after: Liu Gang Inventor after: Ge Weichun Inventor after: Ren Kaiyin Inventor after: Liu Shijin Inventor after: He Xiang Inventor after: Zhang Mingming Inventor after: Huang Gaopan Inventor after: Lei Zhenjiang Inventor after: Lv Xuming Inventor after: Li Zhao Inventor before: Xiang Yuliang Inventor before: Ren Kaiyin Inventor before: Liu Shijin Inventor before: He Xiang Inventor before: Zhang Mingming Inventor before: Huang Gaopan |
|
| COR | Change of bibliographic data | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20160206 Address after: 100031 Xicheng District West Chang'an Avenue, No. 86, Beijing Applicant after: State Grid Corporation of China Applicant after: Nanjing Nari Co., Ltd. Applicant after: NANJING NARI INFORMATION COMMUNICATION SCIENCE & TECHNOLOGY CO., LTD. Applicant after: Information & Telecommunication Branch of State Grid Jiangsu Electric Power Company Applicant after: State Grid Liaoning Electric Power Co., Ltd. information communication company Address before: 100031 Xicheng District West Chang'an Avenue, No. 86, Beijing Applicant before: State Grid Corporation of China Applicant before: Nanjing Nari Co., Ltd. Applicant before: NANJING NARI INFORMATION COMMUNICATION SCIENCE & TECHNOLOGY CO., LTD. Applicant before: Information & Telecommunication Branch of State Grid Jiangsu Electric Power Company |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 100031 Xicheng District West Chang'an Avenue, No. 86, Beijing Co-patentee after: NARI Group Corp. Patentee after: State Grid Corporation of China Co-patentee after: NARI INFORMATION AND COMMUNICATION TECHNOLOGY Co. Co-patentee after: INFORMATION & TELECOMMUNICATION BRANCH OF STATE GRID JIANGSU ELECTRIC POWER Co. Co-patentee after: State Grid Liaoning Electric Power Co.,Ltd. 's information communication common carrier Address before: 100031 Xicheng District West Chang'an Avenue, No. 86, Beijing Co-patentee before: NARI Group CORPORATION STATE GRID ELECTRIC POWER INSTITUTE Patentee before: State Grid Corporation of China Co-patentee before: NARI INFORMATION AND COMMUNICATION TECHNOLOGY Co. Co-patentee before: INFORMATION & TELECOMMUNICATION BRANCH OF STATE GRID JIANGSU ELECTRIC POWER Co. Co-patentee before: State Grid Liaoning Electric Power Co.,Ltd. 's information communication common carrier |
|
| CP01 | Change in the name or title of a patent holder |