CN110866019A - Renewable quasi-real-time BI analysis system - Google Patents
Renewable quasi-real-time BI analysis system Download PDFInfo
- Publication number
- CN110866019A CN110866019A CN201810908921.6A CN201810908921A CN110866019A CN 110866019 A CN110866019 A CN 110866019A CN 201810908921 A CN201810908921 A CN 201810908921A CN 110866019 A CN110866019 A CN 110866019A
- Authority
- CN
- China
- Prior art keywords
- data
- real
- time
- analysis system
- data source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an updatable quasi-real-time BI analysis system, and aims to provide a BI analysis system with a front end capable of supporting visual display of multiple report tools of jdbc/odbc. And (3) logging in the BI system by an administrator, and configuring the data source, wherein the configuration of the data source comprises information such as an address, a port, a database, a table and required fields. After confirming that no error exists, saving and waiting for a system initialization data source; and then configuring model data, including a dimension value and a metric value, and storing after confirming that no errors exist. Building Cube on the configured model, selecting a time range for full construction in the first operation, and configuring a timing period for incremental construction in the subsequent operation; after the construction is completed, the front-end user can use the jdbc and odbc connection system to query data according to the model.
Description
Technical Field
The invention belongs to the field of data processing, and particularly relates to a renewable quasi-real-time BI analysis system.
Background
The bi (business intelligence), which is a complete solution, is used to effectively integrate the existing data in the enterprise, quickly and accurately provide reports and provide decision basis, and help the enterprise make intelligent business operation decision.
The key of business intelligence is to extract useful data from many data from different enterprise operating systems and clean the data to ensure the correctness of the data, then merge the data into an enterprise-level data warehouse through Extraction (Extraction), Transformation (Transformation) and loading (Load), i.e. ETL process, so as to obtain a global view of the enterprise data, analyze and process the data on the basis by using appropriate query and analysis tools, data mining tools, OLAP tools and the like (at this time, information becomes knowledge for assisting decision making), and finally present the knowledge to a manager to provide data support for the decision making process of the manager. The commercial intelligent products and solutions can be roughly classified into data warehouse products, data extraction products, OLAP products, display products, and overall solutions for a certain application integrating the above products, and the like.
At present, chinese patent publication No. CN106815373A discloses a distribution network emergency repair big data display method and system based on BI analysis, which collects the existing work order data of the distribution network emergency repair command platform, the interface data of the power marketing service application system, the interface data of the power utilization information acquisition system, the interface data of the distribution automation, and the like. By sending the data to the BI database, the analysis and comparison of the related information of the distribution network emergency repair command platform are realized on the basis of the existing data analysis algorithm software. And once data abnormity occurs, automatically screening out abnormal information, checking the screened abnormal information, sending the checked abnormal information to the database again for analysis, and displaying an analysis result at the front end to form a distribution network emergency repair command platform service data report and an analysis statistical report.
In a BI analysis scenario with large data volume and high performance requirements, a conventional BI system either performs offline calculation but can support big data association, or performs real-time calculation but cannot support big data association. And scenes with certain requirements on data real-time performance cannot be effectively supported.
Disclosure of Invention
It is an object of the present invention to provide an updatable quasi real-time BI analysis system. The method has the advantages that the data layer work of the complex BI report is completed through simple data source configuration and certain data warehouse experience design, and the front end can support the visual display of various report tools of jdbc/odbc.
The purpose of the invention can be realized by the following technical scheme:
an updatable near real-time BI analysis system comprising a data source, a kafka, an ODS store, a compute engine, and a precomputation engine, the data source comprising a storage administrator-defined data source configuration and a model configuration;
the kafka receives the data sent by the data source for real-time synchronization;
the ODS stores the synchronous data sent by the kafka received by the Kudu and updates in real time;
the calculation engine converts the design of the model configuration into a view and provides data support for a subsequent precomputation engine;
and the pre-calculation engine adopts Kylin to configure the defined model to generate Cube, and Cube is established and finally stored in Hbase for being inquired and used by a front-end user.
Preferably, the data source configuration includes an address, a port, a database, a table, and required field information.
Preferably, the model configuration comprises a dimension value and a metric value.
Preferably, the source data supports postgre and mysql, and other data sources with effective synchronization strategies can also be accessed.
Preferably, the Cuboid is constructed by a combination of dimensional values.
The invention has the beneficial effects that:
a perfect solution is provided for scenes with large data volume, high performance, data updating and real-time requirements, the data layer work of a complex BI report can be completed through simple data source configuration and combination with model design with certain data warehouse experience, and the front end can be visually displayed by using various report tools supporting jdbc/odbc.
Drawings
Fig. 1 is a schematic diagram of a system architecture.
Detailed Description
Example 1: an updatable near real-time BI analysis system includes a data source, a kafka, an ODS store, a calculation engine, and a precomputation engine.
Wherein the data source comprises a storage administrator defined data source configuration and a model configuration.
The model configuration includes a dimension value and a metric value.
The source data supports postgre and mysql, and other data sources with effective synchronization strategies and the like can be accessed.
And the kafka accepts the data sent by the data source for real-time synchronization.
And the ODS stores the synchronous data sent by the kafka received by Kudu and updates the synchronous data in real time.
The calculation engine converts the design of the model configuration into a view, and provides data support for a subsequent precomputation engine.
And the pre-calculation engine adopts Kylin to configure the defined model to generate Cube, and Cube is established through the dimension value and finally stored in Hbase for the front-end user to inquire and use.
And defining a pre-calculation construction period according to the data scale, and triggering a construction task in real time.
The specific operation is as follows:
(1) the administrator logs in the BI system and configures the data source, wherein the data source configuration comprises information such as addresses, ports, databases, tables and required fields. After confirming that no error exists, saving and waiting for a system initialization data source;
(2) and then configuring model data, including a dimension value and a metric value, and storing after confirming that no errors exist.
(3) Building Cube on the configured model, selecting a time range for full construction in the first operation, and configuring a timing period for incremental construction in the subsequent operation;
(4) after the construction is completed, the front-end user can use the jdbc and odbc connection system to query data according to the model.
A perfect solution is provided for scenes with large data volume, high performance, data updating and real-time requirements, the data layer work of a complex BI report can be completed through simple data source configuration and combination with model design with certain data warehouse experience, and the front end can be visually displayed by using various report tools supporting jdbc/odbc.
Finally, it should be noted that: although the present invention has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that: modifications and equivalents may be made thereto without departing from the spirit and scope of the invention, and the appended claims are intended to cover such modifications and equivalents as fall within the true spirit and scope of the invention.
Claims (5)
1. An updatable near real-time BI analysis system, comprising: the system comprises a data source, a kafka, an ODS storage, a calculation engine and a pre-calculation engine, wherein the data source comprises a data source configuration and a model configuration defined by a storage administrator;
the kafka receives the data sent by the data source for real-time synchronization;
the ODS stores the synchronous data sent by the kafka received by the Kudu and updates in real time;
the calculation engine converts the design of the model configuration into a view and provides data support for a subsequent precomputation engine;
and the pre-calculation engine adopts Kylin to configure the defined model to generate Cube, and Cube is established and finally stored in Hbase for being inquired and used by a front-end user.
2. The updatable quasi-real-time BI analysis system of claim 1, wherein: the data source configuration includes address, port, database, table, and required field information.
3. The updatable quasi-real-time BI analysis system of claim 1, wherein: the model configuration includes a dimension value and a metric value.
4. The updatable quasi-real-time BI analysis system of claim 1, wherein: the source data supports postgre and mysql, and other data sources with effective synchronization strategies can be accessed.
5. The updatable quasi-real-time BI analysis system of claim 3, wherein: the Cuboid is constructed by a combination of dimensional values.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810908921.6A CN110866019A (en) | 2018-08-10 | 2018-08-10 | Renewable quasi-real-time BI analysis system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810908921.6A CN110866019A (en) | 2018-08-10 | 2018-08-10 | Renewable quasi-real-time BI analysis system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110866019A true CN110866019A (en) | 2020-03-06 |
Family
ID=69650815
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810908921.6A Pending CN110866019A (en) | 2018-08-10 | 2018-08-10 | Renewable quasi-real-time BI analysis system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110866019A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112485396A (en) * | 2020-11-12 | 2021-03-12 | 电子科技大学中山学院 | Aquaculture water quality monitoring system based on big data |
| CN113076370A (en) * | 2021-04-23 | 2021-07-06 | 上海寒光信息科技有限公司 | Internet data sky-eye agent BI system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120101860A1 (en) * | 2010-10-25 | 2012-04-26 | Ezzat Ahmed K | Providing business intelligence |
| CN107169070A (en) * | 2017-05-08 | 2017-09-15 | 山大地纬软件股份有限公司 | The constructing system and its method in a kind of social security index warehouse based on big data |
| CN107704608A (en) * | 2017-10-17 | 2018-02-16 | 北京览群智数据科技有限责任公司 | A kind of OLAP multidimensional analyses and data digging system |
-
2018
- 2018-08-10 CN CN201810908921.6A patent/CN110866019A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120101860A1 (en) * | 2010-10-25 | 2012-04-26 | Ezzat Ahmed K | Providing business intelligence |
| CN107169070A (en) * | 2017-05-08 | 2017-09-15 | 山大地纬软件股份有限公司 | The constructing system and its method in a kind of social security index warehouse based on big data |
| CN107704608A (en) * | 2017-10-17 | 2018-02-16 | 北京览群智数据科技有限责任公司 | A kind of OLAP multidimensional analyses and data digging system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112485396A (en) * | 2020-11-12 | 2021-03-12 | 电子科技大学中山学院 | Aquaculture water quality monitoring system based on big data |
| CN113076370A (en) * | 2021-04-23 | 2021-07-06 | 上海寒光信息科技有限公司 | Internet data sky-eye agent BI system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105046328B (en) | A kind of three-dimensional visualization bridge defect information acquisition management system and method | |
| CN110493025B (en) | Fault root cause diagnosis method and device based on multilayer digraphs | |
| CN104899295B (en) | A kind of heterogeneous data source data relation analysis method | |
| CN107038162A (en) | Real time data querying method and system based on database journal | |
| CN102254024A (en) | Mass data processing system and method | |
| CN104657387B (en) | A kind of data query method and device | |
| CN112347071B (en) | Power distribution network cloud platform data fusion method and power distribution network cloud platform | |
| CN107145576B (en) | Big data ETL scheduling system supporting visualization and process | |
| CN109190984B (en) | Data processing system and method based on data cube model | |
| CN110866019A (en) | Renewable quasi-real-time BI analysis system | |
| CN111737355A (en) | MongoDB metadata management-based heterogeneous data source synchronization method and system | |
| CN104199978A (en) | System and method for realizing metadata cache and analysis based on NoSQL and method | |
| CN114647716A (en) | Generalization data warehouse | |
| CN112148261A (en) | Data center platform design method of intelligent shipyard digital service platform | |
| CN105373446B (en) | It is a kind of based on the system self-repairing method drilled automatically and device | |
| CN106649873A (en) | Multi-dimensional processing system for FMEA data | |
| CN105630997A (en) | Data parallel processing method, device and equipment | |
| CN114490610A (en) | Data processing method and device for data bin, storage medium and electronic device | |
| CN113610466A (en) | Intelligent warehousing equipment management method and system | |
| CN112256489A (en) | Data acquisition method and device of cloud development platform and data storage architecture | |
| Wang et al. | Design and implementation of an ETL approach in business intelligence project | |
| CN111143406A (en) | Database data comparison method and database data comparison system | |
| CN117762900A (en) | Data modeling method capable of improving calculation performance of risk screening system of publisher | |
| Fattakhova et al. | Ways to Collect Disparate Information in a Single Data Warehouse at a Machine-Building Enterprise | |
| Chen et al. | Design of Big Data Platform for TFT-LCD Line Yield Analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |