CN105320757A - Business intelligent analysis method for quickly processing data - Google Patents
Business intelligent analysis method for quickly processing data Download PDFInfo
- Publication number
- CN105320757A CN105320757A CN201510675705.8A CN201510675705A CN105320757A CN 105320757 A CN105320757 A CN 105320757A CN 201510675705 A CN201510675705 A CN 201510675705A CN 105320757 A CN105320757 A CN 105320757A
- Authority
- CN
- China
- Prior art keywords
- data
- layer
- model
- module
- carried out
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2216/00—Indexing scheme relating to additional aspects of information retrieval not explicitly covered by G06F16/00 and subgroups
- G06F2216/03—Data mining
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention discloses a business intelligent analysis method for quickly processing data. The method comprises the following steps of: S1, storing various key data source information by a data source layer; S2, performing integration on data on the data source layer by a collection layer, and transmitting the data to a storage layer; S3, the storage layer transmitting data in a data source to a calculation layer after integrating and processing the data in the data source; S4, calculating data of the storage layer by a real-time calculation module, an offline calculation module and a parallel calculation module in the calculation layer; and S5, transmitting the data calculated by the calculation layer in the step S4 to a model layer for processing, uploading the data to a service layer by an interface layer after the data are processed by the model layer, and performing reporting, query, mining, extraction and data visualization application on the data processed by the model layer through the service layer. The business intelligent analysis method for quickly processing data, provided by the present invention, is reasonable in design, realizes interaction and free combination between the data, also realizes secondary mining of the data, and can realize a function of quickly processing big data volume.
Description
Technical field
The present invention relates to Intellectual analysis technical field, particularly relate to a kind of Intellectual analysis method of fast processing data.
Background technology
Business intelligence (BusinessIntelligence, hereinafter referred to as BI) system, it is the solution of complete set, is used for data existing in enterprise effectively to integrate, there is provided form fast and accurately and propose decision-making foundation, helping enterprise to make wise business business decision.The key that BI system is set up goes out useful data from many extracting data from different tissues operation system and clear up, to ensure the correctness of data, then through extracting (Extraction), conversion (Transformation) and loading (Load), i.e. ETL process, be merged in the data warehouse of an enterprise-level, thus obtain a global view of business data, utilize suitable inquiry and analysis instrument on this basis, Data Mining Tools, OLAP instrument etc. carries out treatment and analysis (now information becomes the knowledge of aid decision making) to it, finally knowledge is presented to supvr, for supvr's decision process provides support.Existing based on BI system business intelligent analysis method usually from operation flow collection data, for describing business evolve state and setting up index and carrying out Intellectual analysis from bottom to top, there is subject matter and be in this kind of method: can only provide data display, data interaction and independent assortment can not be carried out, the mining again of data can not be carried out, especially can not meet the fast processing of present mass data.
Summary of the invention
For deficiency of the prior art, the invention provides a kind of Intellectual analysis method of fast processing data, it achieves the mutual and independent assortment between data, also achieves the mining again of data and the function of fast processing big data quantity simultaneously.
To achieve these goals, the technical solution used in the present invention is:
An Intellectual analysis method for fast processing data, comprises the following steps:
S1, stores all kinds of critical data source-information by data active layer;
S2, data in data active layer are integrated by acquisition layer, metadata acquisition tool module is provided with in acquisition layer, data cleansing tool model and data processing tools module, data in data active layer are successively by the metadata acquisition tool module on acquisition layer, accumulation layer is passed to after data cleansing tool model and data processing tools resume module, the data of process can be formed unified interface data view by acquisition layer, and be divided into different subject datas according to business demand, also be conducive to acquisition layer simultaneously and reduce the impact of data extraction process for data source layer system,
S3, flow data in data source is carried out collection and integrates by accumulation layer, then flow data is carried out distributed queue, next flow data pre-service is carried out, and the factual data in data source, dimension data and metric data are carried out collection integration by accumulation layer respectively, then factual data, dimension data and metric data are all carried out, in the distributed file system HDFS that distributed storage Hadoop provides, next carrying out batch data pre-service, then sending computation layer to;
S4, the data of accumulation layer are calculated by the real-time computing module in computation layer, calculated off-line module and parallel computation module, parallel computation module realizes the concurrent processing of multinode and multitask by MapReduce, greatly improves computing power, reduces the time of data mart modeling;
S5, in model layer by with data and algorithm for object sets up data model and algorithm model, the data that in above-mentioned steps 4, computation layer is calculated are transferred to data model successively and algorithm model processes, then after the data of algorithm model process are processed by the data analysis frame module in model layer and data mining framework module, operation layer is uploaded to by interface layer, form can be carried out to the data after model layer process by operation layer, inquiry, excavate, extract and data visualization application, described interface layer comprises technical protocol (Http/Socket) and data protocol (SML/JSOM) two kinds of forms, support traditional interface mode and data layout.
Beneficial effect of the present invention: the Intellectual analysis method that the invention provides fast processing data, what efficiently solve the existence of existing BI system can only provide data display, data interaction and independent assortment can not be carried out, the mining again of data can not be carried out, especially the deficiency of the fast processing of present mass data can not be met, having can fast processing mass data, mutual and independent assortment between data, be conducive to business personnel also can independently be analyzed, improve work efficiency, be highly suitable for the business administration analytic system in society, for the management decision of business administration analyst provides strong support.
Accompanying drawing explanation
Fig. 1 is schematic flow sheet of the present invention.
Embodiment
As shown in Figure 1, a kind of Intellectual analysis method of fast processing data, comprises the following steps:
S1, all kinds of critical data source-information is stored by data active layer, data active layer is simply divided into structured database, unstructured data and other forms, then data is simply concluded in the structured database of correspondence, unstructured data and other forms;
S2, data in data active layer are integrated by acquisition layer, metadata acquisition tool module is provided with in acquisition layer, data cleansing tool model and data processing tools module, data in data active layer are successively by the metadata acquisition tool module on acquisition layer, accumulation layer is passed to after data cleansing tool model and data processing tools resume module, the data of process can be formed unified interface data view by acquisition layer, and be divided into different subject datas according to business demand, also be conducive to acquisition layer simultaneously and reduce the impact of data extraction process for data source layer system,
S3, flow data in data source is carried out collection and integrates by accumulation layer, then flow data is carried out distributed queue, next flow data pre-service is carried out, and the factual data in data source, dimension data and metric data are carried out collection integration by accumulation layer respectively, then factual data, dimension data and metric data are all carried out, in the distributed file system HDFS that distributed storage Hadoop provides, next carrying out batch data pre-service, then sending computation layer to;
S4, the data of accumulation layer are calculated by the real-time computing module in computation layer, calculated off-line module and parallel computation module, parallel computation module realizes the concurrent processing of multinode and multitask by MapReduce, greatly improves computing power, reduces the time of data mart modeling;
S5, in model layer by with data and algorithm for object sets up data model and algorithm model, the data that in above-mentioned steps 4, computation layer is calculated are transferred to data model successively and algorithm model processes, then after the data of algorithm model process are processed by the data analysis frame module in model layer and data mining framework module, operation layer is uploaded to by interface layer, form can be carried out to the data after model layer process by operation layer, inquiry, excavate, extract and data visualization application, described interface layer comprises technical protocol (Http/Socket) and data protocol (SML/JSOM) two kinds of forms, support traditional interface mode and data layout.
The present invention has the more analysis of support and data type, support distributed storage and the parallel computation of thousands of computer nodes and PB level mass data, can analyze Volume data, the analysis of more complicated can be simplified simultaneously, meet the large data Intellectual analysis method of business intelligence application, so just can process mass data at short notice, complete calculating and the storage of mass data, the present invention is the data management of a collection, analyze, statistics, monitoring, early warning, be predicted as the multifunctional platform of one, adopt the new technique framework of quick BI system, the present invention can support Distributed Calculation, internal memory calculates, row store, the large data technique such as calculating in storehouse, utilize data mining, intellectual analysis, the data processing technique such as multidimensional analysis and cloud computing, greatly improve the performance of BI system, the present invention carries out high-speed capture and real-time analysis to large data, by the data message of complexity by showing various valuable information after analytical calculation, to obtain the key message needed for core business and strategic decision, for company manager provides the commercial intelligence resolution of specialty, assist the timely adjustable strategies of company manager, thus enterprise operation control and strategic decision level, strengthen the Sustainable Competitiveness of enterprise, the commercial value that final creation is huge.
Beneficial effect of the present invention: the Intellectual analysis method that the invention provides fast processing data, what efficiently solve the existence of existing BI system can only provide data display, data interaction and independent assortment can not be carried out, the mining again of data can not be carried out, especially the deficiency of the fast processing of present mass data can not be met, having can fast processing mass data, mutual and independent assortment between data, be conducive to business personnel also can independently be analyzed, improve work efficiency, be highly suitable for the business administration analytic system in society, for the management decision of business administration analyst provides strong support.
Claims (3)
1. an Intellectual analysis method for fast processing data, is characterized in that, comprise the following steps:
S1, stores all kinds of critical data source-information by data active layer;
S2, the data in data active layer are integrated by acquisition layer, then pass to accumulation layer;
S3, flow data in data source is carried out collection and integrates by accumulation layer, then flow data is carried out distributed queue, next flow data pre-service is carried out, and the factual data in data source, dimension data and metric data are carried out collection integration by accumulation layer respectively, then factual data, dimension data and metric data are all carried out, in the distributed file system HDFS that distributed storage Hadoop provides, next carrying out batch data pre-service, then sending computation layer to;
S4, calculates the data of accumulation layer by the real-time computing module in computation layer, calculated off-line module and parallel computation module;
S5, in model layer by with data and algorithm for object sets up data model and algorithm model, the data that in above-mentioned steps 4, computation layer is calculated are transferred to data model successively and algorithm model processes, then after the data of algorithm model process are processed by the data analysis frame module in model layer and data mining framework module, be uploaded to operation layer by interface layer, form, inquiry, excavation, extraction and data visualization application can be carried out to the data after model layer process by operation layer.
2. the Intellectual analysis method of a kind of fast processing data as claimed in claim 1, is characterized in that, is provided with metadata acquisition tool module, data cleansing tool model and data processing tools module in described acquisition layer.
3. the Intellectual analysis method of a kind of fast processing data as claimed in claim 1, is characterized in that, described interface layer comprises technical protocol and data protocol two kinds of forms.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510675705.8A CN105320757A (en) | 2015-10-19 | 2015-10-19 | Business intelligent analysis method for quickly processing data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510675705.8A CN105320757A (en) | 2015-10-19 | 2015-10-19 | Business intelligent analysis method for quickly processing data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105320757A true CN105320757A (en) | 2016-02-10 |
Family
ID=55248143
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510675705.8A Pending CN105320757A (en) | 2015-10-19 | 2015-10-19 | Business intelligent analysis method for quickly processing data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105320757A (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105787116A (en) * | 2016-03-25 | 2016-07-20 | 南京邮电大学 | Cognitive computing architecture based on context-aware data streams |
| CN106570107A (en) * | 2016-11-01 | 2017-04-19 | 广西电网有限责任公司电力科学研究院 | Big data calculation and analysis scheme fruiting system |
| CN107093019A (en) * | 2017-04-21 | 2017-08-25 | 北京恒冠网络数据处理有限公司 | A kind of big data analysis system for macro adjustments and controls |
| CN107391550A (en) * | 2017-06-06 | 2017-11-24 | 广东广业开元科技有限公司 | A kind of report form generation method and system based on big data mould plate technique |
| CN108268645A (en) * | 2018-01-23 | 2018-07-10 | 广州南方人才资讯科技有限公司 | Big data processing method and system |
| CN108319538A (en) * | 2018-02-02 | 2018-07-24 | 世纪龙信息网络有限责任公司 | The monitoring method and system of big data platform operating status |
| CN108563666A (en) * | 2018-01-05 | 2018-09-21 | 成都兴政电子政务运营服务有限公司 | A kind of data visualization processing system and method based on big data technology |
| CN109271581A (en) * | 2018-08-02 | 2019-01-25 | 北京天元创新科技有限公司 | A kind of quick rendering method of big data based on Dashboard |
| CN109408567A (en) * | 2018-09-11 | 2019-03-01 | 广东布田电子商务有限公司 | A kind of big data processing platform network architecture |
| CN109492130A (en) * | 2018-10-09 | 2019-03-19 | 象翌微链科技发展有限公司 | A kind of data manipulation method and system |
| CN110188088A (en) * | 2019-05-23 | 2019-08-30 | 广东海洋大学 | A kind of marine ships adopt sand behavior big data model |
| CN111143328A (en) * | 2019-12-26 | 2020-05-12 | 山东翰林科技有限公司 | Agile business intelligent data construction method, system, equipment and storage medium |
| CN113076370A (en) * | 2021-04-23 | 2021-07-06 | 上海寒光信息科技有限公司 | Internet data sky-eye agent BI system |
| CN114817264A (en) * | 2022-04-28 | 2022-07-29 | 电子科技大学 | Topology query structure, query method, electronic device and medium for graph computing |
| CN115794044A (en) * | 2023-01-31 | 2023-03-14 | 帆软软件有限公司帆软南京分公司 | Analysis theme system and analysis theme display method of BI tool |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102185927A (en) * | 2011-05-31 | 2011-09-14 | 北京亿赞普网络技术有限公司 | Knowledge-as-service cloud computing platform |
| CN103164476A (en) * | 2011-12-16 | 2013-06-19 | 中国移动通信集团公司 | Execution method and execution device of applying metadata to describe files in business intelligence (BI) |
| CN104182389A (en) * | 2014-07-21 | 2014-12-03 | 安徽华贞信息科技有限公司 | Semantic-based big data analysis business intelligence service system |
| CN104281697A (en) * | 2014-10-15 | 2015-01-14 | 安徽华贞信息科技有限公司 | Semantic-based hadoop system |
| CN104320848A (en) * | 2014-11-14 | 2015-01-28 | 上海美琦浦悦通讯科技有限公司 | Indoor locating implementation system and method based on cloud computing |
| CN104767813A (en) * | 2015-04-08 | 2015-07-08 | 江苏国盾科技实业有限责任公司 | Public bank big data service platform based on openstack |
| CN104933112A (en) * | 2015-06-04 | 2015-09-23 | 浙江力石科技股份有限公司 | Distributed Internet transaction information storage and processing method |
-
2015
- 2015-10-19 CN CN201510675705.8A patent/CN105320757A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102185927A (en) * | 2011-05-31 | 2011-09-14 | 北京亿赞普网络技术有限公司 | Knowledge-as-service cloud computing platform |
| CN103164476A (en) * | 2011-12-16 | 2013-06-19 | 中国移动通信集团公司 | Execution method and execution device of applying metadata to describe files in business intelligence (BI) |
| CN104182389A (en) * | 2014-07-21 | 2014-12-03 | 安徽华贞信息科技有限公司 | Semantic-based big data analysis business intelligence service system |
| CN104281697A (en) * | 2014-10-15 | 2015-01-14 | 安徽华贞信息科技有限公司 | Semantic-based hadoop system |
| CN104320848A (en) * | 2014-11-14 | 2015-01-28 | 上海美琦浦悦通讯科技有限公司 | Indoor locating implementation system and method based on cloud computing |
| CN104767813A (en) * | 2015-04-08 | 2015-07-08 | 江苏国盾科技实业有限责任公司 | Public bank big data service platform based on openstack |
| CN104933112A (en) * | 2015-06-04 | 2015-09-23 | 浙江力石科技股份有限公司 | Distributed Internet transaction information storage and processing method |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105787116A (en) * | 2016-03-25 | 2016-07-20 | 南京邮电大学 | Cognitive computing architecture based on context-aware data streams |
| CN106570107B (en) * | 2016-11-01 | 2019-08-20 | 广西电网有限责任公司电力科学研究院 | A kind of big data calculating analytical plan achievement system |
| CN106570107A (en) * | 2016-11-01 | 2017-04-19 | 广西电网有限责任公司电力科学研究院 | Big data calculation and analysis scheme fruiting system |
| CN107093019A (en) * | 2017-04-21 | 2017-08-25 | 北京恒冠网络数据处理有限公司 | A kind of big data analysis system for macro adjustments and controls |
| CN107391550A (en) * | 2017-06-06 | 2017-11-24 | 广东广业开元科技有限公司 | A kind of report form generation method and system based on big data mould plate technique |
| CN108563666B (en) * | 2018-01-05 | 2022-04-05 | 四川兴政信息技术有限公司 | Data visualization processing system and method based on big data technology |
| CN108563666A (en) * | 2018-01-05 | 2018-09-21 | 成都兴政电子政务运营服务有限公司 | A kind of data visualization processing system and method based on big data technology |
| CN108268645A (en) * | 2018-01-23 | 2018-07-10 | 广州南方人才资讯科技有限公司 | Big data processing method and system |
| CN108319538B (en) * | 2018-02-02 | 2019-11-08 | 世纪龙信息网络有限责任公司 | The monitoring method and system of big data platform operating status |
| CN108319538A (en) * | 2018-02-02 | 2018-07-24 | 世纪龙信息网络有限责任公司 | The monitoring method and system of big data platform operating status |
| CN109271581A (en) * | 2018-08-02 | 2019-01-25 | 北京天元创新科技有限公司 | A kind of quick rendering method of big data based on Dashboard |
| CN109408567A (en) * | 2018-09-11 | 2019-03-01 | 广东布田电子商务有限公司 | A kind of big data processing platform network architecture |
| CN109492130A (en) * | 2018-10-09 | 2019-03-19 | 象翌微链科技发展有限公司 | A kind of data manipulation method and system |
| CN110188088A (en) * | 2019-05-23 | 2019-08-30 | 广东海洋大学 | A kind of marine ships adopt sand behavior big data model |
| CN111143328A (en) * | 2019-12-26 | 2020-05-12 | 山东翰林科技有限公司 | Agile business intelligent data construction method, system, equipment and storage medium |
| CN113076370A (en) * | 2021-04-23 | 2021-07-06 | 上海寒光信息科技有限公司 | Internet data sky-eye agent BI system |
| CN114817264A (en) * | 2022-04-28 | 2022-07-29 | 电子科技大学 | Topology query structure, query method, electronic device and medium for graph computing |
| CN115794044A (en) * | 2023-01-31 | 2023-03-14 | 帆软软件有限公司帆软南京分公司 | Analysis theme system and analysis theme display method of BI tool |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105320757A (en) | Business intelligent analysis method for quickly processing data | |
| Santos et al. | A big data system supporting bosch braga industry 4.0 strategy | |
| CN104820670B (en) | A kind of acquisition of power information big data and storage method | |
| CN107330056B (en) | Wind power plant SCADA system based on big data cloud computing platform and operation method thereof | |
| CN103927346B (en) | Query connection method on basis of data volumes | |
| CN105787059A (en) | Data warehouse based financial data integration method | |
| CN104361022A (en) | Method based on collected data statistics and foreground display | |
| CN108614870B (en) | Rapid batch warehousing system and method for external data information of power equipment | |
| CN115860529A (en) | Supply chain carbon accounting system based on industrial internet | |
| CN112883001A (en) | Data processing method, device and medium based on marketing and distribution through data visualization platform | |
| CN111159152B (en) | Secondary operation and data fusion method based on big data processing technology | |
| CN111125450A (en) | Management method of multilayer topology network resource object | |
| Liu et al. | Research on k-means algorithm based on cloud computing | |
| CN107463151B (en) | A kind of complex surface machining multidimensional knowledge cloud cooperating service method | |
| CN113722564A (en) | Visualization method and device for energy and material supply chain based on space map convolution | |
| CN204790999U (en) | Big data acquisition of industry and processing system | |
| Escobedo et al. | Business intelligence and data analytics (BI&DA) to support the operation of smart grid | |
| CN114047729B (en) | Natural plant processing control method, system, computer device and storage medium | |
| CN104715301A (en) | Sugar manufacturing production index and production technology evaluating system based on cloud storage | |
| Viharos et al. | ” Big Data” Initiative as an IT Solution for Improved Operation and Maintenance of Wind Turbines | |
| Li | Construction of an interactive sharing platform for competitive intelligence data of marine resources under the background of intelligence construction | |
| CN110955521A (en) | Power distribution network wide area distributed type sinking calculation system and method based on big data | |
| Zhou et al. | Research on improving intelligent inspection efficiency of substation based on big data analysis | |
| Lou et al. | A hybrid data storage method for a robotic arm digital twin prototype | |
| Bhardwaj et al. | Data analyzing using Map-Join-Reduce in cloud storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20160210 |