CN105320757A - Business intelligent analysis method for quickly processing data - Google Patents

Business intelligent analysis method for quickly processing data Download PDF

Info

Publication number
CN105320757A
CN105320757A CN201510675705.8A CN201510675705A CN105320757A CN 105320757 A CN105320757 A CN 105320757A CN 201510675705 A CN201510675705 A CN 201510675705A CN 105320757 A CN105320757 A CN 105320757A
Authority
CN
China
Prior art keywords
data
layer
model
module
carried out
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201510675705.8A
Other languages
Chinese (zh)
Inventor
战国科
王真震
陈杰辉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Hualiang Software Co Ltd
Original Assignee
Hangzhou Hualiang Software Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Hualiang Software Co Ltd filed Critical Hangzhou Hualiang Software Co Ltd
Priority to CN201510675705.8A priority Critical patent/CN105320757A/en
Publication of CN105320757A publication Critical patent/CN105320757A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/248Presentation of query results
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2457Query processing with adaptation to user needs
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2216/00Indexing scheme relating to additional aspects of information retrieval not explicitly covered by G06F16/00 and subgroups
    • G06F2216/03Data mining

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The present invention discloses a business intelligent analysis method for quickly processing data. The method comprises the following steps of: S1, storing various key data source information by a data source layer; S2, performing integration on data on the data source layer by a collection layer, and transmitting the data to a storage layer; S3, the storage layer transmitting data in a data source to a calculation layer after integrating and processing the data in the data source; S4, calculating data of the storage layer by a real-time calculation module, an offline calculation module and a parallel calculation module in the calculation layer; and S5, transmitting the data calculated by the calculation layer in the step S4 to a model layer for processing, uploading the data to a service layer by an interface layer after the data are processed by the model layer, and performing reporting, query, mining, extraction and data visualization application on the data processed by the model layer through the service layer. The business intelligent analysis method for quickly processing data, provided by the present invention, is reasonable in design, realizes interaction and free combination between the data, also realizes secondary mining of the data, and can realize a function of quickly processing big data volume.

Description

A kind of Intellectual analysis method of fast processing data
Technical field
The present invention relates to Intellectual analysis technical field, particularly relate to a kind of Intellectual analysis method of fast processing data.
Background technology
Business intelligence (BusinessIntelligence, hereinafter referred to as BI) system, it is the solution of complete set, is used for data existing in enterprise effectively to integrate, there is provided form fast and accurately and propose decision-making foundation, helping enterprise to make wise business business decision.The key that BI system is set up goes out useful data from many extracting data from different tissues operation system and clear up, to ensure the correctness of data, then through extracting (Extraction), conversion (Transformation) and loading (Load), i.e. ETL process, be merged in the data warehouse of an enterprise-level, thus obtain a global view of business data, utilize suitable inquiry and analysis instrument on this basis, Data Mining Tools, OLAP instrument etc. carries out treatment and analysis (now information becomes the knowledge of aid decision making) to it, finally knowledge is presented to supvr, for supvr's decision process provides support.Existing based on BI system business intelligent analysis method usually from operation flow collection data, for describing business evolve state and setting up index and carrying out Intellectual analysis from bottom to top, there is subject matter and be in this kind of method: can only provide data display, data interaction and independent assortment can not be carried out, the mining again of data can not be carried out, especially can not meet the fast processing of present mass data.
Summary of the invention
For deficiency of the prior art, the invention provides a kind of Intellectual analysis method of fast processing data, it achieves the mutual and independent assortment between data, also achieves the mining again of data and the function of fast processing big data quantity simultaneously.
To achieve these goals, the technical solution used in the present invention is:
An Intellectual analysis method for fast processing data, comprises the following steps:
S1, stores all kinds of critical data source-information by data active layer;
S2, data in data active layer are integrated by acquisition layer, metadata acquisition tool module is provided with in acquisition layer, data cleansing tool model and data processing tools module, data in data active layer are successively by the metadata acquisition tool module on acquisition layer, accumulation layer is passed to after data cleansing tool model and data processing tools resume module, the data of process can be formed unified interface data view by acquisition layer, and be divided into different subject datas according to business demand, also be conducive to acquisition layer simultaneously and reduce the impact of data extraction process for data source layer system,
S3, flow data in data source is carried out collection and integrates by accumulation layer, then flow data is carried out distributed queue, next flow data pre-service is carried out, and the factual data in data source, dimension data and metric data are carried out collection integration by accumulation layer respectively, then factual data, dimension data and metric data are all carried out, in the distributed file system HDFS that distributed storage Hadoop provides, next carrying out batch data pre-service, then sending computation layer to;
S4, the data of accumulation layer are calculated by the real-time computing module in computation layer, calculated off-line module and parallel computation module, parallel computation module realizes the concurrent processing of multinode and multitask by MapReduce, greatly improves computing power, reduces the time of data mart modeling;
S5, in model layer by with data and algorithm for object sets up data model and algorithm model, the data that in above-mentioned steps 4, computation layer is calculated are transferred to data model successively and algorithm model processes, then after the data of algorithm model process are processed by the data analysis frame module in model layer and data mining framework module, operation layer is uploaded to by interface layer, form can be carried out to the data after model layer process by operation layer, inquiry, excavate, extract and data visualization application, described interface layer comprises technical protocol (Http/Socket) and data protocol (SML/JSOM) two kinds of forms, support traditional interface mode and data layout.
Beneficial effect of the present invention: the Intellectual analysis method that the invention provides fast processing data, what efficiently solve the existence of existing BI system can only provide data display, data interaction and independent assortment can not be carried out, the mining again of data can not be carried out, especially the deficiency of the fast processing of present mass data can not be met, having can fast processing mass data, mutual and independent assortment between data, be conducive to business personnel also can independently be analyzed, improve work efficiency, be highly suitable for the business administration analytic system in society, for the management decision of business administration analyst provides strong support.
Accompanying drawing explanation
Fig. 1 is schematic flow sheet of the present invention.
Embodiment
As shown in Figure 1, a kind of Intellectual analysis method of fast processing data, comprises the following steps:
S1, all kinds of critical data source-information is stored by data active layer, data active layer is simply divided into structured database, unstructured data and other forms, then data is simply concluded in the structured database of correspondence, unstructured data and other forms;
S2, data in data active layer are integrated by acquisition layer, metadata acquisition tool module is provided with in acquisition layer, data cleansing tool model and data processing tools module, data in data active layer are successively by the metadata acquisition tool module on acquisition layer, accumulation layer is passed to after data cleansing tool model and data processing tools resume module, the data of process can be formed unified interface data view by acquisition layer, and be divided into different subject datas according to business demand, also be conducive to acquisition layer simultaneously and reduce the impact of data extraction process for data source layer system,
S3, flow data in data source is carried out collection and integrates by accumulation layer, then flow data is carried out distributed queue, next flow data pre-service is carried out, and the factual data in data source, dimension data and metric data are carried out collection integration by accumulation layer respectively, then factual data, dimension data and metric data are all carried out, in the distributed file system HDFS that distributed storage Hadoop provides, next carrying out batch data pre-service, then sending computation layer to;
S4, the data of accumulation layer are calculated by the real-time computing module in computation layer, calculated off-line module and parallel computation module, parallel computation module realizes the concurrent processing of multinode and multitask by MapReduce, greatly improves computing power, reduces the time of data mart modeling;
S5, in model layer by with data and algorithm for object sets up data model and algorithm model, the data that in above-mentioned steps 4, computation layer is calculated are transferred to data model successively and algorithm model processes, then after the data of algorithm model process are processed by the data analysis frame module in model layer and data mining framework module, operation layer is uploaded to by interface layer, form can be carried out to the data after model layer process by operation layer, inquiry, excavate, extract and data visualization application, described interface layer comprises technical protocol (Http/Socket) and data protocol (SML/JSOM) two kinds of forms, support traditional interface mode and data layout.
The present invention has the more analysis of support and data type, support distributed storage and the parallel computation of thousands of computer nodes and PB level mass data, can analyze Volume data, the analysis of more complicated can be simplified simultaneously, meet the large data Intellectual analysis method of business intelligence application, so just can process mass data at short notice, complete calculating and the storage of mass data, the present invention is the data management of a collection, analyze, statistics, monitoring, early warning, be predicted as the multifunctional platform of one, adopt the new technique framework of quick BI system, the present invention can support Distributed Calculation, internal memory calculates, row store, the large data technique such as calculating in storehouse, utilize data mining, intellectual analysis, the data processing technique such as multidimensional analysis and cloud computing, greatly improve the performance of BI system, the present invention carries out high-speed capture and real-time analysis to large data, by the data message of complexity by showing various valuable information after analytical calculation, to obtain the key message needed for core business and strategic decision, for company manager provides the commercial intelligence resolution of specialty, assist the timely adjustable strategies of company manager, thus enterprise operation control and strategic decision level, strengthen the Sustainable Competitiveness of enterprise, the commercial value that final creation is huge.
Beneficial effect of the present invention: the Intellectual analysis method that the invention provides fast processing data, what efficiently solve the existence of existing BI system can only provide data display, data interaction and independent assortment can not be carried out, the mining again of data can not be carried out, especially the deficiency of the fast processing of present mass data can not be met, having can fast processing mass data, mutual and independent assortment between data, be conducive to business personnel also can independently be analyzed, improve work efficiency, be highly suitable for the business administration analytic system in society, for the management decision of business administration analyst provides strong support.

Claims (3)

1. an Intellectual analysis method for fast processing data, is characterized in that, comprise the following steps:
S1, stores all kinds of critical data source-information by data active layer;
S2, the data in data active layer are integrated by acquisition layer, then pass to accumulation layer;
S3, flow data in data source is carried out collection and integrates by accumulation layer, then flow data is carried out distributed queue, next flow data pre-service is carried out, and the factual data in data source, dimension data and metric data are carried out collection integration by accumulation layer respectively, then factual data, dimension data and metric data are all carried out, in the distributed file system HDFS that distributed storage Hadoop provides, next carrying out batch data pre-service, then sending computation layer to;
S4, calculates the data of accumulation layer by the real-time computing module in computation layer, calculated off-line module and parallel computation module;
S5, in model layer by with data and algorithm for object sets up data model and algorithm model, the data that in above-mentioned steps 4, computation layer is calculated are transferred to data model successively and algorithm model processes, then after the data of algorithm model process are processed by the data analysis frame module in model layer and data mining framework module, be uploaded to operation layer by interface layer, form, inquiry, excavation, extraction and data visualization application can be carried out to the data after model layer process by operation layer.
2. the Intellectual analysis method of a kind of fast processing data as claimed in claim 1, is characterized in that, is provided with metadata acquisition tool module, data cleansing tool model and data processing tools module in described acquisition layer.
3. the Intellectual analysis method of a kind of fast processing data as claimed in claim 1, is characterized in that, described interface layer comprises technical protocol and data protocol two kinds of forms.
CN201510675705.8A 2015-10-19 2015-10-19 Business intelligent analysis method for quickly processing data Pending CN105320757A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510675705.8A CN105320757A (en) 2015-10-19 2015-10-19 Business intelligent analysis method for quickly processing data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510675705.8A CN105320757A (en) 2015-10-19 2015-10-19 Business intelligent analysis method for quickly processing data

Publications (1)

Publication Number Publication Date
CN105320757A true CN105320757A (en) 2016-02-10

Family

ID=55248143

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510675705.8A Pending CN105320757A (en) 2015-10-19 2015-10-19 Business intelligent analysis method for quickly processing data

Country Status (1)

Country Link
CN (1) CN105320757A (en)

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105787116A (en) * 2016-03-25 2016-07-20 南京邮电大学 Cognitive computing architecture based on context-aware data streams
CN106570107A (en) * 2016-11-01 2017-04-19 广西电网有限责任公司电力科学研究院 Big data calculation and analysis scheme fruiting system
CN107093019A (en) * 2017-04-21 2017-08-25 北京恒冠网络数据处理有限公司 A kind of big data analysis system for macro adjustments and controls
CN107391550A (en) * 2017-06-06 2017-11-24 广东广业开元科技有限公司 A kind of report form generation method and system based on big data mould plate technique
CN108268645A (en) * 2018-01-23 2018-07-10 广州南方人才资讯科技有限公司 Big data processing method and system
CN108319538A (en) * 2018-02-02 2018-07-24 世纪龙信息网络有限责任公司 The monitoring method and system of big data platform operating status
CN108563666A (en) * 2018-01-05 2018-09-21 成都兴政电子政务运营服务有限公司 A kind of data visualization processing system and method based on big data technology
CN109271581A (en) * 2018-08-02 2019-01-25 北京天元创新科技有限公司 A kind of quick rendering method of big data based on Dashboard
CN109408567A (en) * 2018-09-11 2019-03-01 广东布田电子商务有限公司 A kind of big data processing platform network architecture
CN109492130A (en) * 2018-10-09 2019-03-19 象翌微链科技发展有限公司 A kind of data manipulation method and system
CN110188088A (en) * 2019-05-23 2019-08-30 广东海洋大学 A kind of marine ships adopt sand behavior big data model
CN111143328A (en) * 2019-12-26 2020-05-12 山东翰林科技有限公司 Agile business intelligent data construction method, system, equipment and storage medium
CN113076370A (en) * 2021-04-23 2021-07-06 上海寒光信息科技有限公司 Internet data sky-eye agent BI system
CN114817264A (en) * 2022-04-28 2022-07-29 电子科技大学 Topology query structure, query method, electronic device and medium for graph computing
CN115794044A (en) * 2023-01-31 2023-03-14 帆软软件有限公司帆软南京分公司 Analysis theme system and analysis theme display method of BI tool

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102185927A (en) * 2011-05-31 2011-09-14 北京亿赞普网络技术有限公司 Knowledge-as-service cloud computing platform
CN103164476A (en) * 2011-12-16 2013-06-19 中国移动通信集团公司 Execution method and execution device of applying metadata to describe files in business intelligence (BI)
CN104182389A (en) * 2014-07-21 2014-12-03 安徽华贞信息科技有限公司 Semantic-based big data analysis business intelligence service system
CN104281697A (en) * 2014-10-15 2015-01-14 安徽华贞信息科技有限公司 Semantic-based hadoop system
CN104320848A (en) * 2014-11-14 2015-01-28 上海美琦浦悦通讯科技有限公司 Indoor locating implementation system and method based on cloud computing
CN104767813A (en) * 2015-04-08 2015-07-08 江苏国盾科技实业有限责任公司 Public bank big data service platform based on openstack
CN104933112A (en) * 2015-06-04 2015-09-23 浙江力石科技股份有限公司 Distributed Internet transaction information storage and processing method

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102185927A (en) * 2011-05-31 2011-09-14 北京亿赞普网络技术有限公司 Knowledge-as-service cloud computing platform
CN103164476A (en) * 2011-12-16 2013-06-19 中国移动通信集团公司 Execution method and execution device of applying metadata to describe files in business intelligence (BI)
CN104182389A (en) * 2014-07-21 2014-12-03 安徽华贞信息科技有限公司 Semantic-based big data analysis business intelligence service system
CN104281697A (en) * 2014-10-15 2015-01-14 安徽华贞信息科技有限公司 Semantic-based hadoop system
CN104320848A (en) * 2014-11-14 2015-01-28 上海美琦浦悦通讯科技有限公司 Indoor locating implementation system and method based on cloud computing
CN104767813A (en) * 2015-04-08 2015-07-08 江苏国盾科技实业有限责任公司 Public bank big data service platform based on openstack
CN104933112A (en) * 2015-06-04 2015-09-23 浙江力石科技股份有限公司 Distributed Internet transaction information storage and processing method

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105787116A (en) * 2016-03-25 2016-07-20 南京邮电大学 Cognitive computing architecture based on context-aware data streams
CN106570107B (en) * 2016-11-01 2019-08-20 广西电网有限责任公司电力科学研究院 A kind of big data calculating analytical plan achievement system
CN106570107A (en) * 2016-11-01 2017-04-19 广西电网有限责任公司电力科学研究院 Big data calculation and analysis scheme fruiting system
CN107093019A (en) * 2017-04-21 2017-08-25 北京恒冠网络数据处理有限公司 A kind of big data analysis system for macro adjustments and controls
CN107391550A (en) * 2017-06-06 2017-11-24 广东广业开元科技有限公司 A kind of report form generation method and system based on big data mould plate technique
CN108563666B (en) * 2018-01-05 2022-04-05 四川兴政信息技术有限公司 Data visualization processing system and method based on big data technology
CN108563666A (en) * 2018-01-05 2018-09-21 成都兴政电子政务运营服务有限公司 A kind of data visualization processing system and method based on big data technology
CN108268645A (en) * 2018-01-23 2018-07-10 广州南方人才资讯科技有限公司 Big data processing method and system
CN108319538B (en) * 2018-02-02 2019-11-08 世纪龙信息网络有限责任公司 The monitoring method and system of big data platform operating status
CN108319538A (en) * 2018-02-02 2018-07-24 世纪龙信息网络有限责任公司 The monitoring method and system of big data platform operating status
CN109271581A (en) * 2018-08-02 2019-01-25 北京天元创新科技有限公司 A kind of quick rendering method of big data based on Dashboard
CN109408567A (en) * 2018-09-11 2019-03-01 广东布田电子商务有限公司 A kind of big data processing platform network architecture
CN109492130A (en) * 2018-10-09 2019-03-19 象翌微链科技发展有限公司 A kind of data manipulation method and system
CN110188088A (en) * 2019-05-23 2019-08-30 广东海洋大学 A kind of marine ships adopt sand behavior big data model
CN111143328A (en) * 2019-12-26 2020-05-12 山东翰林科技有限公司 Agile business intelligent data construction method, system, equipment and storage medium
CN113076370A (en) * 2021-04-23 2021-07-06 上海寒光信息科技有限公司 Internet data sky-eye agent BI system
CN114817264A (en) * 2022-04-28 2022-07-29 电子科技大学 Topology query structure, query method, electronic device and medium for graph computing
CN115794044A (en) * 2023-01-31 2023-03-14 帆软软件有限公司帆软南京分公司 Analysis theme system and analysis theme display method of BI tool

Similar Documents

Publication Publication Date Title
CN105320757A (en) Business intelligent analysis method for quickly processing data
Santos et al. A big data system supporting bosch braga industry 4.0 strategy
CN104820670B (en) A kind of acquisition of power information big data and storage method
CN107330056B (en) Wind power plant SCADA system based on big data cloud computing platform and operation method thereof
CN103927346B (en) Query connection method on basis of data volumes
CN105787059A (en) Data warehouse based financial data integration method
CN104361022A (en) Method based on collected data statistics and foreground display
CN108614870B (en) Rapid batch warehousing system and method for external data information of power equipment
CN115860529A (en) Supply chain carbon accounting system based on industrial internet
CN112883001A (en) Data processing method, device and medium based on marketing and distribution through data visualization platform
CN111159152B (en) Secondary operation and data fusion method based on big data processing technology
CN111125450A (en) Management method of multilayer topology network resource object
Liu et al. Research on k-means algorithm based on cloud computing
CN113722564A (en) Visualization method and device for energy and material supply chain based on space map convolution
Talib et al. A multi-agent framework for data extraction, transformation and loading in data warehouse
CN107463151A (en) A kind of complex surface machining multidimensional knowledge cloud cooperating service method
CN204790999U (en) Big data acquisition of industry and processing system
Escobedo et al. Business intelligence and data analytics (BI&DA) to support the operation of smart grid
CN114047729B (en) Natural plant processing control method, system, computer device and storage medium
CN104715301A (en) Sugar manufacturing production index and production technology evaluating system based on cloud storage
Viharos et al. ” Big Data” Initiative as an IT Solution for Improved Operation and Maintenance of Wind Turbines
Li Construction of an interactive sharing platform for competitive intelligence data of marine resources under the background of intelligence construction
CN110955521A (en) Power distribution network wide area distributed type sinking calculation system and method based on big data
Zhou et al. Research on improving intelligent inspection efficiency of substation based on big data analysis
Lou et al. A hybrid data storage method for a robotic arm digital twin prototype

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
WD01 Invention patent application deemed withdrawn after publication
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20160210