CN105681474A - System architecture for supporting upper layer applications based on enterprise-level big data platform - Google Patents
System architecture for supporting upper layer applications based on enterprise-level big data platform Download PDFInfo
- Publication number
- CN105681474A CN105681474A CN201610193908.8A CN201610193908A CN105681474A CN 105681474 A CN105681474 A CN 105681474A CN 201610193908 A CN201610193908 A CN 201610193908A CN 105681474 A CN105681474 A CN 105681474A
- Authority
- CN
- China
- Prior art keywords
- data
- platform
- enterprise
- subsystem
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007405 data analysis Methods 0.000 claims abstract description 22
- 238000004458 analytical method Methods 0.000 claims abstract description 14
- 238000012545 processing Methods 0.000 claims abstract description 7
- 238000013079 data visualisation Methods 0.000 claims abstract description 4
- 238000007726 management method Methods 0.000 claims description 29
- 238000005516 engineering process Methods 0.000 claims description 10
- 238000000034 method Methods 0.000 claims description 9
- 238000009412 basement excavation Methods 0.000 claims description 5
- 238000011161 development Methods 0.000 claims description 4
- 230000002776 aggregation Effects 0.000 claims description 3
- 238000004220 aggregation Methods 0.000 claims description 3
- 230000000379 polymerizing effect Effects 0.000 claims description 3
- 238000010276 construction Methods 0.000 abstract description 4
- 238000000926 separation method Methods 0.000 abstract description 2
- 238000012098 association analyses Methods 0.000 description 2
- 238000013523 data management Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005065 mining Methods 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/30—Definitions, standards or architectural aspects of layered protocol stacks
- H04L69/32—Architecture of open systems interconnection [OSI] 7-layer type protocol stacks, e.g. the interfaces between the data link level and the physical level
- H04L69/322—Intralayer communication protocols among peer entities or protocol data unit [PDU] definitions
- H04L69/329—Intralayer communication protocols among peer entities or protocol data unit [PDU] definitions in the application layer [OSI layer 7]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a system architecture for supporting upper layer applications based on an enterprise-level big data platform and belongs to the big data processing field. The objective of the invention is to solve the technical problem of information separation and data duplication between the interior and exterior of an enterprise in the construction of a software system. According to the technical schemes of the invention, the system architecture includes a data center platform, a data analysis platform and a system management platform, wherein the data center platform includes a distributed data processing subsystem, a big data platform, a data warehouse, a data sharing subsystem, a data quality subsystem, a metadata subsystem and a task scheduling subsystem, the data analysis platform includes a data analysis topic portal and a data visualization analysis subsystem, and the system management platform includes an authentication unit, a configuration management unit and a log management unit.
Description
Technical field
The present invention relates to big data processing field, specifically a kind of system framework based on the big data platform upper layer application of enterprise-level.
Background technology
Along with develop rapidly and the information-based propelling of computer network, the collection of data, storage and propagation increase day by day. Between enterprises and enterprise external, the exchange of information is day by day strong, but each department of enterprises, the data source of the information platform between enterprise and enterprise is often independent of one another, there is isomery. When the construction of fore funnel formula software system all generally exists the problems such as information is isolated, Data duplication between enterprises and enterprise external, the problem such as cause data management dispersion, information island serious, data cannot be intersected shared between different systems, thus cause system function duplicate construction, data management difficulty, the wasting of resources. Business analysis as shown in Figure 1, how Data centre initiatively or passive can carry out information acquisition, how can integrate various data source, Various types of data is carried out association mining analysis, realize the application of big data association, thus be various business demand application scene offer data supporting, the business analysis means of variation is brought to enterprise.
Summary of the invention
The technical assignment of the present invention is for above weak point, thering is provided a kind of system framework based on the big data platform upper layer application of enterprise-level, the information between enterprises and enterprise external of building solving software system is isolated and the problem of Data duplication.
The technical solution adopted for the present invention to solve the technical problems is: a kind of system framework based on the big data platform upper layer application of enterprise-level, this system framework comprises Data centre's platform, data analysis platform and system management platform, and Data centre's platform comprises distributed data processing subsystem, big data platform, data warehouse, data sharing subsystems, quality of data subsystem, metadata subsystem and task scheduling sublayer system; Data analysis platform comprises data analysis special topic door and data visualization analyzing subsystem; System management platform comprises authentication, configuration management and daily record management. Wherein, this system framework is based on X86 framework.
As preferably, described Data centre platform realizes with Infrastructure layer and applied layer, data analysis platform realizes with data Layer and service layer, and system management platform realizes with data Layer.Wherein, this system framework is based on the system framework of cloud computing and the Internet model, the decoupling zero separation of optimized integration facility layer (IaaS), data Layer (DaaS), service layer (PaaS) and applied layer (SaaS), each layer all has the service ability of encapsulation simultaneously, builds the system ecology environment of opening and shares. Service layer provides the solution of various exploitation and delivery applications, such as Virtual Service device and operating system.
As preferably, described Infrastructure layer adopts x86 distributed computing and storage resources, it is provided that server, storage and the network hardware the field outside, for realizing easy management, easily expand, characteristic that highly reliable and height is handled up; Data Layer, for providing safe and reliable data open platform, by data unified Modeling, unified stores and unified sharing realizes data polymerizing power fast; Service layer is for providing the infrastructure service unrelated with business; Applied layer is used for quick response service demand and application and development, becomes centralized management and the open platform of multiple application.
As preferably, described Data centre platform, data analysis platform and system management platform all adopt standard interface, it is convenient to upgrading and replaces.
More preferably, described Data centre platform adopts Hadoop, MPP and RDB technology, it is achieved data relation analysis and excavation, and then the data sharing realized between enterprises and enterprise, avoids the phenomenon of data silo.
More preferably, the working process of described system framework comprises the steps:
(1), Data centre's platform completes access and the aggregation process of raw data;
(2), data analysis platform is inquired about from Data centre's platform according to service needed and is presented the data after gathering;
(3), the big data platform of whole enterprise-level is carried out the daily administration of daily record and configuration by system management platform.
Compared to the prior art a kind of system framework based on the big data platform upper layer application of enterprise-level of the present invention, has following useful effect:
1, the present invention's outstanding problems such as information that the software system construction when fore funnel formula exists is isolated, Data duplication that adopt cloud computing, big data technique to solve, by integrating various data source, Various types of data is carried out association mining analysis, thus be various business demand application scene offer data supporting, the business analysis means of variation is brought to enterprise;
2, present invention achieves data opening and shares, break the situation of manufacturer's monopolization data, form open ecological general layout, data main unification, avoid data repeat deposit, reduce the wasting of resources, form application and the architectural framework of data separating, the data that bottom is shared are unified supports the various application in upper strata, thus brings reliable means of numerical analysis to enterprise transformation and innovation;
3, the present invention adopts the system framework of big data processing technique by integrating various data resource based on enterprise-level big data platform, adopt Hadoop, MPP and RDB technology simultaneously, carry out association analysis and the excavation of various data, various types of specific analysis is supported according to business demand, improve the abilities such as company information acquisition, case study and location, bring great help to business operation.
Therefore the present invention has reasonable in design, the feature such as structure is simple, easy to use, one-object-many-purposes, thus, has good value for applications.
Accompanying drawing explanation
Below in conjunction with accompanying drawing, the present invention is further described.
Accompanying drawing 1 is the business graph in background technology;
Accompanying drawing 2 is a kind of structure block diagram divided according to layering based on the system framework of the big data platform upper layer application of enterprise-level;
Accompanying drawing 3 is the structure block diagram of a kind of system framework based on the big data platform upper layer application of enterprise-level according to modular division;
Accompanying drawing 4 is the technology framework figure of accompanying drawing 3.
Embodiment
Below in conjunction with the drawings and specific embodiments, the invention will be further described.
As shown in Figure 3, a kind of system framework based on the big data platform upper layer application of enterprise-level of the present invention, this system framework comprises Data centre's platform, data analysis platform and system management platform according to modular division, and Data centre's platform comprises distributed data processing subsystem, big data platform, data warehouse, data sharing subsystems, quality of data subsystem, metadata subsystem and task scheduling sublayer system; Data analysis platform comprises data analysis special topic door and data visualization analyzing subsystem; System management platform comprises authentication, configuration management and daily record management. Data centre's platform, data analysis platform and system management platform all adopt standard interface, are convenient to upgrading and replace. Data centre's platform adopts Hadoop, MPP and RDB technology, it is achieved data relation analysis and excavation, and then the data sharing realized between enterprises and enterprise, avoids data silo phenomenon.
The working process of this system framework comprises the steps:
(1), Data centre's platform completes access and the aggregation process of raw data;
(2), data analysis platform is inquired about from Data centre's platform according to service needed and is presented the data after gathering;
(3), the big data platform of whole enterprise-level is carried out the daily administration of daily record and configuration by system management platform.
As shown in Figure 2, system framework is divided into Infrastructure layer (IaaS), data Layer (DaaS), service layer (PaaS) and applied layer (SaaS) according to layering; Infrastructure layer (IaaS) adopts x86 distributed computing and storage resources, it is provided that server, storage and the network hardware outside field, for the characteristic realizing easy management, easily expansion, highly reliable and height is handled up; Data Layer (DaaS), for providing safe and reliable data open platform, by data unified Modeling, unified stores and unified sharing realizes data polymerizing power fast; Service layer (PaaS) is for providing the infrastructure service unrelated with business; Applied layer (SaaS), for quick response service demand and application and development, becomes centralized management and the open platform of multiple application. Data centre's platform realizes with Infrastructure layer (IaaS) and applied layer (SaaS), and data analysis platform realizes with data Layer (DaaS) and service layer (PaaS), and system management platform realizes with data Layer (DaaS).
As shown in Figure 4, the working process that this system framework carries out data relation analysis and excavation is as follows:
First, the HDFS of the Hadoop cluster of sorts of systems raw data (i.e. data source) collected Data centre platform, association analysis and calculating through Hadoop generate detailed data, a copy of it is admitted to Hbase cluster and deposits for application query, another part gather formation base data by various business rule and stored in MPP database or RDB for supporting foreground application specific analysis and other routine analyses;
Then, system user obtains the analytical data that Data centre gathers generation from foreground application or data, services module, and the data-guiding business development after analyzing according to data analysis platform;
Finally, the big data platform of enterprise-level is carried out routine maintenance management by system management platform by system manager.
By embodiment above, described those skilled in the art can be easy to realize the present invention. It should be appreciated that the present invention is not limited to above-mentioned a kind of embodiment. On the basis of disclosed enforcement mode, described those skilled in the art can the different technology feature of arbitrary combination, thus realize different technical schemes.
Except the technology feature described in specification sheets, it is the known technology of those skilled in the art.
Claims (6)
1. the system framework based on the big data platform upper layer application of enterprise-level, it is characterized in that: this system framework comprises Data centre's platform, data analysis platform and system management platform, Data centre's platform comprises distributed data processing subsystem, big data platform, data warehouse, data sharing subsystems, quality of data subsystem, metadata subsystem and task scheduling sublayer system; Data analysis platform comprises data analysis special topic door and data visualization analyzing subsystem; System management platform comprises authentication, configuration management and daily record management.
2. a kind of system framework based on the big data platform upper layer application of enterprise-level according to claim 1, it is characterized in that: described Data centre platform realizes with Infrastructure layer and applied layer, data analysis platform realizes with data Layer and service layer, and system management platform realizes with data Layer.
3. a kind of system framework based on the big data platform upper layer application of enterprise-level according to claim 1 and 2, it is characterized in that: described Infrastructure layer adopts x86 distributed computing and storage resources, the outer server in field, storage and the network hardware are provided, for realizing easy management, easily expand, characteristic that highly reliable and height is handled up; Data Layer, for providing safe and reliable data open platform, by data unified Modeling, unified stores and unified sharing realizes data polymerizing power fast; Service layer is for providing the infrastructure service unrelated with business; Applied layer is used for quick response service demand and application and development, becomes centralized management and the open platform of multiple application.
4. a kind of system framework based on the big data platform upper layer application of enterprise-level according to claim 1 and 2, it is characterised in that: described Data centre platform, data analysis platform and system management platform all adopt standard interface.
5. a kind of system framework based on the big data platform upper layer application of enterprise-level according to claim 4, it is characterised in that: described Data centre platform adopts Hadoop, MPP and RDB technology, it is achieved data relation analysis and excavation.
6. a kind of system framework based on the big data platform upper layer application of enterprise-level according to claim 5, it is characterised in that: the working process of described system framework comprises the steps:
(1), Data centre's platform completes access and the aggregation process of raw data;
(2), data analysis platform is inquired about from Data centre's platform according to service needed and is presented the data after gathering;
(3), the big data platform of whole enterprise-level is carried out the daily administration of daily record and configuration by system management platform.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610193908.8A CN105681474A (en) | 2016-03-31 | 2016-03-31 | System architecture for supporting upper layer applications based on enterprise-level big data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610193908.8A CN105681474A (en) | 2016-03-31 | 2016-03-31 | System architecture for supporting upper layer applications based on enterprise-level big data platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105681474A true CN105681474A (en) | 2016-06-15 |
Family
ID=56224847
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610193908.8A Pending CN105681474A (en) | 2016-03-31 | 2016-03-31 | System architecture for supporting upper layer applications based on enterprise-level big data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105681474A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106936843A (en) * | 2017-03-31 | 2017-07-07 | 山东超越数控电子有限公司 | A kind of common protocol layer architecture structure and its management method for managing industrial automation system |
| CN107070890A (en) * | 2017-03-10 | 2017-08-18 | 北京市天元网络技术股份有限公司 | Flow data processing device and communication network major clique system in a kind of communication network major clique system |
| CN107392486A (en) * | 2017-07-30 | 2017-11-24 | 广州云峰信息科技有限公司 | A kind of efficient data analysis and mining algorithm |
| CN107612984A (en) * | 2017-09-04 | 2018-01-19 | 北京天平检验行有限公司 | A kind of big data platform based on internet |
| CN108268529A (en) * | 2016-12-30 | 2018-07-10 | 亿阳信通股份有限公司 | It is a kind of that the data summarization method and system dispatched with multi engine are abstracted based on business |
| CN110737515A (en) * | 2018-07-19 | 2020-01-31 | 阿里巴巴集团控股有限公司 | data task instruction processing method, computer device and storage medium |
| CN110874718A (en) * | 2019-11-12 | 2020-03-10 | 贵阳市绿砼科技服务有限公司 | Concrete enterprise big data management system |
| CN113392116A (en) * | 2021-08-17 | 2021-09-14 | 江苏量界数据科技有限公司 | Distributed data processing system and method |
| CN113822585A (en) * | 2021-09-26 | 2021-12-21 | 云南锡业股份有限公司锡业分公司 | Intelligent smelting factory informatization management system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103491182A (en) * | 2013-09-29 | 2014-01-01 | 成都中科大旗软件有限公司 | Educational informationization open ecology platform based on cloud computing |
| CN104376430A (en) * | 2014-11-28 | 2015-02-25 | 东莞中国科学院云计算产业技术创新与育成中心 | Hidden risk management system based on cloud service platform and implementing method of hidden risk management system |
| CN104767813A (en) * | 2015-04-08 | 2015-07-08 | 江苏国盾科技实业有限责任公司 | Public bank big data service platform based on openstack |
| CN105007314A (en) * | 2015-07-10 | 2015-10-28 | 安徽新华传媒股份有限公司 | Big data processing system oriented to mass reading data of readers |
| CN105373971A (en) * | 2015-12-02 | 2016-03-02 | 国家电网公司 | Method of building energy efficiency management on the basis of big data |
-
2016
- 2016-03-31 CN CN201610193908.8A patent/CN105681474A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103491182A (en) * | 2013-09-29 | 2014-01-01 | 成都中科大旗软件有限公司 | Educational informationization open ecology platform based on cloud computing |
| CN104376430A (en) * | 2014-11-28 | 2015-02-25 | 东莞中国科学院云计算产业技术创新与育成中心 | Hidden risk management system based on cloud service platform and implementing method of hidden risk management system |
| CN104767813A (en) * | 2015-04-08 | 2015-07-08 | 江苏国盾科技实业有限责任公司 | Public bank big data service platform based on openstack |
| CN105007314A (en) * | 2015-07-10 | 2015-10-28 | 安徽新华传媒股份有限公司 | Big data processing system oriented to mass reading data of readers |
| CN105373971A (en) * | 2015-12-02 | 2016-03-02 | 国家电网公司 | Method of building energy efficiency management on the basis of big data |
Non-Patent Citations (5)
| Title |
|---|
| 彭庆: "基于大数据技术的数据共享平台方案研究", 《电信技术》 * |
| 王晖 等: "共享开放的运营商大数据平台架构研究", 《信息通信技术》 * |
| 虞益诚 等: "云计算层次服务关联机制及SaaS模式下数据安全取向的研究", 《软件产业与工程》 * |
| 赵静: "云计算平台Hadoop负载均衡研究", 《中国优秀硕士学位论文全文数据库》 * |
| 陈益 等: "广电"智慧城市"服务云平台规划与设计", 《有线电视技术》 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108268529A (en) * | 2016-12-30 | 2018-07-10 | 亿阳信通股份有限公司 | It is a kind of that the data summarization method and system dispatched with multi engine are abstracted based on business |
| CN107070890A (en) * | 2017-03-10 | 2017-08-18 | 北京市天元网络技术股份有限公司 | Flow data processing device and communication network major clique system in a kind of communication network major clique system |
| CN106936843A (en) * | 2017-03-31 | 2017-07-07 | 山东超越数控电子有限公司 | A kind of common protocol layer architecture structure and its management method for managing industrial automation system |
| CN107392486A (en) * | 2017-07-30 | 2017-11-24 | 广州云峰信息科技有限公司 | A kind of efficient data analysis and mining algorithm |
| CN107612984A (en) * | 2017-09-04 | 2018-01-19 | 北京天平检验行有限公司 | A kind of big data platform based on internet |
| CN107612984B (en) * | 2017-09-04 | 2020-11-10 | 北京天平检验行有限公司 | Big data platform based on internet |
| CN110737515A (en) * | 2018-07-19 | 2020-01-31 | 阿里巴巴集团控股有限公司 | data task instruction processing method, computer device and storage medium |
| CN110737515B (en) * | 2018-07-19 | 2024-04-09 | 阿里巴巴集团控股有限公司 | Processing method of data task instruction, computer equipment and storage medium |
| CN110874718A (en) * | 2019-11-12 | 2020-03-10 | 贵阳市绿砼科技服务有限公司 | Concrete enterprise big data management system |
| CN113392116A (en) * | 2021-08-17 | 2021-09-14 | 江苏量界数据科技有限公司 | Distributed data processing system and method |
| CN113822585A (en) * | 2021-09-26 | 2021-12-21 | 云南锡业股份有限公司锡业分公司 | Intelligent smelting factory informatization management system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105681474A (en) | System architecture for supporting upper layer applications based on enterprise-level big data platform | |
| Barika et al. | Orchestrating big data analysis workflows in the cloud: research challenges, survey, and future directions | |
| CN109074377B (en) | Managed function execution for real-time processing of data streams | |
| CN104767813B (en) | Public's row big data service platform based on openstack | |
| US9020802B1 (en) | Worldwide distributed architecture model and management | |
| US9747127B1 (en) | Worldwide distributed job and tasks computational model | |
| US9280381B1 (en) | Execution framework for a distributed file system | |
| CN104021194A (en) | Mixed type processing system and method oriented to industry big data diversity application | |
| Lai et al. | Towards a framework for large-scale multimedia data storage and processing on Hadoop platform | |
| CN103930875A (en) | Software virtual machine for acceleration of transactional data processing | |
| CN107341205A (en) | A kind of intelligent distribution system based on big data platform | |
| CN109344207B (en) | Big data platform of integrative frequency spectrum all over the sky based on big dipper scanning | |
| CN104102702A (en) | Software and hardware combined application-oriented big data system and method | |
| CN112347212A (en) | Railway cloud GIS platform for BIM application and building method thereof | |
| CN105071994B (en) | A kind of mass data monitoring system | |
| CN107612984B (en) | Big data platform based on internet | |
| CN104363222A (en) | Hadoop-based network security event analyzing method | |
| CN109150964B (en) | Migratable data management method and service migration method | |
| CN105391777A (en) | Algorithm escrow PaaS platform for decoupling logic code and performance code | |
| CN103198099A (en) | Cloud-based data mining application method facing telecommunication service | |
| CN113377344A (en) | Complex information system comprehensive integration method | |
| CN114297173A (en) | Knowledge graph construction method and system for large-scale mass data | |
| KR20130140508A (en) | Apparatus for collecting log information | |
| CN116166191A (en) | Integrated system of lake and storehouse | |
| Hsu | Big data analysis and optimization and platform components |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160615 |