CN114285851A - Cluster management platform based on big data sharing service - Google Patents
Cluster management platform based on big data sharing service Download PDFInfo
- Publication number
- CN114285851A CN114285851A CN202111619597.4A CN202111619597A CN114285851A CN 114285851 A CN114285851 A CN 114285851A CN 202111619597 A CN202111619597 A CN 202111619597A CN 114285851 A CN114285851 A CN 114285851A
- Authority
- CN
- China
- Prior art keywords
- data
- subsystem
- service
- management
- data sharing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 claims abstract description 22
- 230000005540 biological transmission Effects 0.000 claims abstract description 17
- 238000006243 chemical reaction Methods 0.000 claims abstract description 14
- 238000000034 method Methods 0.000 claims abstract description 14
- 238000012423 maintenance Methods 0.000 claims abstract description 11
- 230000008569 process Effects 0.000 claims abstract description 11
- 230000007246 mechanism Effects 0.000 claims abstract description 8
- 238000004806 packaging method and process Methods 0.000 claims abstract description 5
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 238000012545 processing Methods 0.000 abstract description 8
- 230000002457 bidirectional effect Effects 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention relates to the technical field of data sharing, in particular to a cluster management platform based on big data sharing service, which comprises a data bridging subsystem, a data processing subsystem and a data processing subsystem, wherein the data bridging subsystem is used for providing configuration of shared data sources among mechanisms; the data sharing subsystem is used as an information sharing channel between the preposed sharing systems to realize the packaging, conversion, transmission, routing and unpacking of the shared information; the monitoring management subsystem is used for remote deployment and remote system maintenance and simultaneously carrying out system management and monitoring; and the system management subsystem establishes a multi-service domain according to the service requirement so as to provide multi-theme data sharing service. The invention considers the diversity of data sources in the process of big data sharing, ensures the stability of data transmission and the flexibility of data sharing among mechanisms, and ensures that users can safely and quickly realize the function of data sharing among big data.
Description
Technical Field
The invention relates to the technical field of data sharing, in particular to a cluster management platform based on big data sharing service.
Background
The internet big data is mainly grasped in three major data island groups such as governments, operators, internet enterprises and the like, the island groups are mutually split and isolated, and even in the island groups or a specific island (enterprise), the data is not a whole body which can be conveniently circulated, but is in a state of respectively dispersed island-in-island.
Many businesses that do not seem to be related today are silently collecting and storing their own business data, even though they do not know what they can do with the data. The disadvantages of this type of method are that the data communication cost is high, a large amount of development cost is needed, sensitive fields are easy to miss, and the method is not friendly to later expansion.
Therefore, the technical implementation scheme of breaking through a standard for building a big data information island completes data sharing and transmission, and meanwhile, a set of standard data processing process, transmission process and cluster management environment are built to achieve big data sharing service in consideration of the huge quantity characteristic and data security of big data.
Disclosure of Invention
Aiming at the defects of the prior art, the invention discloses a cluster management platform based on big data sharing service, which is used for solving the problems.
The invention is realized by the following technical scheme:
the invention provides a cluster management platform based on big data sharing service, which comprises a data bridging subsystem, a data processing subsystem and a data processing subsystem, wherein the data bridging subsystem is used for providing configuration of sharing data sources among mechanisms;
the data sharing subsystem is used as an information sharing channel between the preposed sharing systems to realize the packaging, conversion, transmission, routing and unpacking of the shared information;
the monitoring management subsystem is used for remote deployment and remote system maintenance and simultaneously carrying out system management and monitoring;
and the system management subsystem establishes a multi-service domain according to the service requirement so as to provide multi-theme data sharing service.
Furthermore, the data bridging subsystem is used for carrying out real-time bidirectional synchronization on the service data and the prepositive sharing system; and a mainstream operating system, a mainstream database, a big database, various structured and unstructured files and various audio and video files are supported.
Furthermore, the data bridging subsystem accesses the bridging object by adopting an adapter component to realize the acquisition and storage of data, and the data bridging subsystem sets a graphical information sharing bridging configuration and management tool to realize bridging index definition and bridging management.
Furthermore, the data bridging subsystem automatically identifies the information which needs to be shared and includes the added, modified or deleted information without modifying the structure of the service information base.
Furthermore, the data sharing subsystem uses HTTP protocol for message transmission and sets message confirmation and message selective retransmission mechanism, and provides message error processing function.
Furthermore, when the data sharing subsystem carries out information conversion, format conversion and content conversion functions based on rule configuration are set, a log management function is set, and RAS digital signatures are used when interfaces used by the data sharing subsystem are connected with other business systems.
Furthermore, the monitoring management subsystem sets the running state, the resource occupation state, the system performance and the system abnormal condition of the running monitoring tool monitoring system.
Furthermore, the monitoring management subsystem carries out comprehensive statistics of shared data and provides remote log management and maintenance.
Furthermore, the monitoring management subsystem provides a system resource configuration tool to realize the dynamic configuration of the business process system, the information service bus system and the application adapter system.
Further, the system management subsystem comprises service domain management, node management, shared service management and data tracking management functions.
The invention has the beneficial effects that:
the invention considers the diversity of data sources in the process of big data sharing, ensures the stability of data transmission and the flexibility of data sharing among mechanisms, and ensures that users can safely and quickly realize the function of data sharing among big data.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a schematic diagram of a cluster management platform based on big data sharing service.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
Referring to fig. 1, the embodiment provides a cluster management platform based on big data sharing service, including
A data bridging subsystem for providing an inter-organization shared data source configuration;
the data sharing subsystem is used as an information sharing channel between the preposed sharing systems to realize the packaging, conversion, transmission, routing and unpacking of the shared information;
the monitoring management subsystem is used for remote deployment and remote system maintenance and simultaneously carrying out system management and monitoring;
and the system management subsystem establishes a multi-service domain according to the service requirement so as to provide multi-theme data sharing service.
The platform of the embodiment supports multi-type, multi-theme and large-volume large data sharing. The method supports cross-department and cross-network data sharing services, ensures safe, reliable, stable and efficient information exchange and transmission among databases, and can configure and manage various information sharing processes meeting business requirements.
The platform of the embodiment realizes the function of connecting each service system, transmits the shared information to the preposed shared database of each system, and receives the processing result fed back by the preposed database.
The flow of the platform data transmission of the embodiment can be configured and managed. And the data sharing network is utilized to realize reliable, stable and safe data transmission between different physical positions, different service systems and the central sharing server.
The platform of the embodiment can dynamically make, manage and monitor the information flow formed according to the service flow.
The platform of the embodiment deploys the unserviced cluster in the data sharing process according to a set strategy, realizes the sharing of the computing resources of the server, and supports dynamic deployment.
The platform of the embodiment can provide the management and maintenance functions of the whole data sharing management platform, realize unified management and maintenance, guarantee system operation, and realize hierarchical management and distributed management.
Example 2
In this embodiment, on the basis of embodiment 1, a data bridging subsystem is provided, which mainly provides a configuration of sharing data sources among organizations.
The data bridging system of the embodiment supports real-time bidirectional synchronization of service data and a preposed sharing system, and supports various mainstream operating systems.
The embodiment supports domestic and foreign mainstream databases and big databases, and the embodiment is preferably selected from SQLServer, ORACLE, DB2, HADOOP and the like.
The embodiment supports various structured and unstructured files, and the embodiment performs optimization, such as TXT, Excel, XML, and the like.
The present embodiment supports various audio and video files, and the present embodiment is preferably performed by AVI, RM, RMVB, WMA, MP3, and the like.
In the embodiment, the adapter component is adopted to access the bridging object, so that the data acquisition and storage are realized, and the function of automatic identification of incremental data is provided.
In the embodiment, under the condition that the structure of the service information base is not modified, the system can automatically identify the information needing to be shared, including the information which is added, modified or deleted.
The present embodiment provides a graphical information sharing bridge configuration and management tool, supporting bridging. The index definition and bridge management function of this embodiment.
Example 3
In this embodiment, on the basis of embodiment 1, a data sharing subsystem is provided as an information sharing channel between pre-configured sharing systems, and functions such as packaging, conversion, transfer, routing, and unpacking of shared information are implemented.
The data sharing transmission system message transmission of the embodiment supports an HTTP protocol and provides a message confirmation and message selective retransmission mechanism to realize a safe and reliable information transfer function between the preposed exchange subsystems.
The embodiment provides a message error processing function, supports various mainstream operating systems, provides an information conversion function, and supports format conversion and content conversion functions based on rule configuration.
The embodiment provides a log management function, can check the transmission condition of the system through the log, and can eliminate error information in the system.
In the security management function of the embodiment, the interfaces used for interfacing with other service systems must ensure security, and RAS digital signatures are used to prevent data theft.
Example 4
In this embodiment, on the basis of embodiment 1, a monitoring management subsystem is provided, which provides a management and monitoring system based on a B/S structure, and simultaneously provides functions of remote deployment and remote system maintenance.
The embodiment provides a system operation monitoring tool for monitoring the operation state, the resource occupation state, the system performance, the system abnormal condition and the like of the system. And an all-round statistical function of shared data is provided.
The embodiment provides a system resource configuration tool, realizes dynamic configuration of a business process system, an information service bus system and an application adapter system, and provides remote log management and maintenance.
In summary, the information sharing system of the present invention is a distributed system, and a complete information sharing process is completed by the cooperation of the distributed sharing services, for example, the basic information summarizing process is composed of the basic information sending service of each sharing unit exchange node and the receiving service of the central node. Therefore, the shared platform provides the function of remote deployment, and reduces the cost of implementation and maintenance.
The information sharing platform of the invention needs to provide a remote deployment function, and sharing service configuration personnel can directly deploy the configured sharing service to the appointed sharing node.
Therefore, the invention considers the diversity of data sources in the process of big data sharing, ensures the stability of data transmission and the flexibility of data sharing among mechanisms, and ensures that users can safely and quickly realize the function of data sharing among big data.
The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A cluster management platform based on big data sharing service is characterized by comprising
The data bridging subsystem is used for providing configuration of a shared data source between organizations;
the data sharing subsystem is used as an information sharing channel between the preposed sharing systems to realize the packaging, conversion, transmission, routing and unpacking of the shared information;
the monitoring management subsystem is used for remote deployment and remote system maintenance and simultaneously carrying out system management and monitoring;
and the system management subsystem establishes a multi-service domain according to the service requirement so as to provide multi-theme data sharing service.
2. The cluster management platform based on big data sharing service as claimed in claim 1, wherein the data bridging subsystem is configured to bi-directionally synchronize the service data with the pre-sharing system in real time; and a mainstream operating system, a mainstream database, a big database, various structured and unstructured files and various audio and video files are supported.
3. The cluster management platform based on the big data sharing service according to claim 2, wherein the data bridging subsystem accesses a bridging object by using an adapter component to achieve data acquisition and storage, and the data bridging subsystem sets a graphical information sharing bridging configuration and management tool to achieve bridging index definition and bridging management.
4. The cluster management platform based on big data sharing service as claimed in claim 2, wherein the data bridging subsystem automatically identifies the information to be shared, including the added, modified or deleted information, without modifying the structure of the service information base.
5. The cluster management platform based on big data sharing service of claim 1, wherein the data sharing subsystem uses HTTP protocol for message transmission and sets message acknowledgement and message selective retransmission mechanism, and provides message error handling function.
6. The cluster management platform based on big data sharing service of claim 5, wherein when the data sharing subsystem performs information conversion, format conversion and content conversion functions based on rule configuration are performed, and a log management function is set, and RAS digital signature is used when interfaces used for interfacing with other business systems are provided.
7. The cluster management platform based on big data sharing service according to claim 1, wherein the monitoring management subsystem sets an operation status, a resource occupation status, a system performance and a system abnormal condition of an operation monitoring tool monitoring system.
8. The cluster management platform based on big data sharing service of claim 7, wherein the monitoring management subsystem performs comprehensive statistics of shared data and provides remote log management and maintenance.
9. The cluster management platform based on big data sharing service according to claim 7, wherein the monitoring management subsystem provides a system resource configuration tool to implement dynamic configuration of a business process system, an information service bus system, and an application adapter system.
10. The big data sharing service based cluster management platform according to claim 1, wherein the system management subsystem comprises service domain management, node management, shared service management and data tracking management functions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111619597.4A CN114285851A (en) | 2021-12-27 | 2021-12-27 | Cluster management platform based on big data sharing service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111619597.4A CN114285851A (en) | 2021-12-27 | 2021-12-27 | Cluster management platform based on big data sharing service |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114285851A true CN114285851A (en) | 2022-04-05 |
Family
ID=80876660
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111619597.4A Pending CN114285851A (en) | 2021-12-27 | 2021-12-27 | Cluster management platform based on big data sharing service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114285851A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103607469A (en) * | 2013-11-28 | 2014-02-26 | 东莞中国科学院云计算产业技术创新与育成中心 | Cloud platform for achieving distributed isomerous data sharing and data sharing method thereof |
| CN105225190A (en) * | 2015-09-30 | 2016-01-06 | 上海安技智能科技股份有限公司 | The large data sharing platform of a kind of warehouse logistics |
| CN107845051A (en) * | 2016-09-21 | 2018-03-27 | 北京航天长峰科技工业集团有限公司 | A kind of E-government information exchanges and sharing method |
| CN110445773A (en) * | 2019-07-24 | 2019-11-12 | 江苏智赢信息科技有限公司 | Fire-fighting command regulator control system based on Internet of Things |
| CN110738454A (en) * | 2019-08-27 | 2020-01-31 | 格局商学教育科技（深圳）有限公司 | enterprise management platform system |
| CN112685385A (en) * | 2020-12-31 | 2021-04-20 | 广西中科曙光云计算有限公司 | Big data platform for smart city construction |
-
2021
- 2021-12-27 CN CN202111619597.4A patent/CN114285851A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103607469A (en) * | 2013-11-28 | 2014-02-26 | 东莞中国科学院云计算产业技术创新与育成中心 | Cloud platform for achieving distributed isomerous data sharing and data sharing method thereof |
| CN105225190A (en) * | 2015-09-30 | 2016-01-06 | 上海安技智能科技股份有限公司 | The large data sharing platform of a kind of warehouse logistics |
| CN107845051A (en) * | 2016-09-21 | 2018-03-27 | 北京航天长峰科技工业集团有限公司 | A kind of E-government information exchanges and sharing method |
| CN110445773A (en) * | 2019-07-24 | 2019-11-12 | 江苏智赢信息科技有限公司 | Fire-fighting command regulator control system based on Internet of Things |
| CN110738454A (en) * | 2019-08-27 | 2020-01-31 | 格局商学教育科技（深圳）有限公司 | enterprise management platform system |
| CN112685385A (en) * | 2020-12-31 | 2021-04-20 | 广西中科曙光云计算有限公司 | Big data platform for smart city construction |
Non-Patent Citations (1)
| Title |
|---|
| 匡晓红;汤丹;刘志峰;: "一种引擎驱动的电子政务数据交换平台", 计算机时代, no. 01, pages 80 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106131213B (en) | Service management method and system | |
| US6535917B1 (en) | Market data domain and enterprise system implemented by a master entitlement processor | |
| US8166185B2 (en) | System and method for enterprise software distribution | |
| CN1288558C (en) | Virtual network having adaptive control program | |
| CN105338086A (en) | Distributed message forwarding method | |
| US20140280988A1 (en) | System and method for parallel multiplexing between servers in a cluster | |
| US20030126196A1 (en) | System for optimizing the invocation of computer-based services deployed in a distributed computing environment | |
| WO2020186807A1 (en) | System and method for power data linking based on blockchain technology | |
| CN102868935A (en) | Scheduling method for responding multiple sources in content distribution network (CDN) | |
| CN105472042A (en) | WEB terminal controlled message middleware system and data transmission method thereof | |
| US6119173A (en) | System and method for communications and process management in a distributed telecommunications switch | |
| CN110096545A (en) | One kind being based on big data platform data processing domain architecting method | |
| CN110837423A (en) | Method and device for automatically acquiring data of guided transport vehicle | |
| CN109345114A (en) | A kind of E-government affairs service system | |
| CN114448686B (en) | Cross-network communication device and method based on micro-service | |
| CN114285851A (en) | Cluster management platform based on big data sharing service | |
| CN116260893B (en) | Message subscription and publishing device of data processing system | |
| CN115225645B (en) | Service updating method, device, system and storage medium | |
| CN110913018A (en) | Distributed regulation and control service system | |
| CN111541667A (en) | Method, equipment and storage medium for intersystem message communication | |
| CN113965608A (en) | Data collection and instruction issuing system of numerical control equipment | |
| CN113378219A (en) | Processing method and system of unstructured data | |
| CN112765294A (en) | Meteorological big data processing and scheduling system | |
| CN104572859A (en) | Distributive complex event processing system | |
| CN117667965B (en) | Method and system for managing big data of battery energy storage system and computing device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |