CN106921755B - Enterprise data integration cloud console, implementation method and system - Google Patents
Enterprise data integration cloud console, implementation method and system Download PDFInfo
- Publication number
- CN106921755B CN106921755B CN201710339811.8A CN201710339811A CN106921755B CN 106921755 B CN106921755 B CN 106921755B CN 201710339811 A CN201710339811 A CN 201710339811A CN 106921755 B CN106921755 B CN 106921755B
- Authority
- CN
- China
- Prior art keywords
- server
- servers
- data
- cloud console
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 35
- 230000010354 integration Effects 0.000 title claims abstract description 24
- 238000004140 cleaning Methods 0.000 claims abstract description 13
- 238000003860 storage Methods 0.000 claims abstract description 10
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 2
- 239000000284 extract Substances 0.000 claims description 2
- 238000012545 processing Methods 0.000 abstract description 11
- 238000012423 maintenance Methods 0.000 abstract description 8
- 238000013075 data extraction Methods 0.000 abstract description 4
- 238000007405 data analysis Methods 0.000 abstract 1
- 238000005516 engineering process Methods 0.000 description 7
- 238000010276 construction Methods 0.000 description 2
- 230000033772 system development Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L67/025—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Debugging And Monitoring (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses an enterprise data integration cloud console, an implementation method and a system, which comprise a cloud console and a server cluster, wherein the cloud console adopts the cloud console structure, the server cluster consists of a master server and a plurality of slave servers, each server in the server cluster is provided with an open source tool button, the cloud console controls all operation tasks to be distributed to the slave servers through the master server, and after all the servers finish the operation tasks, the operation information of relevant servers is displayed in the master server. Compared with the prior art, the enterprise data integration cloud control console, the implementation method and the system overcome the processing bottleneck of a single server, monitor the operation condition of the server in real time, discover and solve problems in advance, greatly reduce the learning cost of operation and maintenance developers, can quickly and effectively complete data extraction, cleaning and storage work, provide the accuracy of integral data, and provide convenience for large data analysis.
Description
Technical Field
The invention relates to the technical field of server clusters, in particular to an enterprise data integration cloud console, an implementation method and an implementation system.
Background
In terms of enterprise informatization, information system construction is usually characterized by stage and distribution, which results in the existence of an information island phenomenon. The information isolated island refers to the situation that data information among different software, especially among different departments cannot be shared, so that a large amount of redundant data and junk data exist in a system, the consistency of the data cannot be guaranteed, and the whole process of enterprise informatization construction is seriously hindered. To solve this problem, people are beginning to pay attention to enterprise data integration research. Data in different systems are attempted to be reprocessed, so that an integrated analysis-oriented environment is formed, and rules can be mined from massive information, knowledge can be extracted, and decision making can be assisted. The appearance of the button can provide corresponding help undoubtedly, but a processing bottleneck exists in comparison with a single server with mass data, and the interface is extremely unfriendly and cannot be remotely controlled, so that the learning cost of operation and maintenance developers is increased, the accuracy of data is reduced, and the overall IT cost is increased.
Based on the method, the invention provides an enterprise data integration cloud console, an implementation method and an implementation system. The cloud console technology provided by the invention can dynamically expand and manage the server cluster in a remote console, and horizontally expand the operation in the server cluster, so that the processing bottleneck of a single server is overcome, the operation condition of the server is monitored in real time, the problems are found and solved in advance, the learning cost of operation and maintenance developers is greatly reduced, the data extraction, cleaning and storage work can be rapidly and effectively completed, the accuracy of the whole data is provided, and the convenience is provided for the analysis of big data.
Disclosure of Invention
The technical task of the invention is to provide an enterprise data integration cloud console, an implementation method and an implementation system aiming at the defects.
An enterprise data integration cloud console structurally comprises the following five modules: the operation expansion, operation display, operation start and stop, operation monitoring and operation statistics are used for realizing remote control of the server cluster and completing operation tasks, wherein,
the operation expansion module is used for dynamically expanding the configuration server cluster and appointing one operation to different servers to complete the operation;
the operation showing module is used for showing all operation information operated by the current server, including operation description creation time, an operation flow chart and operation execution log information;
the operation starting and stopping module is used for remotely controlling the starting and stopping of the operation task;
the operation monitoring module is used for acquiring and displaying the log of the operation task in real time;
and the operation counting module is used for counting the completion condition of the whole operation task.
The cloud console monitors the servers completing the operation tasks in real time, the monitored information comprises network state, CPU utilization rate, process number and disk space information, and early warning notification is carried out in a mode of short messages and mails.
When the operation start-stop module remotely controls the start-stop of the operation task, the relevant information of data acquisition, cleaning and storage in the server bearing the acquisition work is accurately acquired in real time.
An enterprise data integration cloud console implementation method is based on a cloud console and a server cluster, and the implementation process is as follows:
the cloud control console establishes a data acquisition special cluster in the server cluster, wherein the data acquisition special cluster comprises a main server and a plurality of slave servers;
installing an open source tool button for each server of the data acquisition special cluster, and sequentially starting all the servers;
the cloud control console controls all the operation tasks to be distributed into the slave servers through the master server, and after all the servers complete the operation tasks, the operation information of the relevant servers is displayed in the master server.
The cloud console monitors the server bearing the acquisition work in real time, the monitored information comprises network state, CPU utilization rate, process number and disk space information, and early warning notification is carried out in a mode of short messages and mails.
The cloud control console can remotely control the start and stop of the operation task, and accurately acquires relevant information of data acquisition, cleaning and storage in a server bearing acquisition work in real time.
The cloud console is provided with an operation expansion module, the operation expansion module is used for dynamically expanding and configuring a special data acquisition cluster, assigning an operation task to different servers to be completed cooperatively, the master server controls all the slave servers to complete data acquisition cooperatively, and obtains the network state, the CPU utilization rate, the process number and the disk space information of all the servers in real time and gives an early warning in time.
After the job task is completed, job information can be displayed in the cloud console, the job information comprises job description creation time, a job flow chart and job execution log information, and all job information is acquired.
The cloud control platform adopts the cloud control platform structure, the server cluster is composed of a master server and a plurality of slave servers, each server in the server cluster is provided with an open source tool button, the cloud control platform controls all operation tasks to be distributed into the slave servers through the master server, and after all the servers complete the operation tasks, the operation information of the relevant servers is displayed in the master server.
All servers in the server cluster cooperatively complete a certain job task, and a main server acquires the running information of related servers; the cloud console remote control main server extracts data dispersed to heterogeneous data sources of different service systems to a temporary middle layer, then carries out cleaning, conversion and integration, and finally loads the data to a data warehouse or a data mart, wherein the data comprises relationship data and a plane data file.
Compared with the prior art, the enterprise data integration cloud console, the implementation method and the system have the following beneficial effects:
according to the enterprise data integration cloud console, the implementation method and the system, the operation of the button is remotely controlled through a cloud control technology, firstly, the bottleneck of single server processing is overcome, the problem of concurrency of a large number of jobs can be solved in a cluster mode, the complex jobs can be rapidly and effectively carried out, in addition, the robustness of the whole job processing process is increased, and the whole stable operation is not influenced by the downtime of a single server; and secondly, the operation distribution of the graphical interface showing relevant information and remote control operation reduces the learning cost of operation and maintenance developers, the keys can be controlled quickly and effectively, the stability and reliability of whole data extraction are ensured, the practicability is high, and the application range is wide.
Drawings
FIG. 1 is a schematic diagram of the implementation of the method of the present invention.
Detailed Description
The invention is further described with reference to the following figures and specific embodiments.
The enterprise data integration mainly refers to a process of performing re-concentration and re-unified management on business data of an enterprise-distributed information system, and the process comprises data acquisition, cleaning, processing, storage and the like.
The Cloud Control Platform (CCP) is a remote Control Platform with elastically scalable processing capability, and the management mode of the remote Control Platform is simpler and more efficient than that of a physical server based on the Platform for processing data integration work.
The invention aims to remotely control data extraction, cleaning and storage work by using a cloud console technology, overcome the processing bottleneck of a single server, reduce the difficulty of system development and operation and maintenance, reduce the generation of error data and provide more accurate reference for enterprise operation decision.
As shown in fig. 1, an enterprise data integration cloud console structurally includes the following five modules: the operation expansion, operation display, operation start and stop, operation monitoring and operation statistics are used for realizing remote control of the server cluster and completing operation tasks, wherein,
and (3) operation expansion:
the cloud control console can dynamically expand a configuration server cluster, one operation is assigned to different servers for carrying out, the master server controls all the slave servers to cooperatively complete data acquisition work, the processing bottleneck of a single server is overcome, information such as the network state, the CPU utilization rate, the process number and the disk space of the server is acquired in real time, and early warning can be timely carried out.
And (3) operation exhibition:
and displaying all the job information operated by the current server, including job description creation time, job flow chart, job execution log information and the like, and acquiring all the job information quickly.
Operation start and stop:
the start and stop of the operation can be remotely controlled, and the operation can be effective in real time by one key.
Operation monitoring:
the logs of all the operations can be acquired and displayed in real time, and the operation and maintenance personnel can be assisted to troubleshoot problems quickly.
Operation statistics:
through the whole operation statistics, the circulation of different theme data can be shown on a macroscopic level, and the whole situation of all operations can be integrally mastered.
When the operation start-stop module remotely controls the start-stop of the operation task, the relevant information of data acquisition, cleaning and storage in the server bearing the acquisition work is accurately acquired in real time.
An enterprise data integration cloud console implementation method adopts a cloud console technology and can rapidly establish a data acquisition server cluster. The acquisition work can be horizontally expanded, and the basic operation of the acquisition work is supported: starting, stopping, modifying, and replacing servers. The whole work can be simultaneously operated on a plurality of servers, the main server controls all the work to be equally distributed to different servers, and the work is completed in a coordinated mode, so that the purpose of load balancing is achieved. In addition, the server carrying the acquisition work monitors information such as network state, CPU utilization rate, process number, disk space and the like in real time, and can give early warning notification in time, including short messages, mails and the like. The data acquisition and processing tool is remotely controlled, the related interfaces of the data acquisition tool are called, and the purpose of remotely acquiring the information of the running condition of related operation in real time is achieved. By adopting the cloud console technology, a more stable and safer application system can be constructed, the difficulty of system development and operation and maintenance is reduced, the service innovation is concentrated, and meanwhile, the overall IT cost is greatly reduced.
The method is based on a cloud console and a server cluster, and the implementation process is as follows:
the cloud control console establishes a data acquisition special cluster in the server cluster, wherein the data acquisition special cluster comprises a main server and a plurality of slave servers;
installing an open source tool button for each server of the data acquisition special cluster, and sequentially starting all the servers;
the cloud control console controls all the operation tasks to be distributed into the slave servers through the master server, and after all the servers complete the operation tasks, the operation information of the relevant servers is displayed in the master server.
The cloud console monitors the server bearing the acquisition work in real time, the monitored information comprises network state, CPU utilization rate, process number and disk space information, and early warning notification is carried out in a mode of short messages and mails.
The cloud control console can remotely control the start and stop of the operation task, and accurately acquires relevant information of data acquisition, cleaning and storage in a server bearing acquisition work in real time.
The cloud console is provided with an operation expansion module, the operation expansion module is used for dynamically expanding and configuring a special data acquisition cluster, assigning an operation task to different servers to be completed cooperatively, the master server controls all the slave servers to complete data acquisition cooperatively, and obtains the network state, the CPU utilization rate, the process number and the disk space information of all the servers in real time and gives an early warning in time.
After the job task is completed, job information can be displayed in the cloud console, the job information comprises job description creation time, a job flow chart and job execution log information, and all job information is acquired.
The cloud control platform adopts the cloud control platform structure, the server cluster is composed of a master server and a plurality of slave servers, each server in the server cluster is provided with an open source tool button, the cloud control platform controls all operation tasks to be distributed into the slave servers through the master server, and after all the servers complete the operation tasks, the operation information of the relevant servers is displayed in the master server.
All servers in the server cluster cooperatively complete a certain job task, and a main server acquires the running information of related servers; the cloud control console remotely manages a plurality of server clusters, remotely controls a data acquisition tool button to realize horizontal expansion of work in the server clusters, each server cluster consists of a main server and a plurality of slave servers, and the main server is used as a controller of one cluster. And the servers cooperatively complete a certain job task and acquire the running information of the related servers. Data such as relationship data, plane data files and the like which are dispersed to different business systems and in heterogeneous data sources are remotely, quickly and effectively extracted to a temporary middle layer, then cleaning, converting and integrating are carried out, and finally the data are loaded into a data warehouse or a data mart, so that the whole process is real-time and clearly visible.
The cloud console technology mainly adopts an open source tool (Kettle), clusters are deployed on a plurality of servers, one of the servers is a Master, the other servers are Slave, a specific Master server and a Slave server are designated through a configuration file, all the servers are sequentially started, the clusters are set in a graphical interface, the Master server and the Slave server can communicate with each other through an Http open protocol, operation level expansion is achieved, operation is evenly distributed to Slave servers through a Master server, and a plurality of servers cooperatively complete operation tasks. The detailed information of each running server, including information such as network state, CPU utilization rate, process number and disk space, can be acquired through the SlaveServer interface. The operation start and stop are remotely controlled through interfaces such as SlaveServer, TransPainter and KettleDatabaseReposology, and relevant information of data acquisition, cleaning and storage is accurately acquired in real time.
In the invention, firstly, a graphical interface dynamically configures a button server cluster, thereby realizing the horizontal expansion of the operation and solving the problem of slow jamming of the running of complex operation. And secondly, the graphical interface remotely controls the operation of the data acquisition tool button, and the operation information is acquired and displayed in real time, so that the learning cost of operation and maintenance developers is greatly reduced, and convenience is brought to enterprise data integration.
The present invention can be easily implemented by those skilled in the art from the above detailed description. It should be understood, however, that the intention is not to limit the invention to the particular embodiments described. On the basis of the disclosed embodiments, a person skilled in the art can combine different technical features at will, thereby implementing different technical solutions.
In addition to the technical features described in the specification, the technology is known to those skilled in the art.
Claims (4)
1. An enterprise data integration cloud console implementation method is based on an enterprise data integration cloud console, and comprises the following modules: the system comprises operation extension, operation display, operation monitoring and operation statistics, is used for realizing remote control of a server cluster and completing operation tasks, and is characterized in that the realization process comprises the following steps:
the method comprises the following steps that firstly, a cloud control console builds a special data acquisition cluster in a server cluster, wherein the special data acquisition cluster comprises a main server and a plurality of slave servers;
step two, installing an open source tool button for each server of the special data acquisition cluster, and starting all the servers in sequence;
thirdly, the cloud console controls all the operation tasks to be distributed to the slave servers through the master server, the operation expansion module dynamically expands the configuration data acquisition special cluster, one operation task is assigned to different servers to be completed cooperatively, the master server controls all the slave servers to complete data acquisition cooperatively, the network state, the CPU utilization rate, the process number and the disk space information of all the servers are obtained in real time, and early warning is carried out in time;
the operation monitoring module acquires logs of operation tasks in real time, monitors a server bearing acquisition work in real time, and carries out early warning notification in a mode of short messages and mails, wherein the monitored information comprises network state, CPU utilization rate, process number and disk space information;
the operation showing module shows all operation information operated by the current server, wherein the operation information comprises operation description creation time, an operation flow chart and operation execution log information;
and the operation counting module counts the completion condition of the whole operation task, and displays the running information of the relevant server in the main server after all the servers complete the operation task.
2. The method for implementing the enterprise data integration cloud console according to claim 1, wherein an operation start/stop module is further configured in the cloud console, and is used for remotely controlling start/stop of the operation task, and when the operation start/stop module remotely controls start/stop of the operation task, relevant information of data acquisition, cleaning and storage in a server carrying acquisition work is accurately acquired in real time.
3. An enterprise data integration cloud console system is characterized by comprising a cloud console and a server cluster, wherein the cloud console adopts the cloud console structure of claim 1 or 2, the server cluster is composed of a master server and a plurality of slave servers, each server in the server cluster is provided with an open source tool button, the cloud console controls all operation tasks to be distributed into the slave servers through the master server, and after all the servers complete the operation tasks, operation information of relevant servers is displayed in the master server.
4. The enterprise data integration cloud console system of claim 3, wherein each server in the server cluster cooperatively completes a certain job task, and the main server obtains the operation information of the relevant server; the cloud console remote control main server extracts data dispersed to heterogeneous data sources of different service systems to a temporary middle layer, then carries out cleaning, conversion and integration, and finally loads the data to a data warehouse or a data mart, wherein the data comprises relationship data and a plane data file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710339811.8A CN106921755B (en) | 2017-05-15 | 2017-05-15 | Enterprise data integration cloud console, implementation method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710339811.8A CN106921755B (en) | 2017-05-15 | 2017-05-15 | Enterprise data integration cloud console, implementation method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106921755A CN106921755A (en) | 2017-07-04 |
| CN106921755B true CN106921755B (en) | 2020-04-28 |
Family
ID=59567773
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710339811.8A Active CN106921755B (en) | 2017-05-15 | 2017-05-15 | Enterprise data integration cloud console, implementation method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106921755B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111435344B (en) * | 2019-01-15 | 2023-03-21 | 中国石油集团川庆钻探工程有限公司长庆钻井总公司 | Big data-based drilling acceleration influence factor analysis model |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104391989A (en) * | 2014-12-16 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Distributed ETL (extract transform load) all-in-one machine system |
| CN104573071A (en) * | 2015-01-26 | 2015-04-29 | 湖南大学 | Intelligent school situation analysis system and method based on megadata technology |
| CN106549829A (en) * | 2016-10-28 | 2017-03-29 | 北方工业大学 | Big data calculating platform monitoring system and method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120310875A1 (en) * | 2011-06-03 | 2012-12-06 | Prashanth Prahlad | Method and system of generating a data lineage repository with lineage visibility, snapshot comparison and version control in a cloud-computing platform |
-
2017
- 2017-05-15 CN CN201710339811.8A patent/CN106921755B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104391989A (en) * | 2014-12-16 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Distributed ETL (extract transform load) all-in-one machine system |
| CN104573071A (en) * | 2015-01-26 | 2015-04-29 | 湖南大学 | Intelligent school situation analysis system and method based on megadata technology |
| CN106549829A (en) * | 2016-10-28 | 2017-03-29 | 北方工业大学 | Big data calculating platform monitoring system and method |
Non-Patent Citations (1)
| Title |
|---|
| 基于Hadoop的非结构化文本数据ETL系统设计与实现;习云峰;《中国优秀硕士学位论文全文数据库》;20170331;第1章-第4章 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106921755A (en) | 2017-07-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107294772B (en) | Dynamic management monitoring service system combined with Docker | |
| CN110794800B (en) | Intelligent factory information management monitoring system | |
| CN112600891B (en) | Information physical fusion-based edge cloud cooperative system and working method | |
| CN113569987A (en) | Model training method and device | |
| CN111459763B (en) | Cross-kubernetes cluster monitoring system and method | |
| TWI644534B (en) | Cloud platform monitoring method and cloud platform monitoring system | |
| CN108009258B (en) | Data acquisition and analysis platform capable of being configured online | |
| CN107992392B (en) | Automatic monitoring and repairing system and method for cloud rendering system | |
| CN111865682B (en) | Method and device for handling faults | |
| CN103324715B (en) | Disaster recovery backup system availability detection method and device | |
| CN109656690A (en) | Scheduling system, method and storage medium | |
| CN108647886B (en) | Scientific computing process management system | |
| CN114143220B (en) | Real-time data visualization platform | |
| CN105162632A (en) | Automatic processing system for server cluster failures | |
| US20140136668A1 (en) | Real-time self-optimizing system for multitenant based cloud infrastructure | |
| CN102857371A (en) | Dynamic allocation management method for cluster system | |
| CN113176948A (en) | Edge gateway, edge computing system and configuration method thereof | |
| CN110619014A (en) | ETL-based data extraction method | |
| CN106921755B (en) | Enterprise data integration cloud console, implementation method and system | |
| CN114691050B (en) | Cloud native storage method, device, equipment and medium based on kubernets | |
| CN107153679B (en) | Extraction statistical method and system for semi-structured big data | |
| CN104166584A (en) | Server virtualization cluster double-layer redundant architecture and construction method thereof | |
| CN106155859B (en) | monitoring management system, information processing method and high-density server | |
| CN115826955A (en) | Application publishing method, system, electronic device and storage medium | |
| CN112055086B (en) | IIS site, release and management method of Windows service, operation and maintenance system and platform |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |