CN106921755A - A kind of enterprise data integration cloud console, realization method and system - Google Patents
A kind of enterprise data integration cloud console, realization method and system Download PDFInfo
- Publication number
- CN106921755A CN106921755A CN201710339811.8A CN201710339811A CN106921755A CN 106921755 A CN106921755 A CN 106921755A CN 201710339811 A CN201710339811 A CN 201710339811A CN 106921755 A CN106921755 A CN 106921755A
- Authority
- CN
- China
- Prior art keywords
- server
- cloud console
- information
- task
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L67/025—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Debugging And Monitoring (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a kind of enterprise data integration cloud console, realization method and system, including cloud console, server cluster, its medium cloud console uses above-mentioned cloud console structure, server cluster then by master server, it is some from server group into, every server in server cluster is respectively mounted Open-Source Tools Kettle, cloud console controls All Jobs task to assign to from server by master server, and after Servers-all fulfils assignment task, the operation information of associated server is shown in master server.A kind of enterprise data integration cloud console of the invention, realization method and system are compared with prior art, overcome the treatment bottleneck of single server, and monitor in real time is carried out to server ruuning situation, find in advance and solve problem, in addition the learning cost of O&M developer is substantially reduced, data pick-up cleaning and storage work can fast and effectively be completed, there is provided the accuracy of overall data, for big data analysis provides facility.
Description
Technical field
The present invention relates to server cluster technical field, specifically a kind of enterprise data integration cloud console, realization
Method and system.
Background technology
For IT application in enterprises, the characteristics of Information System configuration generally has interim and distributivity, this is resulted in " believes
The presence of breath isolated island " phenomenon.Between " information island " refers to different software, especially different inter-sectional data messages can not be total to
Enjoy, cause there is mass of redundancy data, junk data in system, it is impossible to ensure the uniformity of data, seriously hinder enterprise
The whole progress of informatization.To solve this problem, people begin to focus on enterprise data integration research.Attempt to difference
Data in system are reprocessed, thus formed one it is integrated, towards the environment of analyzing and processing, so as to from these seas
Rule is excavated in the information of amount, knowledge, aid decision is extracted.The appearance of Kettle can undoubtedly provide corresponding help, but
Relative to mass data single server exist treatment bottleneck, the extreme of this outer boundary it is unfriendly and cannot remote control, lead
Cause the learning cost of O&M developer to increase, reduce the accuracy of data, increased overall IT costs.
Based on this, the present invention provides a kind of enterprise data integration cloud console, realization method and system.It is proposed by the present invention
Cloud console technology can manage server cluster in remote console dynamic expansion, and operation carries out level in server cluster
Extension, overcomes the treatment bottleneck of single server, and carries out monitor in real time to server ruuning situation, finds in advance and solves
Problem, substantially reduces the learning cost of O&M developer in addition, can fast and effectively complete data pick-up cleaning and deposit
Storage work, there is provided the accuracy of overall data, for big data analysis provides facility.
The content of the invention
Technical assignment of the invention is directed to above weak point, there is provided a kind of enterprise data integration cloud console, realization
Method and system.
A kind of enterprise data integration cloud console, its structure includes following five modules:Operation extension, operation represent, make
Industry start and stop, monitoring operation, job accounting, for realizing remote control service device cluster, fulfil assignment task, wherein,
Operation expansion module, for dynamic expansion configuration server cluster, different server collaborations is assigned to by an operation
Complete;
Operation display module, the All Jobs information for representing current server operation, including job description creation time, work
Industry flow chart and Job execution log information;
Operation start and stop module, for the start and stop of remotely controlled operation task;
Monitoring operation module, obtains the daily record of job task and represents in real time;
Job accounting module, counts whole job task performance.
The cloud console monitor in real time fulfils assignment the server of task, and the information of the monitoring includes network state, CPU
Utilization rate, enter number of passes and disk space information, and pre-alert notification is carried out by way of including short message, mail.
During operation start and stop module remotely controlled operation task start and stop, in real time in the accurate server for obtaining carrying collecting work
Data acquisition, cleaning, the relevant information of storage.
A kind of enterprise data integration cloud console implementation method, based on cloud console, server cluster, its implementation process
For:
Cloud console sets up the special cluster of data acquisition in server cluster, and the special cluster of the data acquisition includes main service
It is device, some from server;
It is that every server of the special cluster of data acquisition installs Open-Source Tools Kettle, and starts Servers-all successively;
Cloud console controls All Jobs task to assign to from server by master server, and completes to make in Servers-all
After industry task, the operation information of associated server is shown in master server.
The cloud console monitor in real time carries the server of collecting work, and the information of the monitoring includes network state, CPU
Utilization rate, enter number of passes and disk space information, and pre-alert notification is carried out by way of including short message, mail.
The remote controlled job task start and stop of cloud console, in real time in the accurate server for obtaining carrying collecting work
Data acquisition, cleaning, the relevant information of storage.
Operation expansion module is configured with the cloud console, the operation expansion module is adopted for dynamic expansion configuration data
Collect special cluster, a job task is assigned to carries out collaboration completion in different servers, master server control is each from clothes
Business device collaboration data acquisition work, obtains the network state of each server, CPU usage, enters number of passes and disk is empty in real time
Between information, and in time early warning.
After the completion of job task, job information can be represented in cloud console, the job information is created including job description
Time, operation process chart and Job execution log information are built, All Jobs information is obtained.
A kind of enterprise data integration cloud control feature, including cloud console, server cluster, its medium cloud console are used
Above-mentioned cloud console structure, server cluster is then by master server, some from server group into every in server cluster clothes
Business device is respectively mounted Open-Source Tools Kettle, and cloud console controls All Jobs task to assign to from server by master server
In, and after Servers-all fulfils assignment task, the operation information of associated server is shown in master server.
Each server collaboration completes a certain job task in the server cluster, and obtains related service by master server
Device operation information;Cloud console remote control master server will be distributed to different business systems, data in heterogeneous data source
After being drawn into interim intermediate layer, cleaned, changed, it is integrated, be finally loaded into data warehouse or in Data Mart, it is described
Data include relation data, flat data file.
A kind of enterprise data integration cloud console of the invention, realization method and system compared to the prior art, with
Lower beneficial effect:
A kind of enterprise data integration cloud console of the invention, realization method and system, by cloud control technology, remote control
The operation of Kettle, overcomes the bottleneck of single server treatment first, can solve a large amount of operations by trunking mode concurrent
Problem, more can fast and effectively realize the carrying out of complex job, and the robustness of whole operation processing procedure is increased in addition,
The machine of delaying of single server does not influence overall stable operation;Secondly graphic interface represents relevant information and remote control behaviour
The operation distribution of operation reduces O&M developer's learning cost, more can fast and effectively control Kettle, it is ensured that
It is reliable and stable that overall data is extracted, practical, applied widely.
Brief description of the drawings
Accompanying drawing 1 realizes schematic diagram for the inventive method.
Specific embodiment
Below in conjunction with the accompanying drawings and specific embodiment the invention will be further described.
Enterprise data integration, is primarily referred to as the business datum based on the scattered information system of enterprise and is concentrated again, united again
The process of one management, including data acquisition, cleaning, treatment, storage etc..
Cloud console(Cloud Control Platform, CCP)Be a kind of disposal ability can elastic telescopic long-range control
Platform processed, based on this platform processes data integration job, its way to manage is simpler than physical server efficiently.
The purpose of the present invention is to be extracted with cloud console technology remote control data, cleaned, storing work, overcomes separate unit
Server process bottleneck, reduces the difficulty of system development and O&M, the generation of wrong data is reduced, for enterprise management decision-making is provided
Accurately reference.
As shown in Figure 1, a kind of enterprise data integration cloud console, its structure includes following five modules:Operation extension,
Operation represents, operation start and stop, monitoring operation, job accounting, for realizing remote control service device cluster, fulfil assignment task,
Wherein,
Operation extends:
Can be with dynamic expansion configuration server cluster by cloud console, an operation is assigned into different servers is carried out,
Master server control is each to overcome single server to process bottleneck from server collaboration data acquisition work, and clothes are obtained in real time
Business device network state, CPU usage, enter the information such as number of passes and disk space, and can early warning in time.
Operation represents:
Represent the All Jobs information of current server operation, including job description creation time, operation process chart and operation
Execution journal information etc., it is very fast to obtain All Jobs information.
Operation start and stop:
Can in real time be come into force with the start and stop of remotely controlled operation, a key.
Monitoring operation:
The daily record of All Jobs can in real time be obtained and represented, and more can quickly assist operation maintenance personnel to investigate problem.
Job accounting:
By overall job accounting, can macroscopic aspect represent the circulation of different themes data, can integrally hold all works
The overall condition of industry.
During operation start and stop module remotely controlled operation task start and stop, in real time in the accurate server for obtaining carrying collecting work
Data acquisition, cleaning, the relevant information of storage.
A kind of enterprise data integration cloud console implementation method, using cloud console technology, more can rapidly set up data
Acquisition server cluster.Collecting work can be carried out horizontal extension, support the basic operation of collecting work:Start, stop, repairing
Change and change server.Make the whole work can be while operating on multiple servers, master server control all working is divided equally
Onto different servers, collaboration is completed, and reaches the purpose of load balancing.Additionally, the server reality for carrying collecting work
When monitoring network state, CPU usage, enter the information such as number of passes and disk space, and can pre-alert notification, including short message, postal in time
Part etc..Remote control data acquisition process instrument, calls metadata acquisition tool relevant interface, long-range to obtain related operation fortune in real time
Market condition information purpose.More stable, safer application system can be built using cloud console technology, system development and fortune is reduced
The difficulty of dimension and be absorbed in business innovation, while overall IT costs are greatly reduced.
The method of the present invention is based on cloud console, server cluster, and its implementation process is:
Cloud console sets up the special cluster of data acquisition in server cluster, and the special cluster of the data acquisition includes main service
It is device, some from server;
It is that every server of the special cluster of data acquisition installs Open-Source Tools Kettle, and starts Servers-all successively;
Cloud console controls All Jobs task to assign to from server by master server, and completes to make in Servers-all
After industry task, the operation information of associated server is shown in master server.
The cloud console monitor in real time carries the server of collecting work, and the information of the monitoring includes network state, CPU
Utilization rate, enter number of passes and disk space information, and pre-alert notification is carried out by way of including short message, mail.
The remote controlled job task start and stop of cloud console, in real time in the accurate server for obtaining carrying collecting work
Data acquisition, cleaning, the relevant information of storage.
Operation expansion module is configured with the cloud console, the operation expansion module is adopted for dynamic expansion configuration data
Collect special cluster, a job task is assigned to carries out collaboration completion in different servers, master server control is each from clothes
Business device collaboration data acquisition work, obtains the network state of each server, CPU usage, enters number of passes and disk is empty in real time
Between information, and in time early warning.
After the completion of job task, job information can be represented in cloud console, the job information is created including job description
Time, operation process chart and Job execution log information are built, All Jobs information is obtained.
A kind of enterprise data integration cloud control feature, including cloud console, server cluster, its medium cloud console are used
Above-mentioned cloud console structure, server cluster is then by master server, some from server group into every in server cluster clothes
Business device is respectively mounted Open-Source Tools Kettle, and cloud console controls All Jobs task to assign to from server by master server
In, and after Servers-all fulfils assignment task, the operation information of associated server is shown in master server.
Each server collaboration completes a certain job task in the server cluster, and obtains related service by master server
Device operation information;Cloud console remote management multiple servers cluster, remote control data sampling instrument Kettle realizes operation
In server cluster horizontal extension, server cluster by a master server, with some from server group into master server conduct
One controller of cluster.Each server collaboration completes a certain job task, obtains associated server operation information.It is long-range quick
It is effective to be distributed to different business systems, data such as the extraction such as relation data, flat data file in heterogeneous data source
To after interim intermediate layer, cleaned, changed, integrated, being finally loaded into data warehouse or in Data Mart, whole process
It is high-visible in real time.
Cloud console technology is mainly using Open-Source Tools Kettle, above clustered deploy(ment) to multiple servers, wherein one
Platform is Master, and remaining server is Slave, specifies specific Master servers and Slave to service by configuration file
Device, starts Servers-all successively, and cluster is set in graphical interfaces, Master servers and Slave servers
Can be in communication with each other by Http open protocols, realize that level of operation extends, be assigned to from server operation by master server,
Multiple servers cooperate with the task that fulfils assignment.The details of each runtime server can be obtained by SlaveServer interfaces,
Including network state, CPU usage, enter the information such as number of passes and disk space.By SlaveServer, TransPainter,
The interface remotes such as KettleDatabaseRepository control operation start and stop, accurate acquisition data acquisition cleaning storage in real time
Relevant information.
In the present invention, first it is graphic interface dynamic configuration Kettle server clusters, realizes that the level of operation expands
Exhibition, solves the slow stuck problem of complex job operation.The fortune of next graphic interface remote control data sampling instrument Kettle
OK, operation job information being obtained in real time and being represented, for O&M, developer greatly reduces learning cost, is enterprise data integration
Facility is provided.
By specific embodiment above, the those skilled in the art can readily realize the present invention.But should
Work as understanding, the present invention is not limited to above-mentioned specific embodiment.On the basis of disclosed embodiment, the technical field
Technical staff can be combined different technical characteristics, so as to realize different technical schemes.
In addition to the technical characteristic described in specification, the known technology of those skilled in the art is.
Claims (10)
1. a kind of enterprise data integration cloud console, it is characterised in that its structure includes following five modules:Operation extension, work
Industry represents, operation start and stop, monitoring operation, job accounting, for realizing remote control service device cluster, fulfil assignment task, its
In,
Operation expansion module, for dynamic expansion configuration server cluster, different server collaborations is assigned to by an operation
Complete;
Operation display module, the All Jobs information for representing current server operation, including job description creation time, work
Industry flow chart and Job execution log information;
Operation start and stop module, for the start and stop of remotely controlled operation task;
Monitoring operation module, obtains the daily record of job task and represents in real time;
Job accounting module, counts whole job task performance.
2. a kind of enterprise data integration cloud console according to claim 1, it is characterised in that the cloud console is real-time
The server of the task that fulfils assignment is monitored, the information of the monitoring includes network state, CPU usage, enters number of passes and disk space
Information, and pre-alert notification is carried out by way of including short message, mail.
3. a kind of enterprise data integration cloud console according to claim 1, it is characterised in that operation start and stop module is long-range
It is accurate in real time to obtain data acquisition, cleaning, the correlation of storage in the server for carrying collecting work during control job task start and stop
Information.
4. a kind of enterprise data integration cloud console implementation method, it is characterised in that based on cloud console, server cluster, its
Implementation process is:
Cloud console sets up the special cluster of data acquisition in server cluster, and the special cluster of the data acquisition includes main service
It is device, some from server;
It is that every server of the special cluster of data acquisition installs Open-Source Tools Kettle, and starts Servers-all successively;
Cloud console controls All Jobs task to assign to from server by master server, and completes to make in Servers-all
After industry task, the operation information of associated server is shown in master server.
5. a kind of enterprise data integration cloud console implementation method according to claim 4, it is characterised in that the cloud control
Platform monitor in real time processed carries the server of collecting work, the information of the monitoring includes network state, CPU usage, enter number of passes and
Disk space information, and pre-alert notification is carried out by way of including short message, mail.
6. a kind of enterprise data integration cloud console implementation method according to claim 5, it is characterised in that the cloud control
The remote controlled job task start and stop of platform processed, it is accurate in real time to obtain data acquisition in the server for carrying collecting work, clean, deposit
The relevant information of storage.
7. a kind of enterprise data integration cloud console implementation method according to claim 5, it is characterised in that the cloud control
Operation expansion module is configured with platform processed, the operation expansion module is used for dynamic expansion configuration data and gathers special cluster, by one
Individual job task carries out collaboration completion in being assigned to different servers, and master server control is each to complete data from server collaboration
Collecting work, obtains the network state of each server, CPU usage, enters number of passes and disk space information in real time, and pre- in time
It is alert.
8. a kind of enterprise data integration cloud console implementation method according to claim 5, it is characterised in that appoint in operation
After the completion of business, job information can be represented in cloud console, the job information includes job description creation time, operation process chart
And Job execution log information, obtain All Jobs information.
9. a kind of enterprise data integration cloud control feature, it is characterised in that including cloud console, server cluster, its medium cloud
Console uses above-mentioned cloud console structure, and server cluster is then by master server, some from server group into server cluster
In every server be respectively mounted Open-Source Tools Kettle, cloud console controls All Jobs task to assign to by master server
From server, and after Servers-all fulfils assignment task, the operation information of associated server is shown in master server.
10. a kind of enterprise data integration cloud control feature according to claim 9, it is characterised in that the server
Each server collaboration completes a certain job task in cluster, and obtains associated server operation information by master server;Cloud is controlled
Platform remote control master server will be distributed to different business systems, data pick-up in heterogeneous data source to interim intermediate layer
Afterwards, cleaned, changed, it is integrated, be finally loaded into data warehouse or in Data Mart, the data include relation number
According to, flat data file.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201710339811.8A CN106921755B (en) | 2017-05-15 | 2017-05-15 | Enterprise data integration cloud console, implementation method and system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201710339811.8A CN106921755B (en) | 2017-05-15 | 2017-05-15 | Enterprise data integration cloud console, implementation method and system |
Publications (2)
Publication Number | Publication Date |
---|---|
CN106921755A true CN106921755A (en) | 2017-07-04 |
CN106921755B CN106921755B (en) | 2020-04-28 |
Family
ID=59567773
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201710339811.8A Active CN106921755B (en) | 2017-05-15 | 2017-05-15 | Enterprise data integration cloud console, implementation method and system |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN106921755B (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111435344A (en) * | 2019-01-15 | 2020-07-21 | 中国石油集团川庆钻探工程有限公司长庆钻井总公司 | Big data-based drilling acceleration influence factor analysis model |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120310875A1 (en) * | 2011-06-03 | 2012-12-06 | Prashanth Prahlad | Method and system of generating a data lineage repository with lineage visibility, snapshot comparison and version control in a cloud-computing platform |
CN104391989A (en) * | 2014-12-16 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Distributed ETL all-in-one machine system |
CN104573071A (en) * | 2015-01-26 | 2015-04-29 | 湖南大学 | Intelligent school situation analysis system and method based on megadata technology |
CN106549829A (en) * | 2016-10-28 | 2017-03-29 | 北方工业大学 | Big data calculating platform monitoring system and method |
-
2017
- 2017-05-15 CN CN201710339811.8A patent/CN106921755B/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120310875A1 (en) * | 2011-06-03 | 2012-12-06 | Prashanth Prahlad | Method and system of generating a data lineage repository with lineage visibility, snapshot comparison and version control in a cloud-computing platform |
CN104391989A (en) * | 2014-12-16 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Distributed ETL all-in-one machine system |
CN104573071A (en) * | 2015-01-26 | 2015-04-29 | 湖南大学 | Intelligent school situation analysis system and method based on megadata technology |
CN106549829A (en) * | 2016-10-28 | 2017-03-29 | 北方工业大学 | Big data calculating platform monitoring system and method |
Non-Patent Citations (1)
Title |
---|
习云峰: "基于Hadoop的非结构化文本数据ETL系统设计与实现", 《中国优秀硕士学位论文全文数据库》 * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111435344A (en) * | 2019-01-15 | 2020-07-21 | 中国石油集团川庆钻探工程有限公司长庆钻井总公司 | Big data-based drilling acceleration influence factor analysis model |
Also Published As
Publication number | Publication date |
---|---|
CN106921755B (en) | 2020-04-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN107689953B (en) | Multi-tenant cloud computing-oriented container security monitoring method and system | |
CN105631026B (en) | Safety data analysis system | |
CN105357038B (en) | Monitor the method and system of cluster virtual machine | |
CN102414674B (en) | Application efficiency engine | |
US8175863B1 (en) | Systems and methods for analyzing performance of virtual environments | |
EP3977278A1 (en) | Automated cloud-edge streaming workload distribution and bidirectional migration with lossless, once-only processing | |
CN107508722B (en) | Service monitoring method and device | |
CN105653425B (en) | Monitoring system based on complex event processing engine | |
CN105592122A (en) | Cloud platform monitoring method and cloud platform monitoring system | |
CN102591765A (en) | Progress automatic management system | |
CN105162632A (en) | Automatic processing system for server cluster failures | |
CN107992392A (en) | A kind of automatic monitoring repair system and method for cloud rendering system | |
CN108696571B (en) | Cloud storage service system and method, cloud service intelligent equipment and electronic device | |
CN103780696A (en) | Cloud monitoring method, device and system based on distributed pushing | |
CN107645410A (en) | A kind of virtual machine management system and method based on OpenStack cloud platforms | |
CN104935455A (en) | Cluster construction method and device therefor | |
KR101765725B1 (en) | System and Method for connecting dynamic device on mass broadcasting Big Data Parallel Distributed Processing | |
CN112506969A (en) | BMC address query method, system, equipment and readable storage medium | |
CN109213658A (en) | A kind of method for inspecting and device | |
JP5740338B2 (en) | Virtual environment operation support system | |
JP2010128597A (en) | Information processor and method of operating the same | |
CN106921755A (en) | A kind of enterprise data integration cloud console, realization method and system | |
CN116737560B (en) | Intelligent training system based on intelligent guide control | |
CN110674008B (en) | Problem disk log information collection method, device, equipment and medium of SSD | |
CN109995571B (en) | Method and device for matching server configuration and VNF application |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |