CN106921755A - A kind of enterprise data integration cloud console, realization method and system - Google Patents

A kind of enterprise data integration cloud console, realization method and system Download PDF

Info

Publication number
CN106921755A
CN106921755A CN201710339811.8A CN201710339811A CN106921755A CN 106921755 A CN106921755 A CN 106921755A CN 201710339811 A CN201710339811 A CN 201710339811A CN 106921755 A CN106921755 A CN 106921755A
Authority
CN
China
Prior art keywords
server
cloud console
information
task
cluster
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201710339811.8A
Other languages
Chinese (zh)
Other versions
CN106921755B (en
Inventor
林木
高冉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Inspur Software Co Ltd
Original Assignee
Inspur Software Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Inspur Software Co Ltd filed Critical Inspur Software Co Ltd
Priority to CN201710339811.8A priority Critical patent/CN106921755B/en
Publication of CN106921755A publication Critical patent/CN106921755A/en
Application granted granted Critical
Publication of CN106921755B publication Critical patent/CN106921755B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1001Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H04L67/1004Server selection for load balancing
    • H04L67/1008Server selection for load balancing based on parameters of servers, e.g. available memory or workload
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/02Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
    • H04L67/025Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Debugging And Monitoring (AREA)
  • Computer And Data Communications (AREA)

Abstract

The invention discloses a kind of enterprise data integration cloud console, realization method and system, including cloud console, server cluster, its medium cloud console uses above-mentioned cloud console structure, server cluster then by master server, it is some from server group into, every server in server cluster is respectively mounted Open-Source Tools Kettle, cloud console controls All Jobs task to assign to from server by master server, and after Servers-all fulfils assignment task, the operation information of associated server is shown in master server.A kind of enterprise data integration cloud console of the invention, realization method and system are compared with prior art, overcome the treatment bottleneck of single server, and monitor in real time is carried out to server ruuning situation, find in advance and solve problem, in addition the learning cost of O&M developer is substantially reduced, data pick-up cleaning and storage work can fast and effectively be completed, there is provided the accuracy of overall data, for big data analysis provides facility.

Description

A kind of enterprise data integration cloud console, realization method and system
Technical field
The present invention relates to server cluster technical field, specifically a kind of enterprise data integration cloud console, realization Method and system.
Background technology
For IT application in enterprises, the characteristics of Information System configuration generally has interim and distributivity, this is resulted in " believes The presence of breath isolated island " phenomenon.Between " information island " refers to different software, especially different inter-sectional data messages can not be total to Enjoy, cause there is mass of redundancy data, junk data in system, it is impossible to ensure the uniformity of data, seriously hinder enterprise The whole progress of informatization.To solve this problem, people begin to focus on enterprise data integration research.Attempt to difference Data in system are reprocessed, thus formed one it is integrated, towards the environment of analyzing and processing, so as to from these seas Rule is excavated in the information of amount, knowledge, aid decision is extracted.The appearance of Kettle can undoubtedly provide corresponding help, but Relative to mass data single server exist treatment bottleneck, the extreme of this outer boundary it is unfriendly and cannot remote control, lead Cause the learning cost of O&M developer to increase, reduce the accuracy of data, increased overall IT costs.
Based on this, the present invention provides a kind of enterprise data integration cloud console, realization method and system.It is proposed by the present invention Cloud console technology can manage server cluster in remote console dynamic expansion, and operation carries out level in server cluster Extension, overcomes the treatment bottleneck of single server, and carries out monitor in real time to server ruuning situation, finds in advance and solves Problem, substantially reduces the learning cost of O&M developer in addition, can fast and effectively complete data pick-up cleaning and deposit Storage work, there is provided the accuracy of overall data, for big data analysis provides facility.
The content of the invention
Technical assignment of the invention is directed to above weak point, there is provided a kind of enterprise data integration cloud console, realization Method and system.
A kind of enterprise data integration cloud console, its structure includes following five modules:Operation extension, operation represent, make Industry start and stop, monitoring operation, job accounting, for realizing remote control service device cluster, fulfil assignment task, wherein,
Operation expansion module, for dynamic expansion configuration server cluster, different server collaborations is assigned to by an operation Complete;
Operation display module, the All Jobs information for representing current server operation, including job description creation time, work Industry flow chart and Job execution log information;
Operation start and stop module, for the start and stop of remotely controlled operation task;
Monitoring operation module, obtains the daily record of job task and represents in real time;
Job accounting module, counts whole job task performance.
The cloud console monitor in real time fulfils assignment the server of task, and the information of the monitoring includes network state, CPU Utilization rate, enter number of passes and disk space information, and pre-alert notification is carried out by way of including short message, mail.
During operation start and stop module remotely controlled operation task start and stop, in real time in the accurate server for obtaining carrying collecting work Data acquisition, cleaning, the relevant information of storage.
A kind of enterprise data integration cloud console implementation method, based on cloud console, server cluster, its implementation process For:
Cloud console sets up the special cluster of data acquisition in server cluster, and the special cluster of the data acquisition includes main service It is device, some from server;
It is that every server of the special cluster of data acquisition installs Open-Source Tools Kettle, and starts Servers-all successively;
Cloud console controls All Jobs task to assign to from server by master server, and completes to make in Servers-all After industry task, the operation information of associated server is shown in master server.
The cloud console monitor in real time carries the server of collecting work, and the information of the monitoring includes network state, CPU Utilization rate, enter number of passes and disk space information, and pre-alert notification is carried out by way of including short message, mail.
The remote controlled job task start and stop of cloud console, in real time in the accurate server for obtaining carrying collecting work Data acquisition, cleaning, the relevant information of storage.
Operation expansion module is configured with the cloud console, the operation expansion module is adopted for dynamic expansion configuration data Collect special cluster, a job task is assigned to carries out collaboration completion in different servers, master server control is each from clothes Business device collaboration data acquisition work, obtains the network state of each server, CPU usage, enters number of passes and disk is empty in real time Between information, and in time early warning.
After the completion of job task, job information can be represented in cloud console, the job information is created including job description Time, operation process chart and Job execution log information are built, All Jobs information is obtained.
A kind of enterprise data integration cloud control feature, including cloud console, server cluster, its medium cloud console are used Above-mentioned cloud console structure, server cluster is then by master server, some from server group into every in server cluster clothes Business device is respectively mounted Open-Source Tools Kettle, and cloud console controls All Jobs task to assign to from server by master server In, and after Servers-all fulfils assignment task, the operation information of associated server is shown in master server.
Each server collaboration completes a certain job task in the server cluster, and obtains related service by master server Device operation information;Cloud console remote control master server will be distributed to different business systems, data in heterogeneous data source After being drawn into interim intermediate layer, cleaned, changed, it is integrated, be finally loaded into data warehouse or in Data Mart, it is described Data include relation data, flat data file.
A kind of enterprise data integration cloud console of the invention, realization method and system compared to the prior art, with Lower beneficial effect:
A kind of enterprise data integration cloud console of the invention, realization method and system, by cloud control technology, remote control The operation of Kettle, overcomes the bottleneck of single server treatment first, can solve a large amount of operations by trunking mode concurrent Problem, more can fast and effectively realize the carrying out of complex job, and the robustness of whole operation processing procedure is increased in addition, The machine of delaying of single server does not influence overall stable operation;Secondly graphic interface represents relevant information and remote control behaviour The operation distribution of operation reduces O&M developer's learning cost, more can fast and effectively control Kettle, it is ensured that It is reliable and stable that overall data is extracted, practical, applied widely.
Brief description of the drawings
Accompanying drawing 1 realizes schematic diagram for the inventive method.
Specific embodiment
Below in conjunction with the accompanying drawings and specific embodiment the invention will be further described.
Enterprise data integration, is primarily referred to as the business datum based on the scattered information system of enterprise and is concentrated again, united again The process of one management, including data acquisition, cleaning, treatment, storage etc..
Cloud console(Cloud Control Platform, CCP)Be a kind of disposal ability can elastic telescopic long-range control Platform processed, based on this platform processes data integration job, its way to manage is simpler than physical server efficiently.
The purpose of the present invention is to be extracted with cloud console technology remote control data, cleaned, storing work, overcomes separate unit Server process bottleneck, reduces the difficulty of system development and O&M, the generation of wrong data is reduced, for enterprise management decision-making is provided Accurately reference.
As shown in Figure 1, a kind of enterprise data integration cloud console, its structure includes following five modules:Operation extension, Operation represents, operation start and stop, monitoring operation, job accounting, for realizing remote control service device cluster, fulfil assignment task, Wherein,
Operation extends:
Can be with dynamic expansion configuration server cluster by cloud console, an operation is assigned into different servers is carried out, Master server control is each to overcome single server to process bottleneck from server collaboration data acquisition work, and clothes are obtained in real time Business device network state, CPU usage, enter the information such as number of passes and disk space, and can early warning in time.
Operation represents:
Represent the All Jobs information of current server operation, including job description creation time, operation process chart and operation Execution journal information etc., it is very fast to obtain All Jobs information.
Operation start and stop:
Can in real time be come into force with the start and stop of remotely controlled operation, a key.
Monitoring operation:
The daily record of All Jobs can in real time be obtained and represented, and more can quickly assist operation maintenance personnel to investigate problem.
Job accounting:
By overall job accounting, can macroscopic aspect represent the circulation of different themes data, can integrally hold all works The overall condition of industry.
During operation start and stop module remotely controlled operation task start and stop, in real time in the accurate server for obtaining carrying collecting work Data acquisition, cleaning, the relevant information of storage.
A kind of enterprise data integration cloud console implementation method, using cloud console technology, more can rapidly set up data Acquisition server cluster.Collecting work can be carried out horizontal extension, support the basic operation of collecting work:Start, stop, repairing Change and change server.Make the whole work can be while operating on multiple servers, master server control all working is divided equally Onto different servers, collaboration is completed, and reaches the purpose of load balancing.Additionally, the server reality for carrying collecting work When monitoring network state, CPU usage, enter the information such as number of passes and disk space, and can pre-alert notification, including short message, postal in time Part etc..Remote control data acquisition process instrument, calls metadata acquisition tool relevant interface, long-range to obtain related operation fortune in real time Market condition information purpose.More stable, safer application system can be built using cloud console technology, system development and fortune is reduced The difficulty of dimension and be absorbed in business innovation, while overall IT costs are greatly reduced.
The method of the present invention is based on cloud console, server cluster, and its implementation process is:
Cloud console sets up the special cluster of data acquisition in server cluster, and the special cluster of the data acquisition includes main service It is device, some from server;
It is that every server of the special cluster of data acquisition installs Open-Source Tools Kettle, and starts Servers-all successively;
Cloud console controls All Jobs task to assign to from server by master server, and completes to make in Servers-all After industry task, the operation information of associated server is shown in master server.
The cloud console monitor in real time carries the server of collecting work, and the information of the monitoring includes network state, CPU Utilization rate, enter number of passes and disk space information, and pre-alert notification is carried out by way of including short message, mail.
The remote controlled job task start and stop of cloud console, in real time in the accurate server for obtaining carrying collecting work Data acquisition, cleaning, the relevant information of storage.
Operation expansion module is configured with the cloud console, the operation expansion module is adopted for dynamic expansion configuration data Collect special cluster, a job task is assigned to carries out collaboration completion in different servers, master server control is each from clothes Business device collaboration data acquisition work, obtains the network state of each server, CPU usage, enters number of passes and disk is empty in real time Between information, and in time early warning.
After the completion of job task, job information can be represented in cloud console, the job information is created including job description Time, operation process chart and Job execution log information are built, All Jobs information is obtained.
A kind of enterprise data integration cloud control feature, including cloud console, server cluster, its medium cloud console are used Above-mentioned cloud console structure, server cluster is then by master server, some from server group into every in server cluster clothes Business device is respectively mounted Open-Source Tools Kettle, and cloud console controls All Jobs task to assign to from server by master server In, and after Servers-all fulfils assignment task, the operation information of associated server is shown in master server.
Each server collaboration completes a certain job task in the server cluster, and obtains related service by master server Device operation information;Cloud console remote management multiple servers cluster, remote control data sampling instrument Kettle realizes operation In server cluster horizontal extension, server cluster by a master server, with some from server group into master server conduct One controller of cluster.Each server collaboration completes a certain job task, obtains associated server operation information.It is long-range quick It is effective to be distributed to different business systems, data such as the extraction such as relation data, flat data file in heterogeneous data source To after interim intermediate layer, cleaned, changed, integrated, being finally loaded into data warehouse or in Data Mart, whole process It is high-visible in real time.
Cloud console technology is mainly using Open-Source Tools Kettle, above clustered deploy(ment) to multiple servers, wherein one Platform is Master, and remaining server is Slave, specifies specific Master servers and Slave to service by configuration file Device, starts Servers-all successively, and cluster is set in graphical interfaces, Master servers and Slave servers Can be in communication with each other by Http open protocols, realize that level of operation extends, be assigned to from server operation by master server, Multiple servers cooperate with the task that fulfils assignment.The details of each runtime server can be obtained by SlaveServer interfaces, Including network state, CPU usage, enter the information such as number of passes and disk space.By SlaveServer, TransPainter, The interface remotes such as KettleDatabaseRepository control operation start and stop, accurate acquisition data acquisition cleaning storage in real time Relevant information.
In the present invention, first it is graphic interface dynamic configuration Kettle server clusters, realizes that the level of operation expands Exhibition, solves the slow stuck problem of complex job operation.The fortune of next graphic interface remote control data sampling instrument Kettle OK, operation job information being obtained in real time and being represented, for O&M, developer greatly reduces learning cost, is enterprise data integration Facility is provided.
By specific embodiment above, the those skilled in the art can readily realize the present invention.But should Work as understanding, the present invention is not limited to above-mentioned specific embodiment.On the basis of disclosed embodiment, the technical field Technical staff can be combined different technical characteristics, so as to realize different technical schemes.
In addition to the technical characteristic described in specification, the known technology of those skilled in the art is.

Claims (10)

1. a kind of enterprise data integration cloud console, it is characterised in that its structure includes following five modules:Operation extension, work Industry represents, operation start and stop, monitoring operation, job accounting, for realizing remote control service device cluster, fulfil assignment task, its In,
Operation expansion module, for dynamic expansion configuration server cluster, different server collaborations is assigned to by an operation Complete;
Operation display module, the All Jobs information for representing current server operation, including job description creation time, work Industry flow chart and Job execution log information;
Operation start and stop module, for the start and stop of remotely controlled operation task;
Monitoring operation module, obtains the daily record of job task and represents in real time;
Job accounting module, counts whole job task performance.
2. a kind of enterprise data integration cloud console according to claim 1, it is characterised in that the cloud console is real-time The server of the task that fulfils assignment is monitored, the information of the monitoring includes network state, CPU usage, enters number of passes and disk space Information, and pre-alert notification is carried out by way of including short message, mail.
3. a kind of enterprise data integration cloud console according to claim 1, it is characterised in that operation start and stop module is long-range It is accurate in real time to obtain data acquisition, cleaning, the correlation of storage in the server for carrying collecting work during control job task start and stop Information.
4. a kind of enterprise data integration cloud console implementation method, it is characterised in that based on cloud console, server cluster, its Implementation process is:
Cloud console sets up the special cluster of data acquisition in server cluster, and the special cluster of the data acquisition includes main service It is device, some from server;
It is that every server of the special cluster of data acquisition installs Open-Source Tools Kettle, and starts Servers-all successively;
Cloud console controls All Jobs task to assign to from server by master server, and completes to make in Servers-all After industry task, the operation information of associated server is shown in master server.
5. a kind of enterprise data integration cloud console implementation method according to claim 4, it is characterised in that the cloud control Platform monitor in real time processed carries the server of collecting work, the information of the monitoring includes network state, CPU usage, enter number of passes and Disk space information, and pre-alert notification is carried out by way of including short message, mail.
6. a kind of enterprise data integration cloud console implementation method according to claim 5, it is characterised in that the cloud control The remote controlled job task start and stop of platform processed, it is accurate in real time to obtain data acquisition in the server for carrying collecting work, clean, deposit The relevant information of storage.
7. a kind of enterprise data integration cloud console implementation method according to claim 5, it is characterised in that the cloud control Operation expansion module is configured with platform processed, the operation expansion module is used for dynamic expansion configuration data and gathers special cluster, by one Individual job task carries out collaboration completion in being assigned to different servers, and master server control is each to complete data from server collaboration Collecting work, obtains the network state of each server, CPU usage, enters number of passes and disk space information in real time, and pre- in time It is alert.
8. a kind of enterprise data integration cloud console implementation method according to claim 5, it is characterised in that appoint in operation After the completion of business, job information can be represented in cloud console, the job information includes job description creation time, operation process chart And Job execution log information, obtain All Jobs information.
9. a kind of enterprise data integration cloud control feature, it is characterised in that including cloud console, server cluster, its medium cloud Console uses above-mentioned cloud console structure, and server cluster is then by master server, some from server group into server cluster In every server be respectively mounted Open-Source Tools Kettle, cloud console controls All Jobs task to assign to by master server From server, and after Servers-all fulfils assignment task, the operation information of associated server is shown in master server.
10. a kind of enterprise data integration cloud control feature according to claim 9, it is characterised in that the server Each server collaboration completes a certain job task in cluster, and obtains associated server operation information by master server;Cloud is controlled Platform remote control master server will be distributed to different business systems, data pick-up in heterogeneous data source to interim intermediate layer Afterwards, cleaned, changed, it is integrated, be finally loaded into data warehouse or in Data Mart, the data include relation number According to, flat data file.
CN201710339811.8A 2017-05-15 2017-05-15 Enterprise data integration cloud console, implementation method and system Active CN106921755B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710339811.8A CN106921755B (en) 2017-05-15 2017-05-15 Enterprise data integration cloud console, implementation method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710339811.8A CN106921755B (en) 2017-05-15 2017-05-15 Enterprise data integration cloud console, implementation method and system

Publications (2)

Publication Number Publication Date
CN106921755A true CN106921755A (en) 2017-07-04
CN106921755B CN106921755B (en) 2020-04-28

Family

ID=59567773

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710339811.8A Active CN106921755B (en) 2017-05-15 2017-05-15 Enterprise data integration cloud console, implementation method and system

Country Status (1)

Country Link
CN (1) CN106921755B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111435344A (en) * 2019-01-15 2020-07-21 中国石油集团川庆钻探工程有限公司长庆钻井总公司 Big data-based drilling acceleration influence factor analysis model

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120310875A1 (en) * 2011-06-03 2012-12-06 Prashanth Prahlad Method and system of generating a data lineage repository with lineage visibility, snapshot comparison and version control in a cloud-computing platform
CN104391989A (en) * 2014-12-16 2015-03-04 浪潮电子信息产业股份有限公司 Distributed ETL all-in-one machine system
CN104573071A (en) * 2015-01-26 2015-04-29 湖南大学 Intelligent school situation analysis system and method based on megadata technology
CN106549829A (en) * 2016-10-28 2017-03-29 北方工业大学 Big data calculating platform monitoring system and method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120310875A1 (en) * 2011-06-03 2012-12-06 Prashanth Prahlad Method and system of generating a data lineage repository with lineage visibility, snapshot comparison and version control in a cloud-computing platform
CN104391989A (en) * 2014-12-16 2015-03-04 浪潮电子信息产业股份有限公司 Distributed ETL all-in-one machine system
CN104573071A (en) * 2015-01-26 2015-04-29 湖南大学 Intelligent school situation analysis system and method based on megadata technology
CN106549829A (en) * 2016-10-28 2017-03-29 北方工业大学 Big data calculating platform monitoring system and method

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
习云峰: "基于Hadoop的非结构化文本数据ETL系统设计与实现", 《中国优秀硕士学位论文全文数据库》 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111435344A (en) * 2019-01-15 2020-07-21 中国石油集团川庆钻探工程有限公司长庆钻井总公司 Big data-based drilling acceleration influence factor analysis model

Also Published As

Publication number Publication date
CN106921755B (en) 2020-04-28

Similar Documents

Publication Publication Date Title
CN107689953B (en) Multi-tenant cloud computing-oriented container security monitoring method and system
CN105631026B (en) Safety data analysis system
CN105357038B (en) Monitor the method and system of cluster virtual machine
CN102414674B (en) Application efficiency engine
US8175863B1 (en) Systems and methods for analyzing performance of virtual environments
EP3977278A1 (en) Automated cloud-edge streaming workload distribution and bidirectional migration with lossless, once-only processing
CN107508722B (en) Service monitoring method and device
CN105653425B (en) Monitoring system based on complex event processing engine
CN105592122A (en) Cloud platform monitoring method and cloud platform monitoring system
CN102591765A (en) Progress automatic management system
CN105162632A (en) Automatic processing system for server cluster failures
CN107992392A (en) A kind of automatic monitoring repair system and method for cloud rendering system
CN108696571B (en) Cloud storage service system and method, cloud service intelligent equipment and electronic device
CN103780696A (en) Cloud monitoring method, device and system based on distributed pushing
CN107645410A (en) A kind of virtual machine management system and method based on OpenStack cloud platforms
CN104935455A (en) Cluster construction method and device therefor
KR101765725B1 (en) System and Method for connecting dynamic device on mass broadcasting Big Data Parallel Distributed Processing
CN112506969A (en) BMC address query method, system, equipment and readable storage medium
CN109213658A (en) A kind of method for inspecting and device
JP5740338B2 (en) Virtual environment operation support system
JP2010128597A (en) Information processor and method of operating the same
CN106921755A (en) A kind of enterprise data integration cloud console, realization method and system
CN116737560B (en) Intelligent training system based on intelligent guide control
CN110674008B (en) Problem disk log information collection method, device, equipment and medium of SSD
CN109995571B (en) Method and device for matching server configuration and VNF application

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant