CN106254121A - Method for automatically deploying and managing big data cluster - Google Patents
Method for automatically deploying and managing big data cluster Download PDFInfo
- Publication number
- CN106254121A CN106254121A CN201610655078.6A CN201610655078A CN106254121A CN 106254121 A CN106254121 A CN 106254121A CN 201610655078 A CN201610655078 A CN 201610655078A CN 106254121 A CN106254121 A CN 106254121A
- Authority
- CN
- China
- Prior art keywords
- agent
- management
- automatization
- large data
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 27
- 238000012544 monitoring process Methods 0.000 claims abstract description 18
- 230000000007 visual effect Effects 0.000 claims abstract description 7
- 238000009434 installation Methods 0.000 claims description 13
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000010247 heart contraction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
- H04L41/0253—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using browsers or web-pages for accessing management information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/04—Network management architectures or arrangements
- H04L41/042—Network management architectures or arrangements comprising distributed management centres cooperatively managing the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/22—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks comprising specially adapted graphical user interfaces [GUI]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/10—Active monitoring, e.g. heartbeat, ping or trace-route
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/12—Network monitoring probes
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Cardiology (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Debugging And Monitoring (AREA)
- Stored Programmes (AREA)
Abstract
The invention discloses a method for automatically deploying and managing a large data cluster, which comprises a distributed architecture consisting of a Web end, a Server end and a plurality of Agent ends, wherein Java service is realized by the Server and the plurality of agents, the Server end is used for carrying out unified management, and Web end visual operation based on Ember.js/AngularJS is provided. The invention obviously improves the speed of large data cluster deployment and simplifies the deployment process; the Web page can realize the management and configuration tuning of a plurality of clusters; the visual monitoring can intuitively acquire the state information of the cluster in real time; the Kerberos security protocol can effectively improve the security of the cluster.
Description
Technical field
The present invention relates to computer software application technical field, be specifically related to a kind of automatization and dispose and management large data sets
The method of group, a kind of Java of utilization service call Python script is to the method disposing large data sets group.
Background technology
Ember.js be one for create web application JavaScript MVC framework, use based on character string
Handlebars template, support two-way binding, Observer Pattern, computation attribute (relying on other attributes dynamically to change), automatically
More new template, route test, state machine etc..The class that Ember uses self to extend create Ember object, array, character string,
Function, it is provided that big metering method is used for operating with attribute.Ember uses nestable view layer, makes view become to have levels
Metrics is the bag that an indices to JAVA service provides measurement facility, embeds Metrics in JAVA code
Code, in that context it may be convenient to each index of service code is monitored.In conjunction with Metrics, it is achieved that cluster, main frame, application
The monitoring of three aspects.
Kerberos is a kind of network authenticating protocol, and its design object is to be that client/server should by cipher key system
Powerful authentication service is provided by program.The realization of this verification process does not relies on the certification of host operating system, it is not necessary to based on
The trust of host address, it is not required that the physical security of All hosts on network, and suppose that the packet transmitted on network can be by
At random read, revise and insert.
Summary of the invention
The technical problem to be solved in the present invention is: for during large data sets group's installation and deployment, deployment is complicated, it is numerous to configure
Problem trivial, the most manageable, in order to solve in production environment, efficiently completes deployment and the management of multiple large data sets group, with
And the later stage use during configuration tuning, performance monitoring, the present invention provides a kind of automatization to dispose and management large data sets group
Method.
The technical solution adopted in the present invention is:
A kind of automatization disposes and the method for management large data sets group, and described method includes by Web end, Server end and many
The distributed structure/architecture that Agent end is constituted, Java service is realized by Server and multiple Agent, Server end carry out unifying
Management, and web terminal visualized operation based on Ember.js/AngularJS is provided.
Described Web end is mainly responsible for guiding user to carry out big market demand installation, mainly includes adding main frame, selecting to answer
With, distribution main frame, amendment configuration, cluster, main frame and application can also be carried out visual control after installation, support simultaneously
The start stop operation of each application and the tuning of configuration.
Described Server end is mainly responsible for interacting with Web end and Agent end, and described Server end has opened two big classes
API, one type provides monitoring management service for Web end, it is achieved the installation and deployment of application visualize with management and monitoring；Another kind of
For interacting with Agent end, distributed tasks is to the Agent specified, and accept Agent please to Server transmission heart beating simultaneously
Ask, monitor the state of Agent in real time.
Described Agent end is mainly responsible for calling automatization's installation and the prison of Host Status that Python script carries out applying
Control, supports the extension of host number simultaneously.
Described method is by being packaged Metrics, it is achieved that the monitoring to cluster, main frame and application, in conjunction with
Grafana has been better achieved monitoring information visuallization.
Described method is by introducing kerberos agreement, and secure context requires that comparing harsh environment can select according to demand
Select unlatching Kerberos certification to improve the safety of cluster.
The invention have the benefit that
The present invention significantly improves the speed that large data sets group disposes, and simplifies deployment flow process；Web page can realize multiple cluster
Management with configuration tuning；Visual control can obtain the status information of cluster the most in real time；Kerberos security protocol
Can effectively improve the safety of cluster.
Accompanying drawing explanation
Fig. 1 is the configuration diagram that the present invention relates to.
Detailed description of the invention
Below in conjunction with Figure of description, according to detailed description of the invention, the present invention is further described:
Embodiment 1:
As it is shown in figure 1, a kind of automatization disposes and the method for management large data sets group, described method includes by Web end, Server
The distributed structure/architecture that end and multi-Agent end are constituted, Java service is realized by Server and multiple Agent, Server end carry out
Unified management；And web terminal visualized operation based on Ember.js/AngularJS is provided, it is possible to the big data of rapid deployment
Application, supports the installation and deployment of multiple cluster, it is achieved the visual control of cluster, main frame and application simultaneously.
Embodiment 2
On the basis of embodiment 1, Web end described in the present embodiment is mainly responsible for guiding user to carry out big market demand installation, main
Including add main frame, select application, host assignment, amendment configure, after installation can also to cluster, main frame and apply into
Row visual control, supports the tuning of the operations such as each start and stop applied and configuration simultaneously.
Embodiment 3
On the basis of embodiment 1, Server end described in the present embodiment is mainly responsible for interacting with Web end and Agent end, institute
Stating Server end and opened two big classes API, one type provides monitoring management service for Web end, it is achieved the installation and deployment of application
Visualize with management and monitoring；Another kind of for interacting with Agent end, distributed tasks, to the Agent specified, accepts simultaneously
Agent sends heartbeat request to Server, monitors the state of Agent in real time.
Embodiment 4
On the basis of embodiment 1, Agent end described in the present embodiment is mainly responsible for calling that Python script carries out applying is automatic
Change and install and the monitoring of Host Status, support the extension of host number simultaneously.
Embodiment 5
On the basis of 1-4 any embodiment, method described in the present embodiment is by being packaged Metrics, it is achieved that to collection
Group, main frame and the monitoring of application, be better achieved monitoring information visuallization in conjunction with Grafana.
Embodiment 6
On the basis of embodiment 5, method described in the present embodiment is by introducing kerberos agreement, and secure context requires more severe
The environment carved can select to open Kerberos certification to improve the safety of cluster according to demand.
Embodiment is merely to illustrate the present invention, and not limitation of the present invention, about the ordinary skill of technical field
Personnel, without departing from the spirit and scope of the present invention, it is also possible to make a variety of changes and modification, the most all equivalents
Technical scheme fall within scope of the invention, the scope of patent protection of the present invention should be defined by the claims.
Claims (6)
1. an automatization disposes and the method for management large data sets group, it is characterised in that: described method include by Web end,
The distributed structure/architecture that Server end and multi-Agent end are constituted, Java service is realized by Server and multiple Agent, by Server
End carries out unified management, and provides web terminal visualized operation based on Ember.js/AngularJS.
A kind of automatization the most according to claim 1 disposes and the method for management large data sets group, it is characterised in that described
Web end is mainly responsible for guiding user to carry out big market demand installation, mainly includes adding main frame, selection application, distribution main frame, repairing
Change configuration, cluster, main frame and application can also be carried out visual control after installation, support each start and stop applied simultaneously
Operation and the tuning configured.
A kind of automatization the most according to claim 1 disposes and the method for management large data sets group, it is characterised in that described
Server end is mainly responsible for interacting with Web end and Agent end, and described Server end has opened two big classes API, Qi Zhongyi
Class provides monitoring management service for Web end, it is achieved the installation and deployment of application visualize with management and monitoring；Another kind of for
Agent end interacts, and distributed tasks, to the Agent specified, accepts Agent simultaneously and sends heartbeat request to Server, in real time
The state of monitoring Agent.
A kind of automatization the most according to claim 1 disposes and the method for management large data sets group, it is characterised in that described
Agent end is mainly responsible for calling automatization's installation and the monitoring of Host Status that Python script carries out applying, and supports simultaneously
The extension of host number.
5. disposing and the method for management large data sets group according to the arbitrary described a kind of automatization of claim 1-4, its feature exists
In, described method is by being packaged Metrics, it is achieved that the monitoring to cluster, main frame and application, real in conjunction with Grafana
Show monitoring information visuallization.
A kind of automatization the most according to claim 5 disposes and the method for management large data sets group, it is characterised in that described
Method, by introducing kerberos agreement, selects to open Kerberos certification according to demand and improves the safety of cluster.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610655078.6A CN106254121A (en) | 2016-08-11 | 2016-08-11 | Method for automatically deploying and managing big data cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610655078.6A CN106254121A (en) | 2016-08-11 | 2016-08-11 | Method for automatically deploying and managing big data cluster |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106254121A true CN106254121A (en) | 2016-12-21 |
Family
ID=58078091
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610655078.6A Pending CN106254121A (en) | 2016-08-11 | 2016-08-11 | Method for automatically deploying and managing big data cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106254121A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106843981A (en) * | 2017-02-06 | 2017-06-13 | 郑州云海信息技术有限公司 | A kind of Hue automatizations of service configuration treating method based on Apache Ambari |
| CN107239291A (en) * | 2017-06-13 | 2017-10-10 | 浪潮软件股份有限公司 | Method for templated query configuration in mobile application |
| CN107634852A (en) * | 2017-08-17 | 2018-01-26 | 新华三大数据技术有限公司 | The method and apparatus for supervising big data cluster |
| CN107741852A (en) * | 2017-10-16 | 2018-02-27 | 中国电子科技集团公司第二十八研究所 | A kind of service arrangement method based on clustered software |
| CN108628537A (en) * | 2017-03-17 | 2018-10-09 | 北京嘀嘀无限科技发展有限公司 | Monitoring data output method and device |
| CN109165022A (en) * | 2018-07-18 | 2019-01-08 | 山东中创软件商用中间件股份有限公司 | A kind of big data cluster dispositions method, system, medium and equipment |
| CN110209407A (en) * | 2019-06-12 | 2019-09-06 | 浪潮软件股份有限公司 | A kind of big data cluster automatically dispose system and method |
| CN110493061A (en) * | 2019-08-30 | 2019-11-22 | 浪潮软件股份有限公司 | A kind of management method and system of NTP cluster |
| CN110659022A (en) * | 2019-08-19 | 2020-01-07 | 浙江邦盛科技有限公司 | Method for automatically calling Python script based on Java |
| CN110781211A (en) * | 2018-07-31 | 2020-02-11 | 网宿科技股份有限公司 | Data analysis method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103617231A (en) * | 2013-11-26 | 2014-03-05 | 国家电网公司 | Large data management system |
| CN104317610A (en) * | 2014-10-11 | 2015-01-28 | 福建新大陆软件工程有限公司 | Method and device for automatic installation and deployment of hadoop platform |
| CN104462579A (en) * | 2014-12-30 | 2015-03-25 | 浪潮电子信息产业股份有限公司 | Job task management method of large data management platform |
| CN105610987A (en) * | 2016-03-18 | 2016-05-25 | 车智互联（北京）科技有限公司 | Method, application and system for managing server cluster |
| CN105681109A (en) * | 2016-03-15 | 2016-06-15 | 山东超越数控电子有限公司 | Implementation method and framework of high availability cluster management system |
-
2016
- 2016-08-11 CN CN201610655078.6A patent/CN106254121A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103617231A (en) * | 2013-11-26 | 2014-03-05 | 国家电网公司 | Large data management system |
| CN104317610A (en) * | 2014-10-11 | 2015-01-28 | 福建新大陆软件工程有限公司 | Method and device for automatic installation and deployment of hadoop platform |
| CN104462579A (en) * | 2014-12-30 | 2015-03-25 | 浪潮电子信息产业股份有限公司 | Job task management method of large data management platform |
| CN105681109A (en) * | 2016-03-15 | 2016-06-15 | 山东超越数控电子有限公司 | Implementation method and framework of high availability cluster management system |
| CN105610987A (en) * | 2016-03-18 | 2016-05-25 | 车智互联（北京）科技有限公司 | Method, application and system for managing server cluster |
Non-Patent Citations (1)
| Title |
|---|
| 江樱，戴波，王志强: "大数据管理可视化平台设计与实践", 《2014电力行业信息化年会论文集》 * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106843981A (en) * | 2017-02-06 | 2017-06-13 | 郑州云海信息技术有限公司 | A kind of Hue automatizations of service configuration treating method based on Apache Ambari |
| CN108628537A (en) * | 2017-03-17 | 2018-10-09 | 北京嘀嘀无限科技发展有限公司 | Monitoring data output method and device |
| CN107239291A (en) * | 2017-06-13 | 2017-10-10 | 浪潮软件股份有限公司 | Method for templated query configuration in mobile application |
| CN107634852A (en) * | 2017-08-17 | 2018-01-26 | 新华三大数据技术有限公司 | The method and apparatus for supervising big data cluster |
| CN107634852B (en) * | 2017-08-17 | 2018-12-11 | 新华三大数据技术有限公司 | The method and apparatus for supervising big data cluster |
| CN107741852A (en) * | 2017-10-16 | 2018-02-27 | 中国电子科技集团公司第二十八研究所 | A kind of service arrangement method based on clustered software |
| CN109165022A (en) * | 2018-07-18 | 2019-01-08 | 山东中创软件商用中间件股份有限公司 | A kind of big data cluster dispositions method, system, medium and equipment |
| CN110781211A (en) * | 2018-07-31 | 2020-02-11 | 网宿科技股份有限公司 | Data analysis method and device |
| CN110781211B (en) * | 2018-07-31 | 2022-04-05 | 网宿科技股份有限公司 | Data analysis method and device |
| CN110209407A (en) * | 2019-06-12 | 2019-09-06 | 浪潮软件股份有限公司 | A kind of big data cluster automatically dispose system and method |
| CN110659022A (en) * | 2019-08-19 | 2020-01-07 | 浙江邦盛科技有限公司 | Method for automatically calling Python script based on Java |
| CN110659022B (en) * | 2019-08-19 | 2020-09-04 | 浙江邦盛科技有限公司 | Method for automatically calling Python script based on Java |
| CN110493061A (en) * | 2019-08-30 | 2019-11-22 | 浪潮软件股份有限公司 | A kind of management method and system of NTP cluster |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106254121A (en) | Method for automatically deploying and managing big data cluster | |
| US10761913B2 (en) | System and method for real-time asynchronous multitenant gateway security | |
| CN108549580B (en) | Method for automatically deploying Kubernets slave nodes and terminal equipment | |
| US20200250074A1 (en) | Test Orchestration Platform | |
| US11275617B2 (en) | Self-managed intelligent elastic cloud stack | |
| US8713177B2 (en) | Remote management of networked systems using secure modular platform | |
| US8606897B2 (en) | Systems and methods for exporting usage history data as input to a management platform of a target cloud-based network | |
| EP3758297A1 (en) | Network-based resource configuration discovery service | |
| CN111625496B (en) | Deployment method, device and equipment of distributed file system in virtual machine environment | |
| JP5770358B2 (en) | Method and system for dynamically tagging metric data | |
| US20220232101A1 (en) | Virtual workspace experience visualization and optimization | |
| CN101188624A (en) | Grid middleware system based on virtual machine | |
| CN107370796A (en) | A kind of intelligent learning system based on Hyper TF | |
| CN110134564A (en) | A kind of visual Kafka cluster monitoring method | |
| CN109800081A (en) | A kind of management method and relevant device of big data task | |
| CN102821091A (en) | Control device and control method of virtual machine | |
| CN109687986A (en) | A kind of Redis O&M method and system based on privately owned cloud platform | |
| US10122602B1 (en) | Distributed system infrastructure testing | |
| CN110221910A (en) | Method and apparatus for executing MPI operation | |
| Venâncio et al. | Beyond VNFM: Filling the gaps of the ETSI VNF manager to fully support VNF life cycle operations | |
| US20240070123A1 (en) | Using Machine Learning to Provide a Single User Interface for Streamlined Deployment and Management of Multiple Types of Databases | |
| CN107682228A (en) | A kind of method for monitoring state and its device of OpenStack clusters | |
| CN110213043A (en) | A kind of clustering functionality test method based on Java | |
| Castro et al. | Apache Spark usage and deployment models for scientific computing | |
| CN109324820B (en) | Host updating method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20161221 |