CN106921742A - The system and method for storm Distributed Application clusters is built in a kind of K UX3.0 operating systems - Google Patents
The system and method for storm Distributed Application clusters is built in a kind of K UX3.0 operating systems Download PDFInfo
- Publication number
- CN106921742A CN106921742A CN201710138378.1A CN201710138378A CN106921742A CN 106921742 A CN106921742 A CN 106921742A CN 201710138378 A CN201710138378 A CN 201710138378A CN 106921742 A CN106921742 A CN 106921742A
- Authority
- CN
- China
- Prior art keywords
- storm
- data
- clusters
- layer
- application
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/61—Installation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/61—Installation
- G06F8/63—Image based installation; Cloning; Build to order
Abstract
The present invention provides the system and method that storm Distributed Application clusters are built in a kind of K UX3.0 operating systems, build storm Distributed Applications group system and K UX operating systems are connected by SSH standard agreements, Storm clusters are disposed on N+1 node of 64 road servers, including terminal access layer, application layer and data Layer, three layers are attached communication by LAN.Terminal access layer is attached by LAN with application layer and communicated, by ssh protocol remote attended operation systems, remote control Storm application platforms.Application layer is communicated by MQ, Socket, front end system business datum obtains the modes such as API or Log monitoring and obtains data from data access layer, data landing layer is write after being processed by Storm real time processing systems, data access layer can be database, or other storages or apply.
Description
Technical field
The present invention relates to build storm Distributed Applications in computer realm, more particularly to a kind of K-UX3.0 operating systems
The system and method for cluster.
Background technology
The value of data is reduced as time goes by, so event must as soon as possible at them after occurring
Reason, is just processed it at once when best data occur, and an event occurs carries out single treatment, rather than being cached into
One batch processing.Storm technologies enormously simplify parallel real time data processing, developer can be allow rapidly to build a set of
Healthy and strong, easy-to-use real-time streams treatment framework, coordinates SQL products, NoSQL products or MapReduce calculating platforms, it is possible to low
The realtime product of the imagination is difficult into before locally making a lot.
K-UX3.0 operating systems are the host operating systems by the certifications of Unix 03, and K-UX operating systems have extensive
Compatibility, based on X86-based, 64 road servers, with data-handling capacity higher.But, how Storm technologies to be built
It is current assistant officer technical problem to be solved in K-UX3.0 operating systems.
The content of the invention
In order to overcome above-mentioned deficiency of the prior art, the present invention to provide one kind and build storm Distributed Application clusters system
System, including：Terminal access layer, application layer and data Layer；Terminal access layer is attached by network with application layer and communicated, terminal
Access layer passes through ssh protocol remote attended operation systems and remote control Storm application platforms；Application layer by MQ,
Socket communications, front end system business datum obtain API, or Log monitoring, and obtain data from data Layer, by Storm realities
When treatment write-in data；Data Layer includes：Database.
Preferably, terminal access layer includes multiple terminal clientsaconnects, and each terminal clientsaconnect is accessed by VPN and applied
Layer, each terminal clientsaconnect connects cluster application server by application layer, and remote management Strom platforms check cluster service
The deployment way of device, host node are related to the distribution condition of working node, running status, the topological data for the treatment of, task run
Daily record, at the same terminal clientsaconnect conducted interviews for a certain topological task, data processing, user's registration operation, terminal clientsaconnect
Remote control is carried out by remote management Strom platforms.
Preferably, application layer is made up of N+1 node, wherein with a node deployment on Nimbus servers, N number of section
Point is deployed on Supervisor servers, while disposing Zookeeper clusters on all nodes, is realized to Storm clusters
Coherency management, strengthens the reliability and scalability of Storm clusters；Data Layer lands layer two comprising data access layer and data
Part, data access layer obtains API or Log monitor modes using MQ or Socket communications or front end system business datum
Source traffic, data landing layer using HDFS or Mysql data storage device.
Preferably, all node deployments of application layer are in Zookeeper clusters, the fortune for recording Storm cluster servers
Row state, coordination data storage, Nimbus servers coordinate supervisor servers, enter line code point by Zookeeper
Hair task is distributed, and realizes the coherency management to Storm clusters, strengthens the reliability and scalability of Storm clusters.
A kind of method that storm Distributed Application clusters are built in K-UX3.0 operating systems, method includes：
Build Zookeeper clusters；
Java and Python is installed on every server in cluster to rely on；
Build Storm clusters.
Preferably, building Zookeeper cluster steps includes：
Zookeeper is installed on every server, the data catalogues and log catalogues of Zookeeper is created；
The myid files of zk clusters are set up under Data catalogues, myid files are edited, and on the server of corresponding IP
It is input into corresponding numbering；
Zoo_sample.cfg file copies under conf catalogues is a, and it is zoo.cfg to be named as, and changes zoo.cfg
Configuration file；
The bin directory paths of zk are added into environmental variance, it is come into force.
Preferably, Java and Python dependences step is installed on every server in cluster to be included：
Java and Python is installed in cluster on every server to rely on, Jdk and Python files are unziped into phase respectively
Catalogue, and configuration surroundings variable are answered, bin catalogues are loaded into path, it is come into force.
Preferably, building Storm cluster steps includes：
Storm is respectively mounted on Nimbus servers and Supervisor servers, a local data of Storm is created
Catalogue, system path is added by the bin catalogues of Storm, it is come into force；
It is respectively configured the conf/storm.yaml files on Nimbus servers and Supervisor server clusters.
As can be seen from the above technical solutions, the present invention has advantages below：
Build storm Distributed Applications group system and K-UX operating systems, 64 road servers are connected by SSH standard agreements
N+1 node on dispose Storm clusters, including terminal access layer, application layer and data Layer, three layers are connected by LAN
Connect letter.Terminal access layer is attached by LAN with application layer and communicated, by ssh protocol remote attended operation systems,
Remote control Storm application platforms.Application layer is communicated by MQ, Socket, front end system business datum obtains API or Log
The modes such as monitoring obtain data from data access layer, and data landing layer, number are write after being processed by Storm real time processing systems
Can be database, or other storages or application according to Access Layer.
It is more safe and reliable than linux system by K-UX3.0 operating systems, the security of Storm clusters and steady can be lifted
It is qualitative.64 nodes are supported by the maximum of 64 road servers, Storm clusters are built on this platform can save more skies
Between.K-UX operating systems can be managed collectively by BMC management softwares, improve the managerial ability for providing whole Storm clusters.
Brief description of the drawings
In order to illustrate more clearly of technical scheme, will make simple to the accompanying drawing to be used needed for description below
Introduce on ground, it should be apparent that, drawings in the following description are only some embodiments of the present invention, for ordinary skill
For personnel, on the premise of not paying creative work, other accompanying drawings can also be obtained according to these accompanying drawings.
Fig. 1 is the overall schematic of the system that storm Distributed Application clusters are built in K-UX3.0 operating systems；
Fig. 2 is the system cluster structure chart that storm Distributed Application clusters are built in K-UX3.0 operating systems；
Fig. 3 is the method flow diagram that storm Distributed Application clusters are built in K-UX3.0 operating systems.
Specific embodiment
To enable that goal of the invention of the invention, feature, advantage are more obvious and understandable, below will be with specific
Embodiment and accompanying drawing, the technical scheme to present invention protection are clearly and completely described, it is clear that implementation disclosed below
Example is only a part of embodiment of the invention, and not all embodiment.Based on the embodiment in this patent, the common skill in this area
All other embodiment that art personnel are obtained under the premise of creative work is not made, belongs to the model of this patent protection
Enclose.
The system that storm Distributed Application clusters are provided in a kind of K-UX3.0 operating systems of the present embodiment offer, such as Fig. 1,
Shown in Fig. 2, including：Terminal access layer 1, application layer 2 and data Layer 3；Terminal access layer 1 is connected with application layer 2 by network
Letter is connected, terminal access layer 1 passes through ssh protocol remote attended operation systems and remote control Storm application platforms；Application layer
2 are communicated by MQ, Socket, front end system business datum communicates, and obtain API, or Log monitoring, and right in real time by Storm
Data Layer 3 carries out data processing；Data processing method includes：Data write, data modification, data edition, data supplementing editor,
Data deletion etc..Data Layer includes：Database.
In the present embodiment, terminal access layer includes multiple terminal clientsaconnects, and each terminal clientsaconnect is accessed by VPN
Application layer, each terminal clientsaconnect connects cluster application server by application layer, and remote management Strom platforms check cluster
The deployment way of server, the distribution condition of host node and working node, running status, topological data, the task run for the treatment of
Correlation log, at the same terminal clientsaconnect conducted interviews for a certain topological task, data processing, user's registration operation, terminal visitor
Family machine carries out remote control by remote management Strom platforms.
Application layer is made up of N+1 node, wherein with a node deployment on Nimbus servers, N number of node deployment
On Supervisor servers, while disposing Zookeeper clusters on all nodes, the uniformity to Storm clusters is realized
Management, strengthens the reliability and scalability of Storm clusters；Data Layer lands layer two parts comprising data access layer and data,
Data access layer obtains the source number of API or Log monitor modes using MQ or Socket communications or front end system business datum
According to stream, data landing layer uses the data storage device of HDFS or Mysql.
The all node deployments of application layer in Zookeeper clusters, for record Storm cluster servers running status,
Coordination data is stored, and Nimbus servers coordinate supervisor servers by Zookeeper, carries out code distribution task point
Match somebody with somebody, realize the coherency management to Storm clusters, strengthen the reliability and scalability of Storm clusters.
The present invention also provides a kind of method that storm Distributed Application clusters are built in K-UX3.0 operating systems, such as Fig. 3
Shown, method includes：
S1：Build Zookeeper clusters；
S2：Java and Python is installed on every server in cluster to rely on；
S3：Build Storm clusters.
Specifically, build Zookeeper cluster steps including：
Zookeeper is installed on every server, the data catalogues and log catalogues of Zookeeper is created；
The myid files of zk clusters are set up under Data catalogues, myid files are edited, and on the server of corresponding IP
It is input into corresponding numbering；
Zoo_sample.cfg file copies under conf catalogues is a, and it is zoo.cfg to be named as, and changes zoo.cfg
Configuration file；
The bin directory paths of zk are added into environmental variance, it is come into force.
Java and Python dependences step is installed on every server in cluster to be included：
Java and Python is installed in cluster on every server to rely on, Jdk and Python files are unziped into phase respectively
Catalogue, and configuration surroundings variable are answered, bin catalogues are loaded into path, it is come into force.
Building Storm cluster steps includes：
Storm is respectively mounted on Nimbus servers and Supervisor servers, a local data of Storm is created
Catalogue, system path is added by the bin catalogues of Storm, it is come into force；
It is respectively configured the conf/storm.yaml files on Nimbus servers and Supervisor server clusters.
With reference to example once, the present invention will be further described.
Example：Array deposits everyone word, function：
It is random to obtain someone's word；Addition exclamation mark in end if getting！Exclamation mark will be added！
If print.
Implementation method is：
1) enter example root, perform mvn clean, the project target catalogues of original generation are deleted.
2) pack jar in local Repository：Mvn install, the catalogue of generation is in pom.xml
Groupid forms.
3) will
C:Users zhaobj .m2 repository storm storm-samples1 0.0.1-SNAPSHOT mesh
Storm-samples1-0.0.1-SNAPSHOT-jar-with-dependencies.jar under record copies nimbus servers
Under upper bin catalogues.
4) Topology is disposed
Performed under the bin catalogues of Nimbus servers storm and such as issued orders：
./storm jar storm-samples1-0.0.1-SNAPSHOT-jar-with-dependencies.jar
storm.storm_samples1.ExclaimBasicTopo ExclaimBasicTopo。
5) result is checked
Two ways：First, StormUI is checked.2nd ./storm list are run under the bin catalogues of storm.
6) topology is closed
Two ways：First, the topology to be closed, such as ExclaimBasicTopo, Ran Hou are clicked above StormUI
Kill is selected in the Topology actions of new page.2nd, operation ./storm kill ExclaimBasicTopo.
Each embodiment is described by the way of progressive in this specification, and what each embodiment was stressed is and other
The difference of embodiment, identical similar portion reference mutually between each embodiment.
The foregoing description of the disclosed embodiments, enables professional and technical personnel in the field to realize or uses the present invention.
Various modifications to these embodiments will be apparent for those skilled in the art, as defined herein
General Principle can be realized in other embodiments without departing from the spirit or scope of the present invention.Therefore, the present invention
The embodiments shown herein is not intended to be limited to, and is to fit to and principles disclosed herein and features of novelty phase one
The scope most wide for causing.
Claims (8)
1. the system that storm Distributed Application clusters are built in a kind of K-UX3.0 operating systems, it is characterised in that including：Terminal
Access layer, application layer and data Layer；Terminal access layer is attached by network with application layer and communicated, and terminal access layer passes through
Ssh protocol remote attended operation systems and remote control Storm application platforms；Application layer is communicated by MQ, Socket, front end
System business data acquisition API, or Log monitoring, and data are obtained from data Layer, write data by Storm real-time processings；
Data Layer includes：Database.
2. the system that storm Distributed Application clusters are built in K-UX3.0 operating systems according to claim 1, it is special
Levy and be,
Terminal access layer includes multiple terminal clientsaconnects, and each terminal clientsaconnect accesses application layer, each terminal by VPN
Client computer connects cluster application server by application layer, and remote management Strom platforms check the deployment side of cluster server
The distribution condition of formula, host node and working node, running status, topological data, the task run correlation log for the treatment of, while eventually
End client computer conducts interviews for a certain topological task, data processing, user's registration operation, terminal clientsaconnect is by remote management
Strom platforms carry out remote control.
3. the system that storm Distributed Application clusters are built in K-UX3.0 operating systems according to claim 1, it is special
Levy and be,
Application layer is made up of N+1 node, wherein with a node deployment on Nimbus servers, N number of node deployment exists
On Supervisor servers, while disposing Zookeeper clusters on all nodes, the uniformity pipe to Storm clusters is realized
Reason；Data Layer lands layer two parts comprising data access layer and data, and data access layer is using MQ or Socket communications or preceding
End system business datum, obtains the source traffic of API or Log monitor modes, and data landing layer is using HDFS's or Mysql
Data storage device.
4. the system that storm Distributed Application clusters are built in K-UX3.0 operating systems according to claim 3, it is special
Levy and be,
The all node deployments of application layer are in Zookeeper clusters, running status, coordination for recording Storm cluster servers
Data storage, Nimbus servers coordinate supervisor servers by Zookeeper, carry out code distribution task distribution,
Realize the coherency management to Storm clusters.
5. a kind of method that storm Distributed Application clusters are built in K-UX3.0 operating systems, it is characterised in that method includes：
Build Zookeeper clusters；
Java and Python is installed on every server in cluster to rely on；
Build Storm clusters.
6. the method that storm Distributed Application clusters are built in K-UX3.0 operating systems according to claim 5, it is special
Levy and be, building Zookeeper cluster steps includes：
Zookeeper is installed on every server, the data catalogues and log catalogues of Zookeeper is created；
The myid files of zk clusters are set up under Data catalogues, myid files is edited, and be input on the server of corresponding IP
Corresponding numbering；
Zoo_sample.cfg file copies under conf catalogues is a, and it is zoo.cfg to be named as, modification zoo.cfg configurations
File；
The bin directory paths of zk are added into environmental variance, it is come into force.
7. the method that storm Distributed Application clusters are built in K-UX3.0 operating systems according to claim 5, it is special
Levy and be, Java and Python dependences step is installed on every server in cluster to be included：
Java and Python is installed in cluster on every server to rely on, Jdk and Python files are unziped into corresponding mesh respectively
Record, and configuration surroundings variable, bin catalogues are loaded into path, it is come into force.
8. the method that storm Distributed Application clusters are built in K-UX3.0 operating systems according to claim 5, it is special
Levy and be, building Storm cluster steps includes：
Storm is respectively mounted on Nimbus servers and Supervisor servers, a local data mesh of Storm is created
Record, system path is added by the bin catalogues of Storm, it is come into force；
It is respectively configured the conf/storm.yaml files on Nimbus servers and Supervisor server clusters.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710138378.1A CN106921742A (en) | 2017-03-09 | 2017-03-09 | The system and method for storm Distributed Application clusters is built in a kind of K UX3.0 operating systems |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710138378.1A CN106921742A (en) | 2017-03-09 | 2017-03-09 | The system and method for storm Distributed Application clusters is built in a kind of K UX3.0 operating systems |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106921742A true CN106921742A (en) | 2017-07-04 |
Family
ID=59461737
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710138378.1A Pending CN106921742A (en) | 2017-03-09 | 2017-03-09 | The system and method for storm Distributed Application clusters is built in a kind of K UX3.0 operating systems |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106921742A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109379425A (en) * | 2018-10-16 | 2019-02-22 | 浪潮软件集团有限公司 | Distributed cluster deployment management method and device |
| CN111125013A (en) * | 2019-12-26 | 2020-05-08 | 北京锐安科技有限公司 | Data warehousing method, device, equipment and medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104268260A (en) * | 2014-10-10 | 2015-01-07 | 中国科学院重庆绿色智能技术研究院 | Method, device and system for classifying streaming data |
| CN105468720A (en) * | 2015-11-20 | 2016-04-06 | 北京锐安科技有限公司 | Method for integrating distributed data processing systems, corresponding systems and data processing method |
-
2017
- 2017-03-09 CN CN201710138378.1A patent/CN106921742A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104268260A (en) * | 2014-10-10 | 2015-01-07 | 中国科学院重庆绿色智能技术研究院 | Method, device and system for classifying streaming data |
| CN105468720A (en) * | 2015-11-20 | 2016-04-06 | 北京锐安科技有限公司 | Method for integrating distributed data processing systems, corresponding systems and data processing method |
Non-Patent Citations (1)
| Title |
|---|
| 司徒少帅: "基于Storm的交通信息实时处理系统的设计与实现", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109379425A (en) * | 2018-10-16 | 2019-02-22 | 浪潮软件集团有限公司 | Distributed cluster deployment management method and device |
| CN111125013A (en) * | 2019-12-26 | 2020-05-08 | 北京锐安科技有限公司 | Data warehousing method, device, equipment and medium |
| CN111125013B (en) * | 2019-12-26 | 2023-03-17 | 北京锐安科技有限公司 | Data warehousing method, device, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11645309B2 (en) | Discovery of database and related services | |
| US11431568B2 (en) | Discovery of software bus architectures | |
| US10771344B2 (en) | Discovery of hyper-converged infrastructure devices | |
| US7434220B2 (en) | Distributed computing infrastructure including autonomous intelligent management system | |
| US9213581B2 (en) | Method and system for a cloud frame architecture | |
| CN112600891B (en) | Information physical fusion-based edge cloud cooperative system and working method | |
| US7584282B2 (en) | Object-based service oriented architecture method, apparatus and media | |
| US20220070279A1 (en) | Api key security posture scoring for microservices to determine microservice security risks | |
| US7937716B2 (en) | Managing collections of appliances | |
| US20200236187A1 (en) | Discovery and service mapping of application programming interface proxies | |
| US11635752B2 (en) | Detection and correction of robotic process automation failures | |
| US10230567B2 (en) | Management of a plurality of system control networks | |
| US10708755B2 (en) | Discovery and service mapping of serverless resources | |
| US20200348968A1 (en) | Active queue management in a multi-node computing environment | |
| CN109587209A (en) | A kind of server storage cloud platform based on file multiple person cooperational editor | |
| CN115115329A (en) | Manufacturing middleware and cloud manufacturing framework for intelligent production line | |
| CN104363306A (en) | Private cloud management control method for enterprise | |
| CN113486095A (en) | Civil aviation air traffic control cross-network safety data exchange management platform | |
| CN106921742A (en) | The system and method for storm Distributed Application clusters is built in a kind of K UX3.0 operating systems | |
| US11635953B2 (en) | Proactive notifications for robotic process automation | |
| CN104486137A (en) | Data collecting method applied to network monitoring | |
| Hernández | Distributed Infrastructure for Efficient Management of Network Services case: Large company in mining sector in Colombia | |
| US11632440B2 (en) | Integrated operational communications between computational instances of a remote network management platform | |
| US20230140179A1 (en) | Reduced Memory Utilization for Data Analytics Procedures | |
| US10708753B2 (en) | Discovery and service mapping of serverless resources |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170704 |