CN107896230B - System supporting high-concurrency command execution channel - Google Patents
System supporting high-concurrency command execution channel Download PDFInfo
- Publication number
- CN107896230B CN107896230B CN201711434933.1A CN201711434933A CN107896230B CN 107896230 B CN107896230 B CN 107896230B CN 201711434933 A CN201711434933 A CN 201711434933A CN 107896230 B CN107896230 B CN 107896230B
- Authority
- CN
- China
- Prior art keywords
- module
- client
- operation command
- server
- administrator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/10—Network architectures or network communication protocols for network security for controlling access to devices or network resources
- H04L63/101—Access control lists [ACL]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention provides a system for supporting high-concurrency command execution channels, which comprises: the system comprises a receiving module, a load balancing module, a connector module, a publishing and subscribing module, an agent server and a client; the connector module is used for analyzing the operation command to obtain specific operation command information, a client ID required to be operated and an administrator ID when the operation command is received through the HTTP interface; verifying the operation authority of the administrator; the configuration number of the proxy servers is multiple, and each proxy server comprises a proxy module and a server which are bound together. Has the advantages that: the invention solves the problem of weak authority control of the upper layer entrance of the SaltStack, realizes all functions of HTTP interfaces without stateization, is a system supporting a high concurrent command execution channel, and has the advantage of high concurrent command execution efficiency.
Description
Technical Field
The invention belongs to the technical field of information processing, and particularly relates to a system for supporting a high-concurrency command execution channel.
Background
With the continuous expansion of the self-built machine room scale of the internet company, the demand of the server cluster management scheme is larger and larger. The existing server cluster management scheme is basically a C/S architecture and mainly comprises two modes, namely an Agent mode and an Agent mode. The Agent mode refers to: agents need to be deployed at the client to complete the agent work. The communication operation of the server and the client is completely responsible for the agent, and the communication and the authentication are realized mainly based on SSL. The Agentless mode refers to: there is no need to deploy agent at the client. The server communicates with the client through SSH, and the server directly controls the work execution of the client through an SSH protocol.
The Agent mode is realized by the following method: the SaltStack cluster management solution is adopted, and is realized based on the Python language, and a lightweight message queue (zeroMQ) and a Python third-party module (Pyzmq, Pyccrypto, Pyjinjia2, Python-msgpack, PyYAML and the like) are combined to construct. Wherein, the SaltStack is a management tool, which allows an administrator to create a consistent management system for a plurality of operating systems, including a VMware vSphere environment. By deploying a SaltStack environment, execution of commands on thousands of servers can be achieved.
The number of operation entries of the SaltStack is 2, one is a command line mode, and the other is an API interface based on HTTP of a server. And transmitting a command related to the operation of a certain client to the server through the operation inlet, communicating the server with the client through the message queue, transmitting the command to the client, executing the command by the agent of the client, and finally returning the result of the operation command to the server through the message queue to be displayed back to the user. The process is shown in figure 1. In fig. 1, the Salt Master represents: the service end of SaltStack. Salt Minion stands for: client of SaltStack.
The technical scheme mainly has the following defects:
(1) the entrance and the server side binding that the order was issued can't accomplish the stateless extension of server side, therefore, can't adapt to the larger scale cluster.
(2) The entrance authority control is weaker, and the client object operated by the user and the operable function cannot be controlled.
(3) The number of clients that can be managed by a single server is limited, and when the number of clients increases, the performance of the server is affected.
The Agentless mode is specifically realized by the following method: an infrastructure cluster management scheme is adopted, the scheme is also based on python development, SSH communication is used, and a client does not need to deploy Agent. While the anchor is more suitable for management of small server clusters, performance convenience is a major disadvantage for large server clusters. Because of being based on SSH communication, the support of the Windows system is not friendly.
Therefore, this solution mainly has the following disadvantages:
(1) based on SSH communication, when a large number of clients operate simultaneously, the performance of a server is problematic;
(2) the support for windows systems is not good.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a system for supporting a high-concurrency command execution channel, which can effectively solve the problems.
The technical scheme adopted by the invention is as follows:
the invention provides a system for supporting high-concurrency command execution channels, which comprises: the system comprises a receiving module, a load balancing module, a connector module, a publishing and subscribing module, an agent server and a client;
the receiving module is used for receiving an operation command issued by an administrator, wherein the operation command carries specific operation command information, a client ID (identity) to be operated and an administrator ID; then, the receiving module sends the operation command to the load balancing module;
the load balancing module is connected with a plurality of connector modules, each connector module is an independent parallel module, the load balancing module selects the best connector module based on a load balancing strategy, calls an HTTP interface of the selected connector module, and transmits the operation command to the selected connector module through the HTTP interface;
the connector module is used for analyzing the operation command to obtain specific operation command information, a client ID required to be operated and an administrator ID when the operation command is received through the HTTP interface; then, the connector module acquires the corresponding administrator operation authority according to the administrator ID; verifying the operation authority of the administrator, and if the verification fails, returning a notification message for refusing the operation; if the verification is passed, the operation command is sent to the publishing and subscribing module;
the publishing and subscribing module is used for publishing the received operation command from the connector module;
the configuration number of the proxy servers is multiple, and each proxy server comprises a proxy module and a server which are bound together; all the agent modules subscribing to the publish-subscribe module receive the operation command published by the publish-subscribe module; then, the agent module analyzes the received operation command to obtain the ID of the client to be operated and the specific operation command information; then, the agent module further judges whether the client to be operated is in the control authority range of the server bound by the agent module, and if not, ignores the received operation command; if so, the agent module sends the ID of the client to be operated and the specific operation command information to the server bound by the agent module;
the server is used for issuing specific operation command information to the client needing to be operated after receiving the client ID needing to be operated and the specific operation command information sent by the proxy module bound by the server, so that the client needing to be operated executes a specific operation command; after the specific operation command is executed and completed by the client to be operated, the server obtains an operation result, and then the server returns the operation result to the proxy module bound by the server; and the proxy module calls a result collection interface of the connector module again and returns the operation result to the connector module directly through the result collection interface.
Preferably, the connector module verifies the administrator operation authority, specifically:
the connector module firstly obtains a client range which can be operated by an administrator according to the administrator operation authority, then judges whether the ID of the client needing to be operated is in the client range which can be operated by the administrator, and if not, the ID of the client needing to be operated is not verified; if so, the connector module further obtains an operation command range executable by an administrator according to the administrator operation authority, then judges whether the specific operation command information is in the operation command range executable by the administrator, and if not, the verification is not passed; if so, the verification is passed.
Preferably, the publish-subscribe module is a publish-subscribe module of MQ.
Preferably, the agent module is further configured to: the agent module obtains client state monitoring information of each client in the control authority range of the server through the server bound with the agent module, and sends the client state monitoring information to a result collection interface of the connector module.
The system supporting the high-concurrency command execution channel provided by the invention has the following advantages:
the invention solves the problem of weak authority control of the upper layer entrance of the SaltStack, realizes all functions of HTTP interfaces without stateization, is a system supporting a high concurrent command execution channel, and has the advantage of high concurrent command execution efficiency.
Drawings
Fig. 1 is an operation schematic diagram of an Agent mode provided in the prior art.
FIG. 2 is a schematic structural diagram of a system supporting a high-concurrency command execution channel according to the present invention.
Detailed Description
In order to make the technical problems, technical solutions and advantageous effects solved by the present invention more clearly apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention provides a system for supporting high-concurrency command execution channels, and with reference to fig. 2, the system comprises: the system comprises a receiving module, a load balancing module, a connector module, a publishing and subscribing module, an agent server and a client;
the receiving module is used for receiving an operation command issued by an administrator, wherein the operation command carries specific operation command information, a client ID (identity) to be operated and an administrator ID; then, the receiving module sends the operation command to the load balancing module;
the load balancing module is connected with a plurality of connector modules, each connector module is an independent parallel module, the load balancing module selects the best connector module based on a load balancing strategy, calls an HTTP interface of the selected connector module, and transmits the operation command to the selected connector module through the HTTP interface;
the connector module is used for analyzing the operation command to obtain specific operation command information, a client ID required to be operated and an administrator ID when the operation command is received through the HTTP interface; then, the connector module acquires the corresponding administrator operation authority according to the administrator ID; verifying the operation authority of the administrator, and if the verification fails, returning a notification message for refusing the operation; if the verification is passed, the operation command is sent to the publishing and subscribing module; in this step, the connector module verifies the administrator operation authority, specifically: the connector module firstly obtains a client range which can be operated by an administrator according to the operation authority of the administrator, then judges whether the ID of the client which needs to be operated is in the client range which can be operated by the administrator, for example, calls an API of SaltStack to judge whether the ID of the client which needs to be operated is in the client range which can be operated by the administrator, and if not, the verification is not passed; if so, the connector module further obtains an operation command range executable by an administrator according to the administrator operation authority, then judges whether the specific operation command information is in the operation command range executable by the administrator, and if not, the verification is not passed; if so, the verification is passed.
The publishing and subscribing module, specifically an MQ publishing and subscribing module, is configured to publish the received operation command from the connector module;
the configuration number of the proxy servers is multiple, and each proxy server comprises a proxy module and a server which are bound together; all the agent modules subscribing to the publish-subscribe module receive the operation command published by the publish-subscribe module; then, the agent module analyzes the received operation command to obtain the ID of the client to be operated and the specific operation command information; then, the agent module further judges whether the client to be operated is in the control authority range of the server bound by the agent module, and if not, ignores the received operation command; if so, the agent module sends the ID of the client to be operated and the specific operation command information to the server bound by the agent module;
the server is used for issuing specific operation command information to the client needing to be operated after receiving the client ID needing to be operated and the specific operation command information sent by the proxy module bound by the server, so that the client needing to be operated executes a specific operation command; after the client needing to be operated executes the specific operation command, the server obtains an operation result, and then the server returns the operation result to the proxy module bound by the server, specifically, the proxy module obtains the operation result through an API of the SaltStack; and the proxy module calls a result collection interface of the connector module again and returns the operation result to the connector module directly through the result collection interface.
In the present invention, the agent module is further configured to: the agent module obtains client state monitoring information of each client in the control authority range of the server through the server bound with the agent module, and sends the client state monitoring information to a result collection interface of the connector module.
Specifically, the invention is a multi-layer C/S architecture, namely a server-middle agent layer-client architecture. The communication and control from the intermediate proxy layer to the client layer uses the SaltStack, and meets the functional requirements of issuing commands and collecting results of the bottom layer. The invention independently develops a connector module and an agent module, namely: a Connector module and a Proxy module.
The Connector module is used as an entrance for command execution and provides a stateless HTTP interface externally, so that the entrance is separated from the server, and the problem that the entrance is tightly combined with the server and cannot be expanded is solved. In order to improve the concurrent execution capacity, Tornado is adopted as a WEB framework, the non-blocking characteristic of QPS is used for improving the QPS, and the execution permission of the client is limited. Mysql is used as a data persistence scheme for the backend.
The Proxy module calls python-api of the SaltStack server to interact with the Connector module, and mainly achieves the functions of command receiving and execution, result collection and reporting of a survival client (SaltMinion). The Proxy module must be deployed together with the SaltStack server (SaltMaster) since it calls the interface of the SaltStack server (SaltMaster).
The communication between the Connector module and the portal module is as follows: to ensure that all clients (SaltMinion) can receive the issued commands, the publish-subscribe mode of MQ is used.
The system supporting the high-concurrency command execution channel provided by the invention has the following design characteristics:
(1) the Connector module, namely the Connector module, solves the authority control problem of the upper layer, and performs double verification on the operation authority of the administrator, namely, firstly verifying whether the ID of the client needing to be operated is in the range of the client which can be operated by the administrator, and then verifying whether the specific operation command information is in the range of the operation command which can be executed by the administrator, so that the double authority authentication is performed on the range and the function of the administrator. Meanwhile, an HTTP interface inlet of the connector module is separated from a server side, so that the QPS of the inlet is greatly improved. In addition, a plurality of parallel independent stateless connector modules can be flexibly expanded and configured, and the front-end load balancing module selects the optimal connector module based on a load balancing strategy, so that the expansibility and the parallel execution capacity of the system are improved.
(2) The Proxy module, namely the Proxy module, is separated from the upper-layer module, and is linked with the upper-layer Connector module through the MQ, so that the upper layer can be expanded in a stateless manner.
Therefore, the invention solves the problem of weak authority control of the SaltStack upper layer entrance, and realizes all functions of the HTTP interface without stateization.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and improvements can be made without departing from the principle of the present invention, and such modifications and improvements should also be considered within the scope of the present invention.
Claims (4)
1. A system for supporting high concurrency command execution channels, comprising: the system comprises a receiving module, a load balancing module, a connector module, a publishing and subscribing module, an agent server and a client;
the receiving module is used for receiving an operation command issued by an administrator, wherein the operation command carries specific operation command information, a client ID (identity) to be operated and an administrator ID; then, the receiving module sends the operation command to the load balancing module;
the load balancing module is connected with a plurality of connector modules, each connector module is an independent parallel module, the load balancing module selects the best connector module based on a load balancing strategy, calls an HTTP interface of the selected connector module, and transmits the operation command to the selected connector module through the HTTP interface;
the connector module is used for analyzing the operation command to obtain specific operation command information, a client ID required to be operated and an administrator ID when the operation command is received through the HTTP interface; then, the connector module acquires the corresponding administrator operation authority according to the administrator ID; verifying the operation authority of the administrator, and if the verification fails, returning a notification message for refusing the operation; if the verification is passed, the operation command is sent to the publishing and subscribing module; the connector module is used as an entrance for command execution and provides a stateless HTTP interface for the outside;
the publishing and subscribing module is used for publishing the received operation command from the connector module;
the configuration number of the proxy servers is multiple, and each proxy server comprises a proxy module and a server which are bound together; all the agent modules subscribing to the publish-subscribe module receive the operation command published by the publish-subscribe module; then, the agent module analyzes the received operation command to obtain the ID of the client to be operated and the specific operation command information; then, the agent module further judges whether the client to be operated is in the control authority range of the server bound by the agent module, and if not, ignores the received operation command; if so, the agent module sends the ID of the client to be operated and the specific operation command information to the server bound by the agent module; the agent module calls python-api of the SaltStack server to interact with the connector module so as to realize the functions of receiving and executing commands, collecting results and reporting the SaltMinion of the survival client;
the server is used for issuing specific operation command information to the client needing to be operated after receiving the client ID needing to be operated and the specific operation command information sent by the proxy module bound by the server, so that the client needing to be operated executes a specific operation command; after the specific operation command is executed and completed by the client to be operated, the server obtains an operation result, and then the server returns the operation result to the proxy module bound by the server; specifically, the proxy module obtains an operation result through an API of the SaltStack; and the proxy module calls a result collection interface of the connector module again and returns the operation result to the connector module directly through the result collection interface.
2. The system for supporting high concurrency command execution channels according to claim 1, wherein the connector module verifies the administrator operation authority by:
the connector module firstly obtains a client range which can be operated by an administrator according to the administrator operation authority, then judges whether the ID of the client needing to be operated is in the client range which can be operated by the administrator, and if not, the ID of the client needing to be operated is not verified; if so, the connector module further obtains an operation command range executable by an administrator according to the administrator operation authority, then judges whether the specific operation command information is in the operation command range executable by the administrator, and if not, the verification is not passed; if so, the verification is passed.
3. The system supporting high concurrency command execution channels as claimed in claim 1, wherein said publish-subscribe module is a publish-subscribe module of MQ.
4. The system of claim 1, wherein the agent module is further configured to: the agent module obtains client state monitoring information of each client in the control authority range of the server through the server bound with the agent module, and sends the client state monitoring information to a result collection interface of the connector module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711434933.1A CN107896230B (en) | 2017-12-26 | 2017-12-26 | System supporting high-concurrency command execution channel |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711434933.1A CN107896230B (en) | 2017-12-26 | 2017-12-26 | System supporting high-concurrency command execution channel |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107896230A CN107896230A (en) | 2018-04-10 |
| CN107896230B true CN107896230B (en) | 2020-04-10 |
Family
ID=61808686
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711434933.1A Active CN107896230B (en) | 2017-12-26 | 2017-12-26 | System supporting high-concurrency command execution channel |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107896230B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109582315A (en) * | 2018-10-26 | 2019-04-05 | 北京百度网讯科技有限公司 | Service privatization method, apparatus, computer equipment and storage medium |
| CN110427217B (en) * | 2019-07-24 | 2021-07-30 | 上海交通大学 | Content-based publish-subscribe system matching algorithm lightweight parallel method and system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102880475A (en) * | 2012-10-23 | 2013-01-16 | 上海普元信息技术股份有限公司 | Real-time event handling system and method based on cloud computing in computer software system |
| CN107408042A (en) * | 2014-09-26 | 2017-11-28 | 甲骨文国际公司 | For the efficient of Mobile solution and intuitively data are bound |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101741850B (en) * | 2009-12-25 | 2012-05-30 | 北京邮电大学 | Multitask concurrent executive system and method for hybrid network service |
| CN101848236A (en) * | 2010-05-06 | 2010-09-29 | 北京邮电大学 | Real-time data distribution system with distributed network architecture and working method thereof |
| CN103258149A (en) * | 2012-07-27 | 2013-08-21 | 天津中启创科技有限公司 | Online reading system and method based on cloud computing |
| CN104991815B (en) * | 2015-06-19 | 2019-01-18 | 北京奇虎科技有限公司 | The management method and system of Docker container |
| CN105357296B (en) * | 2015-10-30 | 2018-10-23 | 河海大学 | Elastic caching system under a kind of Docker cloud platforms |
| CN105635282B (en) * | 2015-12-30 | 2018-11-16 | 浪潮(北京)电子信息产业有限公司 | A kind of cluster management system and method |
| CN107104956B (en) * | 2017-04-13 | 2019-07-26 | 重庆市环境科学研究院 | A kind of remote sensing distribution method and system based on user's priority |
-
2017
- 2017-12-26 CN CN201711434933.1A patent/CN107896230B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102880475A (en) * | 2012-10-23 | 2013-01-16 | 上海普元信息技术股份有限公司 | Real-time event handling system and method based on cloud computing in computer software system |
| CN107408042A (en) * | 2014-09-26 | 2017-11-28 | 甲骨文国际公司 | For the efficient of Mobile solution and intuitively data are bound |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107896230A (en) | 2018-04-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10031789B2 (en) | Generic distributed processing for multi-agent systems | |
| RU2681351C1 (en) | Method and means of managing an intellectual device for home use to download data, method and device for collecting data unloaded by intelligent device for home use | |
| US11004024B2 (en) | Service and resource orchestration system and method, and apparatus | |
| CN103703724B (en) | Resource payment method | |
| CN104301130B (en) | Monitoring data collection method and system and task the distribution server and acquisition device | |
| CN104021069B (en) | The management method and system of the software performance test based on distributed virtual machine system | |
| CN108011732B (en) | Method, controller and system for configuring service resources | |
| WO2016045403A1 (en) | Runtime environment configuration method, device and system | |
| US20170026306A1 (en) | Systems and methods for autonomous resource discovery, management, and stitching | |
| CN111880902A (en) | Pod creation method, device, equipment and readable storage medium | |
| KR102328938B1 (en) | Management of log data in electronic systems | |
| CN104935672A (en) | High available realizing method and equipment of load balancing service | |
| CN113742031A (en) | Node state information acquisition method and device, electronic equipment and readable storage medium | |
| CN107896230B (en) | System supporting high-concurrency command execution channel | |
| US10206015B2 (en) | System and method for vehicle data communication | |
| CN104919762B (en) | Control method in software defined network, controls equipment and processor | |
| CN110476154B (en) | Proxy server apparatus and method for data collection | |
| CN111625497A (en) | Deployment method, device, equipment and storage medium of distributed file system | |
| US11108673B2 (en) | Extensible, decentralized health checking of cloud service components and capabilities | |
| CN115567383A (en) | Network configuration method, host server, device, and storage medium | |
| US11716382B2 (en) | System and method for replication of storage presentation in application orchestration | |
| CN108089934B (en) | Cluster management method and cluster server | |
| CN108733384A (en) | A kind of operating system remote batch dispositions method and system | |
| CN116634100B (en) | Information interaction method and system based on IPC | |
| CN114697334B (en) | Method and device for executing scheduling task |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |