CN117614963A - Distributed MQTT message server - Google Patents

Distributed MQTT message server Download PDF

Info

Publication number
CN117614963A
CN117614963A CN202311363775.0A CN202311363775A CN117614963A CN 117614963 A CN117614963 A CN 117614963A CN 202311363775 A CN202311363775 A CN 202311363775A CN 117614963 A CN117614963 A CN 117614963A
Authority
CN
China
Prior art keywords
message
mqtt
module
distributed
authentication
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311363775.0A
Other languages
Chinese (zh)
Inventor
杨晴
何程
王骏
卢宇
罗俊杰
郭希恒
黄仁飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yunnan Zhaoxun Technology Co ltd
Original Assignee
Yunnan Zhaoxun Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yunnan Zhaoxun Technology Co ltd filed Critical Yunnan Zhaoxun Technology Co ltd
Priority to CN202311363775.0A priority Critical patent/CN117614963A/en
Publication of CN117614963A publication Critical patent/CN117614963A/en
Pending legal-status Critical Current

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1001Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H04L67/1031Controlling of the operation of servers by a load balancer, e.g. adding or removing servers that serve requests
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00Network architectures or network communication protocols for network security
    • H04L63/08Network architectures or network communication protocols for network security for authentication of entities
    • H04L63/083Network architectures or network communication protocols for network security for authentication of entities using passwords
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00Network architectures or network communication protocols for network security
    • H04L63/10Network architectures or network communication protocols for network security for controlling access to devices or network resources
    • H04L63/101Access control lists [ACL]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1001Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H04L67/1004Server selection for load balancing
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/56Provisioning of proxy services
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/26Special purpose or proprietary protocols or architectures
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/40Network security protocols

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Computing Systems (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer And Data Communications (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention discloses a distributed MQTT message server, which adopts a distributed cluster architecture, the distributed cluster architecture adopts Kubernetes to carry out node discovery and automatic clustering, and the distributed MQTT message server comprises: the system comprises a management console module, a topic monitoring module, an access control module, a sharing subscription module and a reserved message module, wherein the management console module manages and monitors through web pages, the topic monitoring module monitors the message condition of a single topic, the access control module performs safe access control through authentication, authorization and blacklist, the sharing subscription module realizes load balancing among a plurality of subscribers through a subscription group, and the reserved message module stores the latest message for each topic. The distributed MQTT message server realizes high-efficiency data exchange among systems by providing seamless two-way communication among devices, reduces the complexity of system butt joint and the coupling degree among the systems and improves the reliability of the systems.

Description

Distributed MQTT message server
Technical Field
The invention belongs to the technical field of digital communication, and particularly relates to a distributed MQTT message server.
Background
At present, informatization construction is continuously advanced, more and more application programs and systems are introduced into a production link, the application programs and the systems are often different technical platforms and architecture systems and operate independently, data are usually stored in a scattered manner in a conventional business system, data sources exist in different services of a unified system and different services of different systems, data cannot be simply shared, and data security authority limits of complex bottom layers often lead to difficult efficient cooperation among enterprises and departments; in addition, at present, most enterprises have inconsistent digitized construction steps and inconsistent standards, no effective data exchange service is established, the data of each system is like individual islands, the resources are scattered, and the applications cannot cooperate, so that a standard channel for sharing data transmission needs to be established, the data of different systems are connected and transmitted, the scattered data resources are integrated, and the data sharing and communication among different systems are convenient.
Disclosure of Invention
The invention aims to solve the technical problem of providing the distributed MQTT message server which can realize seamless two-way communication between the devices, realize efficient data exchange between systems, reduce the system docking complexity and improve the system reliability.
The technical scheme of the invention is as follows: a distributed MQTT message server, the distributed MQTT message server employing a distributed cluster architecture employing Kubernetes for node discovery and automatic clustering, the distributed MQTT message server comprising: the system comprises a management console module, a topic monitoring module, an access control module, a shared subscription module and a reserved message module, wherein the management console module checks the number of connections, the number of subscriptions and the number of topics accessed into the MQTT service through web pages and displays dynamically updated statistical data when new connection or new subscription topics exist; the topic monitoring module monitors the number of message inflow, the number of message outflow and the rate of the single topic, wherein the rate is the information number per second of the single topic; the access control module comprises: the authentication mechanism adopts password authentication, the authorization mechanism and the blacklist mechanism, and the password authentication comprises the following steps: redis authentication and MySQL authentication, wherein the authorization mechanism comprises two steps: firstly, storing a permission list of an MQTT client into a data source in advance; secondly, inquiring a permission list of the MQTT client from a data source when the MQTT client publishes or subscribes, and allowing or rejecting the current operation according to a matching result; the blacklist mechanism performs access control according to the ID or the user name of the MQTT client: adding a forbidden object through an ID and an IP address user name of the MQTT client; the shared subscription module enables shared subscription by adding a prefix to grouped subscribers before the original subject of the message and forwards the message to subscribers of different groups at the same time; the reservation message module marks the message issued under a single topic as a reservation message, and when the MQTT client subscribes to the topic, if the MQTT server has the reservation message matched with the topic, the reservation message is instantly sent to the MQTT client
Further, the distributed cluster architecture uses a Kubernetes API to perform node discovery and automatic clustering, acquires the node information of the set weights from the Kubernetes API server through the nodes resources and designates the Kubernetes API server for all nodes, and the distributed cluster architecture uses the reverse proxy of ngginx and the load balancing function to realize load balancing for the clusters.
Further, a load balancing policy is configured in an upstream block of the configuration file of the nginnx, and the load balancing policy is a polled load balancing policy.
Further, the password authentication is: redis is used as a first authentication mode, and MySQL is used as a second authentication mode.
Further, the authorization mechanism includes two authorization authentication modes: redis authentication and MySQL authentication, wherein the authorization authentication mode is as follows: redis is used as a first authentication mode, and MySQL is used as a second authentication mode.
Further, only one subscriber of the plurality of subscribers belonging to the same group in the shared subscription module receives the message.
Further, the reserved message page of the management console module displays all reserved messages in the system and is provided with a deletion operation option.
The invention has the beneficial effects that: the distributed MQTT message server provides seamless two-way communication for the devices by adopting the MQTT transmission protocol, realizes high-efficiency data exchange between systems, simplifies the integration of a new system, and ensures that the system is easy to expand; when the architecture is deployed, by adopting the distributed cluster architecture, a large number of clients and messages can be processed while high availability, fault tolerance and expandability are ensured, each node in the cluster can communicate with other nodes, share data such as client connection, subscription, message release and the like, the cluster automatically distributes loads among the nodes and provides high availability when the nodes fail, the cluster provides data redundancy on different nodes through maintaining multiple copies of the data, if one node fails, the data is still available on other nodes in the cluster, and simultaneously new nodes can be added to the cluster along with the increase of services, so that expandability is provided; visual management of the accessed client is realized by using web pages for management and monitoring; by adopting Redis authentication and MySQL authentication as authentication modes of an authentication mechanism, the authentication capability is improved, and meanwhile, the message service access control capability of the access control module is further enhanced by adopting a blacklist system; by setting the shared subscription module and the reserved message module, the latest message state can be immediately obtained while load balancing among a plurality of subscribers is realized, and unexpected time does not need to be waited; in a word, the distributed MQTT message server provided by the invention realizes high-efficiency data exchange among systems by providing seamless two-way communication among devices, reduces complexity of system docking and coupling degree among systems and improves reliability of the systems.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described, and it will be obvious to those skilled in the art that other drawings can be obtained according to these drawings without inventive effort.
FIG. 1 is a schematic diagram of a distributed cluster publish-subscribe process for a distributed MQTT message server of the present invention.
Fig. 2 is a schematic diagram of a client access authentication procedure of a distributed MQTT message server according to the present invention.
Fig. 3 is a schematic diagram of a client-side publish-subscribe authorization check flow of a distributed MQTT message server of the present invention.
Detailed Description
As shown in fig. 1-3, a distributed MQTT message server, where the distributed MQTT message server employs a distributed cluster architecture that employs Kubernetes for node discovery and automatic clustering, the distributed MQTT message server includes: the system comprises a management console module, a topic monitoring module, an access control module, a shared subscription module and a reserved message module, wherein the management console module checks the number of connections, the number of subscriptions and the number of topics accessed into the MQTT service through web pages and displays dynamically updated statistical data (statistical data of connections, subscriptions and topics) when new connection or new subscription topics exist; the topic monitoring module monitors the number of message inflow, the number of message outflow and the rate of the single topic, wherein the rate is the information number per second of the single topic; the access control module comprises: the authentication mechanism adopts password authentication, the authorization mechanism and the blacklist mechanism, and the password authentication comprises the following steps: redis authentication and MySQL authentication, wherein the authorization mechanism comprises two steps: firstly, storing a permission list of an MQTT client into a data source in advance; secondly, inquiring a permission list of the MQTT client from a data source (cache, database and the like) when the MQTT client publishes or subscribes, and allowing or rejecting the current operation according to a matching result; the blacklist mechanism performs access control according to the ID or the user name of the MQTT client: adding a forbidden object through an ID and an IP address user name of the MQTT client; the shared subscription module enables shared subscriptions (group names can be arbitrary character strings) by adding prefixes to groups of subscribers before the original subject of the message and forwards the message to subscribers of different groups at the same time; the reservation message module marks the message issued under a single topic as a reservation message (namely, sets the returned mark as true), and when the MQTT client subscribes to the topic, if the MQTT server side has the reservation message matched with the topic, the reservation message is immediately sent to the MQTT client.
Preferably, the distributed cluster architecture uses a Kubernetes API to perform node discovery and automatic clustering, acquires the node information of the aggregation from a Kubernetes API server through an nodes resource and designates the Kubernetes API server for all nodes, and the distributed cluster architecture uses a reverse proxy and a load balancing function of Nginx to implement load balancing for the cluster (after the Nginx is installed and deployed, a reverse proxy configuration is added in a configuration file server block of the Nginx according to the IP and the port of the back-end MQTT server, and the MQTT connection request from the client is proxied and forwarded to the back-end MQTT server).
Preferably, a load balancing policy is configured in an upstream block of the configuration file of the nginnx, and the load balancing policy is a polled load balancing policy.
Preferably, the password authentication is: redis is used as a first authentication mode, mySQL is used as a second authentication mode (when a client is accessed, password information configured in advance is obtained from Redis according to authentication information sent by the client, password hash is carried out by using SHA256 algorithm through passwords uploaded by the client, and the like, then the password hash is matched with the obtained password, and the matching is passed through the authentication, so that connection is allowed successfully, and if Redis authentication is wrong or is not connected, the authentication is carried out by using MySQL database).
Preferably, the authorization mechanism includes two authorization authentication modes: redis authentication and MySQL authentication, wherein the authorization authentication mode is as follows: redis is used as a first authentication mode, and MySQL is used as a second authentication mode.
Preferably, only one subscriber of the several subscribers belonging to the same group in the shared subscription module receives the message, and load balancing among the subscribers is achieved (e.g., if subscribers s1, s2 are members of group g1, subscribers s3 and s4 are members of group g2, and all subscribers subscribe to the original topic t1, when publishing the message msg1 to the original topic t1, one subscriber of s1, s2 will receive msg1, and one subscriber of s3 and s4 will receive msg 1).
Preferably, the reserved message page of the management console module displays all reserved messages (including theme, qoS, release time and client ID) in the system and is provided with a delete operation option.
The foregoing description of the embodiments of the invention has been presented in conjunction with the drawings and is not intended to limit the scope of the invention. It will be understood by those skilled in the art that non-inventive modifications or equivalent substitutions for the technical solution of the present invention are intended to be encompassed within the scope of the claims of the present invention without departing from the scope of the technical solution.

Claims (7)

1. A distributed MQTT message server, the distributed MQTT message server employing a distributed cluster architecture employing Kubernetes for node discovery and automatic clustering, the distributed MQTT message server comprising: the system comprises a management console module, a topic monitoring module, an access control module, a shared subscription module and a reserved message module, wherein the management console module checks the number of connections, the number of subscriptions and the number of topics accessed into the MQTT service through web pages and displays dynamically updated statistical data when new connection or new subscription topics exist; the topic monitoring module monitors the number of message inflow, the number of message outflow and the rate of the single topic, wherein the rate is the information number per second of the single topic; the access control module comprises: the authentication mechanism adopts password authentication, the authorization mechanism and the blacklist mechanism, and the password authentication comprises the following steps: redis authentication and MySQL authentication, wherein the authorization mechanism comprises two steps: firstly, storing a permission list of an MQTT client into a data source in advance; secondly, inquiring a permission list of the MQTT client from a data source when the MQTT client publishes or subscribes, and allowing or rejecting the current operation according to a matching result; the blacklist mechanism performs access control according to the ID or the user name of the MQTT client: adding a forbidden object through an ID and an IP address user name of the MQTT client; the shared subscription module enables shared subscription by adding a prefix to grouped subscribers before the original subject of the message and forwards the message to subscribers of different groups at the same time; the message issuing under a single theme is marked as a reserved message by the reserved message module, and when the MQTT client subscribes to the theme, if the reserved message matched with the theme exists at the MQTT server, the reserved message is immediately sent to the MQTT client.
2. The distributed MQTT message server of claim 1, wherein the distributed cluster architecture uses Kubernetes API for node discovery and automatic clustering, wherein the distributed cluster architecture obtains the aggregation node information from the Kubernetes API server through the encodings resources and designates the Kubernetes API server for all nodes, and wherein the distributed cluster architecture uses the reverse proxy of nglnx and the load balancing function to implement load balancing for the clusters.
3. The distributed MQTT message server of claim 2, wherein the load balancing policy is configured in an upstream block of the nginnx's configuration file, the load balancing policy being a polled load balancing policy.
4. A distributed MQTT message server in accordance with claim 1, wherein the cryptographic authentication is: redis is used as a first authentication mode, and MySQL is used as a second authentication mode.
5. A distributed MQTT message server in accordance with claim 1, wherein the authorization mechanism comprises two authorization authentication means: redis authentication and MySQL authentication, wherein the authorization authentication mode is as follows: redis is used as a first authentication mode, and MySQL is used as a second authentication mode.
6. A distributed MQTT message server in accordance with claim 1, wherein only one of the subscribers belonging to the same group in the shared subscription module receives messages.
7. A distributed MQTT message server in accordance with claim 1, wherein the management console module retains all retained messages in the message page presentation system and is provided with a delete operation option.
CN202311363775.0A 2023-10-20 2023-10-20 Distributed MQTT message server Pending CN117614963A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311363775.0A CN117614963A (en) 2023-10-20 2023-10-20 Distributed MQTT message server

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311363775.0A CN117614963A (en) 2023-10-20 2023-10-20 Distributed MQTT message server

Publications (1)

Publication Number Publication Date
CN117614963A true CN117614963A (en) 2024-02-27

Family

ID=89946838

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311363775.0A Pending CN117614963A (en) 2023-10-20 2023-10-20 Distributed MQTT message server

Country Status (1)

Country Link
CN (1) CN117614963A (en)

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050021622A1 (en) * 2002-11-26 2005-01-27 William Cullen Dynamic subscription and message routing on a topic between publishing nodes and subscribing nodes
CN106888254A (en) * 2017-01-20 2017-06-23 华南理工大学 A kind of exchange method between container cloud framework based on Kubernetes and its each module
US20180191663A1 (en) * 2017-01-02 2018-07-05 International Business Machines Corporation Cluster assisted MQTT client coverage for fat-pipe cloud applications
US20180191813A1 (en) * 2017-01-02 2018-07-05 International Business Machines Corporation MQTT cluster shared subscription hub for fat-pipe cloud applications
US20180262450A1 (en) * 2017-03-09 2018-09-13 Citrix Systems, Inc. Increasing capacity in a topic-subscription messaging system
US20190068528A1 (en) * 2017-08-30 2019-02-28 Facebook, Inc. Techniques for efficient messaging client communication
CN109587228A (en) * 2018-11-23 2019-04-05 济南浪潮高新科技投资发展有限公司 A kind of publicly-owned agreement platform of internet of things and equipment cut-in method
CN110336736A (en) * 2019-05-27 2019-10-15 四川长虹电器股份有限公司 The shared method subscribed to is realized based on MQTT server cluster
WO2022095435A1 (en) * 2020-11-06 2022-05-12 深圳市爱云信息科技有限公司 Multilevel linkage management platform for aiot device accessing paas and saas
CN114903474A (en) * 2022-05-25 2022-08-16 上海大学 Wearable sweat data monitoring system and sweat data monitoring method
CN116088901A (en) * 2022-12-29 2023-05-09 深圳市有方科技股份有限公司 Firmware upgrading method and device, electronic equipment and computer storage medium

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050021622A1 (en) * 2002-11-26 2005-01-27 William Cullen Dynamic subscription and message routing on a topic between publishing nodes and subscribing nodes
US20180191663A1 (en) * 2017-01-02 2018-07-05 International Business Machines Corporation Cluster assisted MQTT client coverage for fat-pipe cloud applications
US20180191813A1 (en) * 2017-01-02 2018-07-05 International Business Machines Corporation MQTT cluster shared subscription hub for fat-pipe cloud applications
CN106888254A (en) * 2017-01-20 2017-06-23 华南理工大学 A kind of exchange method between container cloud framework based on Kubernetes and its each module
US20180262450A1 (en) * 2017-03-09 2018-09-13 Citrix Systems, Inc. Increasing capacity in a topic-subscription messaging system
US20190068528A1 (en) * 2017-08-30 2019-02-28 Facebook, Inc. Techniques for efficient messaging client communication
CN109587228A (en) * 2018-11-23 2019-04-05 济南浪潮高新科技投资发展有限公司 A kind of publicly-owned agreement platform of internet of things and equipment cut-in method
CN110336736A (en) * 2019-05-27 2019-10-15 四川长虹电器股份有限公司 The shared method subscribed to is realized based on MQTT server cluster
WO2022095435A1 (en) * 2020-11-06 2022-05-12 深圳市爱云信息科技有限公司 Multilevel linkage management platform for aiot device accessing paas and saas
CN114903474A (en) * 2022-05-25 2022-08-16 上海大学 Wearable sweat data monitoring system and sweat data monitoring method
CN116088901A (en) * 2022-12-29 2023-05-09 深圳市有方科技股份有限公司 Firmware upgrading method and device, electronic equipment and computer storage medium

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
王风雷;秦会斌;崔佳冬;: "共享单车服务器及客户端设计与实现", 物联网技术, no. 02, 20 February 2018 (2018-02-20) *

Similar Documents

Publication Publication Date Title
US11184299B2 (en) Hierarchical publish and subscribe system
JP4444518B2 (en) A distributed system that establishes intelligent sessions between anonymous users over various networks
US9391786B1 (en) Systems and methods for setting up a session in a collaborative communication system
US8612527B2 (en) Automatic notification system and process
US20030041101A1 (en) Presence watcher proxy
US20080034078A1 (en) Presence information management system, presence server device, gateway device and client device
US20080208959A1 (en) Hanging request system and method for client/server communication
CN110427266B (en) Data redundancy architecture based on MQTT service
US10348714B1 (en) Systems and methods for setting up a collaborative communication system
US20070011243A1 (en) System and method for storing and transmitting a file data using internet messenger
CN117614963A (en) Distributed MQTT message server
KR20080058409A (en) Endpoint transparent independent messaging scheme system and method
CN113364869A (en) Block chain message transmission method, equipment and storage medium
US20130262583A1 (en) System and method for namespace resolution in peer to peer networks
JP2008537437A (en) Content delivery based on user affinity using a connected endpoint proxy
KR20220063503A (en) Layered IoT service system using message exchange method
CN116962547A (en) MQ-based dynamic data gateway communication method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination