CN116132530A - Method for realizing MQTT Broker server by applying Raft algorithm based on Netty framework - Google Patents

Abstract

The invention provides a method for realizing an MQTT Broker server by applying a Raft algorithm based on a Netty framework. The method for realizing the MQTT Broker server by applying the Raft algorithm based on the Netty framework comprises the following steps: s1, netty is an asynchronous event-driven network application framework, and supports the rapid development of maintainable high-performance protocol-oriented servers and clients; s2, designing and realizing the Netty architecture according to a Reactor mode, wherein the Reactor concurrency model of the Netty is as follows: netty implements and extends the Reactor model. The method for realizing the MQTT Broker server based on the Netty framework by applying the Raft algorithm can avoid the negative influence of commercial authorization, continuously iterate and upgrade, provide data support for the operation and decision of the Broker, ensure the forced consistency of the data in the cluster, ensure that the data can be normally provided after the node is restored even if the node fails, and solve the defects in the prior art.

Description

Method for realizing MQTT Broker server by applying Raft algorithm based on Netty framework

Technical Field

The invention belongs to the technical field of the Internet of things, and particularly relates to a method for realizing an MQTT Broker server by applying a Raft algorithm based on a Netty framework.

Background

Message queue telemetry is a message protocol based on the Publish/Subscribe (ISO/IEC PRF 20922) paradigm under the ISO standard, which can be considered as a "bridge for data transfer" that works on the TCP/IP protocol family and is a Publish/Subscribe message protocol designed for low hardware capability remote devices and poor network conditions, for which a message middleware is required to address the current heavy data transfer protocol.

The MQTT protocol defines two network entities: message broker (message broker) and client (client). The message agent is used for receiving the message from the client and forwarding the message to the target client. The MQTT client may be any device running a MQTT library and connected to the message broker via a network, such as a microcontroller or a large server.

The transmission of information is managed by a theme (topic). When the publisher has data to be distributed, it sends a control message carrying the data to the connected message broker. The agent will distribute this data to clients subscribing to this topic. The publisher does not need to know the subscriber's data and specific location; also, subscribers do not need to configure relevant information of publishers. If a message broker receives a message on a topic and the topic does not have any subscriptions, the broker discards unless the publisher marks it as a reserve message.

Whereas the existing MQTT proxy:

1).Mosquitto

the mosquito is deduced by Eclipse open source community, an open source message proxy software for realizing a message push protocol MQTT v3.1 is provided, a lightweight and published/subscribed message push mode is provided, short message communication between devices is facilitated, C language development is used, and the method is suitable for lightweight MQTT access application situations;

2).EMQX

EMQX is a distributed Internet of things MQTT server which is promoted by Hangzhou Yingyun technology limited, provides an open source version and a business version, is developed by using Erlang language, and supports the linking of mass Internet of things equipment.

However, the prior art has the following disadvantages:

1).Mosquitto

(1) Not supporting security authentication;

(2) Not supporting trunking mode;

(3) Not supporting data persistence;

(4) No service management interface is opened;

2).EMQX

(1) The open source version has limited functions, and a pseudo cluster mode is needed to be realized through a front load;

(2) The business version has complete functions, charges according to the flow or the equipment access number, and can be privately deployed after authorization;

(3) Only commercial version supports data persistence, and when the open source version is down, the data is lost;

(4) And the Erlang is used for development, the expansion modes are more, and the expansion implementation threshold is high.

Therefore, it is necessary to provide a new method for implementing the MQTT Broker server based on the Netty framework by applying the Raft algorithm to solve the above technical problems.

Disclosure of Invention

The invention solves the technical problem of providing a method which can avoid the negative influence of commercial authorization, continuously iterate and upgrade, can provide data support for the operation and decision of a Broker, can ensure the forced consistency of data in a cluster, can normally provide data even if a node fails and after the node is restored, and can solve the defects of the prior art by applying a Raft algorithm based on a Netty framework to realize an MQTT Broker server.

In order to solve the technical problems, the method for realizing the MQTT Broker server based on the Netty framework by applying the Raft algorithm comprises the following steps: s1.Netty

The Netty is an asynchronous event-driven network application framework, and supports the rapid development of maintainable high-performance protocol-oriented servers and clients;

s2.Reactor model

The Netty architecture is designed and implemented according to a Reactor mode, and a Reactor concurrency model of the Netty is as follows:

netty implements and expands the Reactor model, as follows:

1) The acceptors main thread pool is an independent NIO thread pool, which plays the role of acceptors in network links in the form of NIO, and is also called acceptors reactor;

2) After the program is started, randomly selecting a thread from an accepter main thread pool as an accepter thread, wherein the accepter thread is used for binding ports and receiving client connection;

3) The accepter thread receives the connection request and gives the request to other threads of the accepter reactor to process login, handshake and security authentication of the client;

4) After the service layer link is formally established, the request is transferred to a certain thread of the IO sub-thread pool, and IO read-write and coding operations are processed by the thread;

5) Finally, according to the difference of request types, the request is transferred to a specific type of workbench and specific business processing is carried out in a Netty concurrency model, and the concurrency access of a client, the reading and writing of a bottom Socket and the processing of the business request are all multithreaded asynchronous operations, so that Netty endows an application program with a core capability, can respond to network events generated at any time point in any sequence, can continuously multiplex the thread resources of a computer in determined resources, and is suitable for higher throughput and expandability requirements;

S3.Raft

the Raft cluster is composed of a plurality of nodes, and each node has three states: leader, follower and Candidate, the Raft will elect the Leader first, and only one node is the Leader, the other nodes are the Follower, the Leader node processes all client requests, duplicate Log to the Follower node, if the Leader fails, after the election count-down expires, the heartbeat detection of the Leader is still abnormal, the Follower will elect a new Leader again;

S4, realizing an MQTT Broker based on Netty and Raft:

the MQTT Broker is a server directly receiving the network connection of the concurrent clients, and uses a Netty processing network communication part, so that more client connections and more services can be processed by using server resources to the maximum extent, and the Netty self-contained MQTT codec module can be directly used for finishing the codec of MQTT protocol communication;

s5.broker service maintains data consistency through the Raft cluster:

the MQTT Broker service endows data replication characteristics among the MQTT brokers by realizing a Raft algorithm, and forms a cluster based on data synchronization with strong consistency, so that the risk of single-point failure of the system is reduced;

s6, endowing the system with the following characteristics by a Raft algorithm:

1) Persistent session: when the Session identifier 'Clean Session' is set to 0, a persistent Session is created, and when the client is disconnected, the Session still keeps and stores offline information until the Session is logged off overtime, after the persistent Session information and offline MQTT information are stored locally, the Session information and the offline MQTT information are copied to other broaders of the cluster through a Raft algorithm, and after the client is online again, all offline information can be obtained in time, so that waiting is reduced;

2) Subscription data: after the client side subscribes successfully, the subscription relation data is copied to other broaders, broadcasting and issuing instructions are needed, or the client side is replaced with the broader access, so that all the current subscription topic data of the client side can be obtained in time, and the subscription relation data is deleted in the cluster after being deleted locally when the subscription is canceled;

3) Message retransmission for message quality of service (QoS): the MQTT protocol specifies that both the server and the client as communication parties should meet the requirements of their service quality for the messages sent to the opposite end by themselves, i.e. after the MQTT message is sent, the opposite end does not receive a response within a specified time, then the client retransmits the message, meanwhile, if the client sends a reconnection under the condition of maintaining a session, the Broker automatically retransmits the response message to the reconnecting client to ensure the correctness of the QoS flow, and for the above scenario, the message sent by the client but not acknowledged is stored locally and then copied and synchronized to other Broker, whether the message is retried or the publisher reconnects, the message can still be found in other Broker, and the corresponding message is maintained according to the semantic requirement of QoS.

As a further aspect of the present invention, the MQTT Broker service includes the following:

(1) The Broker service consists of a network service layer, an MQTT protocol service layer, a business service layer, a storage module and a bridging module, wherein the network communication and business processing are separated, the storage module consists of a Raft node and a local storage, and the storage module is defined with a plurality of memories;

(2) The network service layer processes the connection of the client and forwards the reported data of the client to the protocol layer for decoding, and meanwhile, the service data coded by the protocol layer is transmitted to the corresponding client;

(3) The protocol layer applies the MQTT protocol coding and decoding module realized by Netty to process the MQTT protocol message required by communication between the client and the Broker, define the processing flow of the corresponding message and realize the triggering processing of protocol service, such as login, authentication and the like and message distribution;

(4) The business service consists of a message issuing module and a data distributing module;

(5) The Raft service realizes network communication, provides a communication function among nodes for the Raft cluster, and supports the operation of the Raft cluster: election, log replication, security check and heartbeat probe activity;

(6) The Raft node consists of a memory for realizing a Raft protocol and a Raft communication service, process data and result data in a service link can be submitted to a Raft Leader according to the need, the data is stored as a Raft Log into a local specific storage engine, and data replication among nodes in a cluster is completed by utilizing Log replication and strong consistency check of the Raft algorithm;

(7) Local storage is used to store data including, but not limited to: client session record, unacknowledged messages (QoS 1, qoS 2), metric values of metrics and configuration data required for program operation;

(8) Stateless bridging services expose REST APIs outward, providing business function management operations including, but not limited to, the following: acquiring the Broker state information, publishing the message and checking the client list.

As a further scheme of the present invention, the message issuing is responsible for capturing user operations, after abstracting into an MQTT message body, the protocol layer processing is shifted to, the data distributing is responsible for capturing the message body decoded by the protocol layer, and the data distributing is responsible for processing including but not limited to: message persistence, inter-device communication, message replay, shared subscription, etc., and distribution of the reported data is accomplished by interfacing the persistence middleware with a specific service interface.

As a further scheme of the present invention, the data replication process specifically includes:

(1) The Leader is in charge of receiving the request of the client, the Leader stores the request as a Raft log entry into a local storage, and after the storage is successful, the Leader initiates a copying action to other servers in parallel so as to copy the log entry;

(2) When the log is copied from the Leader to most of the follow, the Leader opens the log entry to its state machine and returns an execution result, i.e., a global commit copy result, to the client;

(3) The data Block (Block) is the minimum unit of data management and movement of the Raft cluster, and corresponds to an actual Block data interval in the local storage, each Block contains a plurality of data, a plurality of copies exist in the Block, each copy is located in different nodes, and the plurality of blocks on different nodes form an abstract Raft cluster and are copies;

(4) Based on the log consistency check of the Raft algorithm, some folows may not successfully complete log replication, and the Leader may retry replication indefinitely until all folows eventually store all log entries.

As a further scheme of the invention, the MQTT is a single example in operation, a TCP endpoint is exposed to the outside, and at least 3 examples are required to be operated to form a Raft cluster according to the Raft characteristic, and the method comprises the following steps:

(1) The Netty service only can acquire the access of a client for realizing the MQTT protocol, only processes the request meeting the MQTT protocol specification, exposes fixed { ip: port } when the Broker service is started, and exposes a communication endpoint { ip: port } of the Raft node;

(2) The client requests to access a Broker service by using a user name and a password according to a predefined configuration, communication long links are established at two ends after the request is approved, and in the message 'CONNECT' monitoring process, the network service stores the links into a client session list and a client link list by taking a ClientID as a key and updates a routing table;

(3) When the Broker is started, the Raft node is started by a node list in configuration, and a cluster is initially created, and when all nodes are Follower at the initial time, a new Leader election is initiated;

(4) The Netty service begins processing MQTT communication messages with the client, the message types including, but not limited to: subscribing the theme, sending the message and heartbeat probe activity, wherein the client subscribes the theme and sending the message to be limited by the authorization of a manager, and editing and maintaining in a management background are needed;

(5) If the client allows subscribing the theme, after establishing a long link, a SUBSCRIBE message can be initiated, and before returning the SUBACK message, the Broker service stores the relation between the client and a plurality of themes in a local storage by decoding in a form of { ClientID, description }, and copies the relation to other nodes by using a Raft algorithm;

(6) If the client side allows to send the message, after establishing the long link, the client side can initiate a PUBLISH message, and after receiving the message, the Broker processes the information according to the message type in a message processor, and if the information is general, the information is directly forwarded; if the information is reserved, storing the local storage of the reserved information, waiting for retransmission to the client, if the information is QoS 1 and QoS 2, sequentially initiating a plurality of message inter-transmission flows according to the MQTT specification requirements, storing the unacknowledged information in the corresponding local storage, and deleting the unacknowledged information after receiving the PUBCOMP message;

(7) The message issued by the client or the bridging service is decoded by a protocol layer message processor of the Broker service, the message decoded by the processor is transferred to a message distribution module, the destination of the message is determined according to the data characteristics of the message MessageType, messageHeader, qoS and the like (as shown in figure 4), the message can be forwarded after the range of the client is reversely determined through the subscription of the theme, the message is encoded by adopting different encoders according to the data characteristics of the message MessageType, messageHeader, qoS and the like, different message sending flow processes are started, and if delayed sending is required, the message is stored in a delay memory; if QoS2 semantics are required to be achieved, storing the information into a message memory to be confirmed, and matching data processing in a subsequent QoS2 flow node, and if the information is required to be durable, storing the information into a corresponding storage engine through bridging service according to a current durable strategy;

(8) The manager can directly send a request through the operation of the web platform, the web platform is connected with the bridging service, and the external application completes communication of the Broker service through adapting the bridging service, such as adjusting setting, obtaining Broker state information, publishing messages, subscribing and unsubscribing agents, disconnecting designated clients and checking a client list.

As a further scheme of the invention, the Raft algorithm decomposes the consistency problem into three sub-problems, namely election, log replication and security, which are specifically as follows:

(1) Election: when a Leader is down or a cluster is initially created, the Followers can elect a new Leader;

(2) Log replication: the Leader processes the requests of all clients uniformly, and after receiving the data from the clients, the Leader copies the data to other nodes in the cluster in the form of log entries and forces the logs of the other nodes to be consistent with the Leader;

(3) Safety: each node has an explicit log stored in the state machine, the log index position of the node cannot be inconsistent with the log index of the Leader, and the Leader can force the index position of the node to be consistent with the Leader.

As a further scheme of the invention, the Raft algorithm increases two limits for enhancing cluster availability and ensuring strong data consistency: the method is characterized by comprising the following steps of selecting limit and log consistency check:

(1) Election limit: the node in the Candida state can access most nodes after the election is initiated, the own log is compared with the log of each node, the index value and the tenn number of the last log in the two logs are compared to determine which node is newer in log comparison, if the tenn numbers of the last entries of the two logs are different, the log with the large tenn number is updated, if the tenn numbers of the last entries of the two logs are the same, the longer one of the logs is updated, and only the node with the largest log index value and the latest tenn number can win the election;

(2) Log consistency check: under normal conditions, the logs of the Leader and the Follower will keep consistent, under the production running environment, various abnormal conditions will occur in both the Leader and the Follower, the logs of each node will not be consistent, the Raft algorithm agrees that the Leader must save the nextIndex of each Follower, the Leader just selected will initialize the nextIndex to be the Log index value +1 of the current Leader, if the Log of one Follower and the Leader are inconsistent, the consistency check will fail when the Leader synchronizes the Log with the next heartbeat, the Leader will reduce the value of the nextIndex when the nextIndex does not match with the currentIndex of the target Follower, and retry, the nextIndex will eventually coincide with the currentIndex of the Follower at a certain position, because of the election limit, the Leader will save the latest and the most complete load data of the cluster, and the Leader will synchronize the newIndex to the Follower after determining the nextIndex.

Compared with the related art, the method for realizing the MQTT Broker server based on the Netty framework by applying the Raft algorithm has the following beneficial effects:

1. the invention is realized by using Java industrial computer language design, has high universality and reduces the secondary development threshold;

2. The invention can avoid the negative influence of commercial authorization and continuously iterate and upgrade;

3. according to the invention, network access is realized by using Netty, and a single machine 4CoreCPU and 16GRAM can support 50 ten thousand links;

4. the invention can be in an application service form besides a classical Client form by accessing the MQTT Broker role, and is not forced to be accessed as a Client;

5. the system is associated with external services through bridging services, and can provide data support for the operation and decision of a Broker through docking an existing configuration system or a custom configuration system;

6. the invention realizes the lightweight Broker cluster by using the Raft algorithm, can ensure the forced consistency of the data in the cluster, and can normally provide the data after the node is recovered even if the node fails;

7. the invention can theoretically meet the characteristics required by the running of the distributed cluster by continuously upgrading and expanding bridging services, protocol layers and Raft nodes.

Drawings

The present invention is further described below with reference to the accompanying drawings for the convenience of understanding by those skilled in the art.

FIG. 1 is a schematic diagram of a Netty concurrency model in the present invention;

FIG. 2 is a schematic diagram of a Raft node state transition in accordance with the present invention;

FIG. 3 is a schematic diagram of the overall architecture of the MQTT Broker service according to the present invention;

FIG. 4 is a diagram of a message body structure according to the present invention;

FIG. 5 is a schematic diagram of a Raft cluster configuration according to the present invention.

Detailed Description

Referring to fig. 1, fig. 2, fig. 3, fig. 4, and fig. 5 in combination, fig. 1 is a schematic diagram of a Netty concurrency model in the present invention; FIG. 2 is a schematic diagram of a Raft node state transition in accordance with the present invention; FIG. 3 is a schematic diagram of the overall architecture of the MQTT Broker service according to the present invention; FIG. 4 is a diagram of a message body structure according to the present invention; FIG. 5 is a schematic diagram of a Raft cluster configuration according to the present invention. The method for realizing the MQTT Broker server by applying the Raft algorithm based on the Netty framework comprises the following steps: s1.Netty

The Netty is an asynchronous event-driven network application framework, and supports the rapid development of maintainable high-performance protocol-oriented servers and clients;

s2.Reactor model

The Netty architecture is designed and implemented according to a Reactor mode, and a Reactor concurrency model of the Netty is as follows:

as shown in fig. 1, the Netty implements and extends the Reactor model, specifically as follows:

1) The acceptors main thread pool is an independent NIO thread pool, which plays the role of acceptors in network links in the form of NIO, and is also called acceptors reactor;

2) After the program is started, randomly selecting a thread from an accepter main thread pool as an accepter thread, wherein the accepter thread is used for binding ports and receiving client connection;

3) The accepter thread receives the connection request and gives the request to other threads of the accepter reactor to process login, handshake and security authentication of the client;

4) After the service layer link is formally established, the request is transferred to a certain thread of the IO sub-thread pool, and IO read-write and coding operations are processed by the thread;

5) Finally, according to the difference of request types, the request is transferred to a specific type of workbench and specific business processing is carried out in a Netty concurrency model, and the concurrency access of a client, the reading and writing of a bottom Socket and the processing of the business request are all multithreaded asynchronous operations, so that Netty endows an application program with a core capability, can respond to network events generated at any time point in any sequence, can continuously multiplex the thread resources of a computer in determined resources, and is suitable for higher throughput and expandability requirements;

S3.Raft

the Raft cluster is composed of a plurality of nodes, and each node has three states: leader, follower and Candidate (as shown in fig. 2), the Raft will elect a Leader first, only one node is a Leader, the other nodes are all clients, the Leader processes all client requests, the replied Log is copied to the Leader node, if the Leader fails, after the election countdown expires, the heartbeat detection of the Leader is still abnormal, and the Leader re-elects a new Leader;

S4, realizing an MQTT Broker based on Netty and Raft:

as shown in fig. 3, the MQTT Broker is a server directly receiving network connection of concurrent clients, and uses a Netty processing network communication part, so that more client connections can be processed and more services can be processed to the maximum extent by using server resources, and the Netty codec module with the MQTT can be directly used to complete codec of MQTT protocol communication;

s5.broker service maintains data consistency through the Raft cluster:

as shown in fig. 5, the MQTT Broker service gives the data replication characteristics among MQTT broaders by implementing a Raft algorithm, and forms clusters based on data synchronization with strong consistency, so that the risk of single-point failure of the system is reduced;

s6, endowing the system with the following characteristics by a Raft algorithm:

1) Persistent session: when the Session identifier 'Clean Session' is set to 0, a persistent Session is created, and when the client is disconnected, the Session still keeps and stores offline information until the Session is logged off overtime, after the persistent Session information and offline MQTT information are stored locally, the Session information and the offline MQTT information are copied to other broaders of the cluster through a Raft algorithm, and after the client is online again, all offline information can be obtained in time, so that waiting is reduced;

2) Subscription data: after the client side subscribes successfully, the subscription relation data is copied to other broaders, broadcasting and issuing instructions are needed, or the client side is replaced with the broader access, so that all the current subscription topic data of the client side can be obtained in time, and the subscription relation data is deleted in the cluster after being deleted locally when the subscription is canceled;

3) Message retransmission for message quality of service (QoS): the MQTT protocol specifies that both the server and the client as communication parties should meet the requirements of their service quality for the messages sent to the opposite end by themselves, i.e. after the MQTT message is sent, the opposite end does not receive a response within a specified time, then the client retransmits the message, meanwhile, if the client sends a reconnection under the condition of maintaining a session, the Broker automatically retransmits the response message to the reconnecting client to ensure the correctness of the QoS flow, and for the above scenario, the message sent by the client but not acknowledged is stored locally and then copied and synchronized to other Broker, whether the message is retried or the publisher reconnects, the message can still be found in other Broker, and the corresponding message is maintained according to the semantic requirement of QoS.

The MQTT Broker service comprises the following steps:

(1) The Broker service consists of a network service layer, an MQTT protocol service layer, a business service layer, a storage module and a bridging module, wherein the network communication and business processing are separated, the storage module consists of a Raft node and a local storage, and the storage module is defined with a plurality of memories;

(2) The network service layer processes the connection of the client and forwards the reported data of the client to the protocol layer for decoding, and meanwhile, the service data coded by the protocol layer is transmitted to the corresponding client;

(3) The protocol layer applies the MQTT protocol coding and decoding module realized by Netty to process the MQTT protocol message required by communication between the client and the Broker, define the processing flow of the corresponding message and realize the triggering processing of protocol service, such as login, authentication and the like and message distribution;

(4) The business service consists of a message issuing module and a data distributing module;

(5) The Raft service realizes network communication, provides a communication function among nodes for the Raft cluster, and supports the operation of the Raft cluster: election, log replication, security check and heartbeat probe activity;

(6) The Raft node consists of a memory for realizing a Raft protocol and a Raft communication service, process data and result data in a service link can be submitted to a Raft Leader according to the need, the data is stored as a Raft Log into a local specific storage engine, and data replication among nodes in a cluster is completed by utilizing Log replication and strong consistency check of the Raft algorithm;

(7) Local storage is used to store data including, but not limited to: client session record, unacknowledged messages (QoS 1, qoS 2), metric values of metrics and configuration data required for program operation;

(8) Stateless bridging services expose REST APIs outward, providing business function management operations including, but not limited to, the following: acquiring the Broker state information, publishing the message and checking the client list.

The message issuing is responsible for capturing user operation, after being abstracted into an MQTT message body, the message is transferred to the protocol layer for processing, the data distributing is responsible for capturing the decoded message body of the protocol layer, and the data distributing is responsible for processing including but not limited to: message persistence, inter-device communication, message replay, shared subscription, etc., and distribution of the reported data is accomplished by interfacing the persistence middleware with a specific service interface.

The data replication process is specifically as follows:

(1) The Leader is in charge of receiving the request of the client, the Leader stores the request as a Raft log entry into a local storage, and after the storage is successful, the Leader initiates a copying action to other servers in parallel so as to copy the log entry;

(2) When the log is copied from the Leader to most of the follow, the Leader opens the log entry to its state machine and returns an execution result, i.e., a global commit copy result, to the client;

(3) The data Block (Block) is the minimum unit of data management and movement of the Raft cluster, and corresponds to an actual Block data interval in the local storage, each Block contains a plurality of data, a plurality of copies exist in the Block, each copy is located in different nodes, and the plurality of blocks on different nodes form an abstract Raft cluster and are copies;

(4) Based on the log consistency check of the Raft algorithm, some folows may not successfully complete log replication, and the Leader may retry replication indefinitely until all folows eventually store all log entries.

As shown in fig. 3, the MQTT is a separate instance in operation, exposing TCP endpoints to the outside, and according to the Raft characteristics, at least 3 instances need to be operated to form a Raft cluster, which is specifically as follows:

(1) The Netty service only can acquire the access of a client for realizing the MQTT protocol, only processes the request meeting the MQTT protocol specification, exposes fixed { ip: port } when the Broker service is started, and exposes a communication endpoint { ip: port } of the Raft node;

(2) The client requests to access a Broker service by using a user name and a password according to a predefined configuration, communication long links are established at two ends after the request is approved, and in the message 'CONNECT' monitoring process, the network service stores the links into a client session list and a client link list by taking a ClientID as a key and updates a routing table;

(3) When the Broker is started, the Raft node is started by a node list in configuration, and a cluster is initially created, and when all nodes are Follower at the initial time, a new Leader election is initiated;

(4) The Netty service begins processing MQTT communication messages with the client, the message types including, but not limited to: subscribing the theme, sending the message and heartbeat probe activity, wherein the client subscribes the theme and sending the message to be limited by the authorization of a manager, and editing and maintaining in a management background are needed;

(5) If the client allows subscribing the theme, after establishing a long link, a SUBSCRIBE message can be initiated, and before returning the SUBACK message, the Broker service stores the relation between the client and a plurality of themes in a local storage by decoding in a form of { ClientID, description }, and copies the relation to other nodes by using a Raft algorithm;

(6) If the client side allows to send the message, after establishing the long link, the client side can initiate a PUBLISH message, and after receiving the message, the Broker processes the information according to the message type in a message processor, and if the information is general, the information is directly forwarded; if the information is reserved, storing the local storage of the reserved information, waiting for retransmission to the client, if the information is QoS 1 and QoS2, sequentially initiating a plurality of message inter-transmission flows according to the MQTT specification requirements, storing the unacknowledged information in the corresponding local storage, and deleting the unacknowledged information after receiving the PUBCOMP message;

(7) The message issued by the client or the bridging service is decoded by a protocol layer message processor of the Broker service, the message decoded by the processor is transferred to a message distribution module, the destination of the message is determined according to the data characteristics of the message MessageType, messageHeader, qoS and the like (as shown in figure 4), the message can be forwarded after the range of the client is reversely determined through the subscription of the theme, the message is encoded by adopting different encoders according to the data characteristics of the message MessageType, messageHeader, qoS and the like, different message sending flow processes are started, and if delayed sending is required, the message is stored in a delay memory; if QoS2 semantics are required to be achieved, storing the information into a message memory to be confirmed, and matching data processing in a subsequent QoS2 flow node, and if the information is required to be durable, storing the information into a corresponding storage engine through bridging service according to a current durable strategy;

(8) The manager can directly send a request through the operation of the web platform, the web platform is connected with the bridging service, and the external application completes communication of the Broker service through adapting the bridging service, such as adjusting setting, obtaining Broker state information, publishing messages, subscribing and unsubscribing agents, disconnecting designated clients and checking a client list.

The Raft algorithm decomposes the consistency problem into three sub-problems, namely election, log replication and security, which are specifically as follows:

(1) Election: when a Leader is down or a cluster is initially created, the Followers can elect a new Leader;

(2) Log replication: the Leader processes the requests of all clients uniformly, and after receiving the data from the clients, the Leader copies the data to other nodes in the cluster in the form of log entries and forces the logs of the other nodes to be consistent with the Leader;

(3) Safety: each node has an explicit log stored in the state machine, the log index position of the node cannot be inconsistent with the log index of the Leader, and the Leader can force the index position of the node to be consistent with the Leader.

The Raft algorithm increases two limits for enhancing cluster availability and ensuring strong data consistency: the method is characterized by comprising the following steps of selecting limit and log consistency check:

(1) Election limit: the node in the Candida state can access most nodes after the election is initiated, the own log is compared with the log of each node, the index value and the tenn number of the last log in the two logs are compared to determine which node is newer in log comparison, if the tenn numbers of the last entries of the two logs are different, the log with the large tenn number is updated, if the tenn numbers of the last entries of the two logs are the same, the longer one of the logs is updated, and only the node with the largest log index value and the latest tenn number can win the election;

(2) Log consistency check: under normal conditions, the logs of the Leader and the Follower will keep consistent, under the production running environment, various abnormal conditions will occur in both the Leader and the Follower, the logs of each node will not be consistent, the Raft algorithm agrees that the Leader must save the nextIndex of each Follower, the Leader just selected will initialize the nextIndex to be the Log index value +1 of the current Leader, if the Log of one Follower and the Leader are inconsistent, the consistency check will fail when the Leader synchronizes the Log with the next heartbeat, the Leader will reduce the value of the nextIndex when the nextIndex does not match with the currentIndex of the target Follower, and retry, the nextIndex will eventually coincide with the currentIndex of the Follower at a certain position, because of the election limit, the Leader will save the latest and the most complete load data of the cluster, and the Leader will synchronize the newIndex to the Follower after determining the nextIndex.

Compared with the prior art, the invention has the following advantages and effects:

(1) The method is realized by using Java industrial-level computer language design, so that the method is high in universality and the secondary development threshold is reduced;

(2) Negative effects of commercial authorization can be avoided, and the method is continuously iterated and upgraded;

(3) Network access is realized by using Netty, and a single machine 4CoreCPU and 16GRAM can support 50 ten thousand links;

(4) The role of accessing the MQTT Broker can be an application service form besides a classical Client form, and is not forced to be accessed as a Client;

(5) The system is associated with external services through bridging services, and data support can be provided for the operation and decision of a Broker through docking an existing configuration system or a custom configuration system;

(6) The light-weight Broker cluster is realized by using a Raft algorithm, so that the forced consistency of data in the cluster can be ensured, and the data can be normally provided after the nodes are restored even if the nodes are in failure;

(7) By continuously upgrading and expanding bridging services, protocol layers and Raft nodes, the characteristics required by the running of the distributed clusters can be theoretically met.

The foregoing is only a preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art, who is within the scope of the present invention, should make equivalent substitutions or modifications according to the technical scheme of the present invention and the inventive concept thereof, and should be covered by the scope of the present invention.

Claims (7)

1. A method for realizing an MQTT Broker server by applying a Raft algorithm based on a Netty framework is characterized by comprising the following steps:

S1.Netty

the Netty is an asynchronous event-driven network application framework, and supports the rapid development of maintainable high-performance protocol-oriented servers and clients;

s2.Reactor model

The Netty architecture is designed and implemented according to a Reactor mode, and a Reactor concurrency model of the Netty is as follows:

netty implements and expands the Reactor model, as follows:

1) The acceptors main thread pool is an independent NIO thread pool, which plays the role of acceptors in network links in the form of NIO, and is also called acceptors reactor;

2) After the program is started, randomly selecting a thread from an accepter main thread pool as an accepter thread, wherein the accepter thread is used for binding ports and receiving client connection;

3) The accepter thread receives the connection request and gives the request to other threads of the accepter reactor to process login, handshake and security authentication of the client;

4) After the service layer link is formally established, the request is transferred to a certain thread of the IO sub-thread pool, and IO read-write and coding operations are processed by the thread;

5) Finally, according to the difference of request types, the request is transferred to a specific type of workbench and specific business processing is carried out in a Netty concurrency model, and the concurrency access of a client, the reading and writing of a bottom Socket and the processing of the business request are all multithreaded asynchronous operations, so that Netty endows an application program with a core capability, can respond to network events generated at any time point in any sequence, can continuously multiplex the thread resources of a computer in determined resources, and is suitable for higher throughput and expandability requirements;

S3.Raft

The Raft cluster is composed of a plurality of nodes, and each node has three states: leader, follower and Candidate, the Raft will elect the Leader first, and only one node is the Leader, the other nodes are the Follower, the Leader node processes all client requests, duplicate Log to the Follower node, if the Leader fails, after the election count-down expires, the heartbeat detection of the Leader is still abnormal, the Follower will elect a new Leader again;

s4, realizing an MQTT Broker based on Netty and Raft:

the MQTT Broker is a server directly receiving the network connection of the concurrent clients, and uses a Netty processing network communication part, so that more client connections and more services can be processed by using server resources to the maximum extent, and the Netty self-contained MQTT codec module can be directly used for finishing the codec of MQTT protocol communication;

s5.broker service maintains data consistency through the Raft cluster:

the MQTT Broker service endows data replication characteristics among the MQTT brokers by realizing a Raft algorithm, and forms a cluster based on data synchronization with strong consistency, so that the risk of single-point failure of the system is reduced;

s6, endowing the system with the following characteristics by a Raft algorithm:

1) Persistent session: when the Session identifier 'Clean Session' is set to 0, a persistent Session is created, and when the client is disconnected, the Session still keeps and stores offline information until the Session is logged off overtime, after the persistent Session information and offline MQTT information are stored locally, the Session information and the offline MQTT information are copied to other broaders of the cluster through a Raft algorithm, and after the client is online again, all offline information can be obtained in time, so that waiting is reduced;

2) Subscription data: after the client side subscribes successfully, the subscription relation data is copied to other broaders, broadcasting and issuing instructions are needed, or the client side is replaced with the broader access, so that all the current subscription topic data of the client side can be obtained in time, and the subscription relation data is deleted in the cluster after being deleted locally when the subscription is canceled;

3) Message retransmission for message quality of service (QoS): the MQTT protocol specifies that both the server and the client as communication parties should meet the requirements of their service quality for the messages sent to the opposite end by themselves, i.e. after the MQTT message is sent, the opposite end does not receive a response within a specified time, then the client retransmits the message, meanwhile, if the client sends a reconnection under the condition of maintaining a session, the Broker automatically retransmits the response message to the reconnecting client to ensure the correctness of the QoS flow, and for the above scenario, the message sent by the client but not acknowledged is stored locally and then copied and synchronized to other Broker, whether the message is retried or the publisher reconnects, the message can still be found in other Broker, and the corresponding message is maintained according to the semantic requirement of QoS.

2. The method for implementing the MQTT Broker server based on the Netty framework by applying the Raft algorithm according to claim 1, wherein the method comprises the following steps: the MQTT Broker service comprises the following steps:

(1) The Broker service consists of a network service layer, an MQTT protocol service layer, a business service layer, a storage module and a bridging module, wherein the network communication and business processing are separated, the storage module consists of a Raft node and a local storage, and the storage module is defined with a plurality of memories;

(2) The network service layer processes the connection of the client and forwards the reported data of the client to the protocol layer for decoding, and meanwhile, the service data coded by the protocol layer is transmitted to the corresponding client;

(3) The protocol layer applies the MQTT protocol coding and decoding module realized by Netty to process the MQTT protocol message required by communication between the client and the Broker, define the processing flow of the corresponding message and realize the triggering processing of protocol service, such as login, authentication and the like and message distribution;

(4) The business service consists of a message issuing module and a data distributing module;

(5) The Raft service realizes network communication, provides a communication function among nodes for the Raft cluster, and supports the operation of the Raft cluster: election, log replication, security check and heartbeat probe activity;

(6) The Raft node consists of a memory for realizing a Raft protocol and a Raft communication service, process data and result data in a service link can be submitted to a Raft Leader according to the need, the data is stored as a Raft Log into a local specific storage engine, and data replication among nodes in a cluster is completed by utilizing Log replication and strong consistency check of the Raft algorithm;

(7) Local storage is used to store data including, but not limited to: client session record, unacknowledged messages (QoS 1, qoS 2), metric values of metrics and configuration data required for program operation;

(8) Stateless bridging services expose REST APIs outward, providing business function management operations including, but not limited to, the following: acquiring the Broker state information, publishing the message and checking the client list.

3. The method for implementing the MQTT Broker server based on the Netty framework by applying the Raft algorithm according to claim 2, wherein the method comprises the following steps: the message issuing is responsible for capturing user operation, after being abstracted into an MQTT message body, the message is transferred to the protocol layer for processing, the data distributing is responsible for capturing the decoded message body of the protocol layer, and the data distributing is responsible for processing including but not limited to: message persistence, inter-device communication, message replay, shared subscription, etc., and distribution of the reported data is accomplished by interfacing the persistence middleware with a specific service interface.

4. The method for implementing the MQTT Broker server based on the Netty framework by applying the Raft algorithm according to claim 1, wherein the method comprises the following steps: the data replication process is specifically as follows:

(1) The Leader is in charge of receiving the request of the client, the Leader stores the request as a Raft log entry into a local storage, and after the storage is successful, the Leader initiates a copying action to other servers in parallel so as to copy the log entry;

(2) When the log is copied from the Leader to most of the follow, the Leader opens the log entry to its state machine and returns an execution result, i.e., a global commit copy result, to the client;

(3) The data Block (Block) is the minimum unit of data management and movement of the Raft cluster, and corresponds to an actual Block data interval in the local storage, each Block contains a plurality of data, a plurality of copies exist in the Block, each copy is located in different nodes, and the plurality of blocks on different nodes form an abstract Raft cluster and are copies;

(4) Based on the log consistency check of the Raft algorithm, some folows may not successfully complete log replication, and the Leader may retry replication indefinitely until all folows eventually store all log entries.

5. The method for implementing the MQTT Broker server based on the Netty framework by applying the Raft algorithm according to claim 2, wherein the method comprises the following steps: the MQTT is an independent example when running, and exposes TCP endpoints to the outside, and at least 3 examples need to be run to form a Raft cluster according to the Raft characteristics, which is specifically as follows:

(1) The Netty service only can acquire the access of a client for realizing the MQTT protocol, only processes the request meeting the MQTT protocol specification, exposes fixed { ip: port } when the Broker service is started, and exposes a communication endpoint { ip: port } of the Raft node;

(2) The client requests to access a Broker service by using a user name and a password according to a predefined configuration, communication long links are established at two ends after the request is approved, and in the message 'CONNECT' monitoring process, the network service stores the links into a client session list and a client link list by taking a ClientID as a key and updates a routing table;

(3) When the Broker is started, the Raft node is started by a node list in configuration, and a cluster is initially created, and when all nodes are Follower at the initial time, a new Leader election is initiated;

(4) The Netty service begins processing MQTT communication messages with the client, the message types including, but not limited to: subscribing the theme, sending the message and heartbeat probe activity, wherein the client subscribes the theme and sending the message to be limited by the authorization of a manager, and editing and maintaining in a management background are needed;

(5) If the client allows subscribing the theme, after establishing a long link, a SUBSCRIBE message can be initiated, and before returning the SUBACK message, the Broker service stores the relation between the client and a plurality of themes in a local storage by decoding in a form of { ClientID, description }, and copies the relation to other nodes by using a Raft algorithm;

(6) If the client side allows to send the message, after establishing the long link, the client side can initiate a PUBLISH message, and after receiving the message, the Broker processes the information according to the message type in a message processor, and if the information is general, the information is directly forwarded; if the information is reserved, storing the local storage of the reserved information, waiting for retransmission to the client, if the information is QoS 1 and QoS2, sequentially initiating a plurality of message inter-transmission flows according to the MQTT specification requirements, storing the unacknowledged information in the corresponding local storage, and deleting the unacknowledged information after receiving the PUBCOMP message;

(7) The message issued by the client or the bridging service is decoded by a protocol layer message processor of the Broker service, the message decoded by the processor is transferred to a message distribution module, the destination of the message is determined according to the data characteristics of the message MessageType, messageHeader, qoS and the like (as shown in figure 4), the message can be forwarded after the range of the client is reversely determined through the subscription of the theme, the message is encoded by adopting different encoders according to the data characteristics of the message MessageType, messageHeader, qoS and the like, different message sending flow processes are started, and if delayed sending is required, the message is stored in a delay memory; if QoS2 semantics are required to be achieved, storing the information into a message memory to be confirmed, and matching data processing in a subsequent QoS2 flow node, and if the information is required to be durable, storing the information into a corresponding storage engine through bridging service according to a current durable strategy;

(8) The manager can directly send a request through the operation of the web platform, the web platform is connected with the bridging service, and the external application completes communication of the Broker service through adapting the bridging service, such as adjusting setting, obtaining Broker state information, publishing messages, subscribing and unsubscribing agents, disconnecting designated clients and checking a client list.

6. The method for implementing the MQTT Broker server based on the Netty framework by applying the Raft algorithm according to claim 4, wherein the method comprises the following steps: the Raft algorithm decomposes the consistency problem into three sub-problems, namely election, log replication and security, which are specifically as follows:

(1) Election: when a Leader is down or a cluster is initially created, the Followers can elect a new Leader;

(2) Log replication: the Leader processes the requests of all clients uniformly, and after receiving the data from the clients, the Leader copies the data to other nodes in the cluster in the form of log entries and forces the logs of the other nodes to be consistent with the Leader;

(3) Safety: each node has an explicit log stored in the state machine, the log index position of the node cannot be inconsistent with the log index of the Leader, and the Leader can force the index position of the node to be consistent with the Leader.

7. The method for implementing the MQTT Broker server based on the Netty framework by applying the Raft algorithm according to claim 4, wherein the method comprises the following steps: the Raft algorithm increases two limits for enhancing cluster availability and ensuring strong data consistency: the method is characterized by comprising the following steps of selecting limit and log consistency check:

(1) Election limit: the node in the Candida state can access most nodes after the election is initiated, the own log is compared with the log of each node, the index value and the tenn number of the last log in the two logs are compared to determine which node is newer in log comparison, if the tenn numbers of the last entries of the two logs are different, the log with the large tenn number is updated, if the tenn numbers of the last entries of the two logs are the same, the longer one of the logs is updated, and only the node with the largest log index value and the latest tenn number can win the election;

(2) Log consistency check: under normal conditions, the logs of the Leader and the Follower will keep consistent, under the production running environment, various abnormal conditions will occur in both the Leader and the Follower, the logs of each node will not be consistent, the Raft algorithm agrees that the Leader must save the nextIndex of each Follower, the Leader just selected will initialize the nextIndex to be the Log index value +1 of the current Leader, if the Log of one Follower and the Leader are inconsistent, the consistency check will fail when the Leader synchronizes the Log with the next heartbeat, the Leader will reduce the value of the nextIndex when the nextIndex does not match with the currentIndex of the target Follower, and retry, the nextIndex will eventually coincide with the currentIndex of the Follower at a certain position, because of the election limit, the Leader will save the latest and the most complete load data of the cluster, and the Leader will synchronize the newIndex to the Follower after determining the nextIndex.

Images