CN112637301B - Project reader framework-based data transmission method and system - Google Patents
Project reader framework-based data transmission method and system Download PDFInfo
- Publication number
- CN112637301B CN112637301B CN202011483776.5A CN202011483776A CN112637301B CN 112637301 B CN112637301 B CN 112637301B CN 202011483776 A CN202011483776 A CN 202011483776A CN 112637301 B CN112637301 B CN 112637301B
- Authority
- CN
- China
- Prior art keywords
- data
- sensor
- subscriber
- server
- message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/562—Brokering proxy services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/141—Setup of application sessions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/26—Special purpose or proprietary protocols or architectures
Abstract
The invention provides a data transmission method and a system based on Project reader framework, comprising the following steps: the method comprises the steps that an equipment end obtains data detected by a sensor in real time and sends the data detected by the sensor in real time to a data acquisition engine of a cloud platform; the device side and a data acquisition engine of the cloud platform are communicated by adopting an MQTT protocol; a data acquisition engine of the cloud platform is constructed by utilizing a Project reader framework; a data acquisition engine of the Cloud platform utilizes Reactive Spring Cloud Stream to construct a response type message communication assembly, data detected in real time by a sensor is sent to a Kafka message middleware, and the data detected in real time by the sensor in the Kafka message middleware is forwarded to a big data platform or a database; and a Data acquisition engine of the cloud platform utilizes Spring Reactive Data to construct a response type Data access component, and real-time detection Data of the sensor in the Kafka message middleware is forwarded to the MongoDB. The invention is more efficient in terms of data transmission.
Description
Technical Field
The invention belongs to the technical field of data acquisition, and particularly relates to a Project Reactor framework-based data transmission method and system.
Background
The chinese invention application CN201210228761.3 discloses a simple and efficient socket communication framework based on nio. An architecture mode that a TCP connection establishment part and an IO read-write processing part are separated is adopted. In the IO read-write part, a message buffer queue is designed, and the read-write efficiency is improved. But using the nio programming interface is cumbersome.
The Chinese invention application CN201610993370.9 discloses a large-scale concurrent data forwarding method based on Netty, which comprises the construction of a data receiver and a data repeater, wherein the data receiver mainly receives data information of a remote host, and the data repeater mainly takes charge of data transmission to the remote host. Also, the API interface using the Netty framework is complex, has poor network communication capability, only comprises a data receiver and a repeater, and does not provide other functions such as data caching, data storage, device verification, device online and offline notification and the like.
Disclosure of Invention
Aiming at the defects of the prior art, the invention aims to provide a Project Reactor frame-based data transmission method and a Project Reactor frame-based data transmission system, and aims to solve the problems that the prior data transmission method is complicated in NIO programming or a Netty frame interface and poor in network communication capacity.
In order to achieve the above object, in a first aspect, the present invention provides a data transmission method based on Project Reactor framework, including the following steps:
developing an MQTT communication engine based on a Project Reactor framework to receive data sent by a sensor based on an MQTT protocol;
constructing a response message communication component Kafka message middleware by using Reactive Spring Cloud Stream, and sending the received sensor data to the Kafka message middleware; the kafka message middleware is used as a transfer station of sensor data;
constructing a response Data access component MongoDB by using Spring Reactive Data; the MongoDB is used as a database for persistently storing sensor data;
when sensor data needs to be persisted, sending detection data in the kafka message middleware to the MongoDB;
when the sensor data needs to be forwarded, the detection data in the kafka message middleware is sent to a big data platform.
In an optional embodiment, after a publisher of the MQTT protocol establishes a TCP connection with a server of the MQTT protocol, the MQTT protocol connection is performed first, the publisher sends a CONNECT control message to the server to indicate that the publisher wants to CONNECT to the server, and after receiving the CONNECT control message, the server replies a CONNECT control message to the publisher to indicate that the connection is confirmed;
the subscriber of the MQTT protocol is connected with the server through the CONNECT and CONNACK control messages as the subscriber of the MQTT protocol, and at the moment, the subscriber and the subscriber establish the connection based on the MQTT protocol;
a subscriber of the MQTT protocol firstly sends a topic subscription request to a server to indicate that data of the topic subscription is to be received, the subscriber sends a SUBSCRIBE control message to the server to indicate that the subscriber wants to SUBSCRIBE messages, and after receiving the SUBSCRIBE control message, the server replies a SUBSBACK control message to the subscriber to indicate confirmation of the subscription request; the subscribed topic is to acquire real-time sensor data;
the publisher of the MQTT protocol sends a message to a specified subscription topic in the server, and the server receives all sensor data sent by the publisher and forwards all the sensor data to the subscriber in real time; the publisher sends a publish control message and contains sensor real-time data, the server searches for the corresponding subscriber after receiving the data, and the server forwards the data to the corresponding subscriber by using the publish control message.
In a second aspect, the present invention provides a Project Reactor framework-based data transmission system, including: the system comprises an equipment end, a sensor, a data acquisition engine of a cloud platform and a big data platform;
detecting data in real time by a sensor;
the method comprises the steps that an equipment end obtains data detected by a sensor in real time and sends the data detected by the sensor in real time to a data acquisition engine of a cloud platform; the equipment side and a data acquisition engine of the cloud platform are communicated by adopting an MQTT protocol; the equipment terminal is used as a publisher of the MQTT protocol, and the data acquisition engine of the cloud platform is used as a server and a subscriber of the MQTT protocol; a data acquisition engine of the cloud platform is constructed by utilizing a Project Reactor framework;
a data acquisition engine of the Cloud platform utilizes Reactive Spring Cloud Stream to construct a response type message communication assembly, data detected in real time by a sensor is sent to a Kafka message middleware, and the data detected in real time by the sensor in the Kafka message middleware is forwarded to a big data platform or a database; the kafka message middleware is one of response type message communication components and is used as a transfer station for detecting data in real time by a sensor;
a Data acquisition engine of the cloud platform utilizes Spring Reactive Data to construct a response type Data access component, and real-time detection Data of a sensor in Kafka message middleware is forwarded to MongoDB; the MongoDB is one of the responsive data access components and is used as a database for the real-time detection data persistence of the sensor.
In an optional embodiment, the device side creates a corresponding device client for each sensor;
after receiving the real-time data, a certain equipment client of the equipment end establishes TCP connection with a data acquisition engine of the cloud platform; the equipment client side which receives the real-time data serves as a publisher of the MQTT protocol, and the received real-time data is sent to a data acquisition engine of the cloud platform through the MQTT protocol;
the method comprises the steps that an equipment end is provided with a unique equipment code for each equipment client, after the equipment clients establish TCP connection with a data acquisition engine of a cloud platform, the equipment clients send own equipment codes to the data acquisition engine of the cloud platform for verification, and received real-time data are sent to the data acquisition engine of the cloud platform after the verification is passed;
after TCP connection is established between the equipment client and the data acquisition engine of the cloud platform and verification is successful, a heartbeat mechanism is started, and if the data acquisition engine of the cloud platform is not connected, an automatic reconnection mechanism is started to establish TCP connection with the data acquisition engine of the cloud platform again.
In an optional embodiment, after a publisher of the MQTT protocol establishes a TCP connection with a server of the MQTT protocol, the MQTT protocol connection is performed first, the publisher sends a CONNECT control message to the server to indicate that the publisher wants to CONNECT to the server, and after receiving the CONNECT control message, the server replies a CONNECT control message to the publisher to indicate that the connection is confirmed;
the subscriber of the MQTT protocol is connected with the server through the CONNECT and CONNACK control messages as the subscriber of the MQTT protocol, and at the moment, the subscriber and the subscriber establish the connection based on the MQTT protocol;
a subscriber of the MQTT protocol firstly sends a topic subscription request to a server to indicate that data of the topic subscription is to be received, the subscriber sends a SUBSCRIBE control message to the server to indicate that the subscriber wants to SUBSCRIBE messages, and after receiving the SUBSCRIBE control message, the server replies a SUBSBACK control message to the subscriber to indicate confirmation of the subscription request; the subscription topic is used for acquiring real-time sensor data;
the publisher of the MQTT protocol sends a message to a specified subscription topic in the server, and the server receives all sensor data sent by the publisher and forwards all the sensor data to the subscriber in real time; the publisher sends a publish control message and contains sensor real-time data, the server searches for the corresponding subscriber after receiving the data, and the server forwards the data to the corresponding subscriber by using the publish control message.
In an alternative embodiment, the Reactive Spring Cloud Stream responsive message communication mode includes three roles, namely, a publisher, a consumer, and a message communication system of a message;
the sensor real-time data received by the subscriber of the MQTT protocol is used as the message publisher, the message publisher sends the sensor real-time data to a message communication system, and the message communication system can adopt Kafka message middleware;
the Kafka message middleware serves as a data switching center and sends data to consumers for consumption, and the consumers can be large data platforms or databases.
In an optional embodiment, a responsive Data access component is constructed based on Spring Reactive Data, CRUD operation for Data stream is provided, and Data of Kafka message middleware is sent to MongoDB through insertion operation; the MongoDB is a data storage medium that supports a responsive programming model.
Generally, compared with the prior art, the above technical solution conceived by the present invention has the following beneficial effects:
the invention provides a Project reader framework-based data transmission method and a Project reader framework-based data transmission system, wherein a Spring Project reader framework is adopted to develop a response type data acquisition server based on an MQTT protocol, the Spring Project reader manages data processing in a backpressure mechanism mode, and the backpressure mechanism is a mechanism capable of feeding back a flow request upstream and can balance the data transmission rate between a device end and a data acquisition engine of a cloud platform, so that the problem of rate imbalance caused by the fact that a data sender is too fast and a data receiver is too slow is effectively solved, and the strategy is more efficient compared with the strategy of setting an I/O buffer area. The transmission of data is more efficient than a program written using the Netty framework in a traditional programming manner.
The invention provides a data transmission method and system based on Project reader framework, a full stack response type system is constructed, the acquisition, forwarding and storage of data are in accordance with response type specifications, the blocking waiting of each stage of data transmission is reduced, and the data concurrent processing is more efficient compared with a scheme of parallel processing by using a thread pool.
Drawings
Fig. 1 is a flowchart of a data transmission method based on Project Reactor framework according to an embodiment of the present invention;
fig. 2 is a diagram of a data transmission system architecture based on Project Reactor framework according to an embodiment of the present invention;
fig. 3 is a diagram of MQTT protocol message mode provided by the embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention relates to the technical field of data acquisition, and solves the problem of large-scale high-concurrency acquisition and storage of sensor data. The invention discloses a sensor data acquisition method based on project reader framework, which is divided into two parts of equipment end program and cloud platform. The device end program is used for connecting the sensor device, acquiring data information acquired by the sensor, establishing TCP connection with the cloud platform of the invention, and transmitting data by using an MQTT protocol. The cloud platform is used for receiving and persisting sensor data, the data acquisition engine part in the platform adopts a Project reader framework to construct an MQTT server, the MQTT server is used for receiving large-scale massive data in real time and sending the data to a message queue middleware, and the message queue serves as a transfer station for transferring the data to a large data platform and a persisted database.
The invention is suitable for collecting and storing large-scale sensor information, adopts a 'lightweight' MQTT transmission protocol based on a publishing or subscribing mode, and introduces a response programming method, thereby constructing a response data collecting method, greatly improving the processing efficiency of I/O (input/output), and being more suitable for the collecting and storing scenes of multi-source heterogeneous sensors.
Fig. 1 is a flowchart of a data transmission method based on Project Reactor framework according to an embodiment of the present invention, as shown in fig. 1, including the following steps:
s11, detecting data in real time by a sensor;
s12, developing an MQTT communication engine by the cloud platform based on a Project Reactor framework to receive data sent by the sensor based on an MQTT protocol;
s13, the sensor sends the detection data to the cloud platform through the equipment end;
s14, the Cloud platform utilizes Reactive Spring Cloud Stream to construct a response type message communication component Kafka message middleware;
s15, the cloud platform sends the received sensor data to kafka message middleware;
s16, sending the detection data in the kafka message middleware to a big data platform;
s17, the cloud platform utilizes Spring Reactive Data to construct a response type Data access component MongoDB; the MongoDB is used as a database for persistently storing sensor data;
s18, sending the detection data in the kafka message middleware to MongoDB;
s19, MongoDB persists the detection data.
As shown in fig. 2, the architecture diagram of a data transmission system based on Project Reactor framework provided in the embodiment of the present invention includes:
and the equipment end program acquires real-time detection data of the sensor. And then, the equipment end program sends the sensor data to a data acquisition engine of the cloud platform in real time. And the equipment end program and the data acquisition engine of the cloud platform are communicated by adopting an MQTT protocol.
Specifically, the sensor sends real-time data to the device-side program, and the device-side program receives the data in real time. And, the device-side program will create a corresponding device client for each sensor. After receiving the real-time data, a certain device client in the device end program establishes TCP connection with a data acquisition engine in the cloud platform. The equipment client serves as a publisher of the MQTT protocol, and sends real-time data to a data acquisition engine in the cloud platform through the MQTT transmission protocol. The device client side is provided with a unique device code, after connection is established, the device code of the device client side is sent back to the data acquisition engine to be verified, and the data can be sent only after the verification is passed. After the connection between the equipment client and the data acquisition engine is established and the verification is successful, a heartbeat mechanism is started, and if the data acquisition engine is not connected, an automatic reconnection mechanism is started to establish TCP connection with the data acquisition engine again.
And constructing a data acquisition engine of the cloud platform by using the Project Reactor framework, wherein the data acquisition engine is used for receiving mass sensor data sent by a device-side program. A data acquisition engine of the cloud platform and a device-side program form an MQTT-based communication mechanism together, wherein the data acquisition engine serves as a server and a subscriber of an MQTT protocol, and the data acquisition engine serves as a publisher of the MQTT protocol.
A responsive message communication component is constructed using Reactive Spring Cloud Stream. And sending the mass sensor data received by the data acquisition engine to Kafka message middleware, and then forwarding the mass sensor data to a big data platform or a database. The Kafka message middleware is used as a data transfer station of mass sensor information;
a responsive Data access component is built using Spring Reactive Data. And sending the data of the data forwarding transfer station to MongoDB. The MongoDB acts as a persisted database of sensor data.
As a preferred technical solution, a data acquisition engine of the cloud platform includes a server and a subscriber in the MQTT protocol, and a device client (publisher of the MQTT protocol) and the data acquisition engine together form an MQTT protocol message mode, and interact data by using an MQTT transmission protocol based on TCP connection.
After a publisher of the MQTT protocol establishes TCP connection with a server, the publisher needs to perform MQTT protocol connection first, and the publisher sends a CONNECT control message to the server to indicate that the publisher wants to CONNECT with the server. After receiving the CONNECT control message, the server replies a CONNACK control message to the publisher to indicate that the connection is confirmed.
The subscriber of the MQTT protocol and the publisher establish connection with the server through the CONNECT and CONNACK control messages. At this time, the publisher and the subscriber establish a connection based on the MQTT transmission protocol with the server.
A subscriber in MQTT protocol may first send a topic subscription request to the server indicating that data of the topic subscription is to be received. That is, the subscriber will send a SUBSCRIBE control message to the server indicating that the subscriber wants to SUBSCRIBE to the message. After receiving the SUBSCRIBE control message, the server replies a SUBSCRIBE control message to the subscriber to indicate the confirmation of the subscription request.
The publisher of the MQTT protocol can send a message to a specified subscription topic in the server, and the server can receive all sensor data sent by the publisher, so that the server forwards all the sensor data to the subscriber in real time. That is, the publisher sends a publish control message and contains data such as sensor real-time data and subscription topics, the server searches for a corresponding subscriber after receiving the data, and the server forwards the data to the corresponding subscriber by using the publish control message. At this point, the publisher has completed sending sensor data to the server, which has sent the data to the subscriber. The subscriber can then send the data to the data transfer station.
As a preferred technical scheme, the Reactive Spring Cloud Stream responsive message communication mode comprises three roles, namely a publisher, a consumer and a message communication system of messages. Massive sensor data received by a subscriber of the MQTT protocol is used as a message publisher in a response type message communication mode, the message publisher sends the massive sensor data to a message communication system role, and the message communication system can adopt Kafka message middleware. The message middleware is used as a data exchange center and sends the data to a consumer role for consumption. The consumer may be a big data platform or database.
As a preferred technical scheme, a response type Data access model is constructed based on Spring Reactive Data, CRUD operation for Data streams is provided, and Data of a Data exchange center can be sent to a database through insertion operation. The MongoDB supports data storage media of a responsive programming model, and mass sensor data sent from a data forwarding center can be stored in a MongoDB database in a persistent mode.
Specifically, the work flow of the transmission system shown in fig. 2 includes the following steps:
and 1, acquiring real-time detection data of the sensor through a device-side program. And then, the equipment end program sends the sensor data to a data acquisition engine of the cloud platform in real time through a corresponding equipment client. The device client is the publisher in fig. 3.
And 2, a data acquisition engine of the cloud platform is used for receiving mass sensor data, the data acquisition engine of the cloud platform and the device end program in the step 1 form an MQTT protocol message mode, the former is used as a server and a subscriber of the MQTT protocol, and the latter is used as a publisher of the MQTT protocol. The MQTT protocol message mode is shown in fig. 3. The method specifically comprises the following steps:
2.1, defining the corresponding Configuration parameters of the MQTT service end and the equipment end by using the @ Configuration annotation provided by Spring Boot, such as: IP address, port number, heartbeat time, whether to open a log, etc.
2.2, creating a TCP server and a client based on TcpServer and TcpClient provided by Project Reactor API, and setting basic information such as IP address and port number.
2.3, using MQTT as the transmission protocol of the invention, because MQTT is an application layer protocol and needs to be transmitted by means of TCP/IP protocol, using the TCP server and the client in step 2.2 as the communication base.
2.4, constructing an MQTT protocol control message, wherein the control message can be split into: fixed header + variable header + message body. MqtFixedHeader and MqtMessageIdVariableHeader in the io, netty, handler, codec, MQTT packet are used to encapsulate the MQTT fixed header and the variable header respectively, and MqttPublishmessage is used to encapsulate the three parts of the MQTT message. Writing MqttMessageApi tool class, and providing information such as different message types and message grades of MQTT.
2.5, constructing an MQTT protocol server, creating a TCP server side based on TcpServer provided by Project Reactor API, then realizing the MQTT server by combining an MQTT protocol control message, and storing subscription topics and corresponding subscribers by adopting LoadingCache of Google Guava. The major functions of the MQTT protocol server include: 1, receiving real-time information issued by an issuer; 2, processing the subscription and unsubscription of the subscriber; and 3, forwarding the subscription information of the subscriber to the subscriber.
2.6, constructing an MQTT protocol client, creating a TCP client based on TcpServer provided by Project Reactor API, and then realizing the MQTT protocol client by combining an MQTT protocol control message, wherein the MQTT client represents a publisher or a subscriber in the picture 3. The major functions of the MQTT protocol client include: 1 the client can be used as a publisher or a subscriber at the same time; 2, the client can issue information; the 3 client can subscribe or unsubscribe to related topics.
And 2.7, constructing a sensor device check sum online and offline notification, wherein in device management, a device successfully registered generates a unique device code and automatically pushes the unique device code to a data acquisition engine in the figure 2, and a Redis database is used for storing all received device codes. Before sending the real-time sensor data, the equipment client sends respective equipment codes, and after the verification is successful, the equipment client is informed to be on-line and starts to transmit the data in real time.
2.8, constructing a client reconnection mechanism and a heartbeat mechanism of the MQTT protocol, and establishing the client reconnection mechanism and the heartbeat mechanism based on project Reactor API and MQTT protocol specifications to ensure the stability between the equipment end program and the data acquisition engine in the figure 2. If the MQTT protocol server is abnormal in the data transmission process, the MQTT protocol client is disconnected with the MQTT protocol server. At this time, the MQTT protocol client actively reconnects the MQTT server.
And 3, constructing a response message communication mode by using the Reactive Spring Cloud Stream component. And (3) sending the mass sensor data received by the data acquisition engine in the step 2 to Kafka message middleware, and then forwarding the mass sensor data to a big data platform or a database. The Kafka message middleware is used as a sensor data transfer station.
And 4, constructing a response Data access component by utilizing the Spring Reactive Data component, and sending the Data of the Data transfer station in the step 3 to the MongoDB. The MongoDB acts as a persisted database of sensor data.
It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (5)
1. A data transmission method based on a ProjectReactor framework is characterized by comprising the following steps:
developing an MQTT communication engine based on a project Reactor framework to receive data sent by a sensor based on an MQTT protocol; after a publisher of the MQTT protocol establishes TCP connection with a server of the MQTT protocol, the MQTT protocol connection is firstly carried out, the publisher sends a CONNECT control message to the server to indicate that the publisher wants to CONNECT with the server, and after receiving the CONNECT control message, the server replies a CONNACK control message to the publisher to indicate that the connection is confirmed; the subscriber of the MQTT protocol is connected with the server through the CONNECT and CONNACK control messages as the subscriber of the MQTT protocol, and at the moment, the subscriber and the subscriber establish the connection based on the MQTT protocol; a subscriber of the MQTT protocol firstly sends a topic subscription request to a server to indicate that data of the topic subscription is to be received, the subscriber sends a SUBSCRIBE control message to the server to indicate that the subscriber wants to SUBSCRIBE messages, and after receiving the SUBSCRIBE control message, the server replies a SUBSBACK control message to the subscriber to indicate confirmation of the subscription request; the subscription topic is used for acquiring real-time sensor data; the publisher of the MQTT protocol sends a message to a specified subscription topic in the server, and the server receives all sensor data sent by the publisher and forwards all the sensor data to the subscriber in real time; the publisher sends a PUSBACK control message and contains sensor real-time data, the server searches for a corresponding subscriber after receiving the data, and the data is forwarded to the corresponding subscriber by using the PUSBACK control message;
constructing a response type message communication assembly Kafka message middleware by using reactive SpringCloudStream, and sending the received sensor data to the Kafka message middleware; the Kafka message middleware is used as a transfer station of sensor data;
constructing a response type data access component MongoDB by utilizing Springreacteda; the MongoDB is used as a database for persistently storing sensor data;
when sensor data needs to be persisted, sending detection data in the Kafka message middleware to the MongoDB;
when the sensor data needs to be forwarded, the detection data in the Kafka message middleware is sent to a big data platform.
2. A data transmission system based on a project reactor framework, comprising: the system comprises an equipment end, a sensor, a data acquisition engine of a cloud platform and a big data platform;
detecting data in real time by a sensor;
the method comprises the steps that an equipment end obtains data detected by a sensor in real time and sends the data detected by the sensor in real time to a data acquisition engine of a cloud platform; the equipment side and a data acquisition engine of the cloud platform are communicated by adopting an MQTT protocol; the equipment side is used as a publisher of the MQTT protocol, and a data acquisition engine of the cloud platform is used as a server and a subscriber of the MQTT protocol; a data acquisition engine of the cloud platform is constructed by utilizing a project reactor framework; after a publisher of the MQTT protocol establishes TCP connection with a server of the MQTT protocol, the MQTT protocol connection is firstly carried out, the publisher sends a CONNECT control message to the server to indicate that the publisher wants to CONNECT with the server, and after receiving the CONNECT control message, the server replies a CONNACK control message to the publisher to indicate that the connection is confirmed; the subscriber of the MQTT protocol is connected with the server through the CONNECT and CONNACK control messages as the subscriber of the MQTT protocol, and at the moment, the subscriber and the subscriber establish the connection based on the MQTT protocol; a subscriber of the MQTT protocol firstly sends a topic subscription request to a server to indicate that data of the topic subscription is to be received, the subscriber sends a SUBSCRIBE control message to the server to indicate that the subscriber wants to SUBSCRIBE messages, and after receiving the SUBSCRIBE control message, the server replies a SUBSBACK control message to the subscriber to indicate confirmation of the subscription request; the subscription topic is used for acquiring real-time sensor data; the publisher of the MQTT protocol sends a message to a specified subscription topic in the server, and the server receives all sensor data sent by the publisher and forwards all the sensor data to the subscriber in real time; the publisher sends a PUSBACK control message and contains sensor real-time data, the server searches for a corresponding subscriber after receiving the data, and the data is forwarded to the corresponding subscriber by using the PUSBACK control message;
a data acquisition engine of the cloud platform utilizes reactive spring cloud stream to construct a response type message communication assembly, data detected in real time by a sensor is sent to a Kafka message middleware, and the data detected in real time by the sensor in the Kafka message middleware is forwarded to a big data platform or a database; the Kafka message middleware is one of response type message communication components and is used as a transfer station for detecting data in real time by a sensor;
a data acquisition engine of the cloud platform utilizes Springreactedata to construct a response type data access component, and real-time detection data of a sensor in Kafka message middleware is forwarded to MongoDB; the MongoDB is one of the responsive data access components and is used as a database for the real-time detection data persistence of the sensor.
3. The project Reactor framework-based data transmission system according to claim 2, characterized in that the device side creates a corresponding device client for each sensor;
after receiving the real-time data, a certain equipment client of the equipment end establishes TCP connection with a data acquisition engine of the cloud platform; the equipment client side which receives the real-time data serves as a publisher of the MQTT protocol, and the received real-time data is sent to a data acquisition engine of the cloud platform through the MQTT protocol;
the method comprises the steps that the device side is provided with a unique device code for each device client side, after the device client sides are connected with a data acquisition engine of the cloud platform in a TCP mode, the device client sides send own device codes to the data acquisition engine of the cloud platform for verification, and after the verification is passed, received real-time data are sent to the data acquisition engine of the cloud platform;
after TCP connection is established between the equipment client and the data acquisition engine of the cloud platform and verification is successful, a heartbeat mechanism is started, and if the data acquisition engine of the cloud platform is not connected, an automatic reconnection mechanism is started to establish TCP connection with the data acquisition engine of the cloud platform again.
4. The project reader framework-based data transmission system of claim 2, wherein the reactivespring cloudstream responsive message communication mode comprises three roles, namely a message publisher, a consumer, and a message communication system;
the sensor real-time data received by the subscriber of the MQTT protocol is used as the message publisher, the message publisher sends the sensor real-time data to a message communication system, and the message communication system can adopt Kafka message middleware;
the Kafka message middleware serves as a data switching center and sends data to consumers for consumption, and the consumers can be large data platforms or databases.
5. The project Reactor framework-based data transmission system according to claim 2, characterized in that responsive data access components are constructed on the basis of SpringReactiveData, CRUD operations for data streams are provided, and data of Kafka message middleware are sent to MongoDB through insertion operations; the MongoDB is a data storage medium that supports a responsive programming model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011483776.5A CN112637301B (en) | 2020-12-16 | 2020-12-16 | Project reader framework-based data transmission method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011483776.5A CN112637301B (en) | 2020-12-16 | 2020-12-16 | Project reader framework-based data transmission method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112637301A CN112637301A (en) | 2021-04-09 |
| CN112637301B true CN112637301B (en) | 2022-08-05 |
Family
ID=75313517
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011483776.5A Active CN112637301B (en) | 2020-12-16 | 2020-12-16 | Project reader framework-based data transmission method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112637301B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104092767A (en) * | 2014-07-21 | 2014-10-08 | 北京邮电大学 | Posting/subscribing system for adding message queue models and working method thereof |
| CN108848022A (en) * | 2018-06-05 | 2018-11-20 | 华南理工大学 | A kind of information push method based on scene and user behavior |
| CN111818182A (en) * | 2020-08-31 | 2020-10-23 | 四川新网银行股份有限公司 | Micro-service arranging and data aggregating method based on Spring closed gateway |
| CN112051816A (en) * | 2020-09-03 | 2020-12-08 | 珠海格力智能装备有限公司 | Data acquisition system and method |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9927788B2 (en) * | 2011-05-19 | 2018-03-27 | Fisher-Rosemount Systems, Inc. | Software lockout coordination between a process control system and an asset management system |
| CN104639625B (en) * | 2015-01-27 | 2018-05-01 | 华南理工大学 | A kind of data concentrator collection control method, apparatus and system based on MQTT |
| CN108810086A (en) * | 2018-04-27 | 2018-11-13 | 电子科技大学中山学院 | Intelligent equipment communication platform based on MQTT message protocol |
| CN109861978B (en) * | 2018-12-28 | 2021-06-18 | 浙江工业大学 | Internet of things SaaS platform based on MQTT protocol |
-
2020
- 2020-12-16 CN CN202011483776.5A patent/CN112637301B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104092767A (en) * | 2014-07-21 | 2014-10-08 | 北京邮电大学 | Posting/subscribing system for adding message queue models and working method thereof |
| CN108848022A (en) * | 2018-06-05 | 2018-11-20 | 华南理工大学 | A kind of information push method based on scene and user behavior |
| CN111818182A (en) * | 2020-08-31 | 2020-10-23 | 四川新网银行股份有限公司 | Micro-service arranging and data aggregating method based on Spring closed gateway |
| CN112051816A (en) * | 2020-09-03 | 2020-12-08 | 珠海格力智能装备有限公司 | Data acquisition system and method |
Non-Patent Citations (1)
| Title |
|---|
| 如何在React项目中使用MQTT;EMQX;《https://blog.csdn.net/emqx_broker/article/details/109445222》;20201102;正文1-7页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112637301A (en) | 2021-04-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108768826B (en) | Message routing method based on MQTT and Kafka high concurrency scene | |
| WO2019205907A1 (en) | Intelligent device communication platform based on mqtt message protocol | |
| CN102111419B (en) | Message middleware-based client automatic reconnection method | |
| CN110266794A (en) | Method for subscribing and system are acted on behalf of based on MQTT server | |
| CN103036961A (en) | Distributed collection and storage method of journal | |
| CN102869123B (en) | Be applied to virtual peripheral collecting method and the system of intelligent display device | |
| CN104811459A (en) | Processing method, processing device and system for message services and message service system | |
| CN103096271A (en) | Information pushing system and information pushing method | |
| CN102739560B (en) | Instant communication method, system and device | |
| CN113163016B (en) | Network long connection service clustering deployment system and control flow | |
| CN114866528A (en) | Data communication method based on MQTT and Websocket | |
| CN101018349A (en) | A method for sending the SMS of the mobile phone number to the Internet program | |
| CN107071067B (en) | Cgo-based high-performance stock market access system and method | |
| CN111416823A (en) | Data transmission method and device | |
| US8463857B2 (en) | Notification of internet service events using instant messaging service | |
| CN112350914A (en) | Modbus communication method and system based on MQTT cloud platform | |
| KR20070026682A (en) | A method for data transmission between network apparatus | |
| CN112637301B (en) | Project reader framework-based data transmission method and system | |
| US11582320B2 (en) | Message transmitting and receiving method, communication apparatus, and program | |
| CN113132487A (en) | Simplified distributed long-connection data transmission method and system | |
| CN101001213B (en) | Short message transmission system and method | |
| CN106357454A (en) | Log system control device and method | |
| CN116471342A (en) | Multi-protocol access system and method for multi-Internet of things equipment | |
| CN101471927B (en) | Method and device for processing document transmission and document transmission system | |
| CN114844738A (en) | Data management method based on service gateway |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |