CN112948738A - Data processing method and device based on MQTT - Google Patents
Data processing method and device based on MQTT Download PDFInfo
- Publication number
- CN112948738A CN112948738A CN202110290833.6A CN202110290833A CN112948738A CN 112948738 A CN112948738 A CN 112948738A CN 202110290833 A CN202110290833 A CN 202110290833A CN 112948738 A CN112948738 A CN 112948738A
- Authority
- CN
- China
- Prior art keywords
- user
- request
- queue
- sub
- mqtt
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16Y—INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]
- G16Y30/00—IoT infrastructure
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The specification provides a data processing method and device based on MQTT, and the method comprises the following steps: acquiring a plurality of user themes collected by an MQTT server; each user theme carries a user request corresponding to the user theme; acquiring a user request of each user topic in a sharing subscription mode; the method comprises the steps of carrying out batch processing on a plurality of acquired user requests according to the types of user themes to obtain a plurality of request sets; each request set comprises a plurality of user requests, and each request set corresponds to a unique user theme type; reading a database according to the obtained request sets to obtain a reading result set corresponding to each request set; and sending the obtained reading result set to the MQTT server so that the client initiating the request obtains the reading result from the MQTT server according to the subscribed user theme. The scheme of the invention can realize high concurrent processing and can realize the compatibility of user requests with different formats.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method and apparatus based on MQTT.
Background
In the current home environment, the number of intelligent devices is increasing, user requests reported by the devices to the server are increasing, especially in a time period from 8 to 10 (when a large number of users use the devices at home), the problem of too much server pressure is caused, and the condition of request delay or no reaction is caused when the server pressure is too much, that is, high concurrency requirements cannot be realized.
At present, although the patent with application number CN201610225491.9 achieves high concurrency requirement, the format requested by its user is fixed and not compatible with variable parameters.
Therefore, there is a need for a data processing method and apparatus that can achieve high concurrency requirements and can also achieve user requests compatible with different formats.
Disclosure of Invention
The embodiment of the specification provides a data processing method and device based on MQTT, which can realize high concurrent processing and can realize user requests compatible with different formats.
In a first aspect, an embodiment of the present specification provides a data processing method based on MQTT, including:
acquiring a plurality of user themes collected by an MQTT server; each user theme carries a user request corresponding to the user theme;
acquiring a user request of each user topic in a sharing subscription mode;
the method comprises the steps of carrying out batch processing on a plurality of acquired user requests according to the types of user themes to obtain a plurality of request sets; each request set comprises a plurality of user requests, and each request set corresponds to a unique user theme type;
reading a database according to the obtained request sets to obtain a reading result set corresponding to each request set;
and sending the obtained reading result set to the MQTT server so that the client initiating the request obtains the reading result from the MQTT server according to the subscribed user theme.
In a possible design, the batch processing the obtained multiple user requests according to the type of the user theme to obtain multiple request sets includes:
caching the acquired multiple user requests in a queue mode to obtain a cache queue;
classifying the cached queues according to the types of the user themes to obtain a plurality of sub-queues; each sub-queue corresponds to a unique user theme type;
and traversing each sub-queue, and if the number of the user requests in the sub-queue reaches a preset number threshold, packaging the sub-queue into a request set.
In a possible design, the batch processing the obtained multiple user requests according to the type of the user theme to obtain multiple request sets includes:
caching the acquired multiple user requests in a queue mode to obtain a cache queue;
classifying the cached queues according to the types of the user themes to obtain a plurality of sub-queues; each sub-queue corresponds to a unique user theme type;
and traversing each sub-queue, and packaging the sub-queue into a request set if the buffer time of the sub-queue reaches a preset time threshold.
In one possible design, the sending the obtained set of read results to the MQTT server includes:
splitting the obtained reading result set to obtain a plurality of reading results; each user request corresponds to a reading result;
and sending the obtained multiple reading results to the MQTT server in a multi-thread parallel mode.
In a second aspect, embodiments of the present specification provide an MQTT-based data processing apparatus, including:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a plurality of user themes collected by an MQTT server; each user theme carries a user request corresponding to the user theme;
the second acquisition module is used for acquiring the user request of each user theme in a sharing subscription mode;
the batch processing module is used for carrying out batch processing on the obtained multiple user requests according to the types of the user themes to obtain multiple request sets; each request set comprises a plurality of user requests, and each request set corresponds to a unique user theme type;
the reading module is used for reading the database according to the obtained request sets to obtain a reading result set corresponding to each request set;
and the sending module is used for sending the obtained reading result set to the MQTT server so that the client initiating the request obtains the reading result from the MQTT server according to the subscribed user theme.
In one possible design, the batch processing module is configured to perform the following operations:
caching the acquired multiple user requests in a queue mode to obtain a cache queue;
classifying the cached queues according to the types of the user themes to obtain a plurality of sub-queues; each sub-queue corresponds to a unique user theme type;
and traversing each sub-queue, and if the number of the user requests in the sub-queue reaches a preset number threshold, packaging the sub-queue into a request set.
In one possible design, the batch processing module is configured to perform the following operations:
caching the acquired multiple user requests in a queue mode to obtain a cache queue;
classifying the cached queues according to the types of the user themes to obtain a plurality of sub-queues; each sub-queue corresponds to a unique user theme type;
and traversing each sub-queue, and packaging the sub-queue into a request set if the buffer time of the sub-queue reaches a preset time threshold.
In one possible design, the sending module is configured to:
splitting the obtained reading result set to obtain a plurality of reading results; each user request corresponds to a reading result;
and sending the obtained multiple reading results to the MQTT server in a multi-thread parallel mode.
In a third aspect, the present specification provides a computer-readable storage medium, where computer instructions are stored on the computer-readable storage medium, and when executed by a processor, the computer instructions cause the processor to execute the method described above.
In a fourth aspect, embodiments of the present specification provide an MQTT-based data processing apparatus, including: at least one memory and at least one processor;
the at least one memory to store a machine readable program;
the at least one processor is configured to invoke the machine-readable program to perform the method described above.
According to the scheme, the MQTT-based data processing method and the device can realize the sending of user requests in any format by utilizing a theme-subscription mechanism of the MQTT server, namely, the user requests compatible with different formats are realized, so that the universality of data processing is increased; moreover, when the client initiating the request reports the user request to the user theme of the MQTT server, the client does not need to wait for the result of data processing, so that an asynchronous request can be realized, which reduces available resources of the server for data processing, increases the throughput of the server, and thus realizes high concurrent processing. In addition, by processing the acquired multiple user requests in batch, time consumed by network requests and memory consumption can be reduced.
Drawings
In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present specification, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flow chart of a data processing method based on MQTT provided by one embodiment of the present specification;
FIG. 2 is a flow chart of a data processing method based on MQTT provided by another embodiment of the present specification;
FIG. 3 is a schematic diagram of an MQTT-based data processing device provided by one embodiment of the present specification;
FIG. 4 is a schematic diagram of an MQTT-based data processing device provided by one embodiment of the present specification.
Detailed Description
In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure more clear, the technical solution in the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are a part of the embodiments of the present disclosure, but not all embodiments, and all other embodiments obtained by a person of ordinary skill in the art without creative efforts based on the embodiments of the present disclosure belong to the protection scope of the present disclosure.
In the prior art, during a peak period of use of the internet of things device, the mobile phone APP controls the internet of things device to send a user request to the server and receive a request return result based on the HTTP transport protocol, for example, upload a device key or change the device key when the user requests. When the number of user requests is large, the server is stressed more and more, the time for replying the requests is longer and longer, and even the server crashes. In addition, the HTTP-based data processing needs three handshakes and four handoffs before communication can be performed, and the HTTP-based data processing is synchronous processing (for example, if the device key is reported in an HTTP manner, since resources of the server are limited, excessive connection cannot be established, and a subsequent user request needs to be processed after the previous user request is processed, a device end of the internet of things needs to wait, which causes a problem of device jamming or no response), which may result in a long time consumption.
Moreover, HTTP-based data processing, the format of its user request is fixed and not compatible with variable parameters.
Therefore, in order to solve the above technical problems, it is considered to convert the HTTP-based transmission method into the MQTT-based transmission method, so that the user request with long network request time (i.e. high concurrency requirement cannot be met) and format change can be compatible can be solved.
The foregoing is the concept provided by the present invention, and specific implementations of the concept provided by the present invention are described below.
Fig. 1 shows a flow chart of a MQTT-based data processing method according to one embodiment. It is to be appreciated that the method can be performed by any apparatus, device, platform, cluster of devices having computing and processing capabilities. In an embodiment of the invention, the method is performed by a device installed with an operating system.
Referring to fig. 1, the method includes:
step 101: and acquiring a plurality of user themes collected by the MQTT server.
In this step, MQTT is a communication protocol, data transmission and data reception are required to complete communication, and the data transmission and data reception are required to complete conversion through an intermediate server, that is, a server EMQ of MQTT is hereinafter referred to as EMQ. The client side (such as the internet of things device side and the control terminal) initiating the request needs to receive the message from the EMQ and also sends the message to the EMQ, the tool for the client side to send the message to the EMQ and the EMQ to send the message to the client side is MQTT SDK, and the process of sending data to the EMQ through the MQTT SDK and the process of sending data to the MQTT SDK by the EMQ are collectively called MQTT pipelines.
The client initiating the request is connected with the MQTT server in a communication mode of the MQTT pipeline, and the cloud server is connected with the MQTT server in a subscription mode, so that the MQTT server is an intermediate server between the client initiating the request and the cloud server. By means of the topic-subscription mechanism of the MQTT, the user requests in any format can be sent, namely the user requests compatible with different formats can be realized, and therefore the universality of data processing is increased. That is, each user Topic carries a user request corresponding to the user Topic, user requests of different formats are uploaded to the MQTT server by means of the user Topic (i.e., Topic), the cloud server obtains the user requests by means of subscribing the user topics, then a result returned by the cloud server is also issued to the MQTT server by means of the user topics, and a client initiating the request also obtains the corresponding returned results by means of subscribing the user topics.
Moreover, when the client initiating the request reports the user request to the user theme of the MQTT server, the client does not need to wait for the result of data processing, so that an asynchronous request can be realized, which reduces available resources of the server for data processing, increases the throughput of the server, and thus realizes high concurrent processing.
Step 102: and acquiring the user request of each user topic in a sharing subscription mode.
When a large number of MQTT SDKs send mass data to the MQTT server, the received mass data need to be processed, and the data can be received in a shared subscription mode, so that a plurality of servers capable of processing data can be expanded and deployed until mass messages can be received in a peer-to-peer mode.
In this step, there may be a plurality of cloud servers, each cloud server establishes a communication connection with the MQTT server, that is, a plurality of cloud servers subscribe to the same user topic, and when the MQTT server publishes the user topic, only one of the cloud servers will receive the user request in the user topic. By the arrangement, the transverse expansion in the service can be realized, the elastic telescopic framework of the server under the high concurrent request can be realized, and the hot plug of the cloud server can be realized.
Step 103: and carrying out batch processing on the obtained multiple user requests according to the types of the user themes to obtain multiple request sets.
In this embodiment, by performing batch processing on the obtained multiple user requests, time consumed by network requests and memory consumption can be reduced.
In some embodiments, in step 103, the following steps may be specifically included:
step A1, caching the acquired multiple user requests in a queue mode to obtain a cache queue.
The obtained multiple user requests are cached in a queue mode, so that the cloud server can process the user requests in batches, the connection quantity is reduced, and the server pressure is relieved.
Step A2, classifying the cached queues according to the types of the user themes to obtain a plurality of sub-queues; each sub-queue corresponds to a unique user topic type.
By classifying the cached queues according to the types of the user topics, the cloud server can be facilitated to quickly access a database (including a Redis database or a MySQL database, for example). And storing data related to users needing to be read or written frequently by using a Redis database, and storing data related to server states which are not read or written frequently by using a MySQL database. MySQL plays a role in storing service information in the system, and stores collected terminal log data, service processing data and the like. The Redis is an open-source no-sql database based on key-value, data are cached in a memory, performance is greatly improved compared with a traditional relational database, and the Redis is particularly suitable for the condition that requirements on access speed and concurrency are high. Meanwhile, Redis also supports data persistence and supports various data structures such as list, set, hash and the like. Redis is employed herein to store relevant information such as keys for each user and quantity statistics related to topics to handle user requests at large concurrencies.
Step A3, traversing each sub-queue, and if the number of user requests in the sub-queue reaches a preset number threshold, packaging the sub-queue into a request set.
In this embodiment, by setting the buffer queue, when the number of the user requests in the sub-queue reaches a preset number threshold, the sub-queue is packaged into a request set, so that the user requests sent by different clients can be converted into the request set through batch processing, the throughput of the user requests can be increased, and further, the time consumed by network requests and the memory consumption can be reduced.
In other embodiments, step 103 may specifically include the following steps:
and step B1, caching the acquired multiple user requests in a queue mode to obtain a cache queue.
The detailed description may refer to the description in step a 1.
Step B2, classifying the cached queues according to the types of the user themes to obtain a plurality of sub-queues; each sub-queue corresponds to a unique user topic type.
The detailed description may refer to the description in step a 2.
And step B3, traversing each sub-queue, and packaging the sub-queue into a request set if the buffer time of the sub-queue reaches a preset time threshold.
In this embodiment, by setting the buffer queue, when the buffering time of the sub-queue reaches a preset duration threshold, the sub-queue is packaged into a request set, so that the user requests sent by different clients can be converted into the request set through batch processing, the throughput of the user requests can be increased, and further, the time consumed by network requests and the memory consumption can be reduced
Step 104: and reading the database according to the obtained request sets to obtain a reading result set corresponding to each request set.
Each reading result set comprises a plurality of reading results, and each reading result is a returned result requested by a user and obtained after the user accesses the database. For example, if the user request is a report key, the reading result includes report success and report failure; for another example, if the user request is to acquire a key, the read result includes successful acquisition, the acquired key, and failure in acquisition.
Step 105: and sending the obtained reading result set to an MQTT server.
After the obtained reading result set is sent to the MQTT server, the client initiating the request can obtain the reading result from the MQTT server according to the subscribed user topic, thereby completing the data processing.
In step 105, the method may specifically include the following steps:
step C1, splitting the obtained reading result set to obtain a plurality of reading results; each user request corresponds to a read result.
In this step, splitting the plurality of reading results is beneficial to returning a target reading result corresponding to a user request sent by one request-initiating client to the client, so as to realize quick acquisition of the reading result.
And step C2, sending the obtained multiple reading results to the MQTT server in a multi-thread parallel mode.
In this step, the obtained multiple reading results are sent to the MQTT server in a multi-thread parallel manner, so that the speed of sending the reading results to the MQTT server can be increased, and the pressure of the server can be reduced.
It can be seen that, in the process shown in fig. 1, by using the topic-subscription mechanism of the MQTT server, sending of user requests in any format can be realized, that is, user requests compatible with different formats can be realized, so that the universality of data processing is increased; moreover, when the client initiating the request reports the user request to the user theme of the MQTT server, the client does not need to wait for the result of data processing, so that an asynchronous request can be realized, which reduces available resources of the server for data processing, increases the throughput of the server, and thus realizes high concurrent processing. In addition, by processing the acquired multiple user requests in batch, time consumed by network requests and memory consumption can be reduced.
Fig. 2 shows a flow chart of a MQTT-based data processing method according to another embodiment. Referring to fig. 2, the method includes:
step 201: and acquiring a plurality of user themes collected by the MQTT server.
Step 202: and acquiring the user request of each user topic in a sharing subscription mode.
Step 203: and caching the acquired multiple user requests in a queue mode to obtain a cache queue.
Step 204: and classifying the cached queues according to the types of the user themes to obtain a plurality of sub-queues.
Step 205: and traversing each sub-queue, and packaging the sub-queue into a request set if the number of the user requests in the sub-queue reaches a preset number threshold.
Step 206: and traversing each sub-queue, and packaging the sub-queue into a request set if the buffer time of the sub-queue reaches a preset time threshold.
Step 207: and reading the database according to the obtained request sets to obtain a reading result set corresponding to each request set.
Step 208: and splitting the obtained reading result set to obtain a plurality of reading results.
Step 209: and sending the obtained multiple reading results to the MQTT server in a multi-thread parallel mode.
As shown in fig. 3 and 4, the present specification provides an MQTT-based data processing device and an MQTT-based data processing device. The device embodiments may be implemented by software, or by hardware, or by a combination of hardware and software. From a hardware aspect, as shown in fig. 3, a hardware structure diagram of the MQTT-based data processing apparatus provided in the embodiment of the present disclosure is that, in addition to the processor, the memory, the cable interface, and the nonvolatile memory shown in fig. 3, a device in which the apparatus is located in the embodiment may also include other hardware, such as a forwarding chip responsible for processing a packet, and the like. Taking a software implementation as an example, as shown in fig. 4, as a logical apparatus, the apparatus is formed by reading a corresponding computer program instruction in a non-volatile memory into a memory by a CPU of a device in which the apparatus is located and running the computer program instruction.
As shown in fig. 4, the MQTT-based data processing apparatus according to this embodiment is applied to an MQTT-based data processing apparatus, and includes:
a first obtaining module 401, configured to obtain a plurality of user topics collected by an MQTT server; each user theme carries a user request corresponding to the user theme;
a second obtaining module 402, configured to obtain, in a shared subscription manner, a user request of each user topic;
a batch processing module 403, configured to perform batch processing on the obtained multiple user requests according to the types of the user topics to obtain multiple request sets; each request set comprises a plurality of user requests, and each request set corresponds to a unique user theme type;
a reading module 404, configured to read the database according to the obtained request sets, to obtain a reading result set corresponding to each request set;
a sending module 405, configured to send the obtained reading result set to the MQTT server, so that the client initiating the request obtains the reading result from the MQTT server according to the subscribed user topic.
In this embodiment, the first obtaining module 401 may be configured to perform step 101 in the above-described method embodiment, the second obtaining module 402 may be configured to perform step 102 in the above-described method embodiment, the batch processing module 403 may be configured to perform step 103 in the above-described method embodiment, the reading module 404 may be configured to perform step 104 in the above-described method embodiment, and the sending module 405 may be configured to perform step 105 in the above-described method embodiment.
In one embodiment of the present description, the batch processing module is configured to perform the following operations:
caching the acquired multiple user requests in a queue mode to obtain a cache queue;
classifying the cached queues according to the types of the user themes to obtain a plurality of sub-queues; each sub-queue corresponds to a unique user theme type;
and traversing each sub-queue, and packaging the sub-queue into a request set if the number of the user requests in the sub-queue reaches a preset number threshold.
In one embodiment of the present description, the batch processing module is configured to perform the following operations:
caching the acquired multiple user requests in a queue mode to obtain a cache queue;
classifying the cached queues according to the types of the user themes to obtain a plurality of sub-queues; each sub-queue corresponds to a unique user theme type;
and traversing each sub-queue, and packaging the sub-queue into a request set if the buffer time of the sub-queue reaches a preset time threshold.
In one embodiment of the present description, the sending module is configured to perform the following operations:
splitting the obtained reading result set to obtain a plurality of reading results; each user request corresponds to a reading result;
and sending the obtained multiple reading results to the MQTT server in a multi-thread parallel mode.
It is to be understood that the illustrated structure of the embodiments of the present specification does not constitute a specific limitation to the MQTT-based data processing apparatus. In other embodiments of the present description, an MQTT-based data processing apparatus may include more or fewer components than shown, or combine certain components, or split certain components, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
For the information interaction, execution process, and other contents between the modules in the apparatus, the specific contents may refer to the description in the method embodiment of the present specification because the same concept is based on the method embodiment of the present specification, and are not described herein again.
An embodiment of the present specification further provides an MQTT-based data processing apparatus, including: at least one memory and at least one processor;
at least one memory for storing a machine readable program;
at least one processor for invoking a machine readable program to perform the MQTT-based data processing method in any of the embodiments of the present specification.
Embodiments of the present specification also provide a computer-readable medium storing instructions for causing a computer to perform the MQTT-based data processing method as described herein. Specifically, a method or an apparatus equipped with a storage medium on which a software program code that realizes the functions of any of the above-described embodiments is stored may be provided, and a computer (or a CPU or MPU) of the method or the apparatus is caused to read out and execute the program code stored in the storage medium.
In this case, the program code itself read from the storage medium can realize the functions of any of the above-described embodiments, and thus the program code and the storage medium storing the program code constitute a part of this specification.
Examples of the storage medium for supplying the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (e.g., CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD + RW), a magnetic tape, a nonvolatile memory card, and a ROM. Alternatively, the program code may be downloaded from a server computer via a communications network.
Further, it should be clear that the functions of any one of the above-described embodiments can be implemented not only by executing the program code read out by the computer, but also by performing a part or all of the actual operations by an operation method or the like operating on the computer based on instructions of the program code.
In summary, the MQTT-based data processing and apparatus provided by the embodiments of the present disclosure have at least the following advantages:
1. by utilizing a theme-subscription mechanism of the MQTT server, the user requests in any format can be sent, namely the user requests in different formats are compatible, so that the universality of data processing is increased; moreover, when the client initiating the request reports the user request to the user theme of the MQTT server, the client does not need to wait for the result of data processing, so that an asynchronous request can be realized, which reduces available resources of the server for data processing, increases the throughput of the server, and thus realizes high concurrent processing. In addition, by processing the acquired multiple user requests in batch, time consumed by network requests and memory consumption can be reduced.
2. The client initiating the request is connected with the MQTT server in a communication mode of the MQTT pipeline, and the cloud server is connected with the MQTT server in a subscription mode, so that the MQTT server is an intermediate server between the client initiating the request and the cloud server. By means of the topic-subscription mechanism of the MQTT, the user requests in any format can be sent, namely the user requests compatible with different formats can be realized, and therefore the universality of data processing is increased.
3. When the client initiating the request reports the user request to the user theme of the MQTT server, the client does not need to wait for the result of data processing, so that the asynchronous request can be realized, which can reduce the available resources of the server for data processing, increase the throughput of the server, and realize high concurrent processing.
4. The number of the cloud servers can be multiple, each cloud server is in communication connection with the MQTT server, namely the multiple cloud servers subscribe the same user theme, and when the MQTT server publishes the user theme, only one cloud server can receive the user request in the user theme. By the arrangement, the transverse expansion in the service can be realized, the elastic telescopic framework of the server under the high concurrent request can be realized, and the hot plug of the cloud server can be realized.
5. By processing the acquired multiple user requests in batch, time consumed by network requests and memory consumption can be reduced.
6. In this embodiment, splitting the plurality of reading results is beneficial to returning a target reading result corresponding to a user request sent by one request-initiating client to the client, so as to realize quick acquisition of the reading result. By sending the obtained multiple reading results to the MQTT server in a multi-thread parallel mode, the speed of sending the reading results to the MQTT can be increased, and the pressure of the server can be reduced.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The above embodiments are described in further detail for the purpose of illustrating the invention, and it should be understood that the above embodiments are only for illustrative purposes and are not intended to limit the scope of the invention, and any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present invention should be included in the scope of the present invention.
Claims (10)
1. The data processing method based on the MQTT is characterized by comprising the following steps:
acquiring a plurality of user themes collected by an MQTT server; each user theme carries a user request corresponding to the user theme;
acquiring a user request of each user topic in a sharing subscription mode;
the method comprises the steps of carrying out batch processing on a plurality of acquired user requests according to the types of user themes to obtain a plurality of request sets; each request set comprises a plurality of user requests, and each request set corresponds to a unique user theme type;
reading a database according to the obtained request sets to obtain a reading result set corresponding to each request set;
and sending the obtained reading result set to the MQTT server so that the client initiating the request obtains the reading result from the MQTT server according to the subscribed user theme.
2. The method according to claim 1, wherein the batch processing the obtained plurality of user requests according to the type of the user topic to obtain a plurality of request sets comprises:
caching the acquired multiple user requests in a queue mode to obtain a cache queue;
classifying the cached queues according to the types of the user themes to obtain a plurality of sub-queues; each sub-queue corresponds to a unique user theme type;
and traversing each sub-queue, and if the number of the user requests in the sub-queue reaches a preset number threshold, packaging the sub-queue into a request set.
3. The method according to claim 1, wherein the batch processing the obtained plurality of user requests according to the type of the user topic to obtain a plurality of request sets comprises:
caching the acquired multiple user requests in a queue mode to obtain a cache queue;
classifying the cached queues according to the types of the user themes to obtain a plurality of sub-queues; each sub-queue corresponds to a unique user theme type;
and traversing each sub-queue, and packaging the sub-queue into a request set if the buffer time of the sub-queue reaches a preset time threshold.
4. The method according to any one of claims 1-3, wherein the sending the obtained set of read results to the MQTT server comprises:
splitting the obtained reading result set to obtain a plurality of reading results; each user request corresponds to a reading result;
and sending the obtained multiple reading results to the MQTT server in a multi-thread parallel mode.
5. Data processing apparatus based on MQTT, characterized by comprising:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a plurality of user themes collected by an MQTT server; each user theme carries a user request corresponding to the user theme;
the second acquisition module is used for acquiring the user request of each user theme in a sharing subscription mode;
the batch processing module is used for carrying out batch processing on the obtained multiple user requests according to the types of the user themes to obtain multiple request sets; each request set comprises a plurality of user requests, and each request set corresponds to a unique user theme type;
the reading module is used for reading the database according to the obtained request sets to obtain a reading result set corresponding to each request set;
and the sending module is used for sending the obtained reading result set to the MQTT server so that the client initiating the request obtains the reading result from the MQTT server according to the subscribed user theme.
6. The apparatus of claim 5, wherein the batch processing module is configured to:
caching the acquired multiple user requests in a queue mode to obtain a cache queue;
classifying the cached queues according to the types of the user themes to obtain a plurality of sub-queues; each sub-queue corresponds to a unique user theme type;
and traversing each sub-queue, and if the number of the user requests in the sub-queue reaches a preset number threshold, packaging the sub-queue into a request set.
7. The apparatus of claim 5, wherein the batch processing module is configured to:
caching the acquired multiple user requests in a queue mode to obtain a cache queue;
classifying the cached queues according to the types of the user themes to obtain a plurality of sub-queues; each sub-queue corresponds to a unique user theme type;
and traversing each sub-queue, and packaging the sub-queue into a request set if the buffer time of the sub-queue reaches a preset time threshold.
8. The apparatus according to any of claims 5-7, wherein the sending module is configured to:
splitting the obtained reading result set to obtain a plurality of reading results; each user request corresponds to a reading result;
and sending the obtained multiple reading results to the MQTT server in a multi-thread parallel mode.
9. Computer readable storage medium, characterized in that it has stored thereon computer instructions which, when executed by a processor, cause the processor to carry out the method of any one of claims 1 to 4.
10. Data processing apparatus based on MQTT, characterized by comprising: at least one memory and at least one processor;
the at least one memory to store a machine readable program;
the at least one processor, configured to invoke the machine readable program to perform the method of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110290833.6A CN112948738A (en) | 2021-03-18 | 2021-03-18 | Data processing method and device based on MQTT |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110290833.6A CN112948738A (en) | 2021-03-18 | 2021-03-18 | Data processing method and device based on MQTT |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112948738A true CN112948738A (en) | 2021-06-11 |
Family
ID=76226592
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110290833.6A Pending CN112948738A (en) | 2021-03-18 | 2021-03-18 | Data processing method and device based on MQTT |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112948738A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113901091A (en) * | 2021-11-19 | 2022-01-07 | 北京达佳互联信息技术有限公司 | List generation method and device, electronic equipment and computer readable medium |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107809489A (en) * | 2017-11-17 | 2018-03-16 | 南京感度信息技术有限责任公司 | A kind of message push system based on MQTT agreements |
| CN109088723A (en) * | 2018-10-26 | 2018-12-25 | 四川长虹电器股份有限公司 | A kind of long-range control method based on MQTT agreement |
| CN110086636A (en) * | 2019-04-17 | 2019-08-02 | 平安科技（深圳）有限公司 | A kind of message distributing method based on mqtt, server and storage medium |
| CN110460638A (en) * | 2019-07-11 | 2019-11-15 | 深圳职业技术学院 | Information push method and device |
| CN110661871A (en) * | 2019-09-27 | 2020-01-07 | 宁波和利时智能科技有限公司 | Data transmission method and MQTT server |
| CN110673502A (en) * | 2019-10-14 | 2020-01-10 | 珠海格力电器股份有限公司 | Quick response method and system of intelligent device, storage medium and network side device |
| CN110913000A (en) * | 2019-11-27 | 2020-03-24 | 浙江华诺康科技有限公司 | Method, system and computer readable storage medium for processing service information |
| CN111131426A (en) * | 2019-12-19 | 2020-05-08 | 浙江百应科技有限公司 | MQTT data interaction based method, terminal and server |
| CN111131501A (en) * | 2019-12-31 | 2020-05-08 | 郑州信大捷安信息技术股份有限公司 | MQTT protocol-based message pushing system and method |
| CN111400390A (en) * | 2020-04-08 | 2020-07-10 | 上海东普信息科技有限公司 | Data processing method and device |
| CN111416867A (en) * | 2020-03-25 | 2020-07-14 | 上海商米科技集团股份有限公司 | Method for processing messages between different devices, server and computer storage medium |
| CN112422544A (en) * | 2020-11-09 | 2021-02-26 | 深圳市华昊软件有限公司 | Machine room equipment information monitoring system based on MQTT communication protocol |
-
2021
- 2021-03-18 CN CN202110290833.6A patent/CN112948738A/en active Pending
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107809489A (en) * | 2017-11-17 | 2018-03-16 | 南京感度信息技术有限责任公司 | A kind of message push system based on MQTT agreements |
| CN109088723A (en) * | 2018-10-26 | 2018-12-25 | 四川长虹电器股份有限公司 | A kind of long-range control method based on MQTT agreement |
| CN110086636A (en) * | 2019-04-17 | 2019-08-02 | 平安科技（深圳）有限公司 | A kind of message distributing method based on mqtt, server and storage medium |
| CN110460638A (en) * | 2019-07-11 | 2019-11-15 | 深圳职业技术学院 | Information push method and device |
| CN110661871A (en) * | 2019-09-27 | 2020-01-07 | 宁波和利时智能科技有限公司 | Data transmission method and MQTT server |
| CN110673502A (en) * | 2019-10-14 | 2020-01-10 | 珠海格力电器股份有限公司 | Quick response method and system of intelligent device, storage medium and network side device |
| CN110913000A (en) * | 2019-11-27 | 2020-03-24 | 浙江华诺康科技有限公司 | Method, system and computer readable storage medium for processing service information |
| CN111131426A (en) * | 2019-12-19 | 2020-05-08 | 浙江百应科技有限公司 | MQTT data interaction based method, terminal and server |
| CN111131501A (en) * | 2019-12-31 | 2020-05-08 | 郑州信大捷安信息技术股份有限公司 | MQTT protocol-based message pushing system and method |
| CN111416867A (en) * | 2020-03-25 | 2020-07-14 | 上海商米科技集团股份有限公司 | Method for processing messages between different devices, server and computer storage medium |
| CN111400390A (en) * | 2020-04-08 | 2020-07-10 | 上海东普信息科技有限公司 | Data processing method and device |
| CN112422544A (en) * | 2020-11-09 | 2021-02-26 | 深圳市华昊软件有限公司 | Machine room equipment information monitoring system based on MQTT communication protocol |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113901091A (en) * | 2021-11-19 | 2022-01-07 | 北京达佳互联信息技术有限公司 | List generation method and device, electronic equipment and computer readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111641583B (en) | Internet of things resource access system and resource access method | |
| CN106603598B (en) | Method and device for processing service request | |
| CN110677277B (en) | Data processing method, device, server and computer readable storage medium | |
| US11080090B2 (en) | Method and system for scalable job processing | |
| US20080177872A1 (en) | Managing aggregation and sending of communications | |
| CN102307153A (en) | Virtual desktop transmission device and method | |
| CN110768994B (en) | Method for improving SIP gateway performance based on DPDK technology | |
| CN109547162B (en) | Data communication method based on two sets of one-way boundaries | |
| CN109547519B (en) | Reverse proxy method, apparatus and computer readable storage medium | |
| CN112019597B (en) | Distributed data receiving system and data receiving method | |
| CN113641410A (en) | Netty-based high-performance gateway system processing method and system | |
| WO2017185615A1 (en) | Method for determining service status of service processing device and scheduling device | |
| CN111200606A (en) | Deep learning model task processing method, system, server and storage medium | |
| CN112104679B (en) | Method, apparatus, device and medium for processing hypertext transfer protocol request | |
| CN111049915B (en) | Container under-cloud message queue agent system and method | |
| CN116405547A (en) | Message pushing method and device, processor, electronic equipment and storage medium | |
| CN112948738A (en) | Data processing method and device based on MQTT | |
| CN105208004A (en) | Data input method based on OBD equipment | |
| CN115834665B (en) | Network communication method and device | |
| CN105099645A (en) | Multi-user concurrent communication method and device based on half-duplex communication device | |
| CN108234595B (en) | Log transmission method and system | |
| CN108076111B (en) | System and method for distributing data in big data platform | |
| CN111477229B (en) | Voice recognition request processing method and device | |
| CN113992644A (en) | Internet of things gateway system based on non-service technology and data processing method thereof | |
| CN113938296A (en) | Network data access system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210611 |