CN115865990B - High-performance Internet of things platform message engine - Google Patents
High-performance Internet of things platform message engine Download PDFInfo
- Publication number
- CN115865990B CN115865990B CN202310145675.4A CN202310145675A CN115865990B CN 115865990 B CN115865990 B CN 115865990B CN 202310145675 A CN202310145675 A CN 202310145675A CN 115865990 B CN115865990 B CN 115865990B
- Authority
- CN
- China
- Prior art keywords
- message
- data
- queue
- processing module
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention relates to the technical field of equipment message exchange, in particular to a high-performance Internet of things platform message engine which comprises a receiving module, a primary queue module, a processing module, a secondary queue module, a service module and a sending module. The method and the device reduce the packet body size of each message data by packaging and compressing the message data, monitor the real-time total data volume of the first message queue through the processing module, adjust the first message queue, determine whether to adjust the position of the message data in the first message queue according to the compressed packet data volume of any message data, adjust the processing sequence according to the capacity of the first message queue, improve the processing efficiency of the message data, realize high concurrency of message engines, transmit the data through a lightweight protocol, reduce flow transmission, transfer massive uplink data into the first message queue, process the data through the processing module and provide the processed data to a user system, and realize high throughput.
Description
Technical Field
The invention relates to the technical field of equipment message exchange, in particular to a high-performance message engine of an Internet of things platform.
Background
Messages are sent to a message engine, the message engine is a container for storing the messages in the transmission process of the messages, the message engine serves as a middle man when relaying the messages from a source to a target, the main purpose of a queue is to provide routing and guarantee the delivery of the messages, if a receiver is unavailable when sending the messages, the message queue can retain the messages until the messages can be successfully delivered, and when the message engine is connected simultaneously and receives online reported data of a plurality of devices, the messages in the message queue need to be efficiently transferred and processed, otherwise, internal error reporting can be caused, and the whole operation of the message engine is even influenced.
Chinese patent publication No.: CN102654922A, discloses a message management engine; the technical point is that a message management engine is used for managing rendering actions and rendering opportunities of three-dimensional objects in the three-dimensional graphic engine, and the working efficiency of the three-dimensional graphic engine is optimized, so that in the prior art, the message engine mostly adopts priority message data to improve the message processing efficiency of the engine, but when multiple devices report data on line at the same time, a large amount of calculation power is consumed, the situation of engine errors is easy to occur, and the message processing efficiency of the engine cannot be guaranteed.
Disclosure of Invention
Therefore, the invention provides a high-performance message engine of an internet of things platform, which is used for solving the problem of low message processing efficiency of simultaneous online data reporting of multiple devices in the prior art.
In order to achieve the above object, the present invention provides a high-performance internet of things platform message engine, including,
the receiving module is connected with an external equipment end and used for receiving message data sent by the equipment end, packing and compressing the message data and unloading the message data according to the data name of the received message data;
the first-level queue module is connected with the receiving module and used for discharging the message data transferred by the receiving module to the tail end of the first message queue;
the processing module is connected with the first-stage queue module and used for decompressing all message data in the first message queue according to the sequence of the first message queue to form all complete message data and unloading the complete message data, the processing module can judge the real-time total data volume of the first message queue according to the first preset data volume and the second preset data volume which are internally arranged so as to determine whether to adjust the first message queue, and the processing module can also judge the compressed packet data volume of any message data in the first message queue so as to determine whether to adjust the position of the message data in the first message queue;
the second-level queue module is connected with the processing module and used for discharging the complete message data transferred by the processing module into a second message queue;
the business module is connected with the secondary queue module and used for consuming all complete message data in the second message queue according to the sequence of the second message queue and sending out the complete message data;
and the sending module is respectively connected with the service module and the receiving module and is used for receiving the complete message data generated by the service module, and the sending module sends the complete message data after receiving the character confirmation sent by the receiving module.
Further, a first preset data volume L1 and a second preset data volume L2 are arranged in the processing module, wherein L1 is less than L2, before the processing module decompresses each message data in the first message queue, the processing module obtains a real-time total data volume Ls of the first message queue, and compares the real-time total data volume Ls with the first preset data volume L1 and the second preset data volume L2,
when Ls is less than L1, the processing module judges that the real-time total data volume of the first message queue is lower than a first preset data volume, and the processing module does not adjust the first message queue of the primary queue module;
when the Ls is more than or equal to L1 and less than or equal to L2, the processing module determines that the real-time total data volume of the first message queue is between a first preset data volume and a second preset data volume, and the processing module determines the compressed packet data volume of each message data in the first message queue so as to adjust the first message queue of the first-stage queue module;
and when Ls is greater than L2, the processing module judges that the real-time total data volume of the first message queue is higher than a second preset data volume, and the processing module adjusts the first message queue of the first-stage queue module.
Further, a standard message data amount Lb and a standard message data amount difference Δ Lb are arranged in the processing module, when the processing module determines that the real-time total data amount of the first message queue is between the first preset data amount and the second preset data amount, the processing module determines the compressed packet data amount of each message data from the end of the first message queue, for the compressed packet data amount Ly of any message data, the processing module calculates the compressed packet data amount difference Δ Ly, Δ Ly = | Lb-Ly | according to the compressed packet data amount Ly and the standard message data amount Lb, and compares the compressed packet data amount difference Δ Ly with the standard message data amount difference Δ Lb,
when the delta Ly is less than or equal to the delta Lb, the processing module judges that the compressed packet data volume difference delta Ly of the message data does not exceed the standard message data volume difference delta Lb, and the processing module does not adjust the position of the message data in the first message queue;
when delta Ly is larger than delta Lb, the processing module judges that the compressed packet data quantity difference delta Ly of the message data exceeds the standard message data quantity difference delta Lb, and the processing module compares the compressed packet data quantity Ly of the message data with the standard message data quantity Lb to adjust the position of the message data in the first message queue.
Further, when the processing module judges that the compressed packet data quantity difference Delta Ly of the message data exceeds the standard message data quantity difference Delta Lb, the processing module compares the compressed packet data quantity Ly of the message data with the standard message data quantity Lb,
when Ly is less than Lb, the processing module judges that the compressed packet data quantity Ly of the message data is less than the standard message data quantity Lb, the processing module removes the message data in the first message queue, and the first-stage queue module discharges the message data to the front end of the removed message queue;
when Ly is larger than Lb, the processing module judges that the compressed packet data quantity Ly of the message data is higher than the standard message data quantity Lb, and the processing module removes the message data from the first message queue and inserts the message data into the front end of the first message queue.
Further, the processing module is internally provided with a maximum shifted-out data volume Lz, when a shifted-out message queue exists in the primary queue module, the processing module obtains the real-time removed total data volume Lc of the shifted-out message queue and compares the real-time removed total data volume Lc with the maximum shifted-out data volume Lz,
when Lc is less than Lz, the processing module determines that the total real-time removed data volume of the removed message queue does not reach the maximum removed data volume, and the processing module determines whether to decompress and transfer the removed message queue according to the total real-time data volume of the first message queue and a first preset data volume;
when Lc is larger than or equal to Lz, the processing module determines that the real-time total data removal amount of the removed message queue reaches the maximum removed data amount, the processing module stops decompressing and transferring each message data in the first message queue, decompresses and transfers each message data in the removed message queue according to the sequence of the removed message queue, and starts decompressing and transferring each message data in the first message queue until all message data in the removed message queue are decompressed and transferred.
Further, when the processing module determines that the real-time removed total data amount of the removed message queue does not reach the maximum removed data amount, the processing module compares the obtained real-time total data amount Ls' of the first message queue with a first preset data amount L1,
if Ls' is less than L1, the processing module judges that the real-time total data volume of the first message queue does not reach a first preset data volume, the processing module stops decompressing and transferring each message data in the first message queue, decompresses and transfers each message data in the shifted-out message queue according to the sequence of the shifted-out message queue, and starts decompressing and transferring each message data in the first message queue until all message data in the shifted-out message queue are decompressed and transferred;
if Ls' is greater than or equal to L1, the processing module judges that the real-time total data volume of the first message queue reaches a first preset data volume, and the processing module does not decompress and dump the removed message queue.
Further, when the processing module determines that the real-time total data volume of the first message queue is higher than the second preset data volume, the processing module acquires the data volume of each message data in the first message queue, sorts each message data according to the data volume from high to low, and if two or more message data have equal data volume, the processing module acquires the receiving time node of each message data with equal data volume, calculates the time difference between the receiving time node and the current time node, and sorts each message data with equal data volume from high to low according to the time difference.
Further, a preset interception character length Wj and a preset screening character length Ws are arranged in the receiving module, wherein Wj > Ws, a preset interception frequency Q is also arranged in the receiving module, when the receiving module receives message data a sent by the device end, the receiving module intercepts the message data a for Q times, so as to form Q character segments of the message data a, the receiving module combines the Q character segments of the message data a to form a check character segment Ka of the message data a, and if a continuous repeated character segment part with a character length higher than the preset screening character length Ws exists in the check character segment Ka, the receiving module deletes and retains a repeated character segment part.
Furthermore, a preset check character length Wy is also arranged in the receiving module, when the receiving module forms a check character section Ka of the message data A, the receiving module acquires the real-time character length Wa of the check character section Ka and compares the real-time character length Wa with the preset check character length Wy,
when Wa is less than Wy, the receiving module judges that the real-time character length of the check character segment Ka does not reach the preset check character length, deletes the check character segment Ka and repeats the operation of intercepting the character segment with the length Wj for Q times on the message data A;
and when Wa is larger than or equal to Wy, the receiving module judges that the real-time character length of the check character section Ka reaches the preset check character length, and the receiving module acquires the data name Ma of the message data A and stores the data name Ma and the check character section Ka.
Furthermore, a confirmation similarity Gq is also arranged in the receiving module, when the sending module receives the complete message data generated by the service module, the sending module acquires the data name of the complete message data, matches the data name in the data name stored in the receiving module according to the data name, selects the corresponding data name and the check character segment, the receiving module calculates the check similarity Gs between the check character segment and the complete message data, and compares the check similarity Gs with the confirmation similarity Gq,
when Gs is smaller than Gq, the receiving module judges that the check similarity does not reach the confirmation similarity, the receiving module sends error information to the equipment end, and receives message data sent by the equipment end again;
and when Gs is larger than or equal to Gq, the receiving module judges that the check similarity reaches the confirmation similarity, the receiving module sends character confirmation to the sending module, deletes the stored corresponding data name and the check character segment, and the sending module sends the complete message data.
Compared with the prior art, the message data processing method has the advantages that the message data sent by the equipment end are received through the receiving module, the message data are packaged and compressed, the size of the packet body of each message data is reduced, the packaged and compressed message data are discharged into the first message queue through the first-level queue module, the message data are temporarily stored and are subjected to unloading and decompression through the processing module, the real-time total data volume of the first message queue is monitored through the processing module, the first message queue is adjusted, whether the position of the message data in the first message queue is adjusted or not is determined according to the compressed packet data volume of any message data, the processing sequence is adjusted according to the capacity of the first message queue, the message data processing efficiency is improved, high concurrency of message engines is achieved, and the connection number of equipment can be greatly improved due to the characteristic of light-weight processes; meanwhile, the operation error report of a single connection module does not influence the overall operation of the system, each thread does not influence each other, each thread does not depend on shared memory, high fault tolerance is realized, data is transmitted through a lightweight protocol such as MQTT, flow transmission is reduced, massive uplink data is transferred to a first message queue, the data is processed and then provided to a user system, high throughput is realized, the cluster characteristic of Erlang is used, the cross-node subscription message of the equipment is released, the joining and the quitting of a new node are also supported, high expansibility is realized, multi-node deployment is used for forming a service cluster, the equipment selects available nodes through LBS to connect and communicate, and the high availability of a message engine is realized even if a certain node is crashed.
Furthermore, the real-time total data volume of the first message queue in the primary queue module is monitored by setting the first preset data volume and the second preset data volume in the processing module, the stability of the space reservation volume in the primary queue module is guaranteed by adjusting the processing sequence of the first message queue control message data, the processing speed of the processing module is stable, the throughput stability of a message engine is guaranteed, and the processing efficiency of the message data is improved.
Further, when the real-time total data volume of the first message queue is between a first preset data volume and a second preset data volume, the total data volume of the first message queue is about to exceed a standard throughput range of a message engine, so that the first message queue is screened from end messages, a range of a standard message data volume is formed by setting a difference between the standard message data volume and the standard message data volume, the deviation degree of the compression packet data volume of the judged message data and the standard message data volume is represented by calculating a difference between the compression packet data volumes, and when the difference between the compression packet data volumes does not exceed the difference between the standard message data volumes, the message data is represented in the standard range, so that the position of the message data is not adjusted, and the stability of decompression processing of the processing module is guaranteed.
Particularly, when the compressed packet data amount difference of the message data exceeds the standard message data amount difference, the compressed packet data amount and the standard deviation are large, so that the compressed packet data amount of the message data is compared with the standard message data amount to determine the actual size of the compressed packet data amount, when the compressed packet data amount is lower than the standard message data amount, the message data is small, the whole data amount of the first message queue cannot be rapidly reduced by processing the message data, error report caused by the message data amount in the first-stage queue module is not avoided, the message data is not preferentially processed, when the compressed packet data amount is higher than the standard message data amount, the message data is shifted out from the first message queue and inserted into the front end, and because the first message queue is formed according to the receiving sequence of the first-stage queue module, the message data with the large compressed packet data amount are sequentially inserted into the front end of the first message queue, and the timeliness of the message data is guaranteed while priority processing is guaranteed.
Furthermore, whether the decompression and the dump processing are carried out on the removed message queue or not is judged by comparing the real-time removed total data volume with the maximum removed data volume, and since the message data in the removed message queue are all small data volumes, the centralized rapid decompression and the dump processing are adopted, meanwhile, the processing efficiency of the message data can be improved by adding a processing module behind the primary queue module, the decompression and the dump processing are carried out on the removed message queue according to the maximum removed data volume, and the timeliness of all the message data in the removed message queue is guaranteed.
Particularly, when the real-time removal total data volume of the removed message queue is judged not to reach the maximum removed data volume, the current real-time total data volume and the first preset data volume of the first message queue are judged, the state of the first message queue is determined, and if the real-time total data volume of the first message queue does not reach the first preset data volume, the reserved space of the primary queue module is represented for recovery, so that the message data in the removed message queue are processed in a centralized mode through the processing module, and the normal operation of a message engine is guaranteed.
Further, when the real-time total data volume of the first message queue is higher than the second preset data volume, the first-level queue module is represented to be in a saturation state, so that the first message queue in the first-level queue module is forced to be reordered from high to low in data volume through the processing module, message data with large data volume is processed preferentially, the real-time data volume of the first message queue is reduced rapidly, the reserved space of the first-level queue module is replaced by the time of each message data, and the stable operation of a message engine is guaranteed.
Furthermore, the message data is received and the check character segments are intercepted by the receiving module, so that the accuracy of message data verification is guaranteed, meanwhile, when preset repeated parts appear in the intercepted character segments of the song, one repeated part is reserved as a characteristic, other repeated parts are deleted, the data amount of processing is reduced, and the message processing efficiency is improved.
Furthermore, the generated check character segment is judged by setting the length of the preset check character, so that the problem that the accuracy of message data verification is reduced due to the fact that the generated check character segment is too short is solved.
And further, performing similarity verification on the complete message data by using the generated verification character segment to determine that the conversion of the message data is stable, wherein when the verification similarity does not reach the confirmed similarity, the message data in the queue transfer and decompression of the standard message data is damaged to cause message data conversion errors, so that the corresponding message data needs to be acquired again, and when the verification similarity reaches the confirmed similarity, the conversion of the complete message data is completed and the complete message data can be transmitted, so that the accuracy of message data processing is ensured.
Drawings
Fig. 1 is a schematic diagram of a high-performance internet of things platform message engine according to this embodiment.
Detailed Description
In order that the objects and advantages of the invention will be more clearly understood, the invention is further described below with reference to examples; it should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only for explaining the technical principle of the present invention, and do not limit the scope of the present invention.
It should be noted that in the description of the present invention, the terms of direction or positional relationship indicated by the terms "upper", "lower", "left", "right", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, which are only for convenience of description, and do not indicate or imply that the device or element must have a specific orientation, be constructed in a specific orientation, and be operated, and thus, should not be construed as limiting the present invention.
Furthermore, it should be noted that, in the description of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
Referring to fig. 1, which is a schematic diagram of a high-performance platform message engine of the internet of things according to the present embodiment, the present embodiment discloses a high-performance platform message engine of the internet of things, including,
the receiving module is connected with an external equipment end and used for receiving message data sent by the equipment end, packing and compressing the message data and unloading the message data according to the data name of the received message data;
the first-level queue module is connected with the receiving module and used for discharging the message data transferred by the receiving module to the tail end of the first message queue;
the processing module is connected with the first-stage queue module and used for decompressing all message data in the first message queue according to the sequence of the first message queue to form all complete message data and unloading the complete message data, the processing module can judge the real-time total data volume of the first message queue according to the first preset data volume and the second preset data volume which are internally arranged so as to determine whether to adjust the first message queue, and the processing module can also judge the compressed packet data volume of any message data in the first message queue so as to determine whether to adjust the position of the message data in the first message queue;
the second-level queue module is connected with the processing module and used for discharging the complete message data transferred by the processing module into a second message queue;
the business module is connected with the secondary queue module and used for consuming all complete message data in the second message queue according to the sequence of the second message queue and sending out the complete message data;
and the sending module is respectively connected with the service module and the receiving module and is used for receiving the complete message data generated by the service module, and the sending module sends the complete message data after receiving the character confirmation sent by the receiving module.
The message data sent by the equipment end is received by setting a receiving module, the message data is packed and compressed, the size of a bag body of each message data is reduced, the packed and compressed message data is discharged into a first message queue through a first-stage queue module, the message data is temporarily stored and is subjected to dump and decompression by a processing module, the real-time total data volume of the first message queue is monitored through the processing module, the first message queue is adjusted, whether the position of the message data in the first message queue is adjusted or not is determined according to the compressed packet data volume of any message data, the processing sequence is adjusted according to the capacity of the first message queue, the processing efficiency of the message data is improved, high concurrency of message engines is realized, and the connection number of the equipment can be greatly improved due to the characteristic of a lightweight process; meanwhile, the operation error reporting of a single connection module does not influence the overall operation of the system, each thread does not influence each other, and each thread does not depend on shared memory, so that high fault tolerance is realized, data is transmitted through a lightweight protocol such as MQTT, flow transmission is reduced, massive uplink data is transferred to a first message queue, the data is processed and then provided to a user system, high throughput is realized, meanwhile, the cluster characteristic of Erlang is used, the cross-node publishing and subscribing information of the equipment is realized, the adding and exiting of new nodes are also supported, high expansibility is realized, a service cluster is formed by multi-node deployment, the equipment selects available nodes through LBS to connect and communicate, and even if a certain node is crashed, the communication of other equipment is not influenced, so that the high availability of a message engine is realized.
Specifically, a first preset data volume L1 and a second preset data volume L2 are arranged in the processing module, wherein L1 is smaller than L2, before the processing module decompresses each message data in the first message queue, the processing module acquires a real-time total data volume Ls of the first message queue, compares the real-time total data volume Ls with the first preset data volume L1 and the second preset data volume L2,
when Ls is less than L1, the processing module judges that the real-time total data volume of the first message queue is lower than a first preset data volume, and the processing module does not adjust the first message queue of the primary queue module;
when the Ls is more than or equal to L1 and less than or equal to L2, the processing module determines that the real-time total data volume of the first message queue is between a first preset data volume and a second preset data volume, and the processing module determines the compressed packet data volume of each message data in the first message queue so as to adjust the first message queue of the first-stage queue module;
and when Ls is greater than L2, the processing module judges that the real-time total data volume of the first message queue is higher than a second preset data volume, and the processing module adjusts the first message queue of the first-stage queue module.
The real-time total data volume of the first message queue in the primary queue module is monitored by setting the first preset data volume and the second preset data volume in the processing module, the stability of the space reservation volume in the primary queue module is guaranteed by adjusting the processing sequence of the first message queue control message data, the processing speed stability of the processing module can be guaranteed, the throughput stability of a message engine is guaranteed, and the processing efficiency of the message data is improved.
Specifically, a standard message data amount Lb and a standard message data amount difference Δ Lb are arranged in the processing module, when the processing module determines that the real-time total data amount of the first message queue is between a first preset data amount and a second preset data amount, the processing module determines the compressed packet data amount of each message data from the end of the first message queue, for the compressed packet data amount Ly of any message data, the processing module calculates the compressed packet data amount difference Δ Ly, Δ Ly = | Lb-Ly | according to the compressed packet data amount Ly and the standard message data amount Lb, and compares the compressed packet data amount difference Δ Ly with the standard message data amount difference Δ Lb,
when the delta Ly is less than or equal to the delta Lb, the processing module judges that the compressed packet data volume difference delta Ly of the message data does not exceed the standard message data volume difference delta Lb, and the processing module does not adjust the position of the message data in the first message queue;
when delta Ly is larger than delta Lb, the processing module judges that the compressed packet data quantity difference delta Ly of the message data exceeds the standard message data quantity difference delta Lb, and the processing module compares the compressed packet data quantity Ly of the message data with the standard message data quantity Lb to adjust the position of the message data in the first message queue.
When the real-time total data volume of the first message queue is between a first preset data volume and a second preset data volume, the total data volume of the first message queue is shown to exceed the standard throughput range of a message engine, therefore, the first message queue is screened from the end message, the range of the standard message data volume is formed by setting the difference between the standard message data volume and the standard message data volume, the deviation degree of the compression packet data volume of the judged message data and the standard message data volume is shown by calculating the difference between the compression packet data volumes, and when the difference between the compression packet data volumes does not exceed the difference between the standard message data volumes, the message data is shown to be in the standard range, so that the position adjustment is not carried out, and the stability of the decompression processing of the processing module is guaranteed.
Specifically, when the processing module judges that the compressed packet data amount difference Δ Ly of the message data exceeds the standard message data amount difference Δ Lb, the processing module compares the compressed packet data amount Ly of the message data with the standard message data amount Lb,
when Ly is less than Lb, the processing module judges that the compressed packet data quantity Ly of the message data is less than the standard message data quantity Lb, the processing module removes the message data in the first message queue, and the first-stage queue module discharges the message data to the front end of the removed message queue;
when Ly is larger than Lb, the processing module judges that the compressed packet data quantity Ly of the message data is higher than the standard message data quantity Lb, and the processing module removes the message data from the first message queue and inserts the message data into the front end of the first message queue.
When the compressed packet data amount difference of the message data exceeds the standard message data amount difference, the compressed packet data amount and the standard deviation are larger, so that the compressed packet data amount of the message data is compared with the standard message data amount, the actual size of the compressed packet data amount is determined, when the compressed packet data amount is lower than the standard message data amount, the message data is smaller, the whole data amount of the first message queue cannot be rapidly reduced by processing the message data, error report caused by the message data amount in the first-stage queue module is not avoided, the message data is not preferentially processed, when the compressed packet data amount is higher than the standard message data amount, the message data is shifted out from the first message queue and inserted into the front end, and because the first message queue is formed according to the receiving sequence of the first-stage queue module, the message data with the larger compressed packet data amount are sequentially inserted into the front end of the first message queue, and the timeliness of the message data is guaranteed while priority processing is guaranteed.
Specifically, the processing module is internally provided with a maximum removed data volume Lz, when a removed message queue exists in the primary queue module, the processing module obtains a real-time removed total data volume Lc of the removed message queue and compares the real-time removed total data volume Lc with the maximum removed data volume Lz,
when Lc is less than Lz, the processing module judges that the real-time removal total data volume of the removed message queue does not reach the maximum removed data volume, and the processing module judges whether to decompress and dump the removed message queue according to the real-time total data volume of the first message queue and a first preset data volume;
when Lc is larger than or equal to Lz, the processing module determines that the real-time total data removal amount of the removed message queue reaches the maximum removed data amount, the processing module stops decompressing and unloading each message data in the first message queue, decompresses and unloads each message data in the removed message queue according to the sequence of the removed message queue, and starts decompressing and unloading each message data in the first message queue until all message data in the removed message queue are decompressed and unloaded.
The total data volume removed in real time is compared with the maximum removed data volume, whether decompression and unloading processing is carried out on the removed message queue or not is judged, since the message data in the removed message queue are all small data volumes, concentrated rapid decompression and unloading processing is adopted, meanwhile, a processing module can be added behind a primary queue module to improve the processing efficiency of the message data, decompression and unloading processing is carried out on the removed message queue according to the maximum removed data volume, and timeliness of all message data in the removed message queue is guaranteed.
Specifically, when the processing module determines that the real-time removed total data amount of the removed message queue does not reach the maximum removed data amount, the processing module compares the acquired real-time total data amount Ls' of the first message queue with a first preset data amount L1,
if Ls' is less than L1, the processing module judges that the real-time total data volume of the first message queue does not reach a first preset data volume, the processing module stops decompressing and transferring each message data in the first message queue, decompresses and transfers each message data in the shifted-out message queue according to the sequence of the shifted-out message queue, and starts decompressing and transferring each message data in the first message queue until all message data in the shifted-out message queue are decompressed and transferred;
if Ls' is greater than or equal to L1, the processing module judges that the real-time total data volume of the first message queue reaches a first preset data volume, and the processing module does not decompress and dump the removed message queue.
When the real-time removal total data volume of the removed message queue is judged not to reach the maximum removed data volume, the current real-time total data volume and the first preset data volume of the first message queue are judged, the state of the first message queue is determined, and if the real-time total data volume of the first message queue does not reach the first preset data volume, the reserved space of the first-stage queue module is represented for recovery, so that the message data in the removed message queue are processed in a centralized mode through the processing module, and the normal operation of a message engine is guaranteed.
Specifically, when the processing module determines that the real-time total data volume of the first message queue is higher than the second preset data volume, the processing module acquires the data volume of each message data in the first message queue and sorts each message data according to the data volume from high to low, and if two or more message data have equal data volumes, the processing module acquires the receiving time nodes of each message data with equal data volumes, calculates the time difference between the receiving time nodes and the current time node, and sorts each message data with equal data volumes from high to low according to the time difference.
When the real-time total data volume of the first message queue is higher than the second preset data volume, the first-level queue module is represented to be in a saturation state, so that the first message queue in the first-level queue module is forced to be reordered from high to low in data volume through the processing module, message data with larger data volume is processed preferentially, the real-time data volume of the first message queue is reduced rapidly, the reserved space of the first-level queue module is replaced by the time of each message data, and the stable operation of a message engine is guaranteed.
Specifically, a preset interception character length Wj and a preset screening character length Ws are arranged in the receiving module, wherein Wj > Ws, a preset interception frequency Q is also arranged in the receiving module, when the receiving module receives message data a sent by an equipment end, the receiving module intercepts the message data a for Q times, so as to form Q character segments of the message data a, the receiving module combines the Q character segments of the message data a to form a check character segment Ka of the message data a, and if a continuous repeated character segment part with a character length higher than the preset screening character length Ws exists in the check character segment Ka, the receiving module deletes and retains a repeated character segment part.
The message data are received and the character segments are checked through the receiving module, so that the accuracy of message data verification is guaranteed, meanwhile, when preset repeated parts appear in each character segment of the intercepted song, one repeated part is reserved as a characteristic, other repeated parts are deleted, the data amount of processing is reduced, and the message processing efficiency is improved.
Specifically, a preset check character length Wy is also arranged in the receiving module, when the receiving module forms the check character section Ka of the message data A, the receiving module acquires the real-time character length Wa of the check character section Ka and compares the real-time character length Wa with the preset check character length Wy,
when Wa is less than Wy, the receiving module judges that the real-time character length of the check character segment Ka does not reach the preset check character length, deletes the check character segment Ka and repeats the operation of intercepting the character segment with the length Wj for Q times on the message data A;
and when Wa is larger than or equal to Wy, the receiving module judges that the real-time character length of the check character segment Ka reaches the preset check character length, and the receiving module acquires the data name Ma of the message data A and stores the data name Ma and the check character segment Ka.
The generated check character segment is judged by setting the length of the preset check character, so that the problem that the accuracy of message data verification is reduced due to the fact that the generated check character segment is too short is solved.
Specifically, the receiving module is further provided with a confirmation similarity Gq, when receiving the complete message data generated by the service module, the sending module acquires the data name of the complete message data, matches the data name in the data name stored in the receiving module according to the data name, selects the corresponding data name and the check character segment, the receiving module calculates the check similarity Gs between the check character segment and the complete message data, and compares the check similarity Gs with the confirmation similarity Gq,
when Gs is smaller than Gq, the receiving module judges that the check similarity does not reach the confirmation similarity, the receiving module sends error information to the equipment end, and receives message data sent by the equipment end again;
and when Gs is larger than or equal to Gq, the receiving module judges that the check similarity reaches the confirmation similarity, the receiving module sends character confirmation to the sending module, deletes the stored corresponding data name and the check character segment, and the sending module sends the complete message data.
And when the verification similarity reaches the confirmation similarity, the conversion of the complete message data is finished and can be sent out, so that the accuracy of message data processing is ensured.
So far, the technical solutions of the present invention have been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of the present invention is obviously not limited to these specific embodiments. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can fall into the protection scope of the invention.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention; various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A high-performance Internet of things platform message engine is characterized by comprising,
the receiving module is connected with an external equipment end and used for receiving message data sent by the equipment end, packing and compressing the message data and unloading the message data according to the data name of the received message data;
the first-level queue module is connected with the receiving module and used for discharging the message data transferred by the receiving module to the tail end of the first message queue;
the processing module is connected with the first-stage queue module and used for decompressing all message data in the first message queue according to the sequence of the first message queue to form all complete message data and unloading the complete message data, the processing module can judge the real-time total data volume of the first message queue according to the first preset data volume and the second preset data volume which are internally arranged so as to determine whether to adjust the first message queue, and the processing module can also judge the compressed packet data volume of any message data in the first message queue so as to determine whether to adjust the position of the message data in the first message queue;
the second-level queue module is connected with the processing module and used for discharging the complete message data transferred by the processing module into a second message queue;
the business module is connected with the secondary queue module and used for consuming all complete message data in the second message queue according to the sequence of the second message queue and sending out the complete message data;
and the sending module is respectively connected with the service module and the receiving module and is used for receiving the complete message data sent by the service module, and the sending module sends the complete message data after receiving the character confirmation sent by the receiving module.
2. The high-performance internet of things platform message engine according to claim 1, wherein a first preset data amount L1 and a second preset data amount L2 are provided in the processing module, wherein L1 is smaller than L2, before the processing module decompresses each message data in the first message queue, the processing module obtains a total real-time data amount Ls of the first message queue and compares the total real-time data amount Ls with the first preset data amount L1 and the second preset data amount L2,
when Ls is less than L1, the processing module judges that the real-time total data volume of the first message queue is lower than a first preset data volume, and the processing module does not adjust the first message queue of the primary queue module;
when the Ls is more than or equal to L1 and less than or equal to L2, the processing module determines that the real-time total data volume of the first message queue is between a first preset data volume and a second preset data volume, and the processing module determines the compressed packet data volume of each message data in the first message queue so as to adjust the first message queue of the first-stage queue module;
and when Ls is greater than L2, the processing module judges that the real-time total data volume of the first message queue is higher than a second preset data volume, and the processing module adjusts the first message queue of the first-stage queue module.
3. The internet of things platform message engine with high performance according to claim 2, wherein a standard message data amount Lb and a standard message data amount difference Δ Lb are disposed in the processing module, when the processing module determines that the real-time total data amount of the first message queue is between a first preset data amount and a second preset data amount, the processing module determines a compressed packet data amount of each message data from the end of the first message queue, for a compressed packet data amount Ly of any message data, the processing module calculates a compressed packet data amount difference Δ Ly, Δ Ly = | Lb-Ly | according to the compressed packet data amount Ly and the standard message data amount Lb, and compares the compressed packet data amount difference Δ Ly with the standard message data amount difference Δ Lb,
when the delta Ly is less than or equal to the delta Lb, the processing module judges that the compressed packet data volume difference delta Ly of the message data does not exceed the standard message data volume difference delta Lb, and the processing module does not adjust the position of the message data in the first message queue;
when delta Ly is larger than delta Lb, the processing module judges that the compressed packet data quantity difference delta Ly of the message data exceeds the standard message data quantity difference delta Lb, and the processing module compares the compressed packet data quantity Ly of the message data with the standard message data quantity Lb to adjust the position of the message data in the first message queue.
4. The IOT platform message engine of claim 3, wherein the processing module compares the compressed packet data amount Ly of the message data with the standard message data amount Lb when determining that the compressed packet data amount difference Δ Ly of the message data exceeds the standard message data amount DeltaLb,
when Ly is less than Lb, the processing module judges that the compressed packet data quantity Ly of the message data is less than the standard message data quantity Lb, the processing module removes the message data in the first message queue, and the first-stage queue module discharges the message data to the front end of the removed message queue;
when Ly is larger than Lb, the processing module judges that the compressed packet data quantity Ly of the message data is higher than the standard message data quantity Lb, and the processing module removes the message data from the first message queue and inserts the message data into the front end of the first message queue.
5. The high-performance internet of things platform message engine according to claim 4, wherein a maximum shifted-out data amount Lz is provided in the processing module, when there is a shifted-out message queue in the primary queue module, the processing module obtains a total real-time shifted-out data amount Lc of the shifted-out message queue and compares the total real-time shifted-out data amount Lc with the maximum shifted-out data amount Lz,
when Lc is less than Lz, the processing module judges that the real-time removal total data volume of the removed message queue does not reach the maximum removed data volume, and the processing module judges whether to decompress and dump the removed message queue according to the real-time total data volume of the first message queue and a first preset data volume;
when Lc is larger than or equal to Lz, the processing module determines that the real-time total data removal amount of the removed message queue reaches the maximum removed data amount, the processing module stops decompressing and unloading each message data in the first message queue, decompresses and unloads each message data in the removed message queue according to the sequence of the removed message queue, and starts decompressing and unloading each message data in the first message queue until all message data in the removed message queue are decompressed and unloaded.
6. The high-performance platform message engine of internet of things of claim 5, wherein the processing module compares the obtained real-time total data amount Ls' of the first message queue with a first preset data amount L1 when determining that the real-time removed total data amount of the removed message queue does not reach the maximum removed data amount,
if Ls' is less than L1, the processing module determines that the total real-time data amount of the first message queue does not reach a first preset data amount, the processing module stops decompressing and transferring each message data in the first message queue, decompresses and transfers each message data in the removed message queue according to the sequence of the removed message queue, and starts decompressing and transferring each message data in the first message queue until all message data in the removed message queue are decompressed and transferred;
if Ls' is greater than or equal to L1, the processing module judges that the real-time total data volume of the first message queue reaches a first preset data volume, and the processing module does not decompress and dump the removed message queue.
7. The high-performance internet of things platform message engine according to claim 2, wherein when the processing module determines that the real-time total data amount of the first message queue is higher than the second preset data amount, the processing module acquires the data amount of each message data in the first message queue and sorts the message data according to the data amount from high to low, and if two or more message data have equal data amounts, the processing module acquires the receiving time nodes of each message data with equal data amounts, calculates the time difference between the receiving time node and the current time node, and sorts the message data with equal data amounts from high to low.
8. The internet of things platform message engine with high performance according to claim 1, wherein the receiving module is provided with a preset interception character length Wj and a preset screening character length Ws, where Wj > Ws, the receiving module is further provided with a preset interception frequency Q, when the receiving module receives message data a sent by the device, the receiving module intercepts the message data a Q times of character segments with a length of Wj to form Q character segments of the message data a, the receiving module combines the Q character segments of the message data a to form a check character segment Ka of the message data a, and if a continuous repeated character segment part with a character length higher than the preset screening character length Ws exists in the check character segment Ka, the receiving module deletes and retains a repeated character segment part.
9. The message engine of the internet of things platform with high performance as claimed in claim 8, wherein the receiving module further comprises a preset check character length Wy, and when the receiving module forms the check character segment Ka of the message data a, the receiving module obtains the real-time character length Wa of the check character segment Ka and compares the real-time character length Wa with the preset check character length Wy,
when Wa is less than Wy, the receiving module judges that the real-time character length of the check character segment Ka does not reach the preset check character length, the receiving module deletes the check character segment Ka and repeats the operation of intercepting the character segment with the length Wj for Q times on the message data A;
and when Wa is larger than or equal to Wy, the receiving module judges that the real-time character length of the check character section Ka reaches the preset check character length, and the receiving module acquires the data name Ma of the message data A and stores the data name Ma and the check character section Ka.
10. The message engine of the platform of internet of things with high performance according to claim 9, wherein the receiving module is further provided with a confirmation similarity Gq, the sending module acquires a data name of the complete message data when receiving the complete message data sent by the service module, matches the data name with a data name stored in the receiving module according to the data name, selects a corresponding data name and a check character segment, the receiving module calculates a check similarity Gs between the check character segment and the complete message data, and compares the check similarity Gs with the confirmation similarity Gq,
when Gs is smaller than Gq, the receiving module judges that the check similarity does not reach the confirmation similarity, the receiving module sends error information to the equipment end, and receives message data sent by the equipment end again;
and when Gs is larger than or equal to Gq, the receiving module judges that the check similarity reaches the confirmation similarity, the receiving module sends character confirmation to the sending module, deletes the stored corresponding data name and the check character segment, and the sending module sends the complete message data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310145675.4A CN115865990B (en) | 2023-02-22 | 2023-02-22 | High-performance Internet of things platform message engine |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310145675.4A CN115865990B (en) | 2023-02-22 | 2023-02-22 | High-performance Internet of things platform message engine |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115865990A CN115865990A (en) | 2023-03-28 |
| CN115865990B true CN115865990B (en) | 2023-04-18 |
Family
ID=85658608
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310145675.4A Active CN115865990B (en) | 2023-02-22 | 2023-02-22 | High-performance Internet of things platform message engine |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115865990B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111031094A (en) * | 2019-11-06 | 2020-04-17 | 远景智能国际私人投资有限公司 | Data transmission method, device, equipment and storage medium in IoT system |
| CN114726792A (en) * | 2022-04-19 | 2022-07-08 | 中国联合网络通信集团有限公司 | Processing method, device, equipment and medium based on json message |
| CN114756390A (en) * | 2022-04-21 | 2022-07-15 | 北京沃东天骏信息技术有限公司 | Message queue consumption method and system, electronic equipment, readable medium and product |
| CN114885018A (en) * | 2022-06-22 | 2022-08-09 | 中国平安财产保险股份有限公司 | Message pushing method, device, equipment and storage medium based on double queues |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013049687A1 (en) * | 2011-09-30 | 2013-04-04 | Mail Bypass, Inc. | Message delivery systems and methods |
-
2023
- 2023-02-22 CN CN202310145675.4A patent/CN115865990B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111031094A (en) * | 2019-11-06 | 2020-04-17 | 远景智能国际私人投资有限公司 | Data transmission method, device, equipment and storage medium in IoT system |
| CN114726792A (en) * | 2022-04-19 | 2022-07-08 | 中国联合网络通信集团有限公司 | Processing method, device, equipment and medium based on json message |
| CN114756390A (en) * | 2022-04-21 | 2022-07-15 | 北京沃东天骏信息技术有限公司 | Message queue consumption method and system, electronic equipment, readable medium and product |
| CN114885018A (en) * | 2022-06-22 | 2022-08-09 | 中国平安财产保险股份有限公司 | Message pushing method, device, equipment and storage medium based on double queues |
Non-Patent Citations (1)
| Title |
|---|
| 耿云飞..融合通信消息引擎的设计与实现.《中国优秀硕士学位论文全文数据库信息科技》.2016,(第11期),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115865990A (en) | 2023-03-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7382726B2 (en) | Node system, dual ring communication system using node system, and communication method thereof | |
| CN108512760B (en) | Routing method based on SDN guarantee service QoS | |
| CN100411327C (en) | Appts. and method for delivery of packets in multi-hop wireless networks | |
| US20050021832A1 (en) | Deferred acknowledgment communications and alarm management | |
| CN101553998B (en) | A kind of wireless mesh network with locally activated fast active scheduling of wireless messages | |
| EP3756316A1 (en) | Satellite and terrestrial load balancing | |
| KR101769114B1 (en) | Sending node and buffer status reporting method | |
| CN109756536A (en) | A kind of method, apparatus and system of data transmission | |
| CN101305577B (en) | Device and method correlating message transmission | |
| CN115865990B (en) | High-performance Internet of things platform message engine | |
| WO2019183598A1 (en) | Coordinated data sharing in virtualized networking environments | |
| CN101341692B (en) | Admission control using backup link based on access network in Ethernet | |
| Al-Tabbakh | Novel technique for data aggregation in wireless sensor networks | |
| JP2010093387A (en) | Wireless communication system, transmission terminal, relay terminal, data transmission method, data reception method, and computer program | |
| Yin et al. | NC-OLSR: A network coding based OLSR multipath transmission scheme for FANETs | |
| CN105282045B (en) | A kind of distributed computing and storage method based on consistency hash algorithm | |
| EP4189936A1 (en) | Hybrid software-defined networking and mobile ad-hoc networking routing in mesh networks | |
| CN117240734A (en) | Cloud edge cooperation method, cloud edge cooperation system, computer equipment and storage medium | |
| CN114095971B (en) | Method, system, device, equipment and storage medium for processing communication data packet | |
| KR101186880B1 (en) | Content-based routing method in mobile ad-hoc networks and apparatus thereof | |
| KR102135858B1 (en) | Apparatus and Method for Managing Data Packet Queue in Industrial IoT Wireless Sensor Network | |
| KR20220168001A (en) | User Equipment for Dynamically Offloading Tasks Based on Multi-Radio Access Technology for IIot Service and Offloading Method Thereof | |
| CN116527673B (en) | Cluster service implementation method based on decentralization | |
| Yang et al. | Featherlight information network with delay-endurable RFID support (FINDERS) | |
| CN105900383A (en) | Communication system, control node and communication method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |