CN111949422A - Data multi-level caching and high-speed transmission recording method based on MQ and asynchronous IO - Google Patents
Data multi-level caching and high-speed transmission recording method based on MQ and asynchronous IO Download PDFInfo
- Publication number
- CN111949422A CN111949422A CN202010868475.8A CN202010868475A CN111949422A CN 111949422 A CN111949422 A CN 111949422A CN 202010868475 A CN202010868475 A CN 202010868475A CN 111949422 A CN111949422 A CN 111949422A
- Authority
- CN
- China
- Prior art keywords
- data
- cache
- sending
- asynchronous
- thread
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 51
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000012545 processing Methods 0.000 claims abstract description 46
- 230000008569 process Effects 0.000 claims abstract description 12
- 238000005516 engineering process Methods 0.000 claims abstract description 10
- 238000012544 monitoring process Methods 0.000 claims description 8
- 238000004891 communication Methods 0.000 claims description 7
- 238000003672 processing method Methods 0.000 claims description 5
- 230000008878 coupling Effects 0.000 claims description 3
- 238000010168 coupling process Methods 0.000 claims description 3
- 238000005859 coupling reaction Methods 0.000 claims description 3
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/543—User-generated data transfer, e.g. clipboards, dynamic data exchange [DDE], object linking and embedding [OLE]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Signal Processing For Digital Recording And Reproducing (AREA)
Abstract
The invention provides a data multilevel caching and high-speed transmission recording method based on MQ and asynchronous IO, which can realize high-speed stable network transmission by utilizing a multilevel caching technology and a zeroMQ technology and furthest utilize network bandwidth resources in a transmission level. On the aspect of data recording, on one hand, asynchronous file writing is carried out through asynchronous IO, frequent switching between a user mode (UserMode) and a kernel mode (KernelMode) of a CPU is reduced, and data writing speed is improved; on the other hand, by sharing data receiving and asynchronous IO data caching, memory copy in the processing process is reduced, cache consumption is reduced, and the data writing speed is improved. By optimizing and improving the two aspects of the transmission layer and the data recording layer, the speed and the stability of the transmission recording of the whole system are improved.
Description
Technical Field
The invention relates to the field of high-speed recording of the Internet of things, in particular to a data multi-level caching and high-speed transmission recording method based on MQ and asynchronous IO.
Background
In the field of internet of things high-speed recording, high-speed transmission and high-speed recording of various massive internet of things data are required. In the transmission layer, TCP or UDP is usually used for transmission, but the transmission speed of TCP is not high enough, the transmission speed of UDP is fast, but the problem of packet loss is difficult to avoid, and a method for stable transmission without packet loss and capable of fully utilizing the network bandwidth is required. On the data receiving and recording storage layer, a hardware board card can be used for high-speed writing, but the flexibility is slightly insufficient, and the cost is higher; the flexibility is improved by utilizing software to receive and write in at a high speed, but the scheduling of a CPU and a disk IO has considerable difficulty, and the packet loss of data is aggravated once data congestion occurs;
therefore, the invention discloses a data multi-level caching and high-speed transmission recording method based on MQ and asynchronous IO.
Disclosure of Invention
In order to achieve the purpose, the invention is realized by the following technical scheme: the data multi-level caching and high-speed transmission recording method based on MQ and asynchronous IO is suitable for an integral system architecture, and the integral system architecture consists of a data sending end, a data receiving end and a multi-level cache;
the data sending end consists of a data sending interface unit and a high-speed data sending unit;
the data receiving end consists of a high-speed data receiving unit and an asynchronous IO recording unit;
the multi-level cache comprises a multi-level cache of a sending data cache and a multi-level cache of a receiving data cache.
Further, the data sending terminal comprises a third-level cache, the third-level cache consists of a sending terminal first-level cache, a sending terminal second-level cache and a sending terminal third-level cache, communication is carried out between the data preparation thread and the data sending thread through an inter-thread communication mode inproc protocol of zeroMQ, data caches are arranged at both a PUSH terminal and a PULL terminal of the zeroMQ, and the data caches comprise the sending terminal first-level cache and the sending terminal second-level cache; when the high-speed data sending unit sends data to a receiver, a DEALER mode of a TCP protocol of zeroMQ is adopted for sending, and in order to enhance the transmission stability of the data, a sending end cache is set and is a three-level cache of the sending end.
Further, the data receiving end comprises a two-level cache, the two-level cache is composed of a receiving end first-level cache and a receiving end second-level cache, data are received through the high-speed data receiving unit, the data are received in a DEALER mode of a TCP protocol of ZeroMQ, the stability of the data is improved by arranging the TCP receiving cache, the TCP receiving cache is the receiving end first-level cache, and a recording data cache is arranged between the high-speed data receiving unit and the asynchronous IO recording unit so as to improve the performance and reduce the coupling degree between modules and is the receiving end second-level cache.
The data sending end processing method comprises the following steps:
data transmission processing flow: the data sending processing is divided into two threads, a data preparation thread and a data sending thread, and the two threads buffer data through a sending data queue and exchange data. The data preparation thread continuously writes prepared data into the data queue through the inproc protocol of the zeroMQ, and the data sending thread continuously obtains the data from the data queue through the inproc protocol of the zeroMQ and sends the data out through the TCP protocol of the zeroMQ.
Access mode of sending data: the method provides a flexible data transmission scene for a data transmission module, and supports two data access modes:
an access mode 1: providing a data access mode of an API (application programming interface), and finishing data transmission by directly calling the API by an external application;
and an access mode 2: providing relevant parameters of a data transmission queue, and writing data into the data transmission queue for transmission by using the corresponding setting of the zeroMQ by external application;
s1: the data preparation thread provides an external API interface, and the external API calls the triggering processing logic to perform the following detailed processing:
calling a sending interface unit, writing data into a sending interface, sending the data by adopting a PUSH mode of an inproc protocol through ZMQ by the sending interface, setting a sending data cache, writing the sending data into the data cache before sending the data, and sending the data to a data sending thread through the cache;
s2: the data sending thread monitors a data sending queue, and data in the queue triggers data sending logic processing, wherein the detailed processing is as follows: monitoring ZMQ, adopting a PULL mode of an inproc protocol to monitor and receive data, setting ZMQ to receive a cache, storing the data into the cache after monitoring the data, then acquiring the data from the cache to prevent backlog and loss caused by overlarge data flow, packaging the acquired data, writing the data into a ZMQ data cache which is sent outwards, and sending the data in the cache out through a DEALER mode of a TCP protocol of a zeroMQ.
A data receiving processing method, the method comprising the steps of:
data receiving and processing flow: the data sending and processing is divided into two threads, namely a data receiving thread and a data recording thread, the two threads share data through an asynchronous IO data channel and exchange data, the data receiving thread continuously and asynchronously obtains data from a data queue through a TCP protocol of a zeroMQ and writes the data into the asynchronous IO data channel, the data recording thread obtains data from the asynchronous IO data channel, a file is written in through an asynchronous IO technology and records the data, and by sharing data cache addresses in the asynchronous IO data channel of data receiving and asynchronous IO, memory copy in the processing process is reduced, cache consumption is reduced, and the data writing speed is improved;
step one, a data receiving thread monitors a data sending queue, and data in the queue triggers data receiving logic processing, wherein the detailed processing is as follows:
s1: initializing context, iocb, IO _ event and other related contexts related to asynchronous IO;
s2: initializing a data cache, an idle list lock, a used list, a list lock and the like for asynchronous IO channel queue sharing;
s3: starting a data recording thread and sharing the contents with the data recording thread;
s4: monitoring ZMQ, adopting DEALER mode of TCP protocol to monitor and receive data;
s5: ZMQ is set to receive the cache, and after the data is monitored, the data is stored in the cache and then is obtained from the cache, so as to prevent backlog and loss caused by overlarge data flow;
s6: judging whether the asynchronous IO has an idle channel, if so, writing the data into the idle channel (sharing the memory address in the cache with the data recording thread);
s7: in order to prevent the conflict of the shared variables of the data receiving thread and the data recording thread, the shared variables are locked before operation, and then unlocked after the operation is finished.
Step two, the data recording thread circularly processes the asynchronous IO channel, the asynchronous IO writing-in processing is carried out when data are found in the queue, the data in the shared cache is released after the data are written in, so that the data receiving thread can write the data into the idle unit, and the detailed processing is as follows:
s1: circularly judging whether data exist in a queue of the asynchronous IO channel, and if the data exist, performing the following processing;
s2: locking the used queue, acquiring a data position needing to be processed from the used queue, and then unlocking;
s3: acquiring data of a corresponding data position from a buffer queue according to the acquired data position;
s4: according to the acquired data position, a corresponding available channel in iocb of the asynchronous IO is designated;
s5: calling IO _ prep _ write to write data into asynchronous IO;
s6: invoking IO _ submit to submit the asynchronous IO;
s7: updating the idle queue;
s8: and judging whether the idle queue is smaller than half of the length of the whole queue, if so, calling io _ getevents to acquire an idle unit, and updating the idle queue.
The invention provides a data multi-level caching and high-speed transmission recording method based on MQ and asynchronous IO. The method has the following beneficial effects: the invention utilizes multilevel cache technology and zeroMQ technology to realize high-speed stable network transmission and furthest utilizes network bandwidth resources in the transmission level. On the aspect of data recording, on one hand, asynchronous file writing is carried out through asynchronous IO, frequent switching of a User Mode (User Mode) and a Kernel Mode (Kernel Mode) of a CPU is reduced, and data writing speed is improved; on the other hand, by sharing data receiving and asynchronous IO data caching, memory copy in the processing process is reduced, cache consumption is reduced, and the data writing speed is improved. By optimizing and improving the two aspects of the transmission layer and the data recording layer, the speed and the stability of the transmission recording of the whole system are improved.
Drawings
Fig. 1 is an overall architecture diagram of the system of the present invention.
FIG. 2 is a diagram of a system cache architecture according to the present invention.
Fig. 3 is a flow chart of the data transmitting end processing according to the present invention.
FIG. 4 is a flow chart of the data receiving end processing according to the present invention.
FIG. 5 is a detailed flow chart of the data receiving thread according to the present invention.
FIG. 6 is a detailed flow chart of the data logging thread of the present invention.
Detailed Description
According to a first aspect of the present invention, the present invention provides a data multi-level caching and high-speed transmission recording method based on MQ and asynchronous IO, as shown in fig. 1 to 6, the present invention is composed of four units of a transmitting end and a receiving end and a multi-level cache, and the data transmitting end is composed of a data transmitting interface unit and a high-speed data transmitting unit.
The data receiving end consists of a high-speed data receiving unit and an asynchronous IO recording unit.
The buffer comprises a multi-level buffer of a sending data buffer and a multi-level buffer of a receiving data buffer (see figure 2 in detail).
In order to improve the data transmission stability, the system designs a multi-level cache architecture system.
The sending end comprises three levels of caches: in order to decouple the data preparation thread (data sending interface unit) and the data sending thread (high-speed data sending unit), communication is carried out between the data preparation thread and the data sending thread through an inter-thread communication mode inproc protocol of zeroMQ; in order to improve the stability of data communication, data caches, namely a first-level cache of a sending terminal and a second-level cache of the sending terminal in the following figure, are arranged at both the PUSH end and the PULL end of the zeroMQ. When the high-speed data sending unit sends data to the receiver, the data is sent by adopting a DEALER mode of a TCP protocol of zeroMQ, and in order to enhance the transmission stability of the data, a sending end cache, namely a sending end three-level cache in the following figure, is set.
The receiving end comprises two levels of caches: the data is received by a high-speed data receiving unit, the data is received by adopting a DEALER mode of a TCP protocol of zeroMQ, and the stability of the data is improved by setting a TCP receiving cache, namely the receiving end first-level cache in the next figure. Between the high-speed data receiving unit and the asynchronous IO recording unit, in order to improve the performance and reduce the coupling degree between the modules, a recording data cache is provided, i.e. a receiving end secondary cache of the next figure.
Meanwhile, the invention adopts the following key technologies:
zeroMQ technology model selection
Compared with the traditional TCP protocol, the zeroMQ provides various modes to realize the socket connection of N to N, makes up the limitation that the TCP can only transmit one to one, shields the problems of sticky packets and half packets which need to be processed during streaming transmission, and has better expansibility; compared with the traditional UDP protocol, the zeroMQ can provide stable transmission, and the problem of packet loss of the UDP protocol is solved.
Compared with other message queues such as a RabbitMQ, a RocktMQ, a Kafka and the like, the zero MQ has the advantages of high concurrency, high throughput, low delay and the like, the single-machine throughput is 100 ten thousand grade, the delay is microsecond grade, and the zero MQ is more suitable for the high-speed stable transmission scene of mass data. The zeroMQ core is written by C language, supports APIs of various programming languages such as C, C + +, java and the like, has natural advantages in integration with the C + + language, and can run on most operating systems.
Asynchronous IO technology model selection
In the data transmission and recording process, the speed of the CPU is far faster than IO such as a disk, a network and the like. In one thread, the speed of executing the code by the CPU is extremely fast, however, once an IO operation is encountered, such as reading and writing a file, and sending network data, it is necessary to wait for the completion of the IO operation before proceeding to the next operation. During the IO operation, the current thread is suspended, and other code that needs to be executed by the CPU cannot be executed by the current thread.
Because one IO operation blocks the current thread, causing other code to be unable to execute, we must use multiple threads or multiple processes to concurrently execute code to serve multiple users. Each user is assigned a thread, and if IO is encountered, the thread is suspended, and the threads of other users are not affected.
Multi-threading and multi-process models, while solving the concurrency problem, the system cannot add threads without an upper bound. Because the overhead of system thread switching is also large, once the number of threads is too large, the time of the CPU is spent on thread switching, the time for actually running the code is short, and the performance is seriously reduced as a result.
Asynchronous IO may solve the above IO problem. When IO operation is met, the code is only responsible for sending an IO request, does not wait for an IO result, then directly ends the message processing of the current round, and enters the next message processing process. After the IO operation is completed, an IO completion message is received, and the IO operation result can be directly obtained when the message is processed.
In the period from the "sending of the IO request" to the "receiving of the IO completion", the main thread can only be suspended in the synchronous IO model, but the main thread does not rest in the asynchronous IO model, and continues to process other messages in the message loop. Thus, under the asynchronous IO model, one thread can process multiple IO requests simultaneously, and the operation of the thread is not switched. For most IO-intensive applications, the use of asynchronous IO will greatly increase the multitasking capability of the system.
When data is sent and processed, the data sending and processing are divided into two threads, namely a data preparation thread and a data sending thread, and the two threads cache data through a sending data queue and exchange data. The data preparation thread continuously writes prepared data into the data queue through the inproc protocol of the zeroMQ, and the data sending thread continuously obtains the data from the data queue through the inproc protocol of the zeroMQ and sends the data out through the TCP protocol of the zeroMQ.
Access mode for transmitting data
The method provides a flexible data transmission scene for a data transmission module, and supports two data access modes:
an access mode 1: providing a data access mode of an API (application programming interface), and finishing data transmission by directly calling the API by an external application; (see access mode 1 of FIG. 3)
And an access mode 2: and providing related parameters of a transmission data queue, and writing data into the transmission data queue for transmission by using the corresponding setting of the zeroMQ by an external application. (see access mode 2 of FIG. 3)
The data preparation thread provides an external API interface, and the external API calls the triggering processing logic to perform the following detailed processing:
calling the sending interface unit and writing data into the sending interface
The sending interface sends the data in PUSH mode of inproc protocol via ZMQ.
And setting a data sending cache, writing the data sending cache with the data sending cache before sending the data, and sending the data to the data sending thread through the cache.
The data sending thread monitors a data sending queue, and data in the queue triggers data sending logic processing, wherein the detailed processing is as follows:
listen ZMQ, listen and receive data using the PULL mode of the inproc protocol.
The device ZMQ receives the cache, and after the data is monitored, the data is stored in the cache and then is obtained from the cache, so as to prevent backlog and loss caused by overlarge data traffic.
Packaging the obtained data and writing the data into ZMQ data cache sent out
Sending out the data in the cache through a DEALER mode of a TCP protocol of a zeroMQ;
when data is received and processed, data sending and processing are divided into two threads, namely a data receiving thread and a data recording thread, and the two threads share data through an asynchronous IO data channel and exchange data. And the data receiving thread continuously and asynchronously obtains data from the data queue through a TCP protocol of the zeroMQ and writes the data into an asynchronous IO data channel. And the data recording thread acquires data from the asynchronous IO data channel, writes the data into a file through an asynchronous IO technology, and records the data. By sharing data cache addresses in the asynchronous IO data channel of the data receiving and asynchronous IO, memory copy in the processing process is reduced, cache consumption is reduced, and the data writing speed is improved.
The data receiving thread monitors a data sending queue, and data in the queue triggers data receiving logic processing, wherein the detailed processing is as follows:
initializing asynchronous IO related context, iocb, IO _ event, etc.
Data caches, free lists, free list locks, used lists, list locks, etc. for asynchronous IO channel queue sharing are initialized.
And starting the data recording thread and sharing the data recording thread with the data recording thread.
And monitoring ZMQ, and monitoring and receiving data by adopting a DEALER mode of a TCP protocol.
The device ZMQ receives the cache, and after the data is monitored, the data is stored in the cache and then is obtained from the cache, so as to prevent backlog and loss caused by overlarge data traffic.
And judging whether the asynchronous IO has an idle channel or not, and if so, writing the data into the idle channel (sharing the memory address in the cache with the data recording thread).
In order to prevent the conflict of the shared variables of the data receiving thread and the data recording thread, the shared variables are locked before operation, and then unlocked after the operation is finished.
And circularly processing the asynchronous IO channel by the data recording thread, performing asynchronous IO writing processing when data are found in the queue, and releasing the data in the shared cache after writing so that the data receiving thread can write the data into the idle unit. The detailed processing is as follows:
and circularly judging whether data exists in the queue of the asynchronous IO channel, and if so, performing the following processing.
And locking the used queue, acquiring a data position needing to be processed from the used queue, and unlocking.
And acquiring data of the corresponding data position from the buffer queue according to the acquired data position.
According to the obtained data position, appointing a corresponding available channel in iocb of asynchronous IO
Calling IO _ prep _ write to write data to asynchronous IO
Call IO _ submit commit to asynchronous IO
Updating an idle queue
And judging whether the idle queue is smaller than half of the length of the whole queue, if so, calling io _ getevents to acquire an idle unit, and updating the idle queue.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (5)
1. The data multilevel caching and high-speed transmission recording method based on MQ and asynchronous IO is suitable for the whole system architecture and is characterized in that: the whole system architecture consists of a data sending end, a data receiving end and a multi-level cache;
the data sending end consists of a data sending interface unit and a high-speed data sending unit;
the data receiving end consists of a high-speed data receiving unit and an asynchronous IO recording unit;
the multi-level cache comprises a multi-level cache of a sending data cache and a multi-level cache of a receiving data cache.
2. The overall system architecture of claim 1, wherein: the data sending terminal comprises a third-level cache, the third-level cache consists of a first-level cache of the sending terminal, a second-level cache of the sending terminal and a third-level cache of the sending terminal, communication is carried out between a data preparation thread and a data sending thread through an inter-thread communication mode inproc protocol of a zeroMQ, data caches are arranged at a PUSH terminal and a PULL terminal of the zeroMQ, and the first-level cache of the data caching sending terminal and the second-level cache of the sending terminal are arranged; when the high-speed data sending unit sends data to a receiver, a DEALER mode of a TCP protocol of zeroMQ is adopted for sending, and in order to enhance the transmission stability of the data, a sending end cache is set and is a three-level cache of the sending end.
3. The overall system architecture of claim 1, wherein: the data receiving terminal comprises a two-stage cache, the two-stage cache comprises a first-stage cache of the receiving terminal and a second-stage cache of the receiving terminal, data are received through a high-speed data receiving unit, data are received in a DEALER mode of a TCP protocol of ZeroMQ, the data stability is improved by setting the TCP receiving cache, the TCP receiving cache is the first-stage cache of the receiving terminal, and a recording data cache is set between the high-speed data receiving unit and an asynchronous IO recording unit for improving the performance and reducing the coupling degree between modules and is the second-stage cache of the receiving terminal.
4. The data transmitting-end processing method according to claim 1, characterized in that:
data transmission processing flow: the data sending processing is divided into two threads, a data preparation thread and a data sending thread, and the two threads buffer data through a sending data queue and exchange data. The data preparation thread continuously writes prepared data into a data queue through an inproc protocol of the zeroMQ, the data sending thread continuously acquires the data from the data queue through the inproc protocol of the zeroMQ and sends the data out through a TCP protocol of the zeroMQ, and the data sending access mode is as follows: the method provides a flexible data transmission scene for a data transmission module, and supports two data access modes:
an access mode 1: providing a data access mode of an API (application programming interface), and finishing data transmission by directly calling the API by an external application;
and an access mode 2: providing relevant parameters of a data transmission queue, and writing data into the data transmission queue for transmission by using the corresponding setting of the zeroMQ by external application;
the processing method comprises the following steps:
s1: the data preparation thread provides an external API interface, and the external API calls the triggering processing logic to perform the following detailed processing:
calling a sending interface unit, writing data into a sending interface, sending the data by adopting a PUSH mode of an inproc protocol through ZMQ by the sending interface, setting a sending data cache, writing the sending data into the data cache before sending the data, and sending the data to a data sending thread through the cache;
s2: the data sending thread monitors a data sending queue, and data in the queue triggers data sending logic processing, wherein the detailed processing is as follows: monitoring ZMQ, adopting a PULL mode of an inproc protocol to monitor and receive data, setting ZMQ to receive a cache, storing the data into the cache after monitoring the data, then acquiring the data from the cache to prevent backlog and loss caused by overlarge data flow, packaging the acquired data, writing the data into a ZMQ data cache which is sent outwards, and sending the data in the cache out through a DEALER mode of a TCP protocol of a zeroMQ.
5. The data reception processing method according to claim 1, wherein:
the data receiving and processing flow is divided into two threads, namely a data receiving thread and a data recording thread, the two threads share data through an asynchronous IO data channel and exchange data, the data receiving thread continuously and asynchronously obtains data from a data queue through a TCP protocol of a zeroMQ and writes the data into the asynchronous IO data channel, the data recording thread obtains the data from the asynchronous IO data channel, a file is written in through an asynchronous IO technology and records the data, and by sharing data cache addresses in the asynchronous IO data channel of data receiving and asynchronous IO, memory copy in the processing process is reduced, cache consumption is reduced, and the data writing speed is improved;
the method comprises the following steps:
step one, a data receiving thread monitors a data sending queue, and data in the queue triggers data receiving logic processing, wherein the detailed processing is as follows:
s1: initializing context, iocb, IO _ event and other related contexts related to asynchronous IO;
s2: initializing a data cache, an idle list lock, a used list, a list lock and the like for asynchronous IO channel queue sharing;
s3: starting a data recording thread and sharing the contents with the data recording thread;
s4: monitoring ZMQ, adopting DEALER mode of TCP protocol to monitor and receive data;
s5: ZMQ is set to receive the cache, and after the data is monitored, the data is stored in the cache and then is obtained from the cache, so as to prevent backlog and loss caused by overlarge data flow;
s6: judging whether the asynchronous IO has an idle channel, if so, writing the data into the idle channel (sharing the memory address in the cache with the data recording thread);
s7: in order to prevent the conflict of the shared variables of the data receiving thread and the data recording thread, the shared variables are locked before operation, and then unlocked after the operation is finished.
Step two, the data recording thread circularly processes the asynchronous IO channel, the asynchronous IO writing-in processing is carried out when data are found in the queue, the data in the shared cache is released after the data are written in, so that the data receiving thread can write the data into the idle unit, and the detailed processing is as follows:
s1: circularly judging whether data exist in a queue of the asynchronous IO channel, and if the data exist, performing the following processing;
s2: locking the used queue, acquiring a data position needing to be processed from the used queue, and then unlocking;
s3: acquiring data of a corresponding data position from a buffer queue according to the acquired data position;
s4: according to the acquired data position, a corresponding available channel in iocb of the asynchronous IO is designated;
s5: calling IO _ prep _ write to write data into asynchronous IO;
s6: invoking IO _ submit to submit the asynchronous IO;
s7: updating the idle queue;
s8: and judging whether the idle queue is smaller than half of the length of the whole queue, if so, calling io _ getevents to acquire an idle unit, and updating the idle queue.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010868475.8A CN111949422B (en) | 2020-08-26 | 2020-08-26 | Data multi-level cache and high-speed transmission recording method based on MQ and asynchronous IO |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010868475.8A CN111949422B (en) | 2020-08-26 | 2020-08-26 | Data multi-level cache and high-speed transmission recording method based on MQ and asynchronous IO |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111949422A true CN111949422A (en) | 2020-11-17 |
| CN111949422B CN111949422B (en) | 2024-04-16 |
Family
ID=73366538
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010868475.8A Active CN111949422B (en) | 2020-08-26 | 2020-08-26 | Data multi-level cache and high-speed transmission recording method based on MQ and asynchronous IO |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111949422B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113946362A (en) * | 2021-09-02 | 2022-01-18 | 北京房江湖科技有限公司 | Consumption data processing method and storage medium |
| CN115334156A (en) * | 2021-04-26 | 2022-11-11 | 深信服科技股份有限公司 | Message processing method, device, equipment and storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1998034391A2 (en) * | 1997-02-03 | 1998-08-06 | Mci Worldcom, Inc. | A communication system architecture |
| CN102622426A (en) * | 2012-02-27 | 2012-08-01 | 杭州闪亮科技有限公司 | Database writing system and database writing method |
| US20180287972A1 (en) * | 2017-03-31 | 2018-10-04 | Bmc Software, Inc. | Systems and methods for intercepting access to messaging systems |
| CN108696578A (en) * | 2018-04-26 | 2018-10-23 | 昆明理工大学 | The communications framework design method that multiple machine distributing based on ZeroMQ calculates |
| US20180322606A1 (en) * | 2017-05-05 | 2018-11-08 | Intel Corporation | Data parallelism and halo exchange for distributed machine learning |
| CN108829713A (en) * | 2018-05-04 | 2018-11-16 | 华为技术有限公司 | Distributed cache system, cache synchronization method and device |
| CN110135575A (en) * | 2017-12-29 | 2019-08-16 | 英特尔公司 | Communication optimization for distributed machines study |
| CN110166562A (en) * | 2019-05-24 | 2019-08-23 | 东软集团股份有限公司 | Data synchronous method, apparatus, storage medium and electronic equipment |
| US20200026787A1 (en) * | 2018-07-23 | 2020-01-23 | Wipro Limited | METHOD, SYSTEM, AND FRAMEWORK FOR IMPLEMENTING INTERNET OF THINGS (IoT) APPLICATIONS |
| CN111104232A (en) * | 2019-11-09 | 2020-05-05 | 苏州浪潮智能科技有限公司 | Method, device and medium for accelerating message writing of message queue |
-
2020
- 2020-08-26 CN CN202010868475.8A patent/CN111949422B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1998034391A2 (en) * | 1997-02-03 | 1998-08-06 | Mci Worldcom, Inc. | A communication system architecture |
| CN102622426A (en) * | 2012-02-27 | 2012-08-01 | 杭州闪亮科技有限公司 | Database writing system and database writing method |
| US20180287972A1 (en) * | 2017-03-31 | 2018-10-04 | Bmc Software, Inc. | Systems and methods for intercepting access to messaging systems |
| US20180322606A1 (en) * | 2017-05-05 | 2018-11-08 | Intel Corporation | Data parallelism and halo exchange for distributed machine learning |
| CN110135575A (en) * | 2017-12-29 | 2019-08-16 | 英特尔公司 | Communication optimization for distributed machines study |
| CN108696578A (en) * | 2018-04-26 | 2018-10-23 | 昆明理工大学 | The communications framework design method that multiple machine distributing based on ZeroMQ calculates |
| CN108829713A (en) * | 2018-05-04 | 2018-11-16 | 华为技术有限公司 | Distributed cache system, cache synchronization method and device |
| US20200026787A1 (en) * | 2018-07-23 | 2020-01-23 | Wipro Limited | METHOD, SYSTEM, AND FRAMEWORK FOR IMPLEMENTING INTERNET OF THINGS (IoT) APPLICATIONS |
| CN110166562A (en) * | 2019-05-24 | 2019-08-23 | 东软集团股份有限公司 | Data synchronous method, apparatus, storage medium and electronic equipment |
| CN111104232A (en) * | 2019-11-09 | 2020-05-05 | 苏州浪潮智能科技有限公司 | Method, device and medium for accelerating message writing of message queue |
Non-Patent Citations (2)
| Title |
|---|
| REN SIKE 等: "Architecture Design Of Online Education System", 《2017IEEE/ACIS 16TH INTERNATIONAL CONFERENCE ON COMPUTER AND INFORMATION SCIENCE(ICIS)》, pages 1 - 4 * |
| 胡惠: "多通道超高速数据采集与记录系统的设计与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》, pages 140 - 579 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115334156A (en) * | 2021-04-26 | 2022-11-11 | 深信服科技股份有限公司 | Message processing method, device, equipment and storage medium |
| CN113946362A (en) * | 2021-09-02 | 2022-01-18 | 北京房江湖科技有限公司 | Consumption data processing method and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111949422B (en) | 2024-04-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108628684B (en) | DPDK-based message processing method and computer equipment | |
| US8751737B2 (en) | Method and apparatus for using a shared ring buffer to provide thread synchronization in a multi-core processor system | |
| KR102011949B1 (en) | System and method for providing and managing message queues for multinode applications in a middleware machine environment | |
| US7603502B2 (en) | Resource accessing with locking | |
| CN109729024B (en) | Data packet processing system and method | |
| US9444737B2 (en) | Packet data processor in a communications processor architecture | |
| CN111459418B (en) | RDMA (remote direct memory Access) -based key value storage system transmission method | |
| US20060026169A1 (en) | Communication method with reduced response time in a distributed data processing system | |
| EP1390847B1 (en) | Reducing latency and message traffic during data and lock transfer in a multi-node system | |
| CN111949422B (en) | Data multi-level cache and high-speed transmission recording method based on MQ and asynchronous IO | |
| CN111404931B (en) | Remote data transmission method based on persistent memory | |
| CN112559436B (en) | Context access method and system of RDMA communication equipment | |
| CN111209123A (en) | Local storage IO protocol stack data interaction method and device | |
| CN109831394B (en) | Data processing method, terminal and computer storage medium | |
| WO2023046141A1 (en) | Acceleration framework and acceleration method for database network load performance, and device | |
| CN111124270A (en) | Method, apparatus and computer program product for cache management | |
| US20140348101A1 (en) | Buffer resource management method and telecommunication equipment | |
| US8135851B2 (en) | Object request broker for accelerating object-oriented communications and method | |
| CN112486702B (en) | Global message queue implementation method based on multi-core multi-processor parallel system | |
| US7853713B2 (en) | Communication interface device and communication method | |
| WO2023231723A1 (en) | Streaming media data processing method and system | |
| CN111131081B (en) | Method and device for supporting high-performance one-way transmission of multiple processes | |
| CN108075989B (en) | Extensible protocol-based load balancing network middleware implementation method | |
| CN114691382A (en) | RDMA-based communication method, node, system and medium | |
| US20070230347A1 (en) | Communication interface device and communication method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |