CN111949422B - Data multi-level cache and high-speed transmission recording method based on MQ and asynchronous IO - Google Patents
Data multi-level cache and high-speed transmission recording method based on MQ and asynchronous IO Download PDFInfo
- Publication number
- CN111949422B CN111949422B CN202010868475.8A CN202010868475A CN111949422B CN 111949422 B CN111949422 B CN 111949422B CN 202010868475 A CN202010868475 A CN 202010868475A CN 111949422 B CN111949422 B CN 111949422B
- Authority
- CN
- China
- Prior art keywords
- data
- asynchronous
- thread
- sending
- receiving
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 66
- 238000000034 method Methods 0.000 title claims abstract description 38
- 239000000872 buffer Substances 0.000 claims abstract description 57
- 238000012545 processing Methods 0.000 claims abstract description 37
- 230000008569 process Effects 0.000 claims abstract description 23
- 238000005516 engineering process Methods 0.000 claims abstract description 10
- 238000012544 monitoring process Methods 0.000 claims description 14
- 238000004891 communication Methods 0.000 claims description 7
- 230000008878 coupling Effects 0.000 claims description 3
- 238000010168 coupling process Methods 0.000 claims description 3
- 238000005859 coupling reaction Methods 0.000 claims description 3
- 230000002708 enhancing effect Effects 0.000 claims description 3
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 238000003672 processing method Methods 0.000 claims description 3
- 230000003139 buffering effect Effects 0.000 claims 1
- 239000008186 active pharmaceutical agent Substances 0.000 description 9
- 238000010586 diagram Methods 0.000 description 3
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/543—User-generated data transfer, e.g. clipboards, dynamic data exchange [DDE], object linking and embedding [OLE]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Signal Processing For Digital Recording And Reproducing (AREA)
Abstract
The invention provides a data multi-level buffer and high-speed transmission recording method based on MQ and asynchronous IO, which can realize high-speed stable network transmission by utilizing a multi-level buffer technology and a zeroMQ technology and maximally utilize network bandwidth resources at the transmission level. On the data recording layer, on one hand, asynchronous writing files are carried out through asynchronous IO, frequent switching of a user mode (UserMode) and a kernel mode (Kernel Mode) of a CPU is reduced, and the data writing speed is improved; on the other hand, through sharing data receiving and data caching of asynchronous IO, memory copying in the processing process is reduced, caching consumption is reduced, and data writing speed is improved. By optimizing and improving the transmission layer and the data recording layer, the speed and the stability of the transmission record of the whole system are improved.
Description
Technical Field
The invention relates to the field of high-speed recording of the Internet of things, in particular to a data multi-level cache and high-speed transmission recording method based on MQ and asynchronous IO.
Background
In the field of high-speed recording of the internet of things, high-speed transmission and high-speed recording of various mass data of the internet of things are required. In the transmission layer, transmission is usually performed by using TCP or UDP, but the transmission speed of TCP is not high enough, the transmission speed of UDP is high, but the problem of packet loss is difficult to avoid, and a method capable of fully utilizing network bandwidth and stably transmitting without packet loss is needed. In the data receiving and recording storage layer, the hardware board card can be used for writing at high speed, but the flexibility is slightly insufficient, and the cost is relatively high; the flexibility is improved by utilizing software to receive and write at high speed, but the scheduling of the CPU and the disk IO has considerable difficulty, and the packet loss of the data is aggravated once the data congestion occurs;
therefore, the invention discloses a data multi-level cache and high-speed transmission recording method based on MQ and asynchronous IO.
Disclosure of Invention
In order to achieve the above purpose, the invention is realized by the following technical scheme: the method is applicable to an overall system architecture, and the overall system architecture consists of a data sending end, a data receiving end and a multi-level cache;
the data transmitting end consists of a data transmitting interface unit and a high-speed data transmitting unit;
the data receiving end consists of a high-speed data receiving unit and an asynchronous IO recording unit;
the multi-level caches include a multi-level cache that sends data caches and a multi-level cache that receives data caches.
Further, the data sending end comprises a third-level cache, the third-level cache consists of a first-level cache of the sending end, a second-level cache of the sending end and a third-level cache of the sending end, communication is carried out between a data preparation thread and a data sending thread through an inter-thread communication mode inproc protocol of the zeroMQ, the data cache is arranged at a PUSH end and a PULL end of the zeroMQ, and the first-level cache and the second-level cache of the sending end of the data cache are arranged; when the high-speed data transmitting unit transmits data to the receiving party, a DEALER mode of a TCP protocol of ZeroMQ is adopted for transmitting, a transmitting end buffer memory is arranged for enhancing the transmission stability of the data, and the transmitting end buffer memory is a transmitting end three-level buffer memory.
Further, the data receiving end comprises two-level caches, the two-level caches are composed of a first-level cache of the receiving end and a second-level cache of the receiving end, data are received through a high-speed data receiving unit, a DEALER mode of a TCP protocol of zeroMQ is adopted to receive the data, the stability of the data is improved through setting the TCP receiving cache, the TCP receiving cache is the first-level cache of the receiving end, and a record data cache is set between the high-speed data receiving unit and the asynchronous IO recording unit for improving performance and reducing the coupling degree between modules.
The data transmitting end processing method comprises the following steps:
the data transmission processing flow is as follows: the data transmission process is divided into two threads, namely a data preparation thread and a data transmission thread, and the two threads perform data caching through a data transmission queue to exchange data. The data preparation thread continuously writes the prepared data into the data queue through the inproc protocol of the zeroMQ, and the data sending thread continuously acquires the data from the data queue through the inproc protocol of the zeroMQ and sends the data out through the TCP protocol of the zeroMQ.
Access mode of sending data: flexible data transmission scenes are provided for the data transmission module, and two data access modes are supported:
access mode 1: providing a data access mode of an API interface, and enabling an external application to directly call the API interface to finish data transmission;
access mode 2: providing relevant parameters of a data transmission queue, and enabling an external application to write data into the data transmission queue by using corresponding setting of zeroMQ for transmission;
s1: the data preparation thread provides an external API interface, and the external API call triggers processing logic, and the detailed processing is as follows:
calling a sending interface unit, writing data into a sending interface, sending the data by the sending interface through ZMQ by adopting a PUSH mode of an inproc protocol, setting a sending data buffer, writing the sending data into the data buffer before sending the data, and then sending the data to a data sending thread through the buffer;
s2: the data sending thread monitors a data sending queue, and the data in the queue triggers the data sending logic to process, wherein the detailed processing is as follows: monitoring ZMQ, monitoring and receiving data by adopting a PULL mode of an inproc protocol, setting ZMQ a receiving buffer, storing the data in the buffer after monitoring the data, then acquiring the data from the buffer to prevent backlog and loss caused by overlarge data flow, packaging the acquired data, writing the acquired data into a ZMQ data buffer which is transmitted outwards, and transmitting the data in the buffer through a DEALER mode of a TCP protocol of ZeroMQ.
A data reception processing method, the method comprising the steps of:
the data receiving processing flow comprises the following steps: the data sending process is divided into two threads, namely a data receiving thread and a data recording thread, wherein the two threads share data through an asynchronous IO data channel, exchange data, the data receiving thread continuously and asynchronously acquire data from a data queue through a TCP protocol of ZeroMQ, write the data into the asynchronous IO data channel, the data recording thread acquires the data from the asynchronous IO data channel, writes the data into a file through an asynchronous IO technology, carries out data recording, and reduces memory copying in the processing process, reduces the consumption of the cache and improves the writing speed of the data through sharing data receiving and data cache addresses in the asynchronous IO data channel of the asynchronous IO;
step one, a data receiving thread monitors a data sending queue, and when data exists in the queue, the data receiving thread triggers data receiving logic processing, and the detailed processing is as follows:
s1: initializing relevant contexts such as context, iocb, io_event and the like related to asynchronous IO;
s2: initializing a data cache, an idle list lock, a used list, a list lock and the like for sharing by an asynchronous IO channel queue;
s3: starting a data recording thread and sharing the content with the data recording thread;
s4: monitoring ZMQ, monitoring and receiving data by adopting a DEALER mode of the TCP protocol;
s5: setting ZMQ to receive the buffer, storing the data in the buffer after monitoring the data, and then acquiring the data from the buffer to prevent backlog and loss caused by overlarge data flow;
s6: judging whether the asynchronous IO has an idle channel, if so, writing data into the idle channel (sharing a memory address in a cache with a data recording thread);
s7: in order to prevent the conflict of the shared variable operation of the data receiving thread and the data recording thread, the shared variable is locked before the shared variable operation is finished, and then unlocked.
Step two, the data recording thread circularly processes the asynchronous IO channel, the asynchronous IO writing process is performed when the data in the queue is found, the data in the shared buffer memory is released after writing, so that the data receiving thread writes the data into the idle unit, and the detailed process is as follows:
s1: circularly judging whether data exists in a queue of the asynchronous IO channel, and performing the following processing when the data exists;
s2: locking the used queue, acquiring the data position to be processed from the used queue, and then unlocking;
s3: acquiring data of the corresponding data position from the cache queue according to the acquired data position;
s4: according to the acquired data position, designating a corresponding available channel in the iocb of the asynchronous IO;
s5: calling io_prep_write to write data into the asynchronous IO;
s6: calling io_submit to submit asynchronous IO;
s7: updating the idle queue;
s8: and judging whether the idle queue is smaller than half of the whole queue length, if so, calling io_getes to acquire an idle unit, and updating the idle queue.
The invention provides a data multi-level cache and high-speed transmission recording method based on MQ and asynchronous IO. The beneficial effects are as follows: the invention utilizes the multi-level caching technology and the zeroMQ technology to realize high-speed stable network transmission and maximally utilizes network bandwidth resources at the transmission level. On the data recording layer, on one hand, asynchronous writing files are carried out through asynchronous IO, frequent switching of a User Mode and a Kernel Mode of a CPU is reduced, and the data writing speed is improved; on the other hand, through sharing data receiving and data caching of asynchronous IO, memory copying in the processing process is reduced, caching consumption is reduced, and data writing speed is improved. By optimizing and improving the transmission layer and the data recording layer, the speed and the stability of the transmission record of the whole system are improved.
Drawings
FIG. 1 is a diagram of the overall architecture of the system of the present invention.
FIG. 2 is a diagram of a system cache architecture according to the present invention.
Fig. 3 is a flow chart of the data transmitting end processing of the present invention.
Fig. 4 is a flow chart of the data receiving end processing of the present invention.
FIG. 5 is a detailed flow chart of a data receiving thread according to the present invention.
FIG. 6 is a detailed flow chart of a data recording thread of the present invention.
Detailed Description
According to a first aspect of the present invention, the present invention provides a data multi-level buffer and high-speed transmission recording method based on MQ and asynchronous IO, as shown in fig. 1-6, the present invention comprises two parts of four units and multi-level buffer at a transmitting end and a receiving end, and the data transmitting end comprises a data transmitting interface unit and a high-speed data transmitting unit.
The data receiving end is composed of a high-speed data receiving unit and an asynchronous IO recording unit.
The buffers include a multi-level buffer composition of a transmit data buffer and a multi-level buffer composition of a receive data buffer (see fig. 2 for details).
In order to improve the stability of data transmission, the system designs a multi-level cache architecture.
The sending end comprises three levels of caches: in order to decouple the data preparation thread (data transmission interface unit) and the data transmission thread (high-speed data transmission unit), communication is carried out between the data preparation thread and the data transmission thread by means of an inter-thread communication mode inproc protocol of ZeroMQ; in order to improve stability of data communication, data caches are set at both the PUSH end and the PULL end of ZeroMQ, namely a first-level cache of a transmitting end and a second-level cache of the transmitting end in the following diagram. When the high-speed data transmitting unit transmits data to the receiving party, the data is transmitted by adopting a DEALER mode of the TCP protocol of the zeroMQ, and a transmitting end buffer memory, namely a transmitting end three-level buffer memory in the following figure, is arranged for enhancing the transmission stability of the data.
The receiving end comprises two levels of caches: the data is received by a high-speed data receiving unit, the data is received by adopting a DEALER mode of a TCP protocol of zeroMQ, and the stability of the data is improved by setting a TCP receiving buffer, namely a receiving end first-level buffer in the following figure. In order to improve the performance and reduce the coupling degree between the modules, a record data buffer, namely a receiving end secondary buffer of the lower graph, is arranged between the high-speed data receiving unit and the asynchronous IO recording unit.
Meanwhile, the invention adopts the following key technology:
ZeroMQ technology selection
ZeroMQ provides various transmission tools such as sockets for carrying out atomic message transmission in processes, among processes, in TCP and in multicasting, compared with the traditional TCP protocol, zeroMQ provides various modes for realizing N-to-N socket connection, thereby making up the limitation that TCP can only carry out one-to-one transmission, shielding the problems of packet sticking and half packet processing during streaming and having better expansibility; compared with the traditional UDP protocol, the zeroMQ can provide stable transmission, and solves the packet loss problem of the UDP protocol.
Compared with other message queues such as ZeroMQ, rabbitMQ, rocketMQ, kafka and the like, the method has the advantages of high concurrency, high throughput, low delay and the like, single-machine throughput is 100 ten thousand, delay is microsecond, and the method is more suitable for a high-speed stable transmission scene of mass data. The ZeroMQ core is written in C language, supports APIs for multiple programming languages such as C, C ++, java, etc., and the c++ language integrates the natural advantages and can run on most operating systems.
Asynchronous IO technology selection
In the process of data transmission recording, the speed of the CPU is far faster than IO of a disk, a network and the like. In one thread, the speed of executing codes by the CPU is extremely high, however, once IO operations such as reading and writing files and sending network data are encountered, the completion of the IO operations needs to be waited for before the next operation can be continued. During IO operations, the current thread is suspended, and other code that requires CPU execution cannot be executed by the current thread.
Because one IO operation blocks the current thread, causing other code to be unable to execute, we must use multithreading or multiprocessing to concurrently execute code to serve multiple users. Each user will be assigned a thread, and if an IO is encountered, the threads are suspended, and the threads of other users are not affected.
The multithreading and multiprocessing model, while solving the concurrency problem, does not allow the system to add threads without any upper limit. Since the overhead of switching threads is also large, once the number of threads is excessive, the CPU time is spent on line Cheng Qie, and the time to actually run code is small, resulting in a serious performance degradation.
Asynchronous IO may solve the IO problems described above. When encountering IO operation, the code is only responsible for sending out IO requests, does not wait for IO results, then directly ends the round of message processing, and enters the next round of message processing process. When the IO operation is completed, an IO completed message is received, and the IO operation result can be directly obtained when the message is processed.
In the period from sending IO request to receiving IO completion, the main thread can only be suspended under the synchronous IO model, but under the asynchronous IO model, the main thread does not rest, and other messages continue to be processed in the message cycle. Thus, under the asynchronous IO model, one thread can process multiple IO requests simultaneously, and no thread operation is switched. For most IO intensive applications, the use of asynchronous IOs will greatly increase the multitasking capability of the system.
During data transmission processing, firstly, the data transmission processing is divided into two threads, namely a data preparation thread and a data transmission thread, and the two threads perform data caching through a data transmission queue to exchange data. The data preparation thread continuously writes the prepared data into the data queue through the inproc protocol of the zeroMQ, and the data sending thread continuously acquires the data from the data queue through the inproc protocol of the zeroMQ and sends the data out through the TCP protocol of the zeroMQ.
Access mode for transmitting data
Flexible data transmission scenes are provided for the data transmission module, and two data access modes are supported:
access mode 1: providing a data access mode of an API interface, and enabling an external application to directly call the API interface to finish data transmission; (see Access mode 1 of FIG. 3)
Access mode 2: and providing relevant parameters of a transmission data queue, and enabling an external application to write data into the transmission data queue by using corresponding setting of zeroMQ for transmission. (see access mode 2 of FIG. 3)
The data preparation thread provides an external API interface, and the external API call triggers processing logic, and the detailed processing is as follows:
calling a transmitting interface unit to write data into the transmitting interface
The sending interface sends data by ZMQ in PUSH mode of inproc protocol.
Setting a sending data buffer, writing the sending data into the data buffer before sending the data, and then sending the data to a data sending thread through the buffer.
The data sending thread monitors a data sending queue, and the data in the queue triggers the data sending logic to process, wherein the detailed processing is as follows:
listening ZMQ, the push mode using the inproc protocol listens for and receives data.
The ZMQ is arranged to receive the buffer memory, store the data in the buffer memory after monitoring the data, and then acquire the data from the buffer memory so as to prevent backlog and loss caused by overlarge data flow.
Packaging the obtained data, and writing the obtained data into a ZMQ data cache which is sent outwards
Transmitting the data in the buffer memory through a DEALER mode of a TCP protocol of the zeroMQ;
during data receiving processing, data sending processing is divided into two threads, namely a data receiving thread and a data recording thread, and the two threads share data and exchange data through an asynchronous IO data channel. The data receiving thread continuously and asynchronously acquires data from the data queue through the TCP protocol of the zeroMQ and writes the data into the asynchronous IO data channel. The data recording thread acquires data from the asynchronous IO data channel, writes the data into the file through the asynchronous IO technology, and records the data. By sharing data receiving and data buffer addresses in an asynchronous IO data channel of an asynchronous IO, memory copying in the processing process is reduced, buffer consumption is reduced, and data writing speed is improved.
The data receiving thread monitors a data sending queue, and when data exists in the queue, the data receiving thread triggers the data receiving logic to process, and the detailed processing is as follows:
and initializing relevant contexts such as context, iocb, io_event and the like related to the asynchronous IO.
Initializing data cache for sharing by the asynchronous IO channel queue, free list lock, used list, list lock and the like.
And starting the data recording thread and sharing the content with the data recording thread.
Listening ZMQ, the data is listened to and received using the DEALER mode of the TCP protocol.
The ZMQ is arranged to receive the buffer memory, store the data in the buffer memory after monitoring the data, and then acquire the data from the buffer memory so as to prevent backlog and loss caused by overlarge data flow.
Judging whether the asynchronous IO has a free channel, if so, writing the data into the free channel (sharing the memory address in the cache with the data recording thread).
In order to prevent the conflict of the shared variable operation of the data receiving thread and the data recording thread, the shared variable is locked before the shared variable operation is finished, and then unlocked.
The data recording thread circularly processes the asynchronous IO channel, and finds that the data in the queue is subjected to asynchronous IO writing processing, and releases the data in the shared cache after writing so that the data receiving thread writes the data into the idle unit. The detailed process is as follows:
and circularly judging whether data exists in the queue of the asynchronous IO channel, and performing the following processing when the data exists.
The used queue is locked, the data position needing to be processed is obtained from the used queue, and then the data is unlocked.
And acquiring the data of the corresponding data position from the cache queue according to the acquired data position.
Designating a corresponding available channel in the iocb of the asynchronous IO according to the acquired data position
Invoking io_prep_pwrite to write data to asynchronous IO
Invoking io_submit to asynchronous IO
Updating an idle queue
And judging whether the idle queue is smaller than half of the whole queue length, if so, calling io_getes to acquire an idle unit, and updating the idle queue.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (2)
1. The data multi-level caching and high-speed transmission recording method based on MQ and asynchronous IO is suitable for the whole system architecture, and is characterized in that: the whole system architecture consists of a data sending end, a data receiving end and a multi-level cache;
the data transmitting end consists of a data transmitting interface unit and a high-speed data transmitting unit;
the data receiving end consists of a high-speed data receiving unit and an asynchronous IO recording unit;
the multi-level cache comprises a multi-level cache of a sending data cache and a multi-level cache of a receiving data cache;
the data sending end comprises a third-level cache, the third-level cache consists of a first-level cache of the sending end, a second-level cache of the sending end and a third-level cache of the sending end, communication is carried out between a data preparation thread and a data sending thread through an inter-thread communication mode inproc protocol of the zeroMQ, the data caches are arranged at a PUSH end and a PULL end of the zeroMQ, and the first-level cache and the second-level cache of the sending end of the data cache are arranged; when the high-speed data transmitting unit transmits data to a receiving party, a DEALER mode of a TCP protocol of ZeroMQ is adopted for transmitting, a transmitting end buffer memory is arranged for enhancing the transmission stability of the data, and the transmitting end buffer memory is a transmitting end three-level buffer memory;
the data receiving end comprises two levels of caches, the two levels of caches are composed of a first-level cache of the receiving end and a second-level cache of the receiving end, data are received through a high-speed data receiving unit, a DEALER mode of a TCP protocol of zeroMQ is adopted for receiving the data, the stability of the data is improved by setting a TCP receiving cache, the TCP receiving cache is the first-level cache of the receiving end, and a record data cache is arranged between the high-speed data receiving unit and an asynchronous IO recording unit, so that the performance is improved and the coupling degree between modules is reduced, and the record data cache is the second-level cache of the receiving end;
the data receiving processing flow is divided into two threads, namely a data receiving thread and a data recording thread, wherein the two threads share data through an asynchronous IO data channel, exchange data, the data receiving thread continuously acquires data asynchronously from a data queue through a TCP protocol of zeroMQ, writes the data into the asynchronous IO data channel, the data recording thread acquires the data from the asynchronous IO data channel, writes the data into a file through an asynchronous IO technology, carries out data recording, and reduces memory copying in the processing process, reduces cache consumption and improves the writing speed of the data through sharing data receiving and data cache addresses in the asynchronous IO data channel of the asynchronous IO;
the method comprises the following steps:
step one, a data receiving thread monitors a data sending queue, and when data exists in the queue, the data receiving thread triggers data receiving logic processing, and the detailed processing is as follows:
s1: initializing relevant contexts such as context, iocb, io_event and the like related to asynchronous IO;
s2: initializing data cache, idle list lock, used list and list lock shared by the asynchronous IO channel queues;
s3: starting a data recording thread and sharing the data recording thread;
s4: monitoring ZMQ, monitoring and receiving data by adopting a DEALER mode of the TCP protocol;
s5: setting ZMQ to receive the buffer, storing the data in the buffer after monitoring the data, and then acquiring the data from the buffer to prevent backlog and loss caused by overlarge data flow;
s6: judging whether the asynchronous IO has an idle channel, if so, writing data into the idle channel;
s7: in order to prevent the conflict of the shared variable operation of the data receiving thread and the data recording thread, locking is performed before the shared variable operation, and unlocking is performed after the operation is completed;
step two, the data recording thread circularly processes the asynchronous IO channel, the asynchronous IO writing process is performed when the data in the queue is found, the data in the shared buffer memory is released after writing, so that the data receiving thread writes the data into the idle unit, and the detailed process is as follows:
s1: circularly judging whether data exists in a queue of the asynchronous IO channel, and performing the following processing when the data exists;
s2: locking the used queue, acquiring the data position to be processed from the used queue, and then unlocking;
s3: acquiring data of the corresponding data position from the cache queue according to the acquired data position;
s4: according to the acquired data position, designating a corresponding available channel in the iocb of the asynchronous IO;
s5: calling io_prep_write to write data into the asynchronous IO;
s6: calling io_submit to submit asynchronous IO;
s7: updating the idle queue;
s8: and judging whether the idle queue is smaller than half of the whole queue length, if so, calling io_getes to acquire an idle unit, and updating the idle queue.
2. The MQ and asynchronous IO based data multi-level buffering and high-speed transmission recording method of claim 1, wherein:
the data transmission processing flow is as follows: the data transmission processing is divided into two threads, namely a data preparation thread and a data transmission thread, wherein the two threads perform data caching through a data transmission queue and exchange data; the data preparation thread continuously writes the prepared data into the data queue through the inproc protocol of the zeroMQ, the data sending thread continuously acquires the data from the data queue through the inproc protocol of the zeroMQ, the data is sent out through the TCP protocol of the zeroMQ, and the access mode of the data sending is as follows: flexible data transmission scenes are provided for the data transmission module, and two data access modes are supported:
access mode 1: providing a data access mode of an API interface, and enabling an external application to directly call the API interface to finish data transmission;
access mode 2: providing relevant parameters of a data transmission queue, and enabling an external application to write data into the data transmission queue by using corresponding setting of zeroMQ for transmission;
the processing method comprises the following steps:
s1: the data preparation thread provides an external API interface, and the external API call triggers processing logic, and the detailed processing is as follows:
calling a sending interface unit, writing data into a sending interface, sending the data by the sending interface through ZMQ by adopting a PUSH mode of an inproc protocol, setting a sending data buffer, writing the sending data into the data buffer before sending the data, and then sending the data to a data sending thread through the buffer;
s2: the data sending thread monitors a data sending queue, and the data in the queue triggers the data sending logic to process, wherein the detailed processing is as follows: monitoring ZMQ, monitoring and receiving data by adopting a PULL mode of an inproc protocol, setting ZMQ a receiving buffer, storing the data in the buffer after monitoring the data, then acquiring the data from the buffer to prevent backlog and loss caused by overlarge data flow, packaging the acquired data, writing the acquired data into a ZMQ data buffer which is transmitted outwards, and transmitting the data in the buffer through a DEALER mode of a TCP protocol of ZeroMQ.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010868475.8A CN111949422B (en) | 2020-08-26 | 2020-08-26 | Data multi-level cache and high-speed transmission recording method based on MQ and asynchronous IO |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010868475.8A CN111949422B (en) | 2020-08-26 | 2020-08-26 | Data multi-level cache and high-speed transmission recording method based on MQ and asynchronous IO |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111949422A CN111949422A (en) | 2020-11-17 |
| CN111949422B true CN111949422B (en) | 2024-04-16 |
Family
ID=73366538
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010868475.8A Active CN111949422B (en) | 2020-08-26 | 2020-08-26 | Data multi-level cache and high-speed transmission recording method based on MQ and asynchronous IO |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111949422B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115334156A (en) * | 2021-04-26 | 2022-11-11 | 深信服科技股份有限公司 | Message processing method, device, equipment and storage medium |
| CN113946362B (en) * | 2021-09-02 | 2022-08-16 | 贝壳找房(北京)科技有限公司 | Consumption data processing method and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1998034391A2 (en) * | 1997-02-03 | 1998-08-06 | Mci Worldcom, Inc. | A communication system architecture |
| CN102622426A (en) * | 2012-02-27 | 2012-08-01 | 杭州闪亮科技有限公司 | Database writing system and database writing method |
| CN108696578A (en) * | 2018-04-26 | 2018-10-23 | 昆明理工大学 | The communications framework design method that multiple machine distributing based on ZeroMQ calculates |
| CN108829713A (en) * | 2018-05-04 | 2018-11-16 | 华为技术有限公司 | Distributed cache system, cache synchronization method and device |
| CN110135575A (en) * | 2017-12-29 | 2019-08-16 | 英特尔公司 | Communication optimization for distributed machines study |
| CN110166562A (en) * | 2019-05-24 | 2019-08-23 | 东软集团股份有限公司 | Data synchronous method, apparatus, storage medium and electronic equipment |
| CN111104232A (en) * | 2019-11-09 | 2020-05-05 | 苏州浪潮智能科技有限公司 | Method, device and medium for accelerating message writing of message queue |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10523603B2 (en) * | 2017-03-31 | 2019-12-31 | Bmc Software, Inc. | Systems and methods for intercepting access to messaging systems |
| US11373266B2 (en) * | 2017-05-05 | 2022-06-28 | Intel Corporation | Data parallelism and halo exchange for distributed machine learning |
| US20200026787A1 (en) * | 2018-07-23 | 2020-01-23 | Wipro Limited | METHOD, SYSTEM, AND FRAMEWORK FOR IMPLEMENTING INTERNET OF THINGS (IoT) APPLICATIONS |
-
2020
- 2020-08-26 CN CN202010868475.8A patent/CN111949422B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1998034391A2 (en) * | 1997-02-03 | 1998-08-06 | Mci Worldcom, Inc. | A communication system architecture |
| CN102622426A (en) * | 2012-02-27 | 2012-08-01 | 杭州闪亮科技有限公司 | Database writing system and database writing method |
| CN110135575A (en) * | 2017-12-29 | 2019-08-16 | 英特尔公司 | Communication optimization for distributed machines study |
| CN108696578A (en) * | 2018-04-26 | 2018-10-23 | 昆明理工大学 | The communications framework design method that multiple machine distributing based on ZeroMQ calculates |
| CN108829713A (en) * | 2018-05-04 | 2018-11-16 | 华为技术有限公司 | Distributed cache system, cache synchronization method and device |
| CN110166562A (en) * | 2019-05-24 | 2019-08-23 | 东软集团股份有限公司 | Data synchronous method, apparatus, storage medium and electronic equipment |
| CN111104232A (en) * | 2019-11-09 | 2020-05-05 | 苏州浪潮智能科技有限公司 | Method, device and medium for accelerating message writing of message queue |
Non-Patent Citations (2)
| Title |
|---|
| Architecture Design Of Online Education System;Ren Sike 等;《2017IEEE/ACIS 16th International Conference on Computer and Information Science(ICIS)》;1-4 * |
| 多通道超高速数据采集与记录系统的设计与实现;胡惠;《中国优秀硕士学位论文全文数据库 信息科技辑》;I140-579 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111949422A (en) | 2020-11-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110691062B (en) | Data writing method, device and equipment | |
| US6577542B2 (en) | Scratchpad memory | |
| CN109729024B (en) | Data packet processing system and method | |
| KR102011949B1 (en) | System and method for providing and managing message queues for multinode applications in a middleware machine environment | |
| US8505013B2 (en) | Reducing data read latency in a network communications processor architecture | |
| CN111949422B (en) | Data multi-level cache and high-speed transmission recording method based on MQ and asynchronous IO | |
| US8514874B2 (en) | Thread synchronization in a multi-thread network communications processor architecture | |
| EP1390847B1 (en) | Reducing latency and message traffic during data and lock transfer in a multi-node system | |
| CN111459418B (en) | RDMA (remote direct memory Access) -based key value storage system transmission method | |
| US20110225372A1 (en) | Concurrent, coherent cache access for multiple threads in a multi-core, multi-thread network processor | |
| US8910171B2 (en) | Thread synchronization in a multi-thread network communications processor architecture | |
| US20130282853A1 (en) | Apparatus and method for processing data in middleware for data distribution service | |
| CN111124270B (en) | Method, apparatus and computer program product for cache management | |
| US11537453B2 (en) | Multithreaded lossy queue protocol | |
| CN111209123A (en) | Local storage IO protocol stack data interaction method and device | |
| CN113032162B (en) | Multi-process communication method based on shared memory backup mechanism | |
| US20130067160A1 (en) | Producer-consumer data transfer using piecewise circular queue | |
| CN112698959A (en) | Multi-core communication method and device | |
| CN109831394B (en) | Data processing method, terminal and computer storage medium | |
| JP2003248622A (en) | Memory system for increased bandwidth | |
| CN112486702B (en) | Global message queue implementation method based on multi-core multi-processor parallel system | |
| CN112148441A (en) | Embedded message queue realizing method of dynamic storage mode | |
| CN110413689B (en) | Multi-node data synchronization method and device for memory database | |
| CN111638979A (en) | Call request processing method and device, electronic equipment and readable storage medium | |
| Faraji et al. | Design considerations for GPU‐aware collective communications in MPI |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |