CN106293973B - lock-free message queue communication method and system - Google Patents
lock-free message queue communication method and system Download PDFInfo
- Publication number
- CN106293973B CN106293973B CN201610681120.1A CN201610681120A CN106293973B CN 106293973 B CN106293973 B CN 106293973B CN 201610681120 A CN201610681120 A CN 201610681120A CN 106293973 B CN106293973 B CN 106293973B
- Authority
- CN
- China
- Prior art keywords
- message
- queue
- lock
- free
- read
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/543—Local
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Abstract
The invention discloses a lock-free message queue communication method and a lock-free message queue communication system. The lock-free message queue communication method comprises the following steps: creating a lock-free message queue in the shared memory, wherein the lock-free message queue comprises at least one message body arranged according to a first-in first-out principle; receiving delivery messages sent by at least one writing process, and writing message bodies corresponding to the delivery messages of each writing process into the tail part of a queue of a non-lock message queue in sequence according to the sequence of the received delivery messages; and receiving the read messages sent by at least one read process, and sending the message bodies to each read process one by one from the head of the queue of the lock-free message queue according to the sequence of the received read messages. All message bodies in the lock-free message queue communication method and system are stored in a shared memory according to a first-in first-out principle, and concurrent safe communication of a large amount of data interaction among a plurality of processes in the same equipment can be realized through a writing process and a reading process.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a lock-free message queue communication method and system.
Background
The conventional interprocess communication common modes comprise pipelines, named pipelines, semaphores, message queues, signals, shared memories and sockets. The pipeline is a half-duplex communication mode, data can only flow in a single direction, and the pipeline can only be used between parent and child processes. Named pipes are half-duplex communication methods, and allow communication between unrelated processes. The semaphore is a counter that controls access to the shared resource by multiple processes; semaphores act as a lock mechanism to prevent a process from accessing a shared resource while other processes are accessing the resource. A signal is a relatively complex communication means for informing a receiving process that an event has occurred. The message queue is a linked list of messages, stored in the kernel and identified by a message queue identifier. The shared memory is a memory which is mapped and can be accessed by other processes, the shared memory is created by one process, but a plurality of processes can access the shared memory; the shared memory is the fastest IPC mode and is specially designed for low running efficiency of other interprocess communication modes; shared memory is often used in conjunction with other communication mechanisms to achieve synchronization and communication between processes. Socket is an inter-process communication mechanism, can be used for inter-process communication between different processes, and is a bottom layer communication mechanism of a network layer. The pipeline and the named pipeline are only limited to the communication between the parent and child processes; signals and semaphores are not suitable for application scenarios where a large amount of data is interacted between processes; sockets are a communication mechanism of a network layer, and the communication efficiency is lost; the message queue and the shared memory have the process safety problem when multi-process concurrent reading and writing is carried out.
disclosure of Invention
the technical problem to be solved by the present invention is to provide a lock-free message queue communication method and system based on a shared memory, aiming at the defects of the prior art, so as to solve the problem of concurrency security existing in the existing interaction of a large amount of data among multiple processes.
The technical scheme adopted by the invention for solving the technical problems is as follows: a lock-free message queue communication method, comprising:
creating a lock-free message queue in a shared memory, wherein the lock-free message queue comprises at least one message body arranged according to a first-in first-out principle;
Receiving delivery messages sent by at least one writing process, and writing message bodies corresponding to the delivery messages of each writing process into the tail part of the queue of the non-lock message queue in sequence according to the sequence of the received delivery messages;
And receiving the read messages sent by at least one read process, and sending the message bodies to each read process one by one from the head of the queue of the lock-free message queue according to the sequence of the received read messages.
preferably, the lock-free message queue includes a queue head memory block for storing head information, a message index memory block for storing a message index, and a message content memory block for storing message content;
The receiving at least one delivery message sent by the writing process, and writing the message body corresponding to the delivery message of each writing process into the tail part of the queue of the non-lock message queue according to the sequence of the received delivery messages, includes:
At least one writing process acquires queue writing authority and seizes message index writing authority in sequence, and sends delivery messages to the lock-free message queue;
determining the sequence of the received delivery messages according to the preemption message index writing permission of each writing process;
Writing the message body corresponding to the delivery message of each write process into the message content memory block at the tail part of the queue of the lock-free message queue in sequence;
updating the message index in the message index memory block, and releasing the writing permission of the preemptive message index;
and updating the head information in the memory block at the head of the queue, and releasing the write permission of the queue.
preferably, the receiving the read message sent by at least one read process, and sending the message body to each read process one by one from the queue head of the lock-free message queue according to the sequence of the received read messages includes:
at least one reading process acquires queue reading permission and sends a reading message to the lock-free message queue;
Traversing all message indexes in the message index memory block based on the queue reading permission to acquire unread message contents;
Each reading process acquires the preemptive message index reading authority and determines the sequence of the received read messages according to the preemptive message index reading authority;
Sending the unread message contents to a reading process corresponding to the preemptive message index reading right one by one from the head of the queue of the lock-free message queue;
Updating the message index in the message index memory block, and releasing the read permission of the preemptive message index;
and updating the head information in the memory block at the head of the queue, and releasing the queue reading permission.
preferably, the writing process acquires the queue writing permission by adopting an atomic function; and the reading process acquires the queue reading permission by adopting an atomic function.
Preferably, the writing the message body corresponding to the delivery message of each write process into the queue tail of the lock-free message queue in sequence further includes: issuing a message queue-entering notification to all read processes;
Said sequentially sending message bodies from the head of said lock-free message queue to each of said read processes one by one, further comprising: a message dequeue notification is sent to all write processes.
the invention also provides a lock-free message queue communication system, comprising:
the device comprises a lock-free message queue creating module, a lock-free message queue creating module and a lock-free message queue processing module, wherein the lock-free message queue creating module is used for creating a lock-free message queue in a shared memory, and the lock-free message queue comprises at least one message body which is arranged according to a first-in first-out principle;
The write process communication module is used for receiving delivery messages sent by at least one write process and writing message bodies corresponding to the delivery messages of each write process into the tail part of the lock-free message queue in sequence according to the sequence of the received delivery messages;
And the read process communication module is used for receiving the read messages sent by at least one read process and sending the message bodies to each read process one by one from the head of the queue of the lock-free message queue according to the sequence of the received read messages.
preferably, the lock-free message queue includes a queue head memory block for storing head information, a message index memory block for storing a message index, and a message content memory block for storing message content;
the write process communication module comprises:
the write permission acquisition submodule is used for enabling at least one write process to successively acquire queue write permission and seize message index write permission and sending a delivery message to the lock-free message queue;
The writing process sequence determining submodule is used for determining the sequence of the received delivery messages according to the preemption message index writing authority of each writing process;
A message body writing sub-module, configured to sequentially write a message body corresponding to the delivery message of each write process into the message content memory block at the tail of the lock-free message queue;
A message index write permission release module, configured to update the message index in the message index memory block, and release the write permission of the preemption message index;
And the queue write permission releasing module is used for updating the head information in the memory block at the head of the queue and releasing the queue write permission.
preferably, the read process communication module includes:
a queue read permission obtaining submodule, configured to enable at least one read process to obtain a queue read permission, and send a read message to the lock-free message queue;
A message content obtaining sub-module, configured to traverse all message indexes in the message index memory block based on the queue read permission to obtain unread message content;
A reading process sequence determining submodule, configured to enable each reading process to acquire a preemption message index reading permission, and determine a sequence of received reading messages according to the preemption message index reading permission;
The message body reading submodule is used for sequentially sending the unread message contents to a reading process corresponding to the message index reading permission from the head of the lock-free message queue one by one;
a message index read permission release submodule, configured to update the message index in the message index memory block, and release the preemption message index read permission;
and the queue reading permission releasing submodule is used for updating the head information in the memory block at the head of the queue and releasing the queue reading permission.
preferably, the writing process acquires the queue writing permission by adopting an atomic function; and the reading process acquires the queue reading permission by adopting an atomic function.
Preferably, the write process communication module further includes a message in-queue notification issuing sub-module, configured to issue message in-queue notifications to all read processes;
The read process communication module also comprises a message dequeue notification issuing submodule used for sending message dequeue notification to all the write processes.
Compared with the prior art, the invention has the following advantages: in the communication method of the lock-free message queue, the lock-free message queue is created in the shared memory, so that the lock-free message queue can receive the delivery message sent by at least one writing process to finish the writing of the message body; and can receive the read message that at least one read process sent, finish the reading of the message body. All message bodies in the lock-free message queue communication method are stored in a shared memory according to a first-in first-out principle, and concurrent safe communication of a large amount of data interaction among a plurality of processes in the same equipment can be realized through a writing process and a reading process.
in the communication system of the lockless message queue provided by the invention, the lockless message queue is established in the shared memory through the lockless message queue establishing module; the lock-free message queue can receive delivery messages sent by at least one writing process through the writing process communication module to finish the writing of the message body; and the lock-free message queue can receive the read message sent by at least one read process through the read process communication module to finish the reading of the message body. All message bodies in the lock-free message queue communication system are stored in a shared memory according to a first-in first-out principle, and concurrent safe communication of a large amount of data interaction among a plurality of processes in the same equipment can be realized through a writing process and a reading process.
drawings
The invention will be further described with reference to the accompanying drawings and examples, in which:
fig. 1 is a schematic diagram of a lock-free message queue communication method in embodiment 1 of the present invention.
Fig. 2 is a flowchart of a lock-free message queue communication method according to embodiment 1 of the present invention.
fig. 3 is a partial flowchart of a lock-free message queue communication method in embodiment 1 of the present invention.
Fig. 4 is another partial flowchart of a lock-free message queue communication method in embodiment 1 of the present invention.
fig. 5 is a schematic block diagram of a lock-free message queue communication system according to embodiment 2 of the present invention.
fig. 6 is a schematic block diagram of a write process communication module in embodiment 2 of the present invention.
fig. 7 is a schematic block diagram of a read process communication module in embodiment 2 of the present invention.
in the figure: 1. a lock-free message queue creation module; 2. a write process communication module; 21. a write permission obtaining submodule; 22. determining a submodule of the writing process sequence; 23. the message body is written into the submodule; 24. a message index write permission release module; 25. a queue write permission release module; 26. the message enqueue informing and issuing submodule; 3. a read process communication module; 31. a queue reading permission obtaining submodule; 32. a message content acquisition submodule; 33. a process reading sequence determining submodule; 34. a message body reading submodule; 35. a message index reading authority release submodule; 36. a queue read permission release submodule; 37. the message dequeue notification issuing submodule; 4. and a lock-free message queue destroying module.
Detailed Description
for a more clear understanding of the technical features, objects and effects of the present invention, embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
Example 1
Fig. 1 and 2 show a lock-free message queue communication method in the present embodiment. As shown in fig. 1 and 2, the lock-free message queue communication method includes:
S1: the host process creates a lock-free message queue in the shared memory, the lock-free message queue including at least one message body arranged according to a first-in-first-out principle. Specifically, the main process creates an instance of a lock-free message queue class (CMsgQueueShm), calls an OPEN operation to complete the creation of the lock-free message queue, and completes the initialization operation of the queue header information. In this embodiment, all the message bodies are stored in the shared memory, and data interaction can be completed through read-write operations on the message bodies in the shared memory. Specifically, the lock-free message queue includes a queue head memory block for storing head information, a message index memory block for storing a message index, and a message content memory block for storing message content.
the memory block at the head of the queue reserves basic information of the queue, including queue capacity (i.e. the maximum number of messages to be saved), current usage number (i.e. the number of write processes and the number of read processes), number of messages to be queued, number of messages to be dequeued, number of writable messages, number of readable messages, data write position (i.e. write position offset of message content area), and data read position (i.e. read position offset of message content area). The message index memory block retains index information of each message, including whether the current message is being used (i.e. being read by a reading process or being written by a writing process normally), the number of current writing processes, the number of current reading processes, the length of message content, the number of message indexes, and the starting position of the message in the message content area. The message content memory block is used for reserving the specific content of the message body and consists of a whole large memory block.
s2: and receiving delivery messages sent by at least one writing process, and writing the message bodies corresponding to the delivery messages of each writing process into the tail part of the queue of the non-lock message queue in sequence according to the sequence of the received delivery messages. In this embodiment, the lock-free message queue may receive a delivery message sent by at least one write process, complete a concurrent operation of the at least one write process, and a message body written into the lock-free message queue by the write process strictly follows a first-in first-out principle, and only one message body may be written into the lock-free message queue by one write process. As shown in fig. 3, step S2 specifically includes:
S21: at least one writing process acquires the queue writing authority and the preemptive message index writing authority in sequence, and sends delivery messages to the lock-free message queue. Wherein, the queue write permission is the permission that the write process can write the message body into the lock-free message queue. Preempting the message index write permission refers to permission of a write process to preempt the priority of the message index in the shared memory. In the embodiment, the writing process adopts the atomic function to replace the lock function of the system critical area to acquire the queue writing authority, so that the data security is ensured, the system call can be reduced, and the concurrency performance of the lock-free message queue is improved.
s22: and determining the sequence of the received delivery messages according to the preemption message index writing permission of each writing process. It can be understood that, if the delivery messages of the two writing processes do not reach the lock-free message queue at the same time, the sequence of the delivery messages can be directly determined according to the time of the delivery messages reaching the lock-free message queue. When two or more delivery messages arrive at the lock-free message queue at the same time, the sequence of the received delivery messages can be determined according to the priority of the writing authority of the preemptive message index.
S23: and writing the message bodies corresponding to the delivery messages of each writing process into the message content memory block at the tail part of the queue of the lock-free message queue in sequence. Specifically, the message body corresponding to the delivery message of each write process is written into the lock-free message queue at the tail of the lock-free message queue according to the sequence of the delivery messages determined in step S22, so as to complete the message write (PUT) operation. Step S23 writes the message body of the write process into the lock-free message queue strictly following the first-in first-out principle, and one message body can only be written into the lock-free message queue by one write process. It can be understood that the length of the message body corresponding to the delivery message of each write process is not fixed, as long as the length of the message body is not greater than the length of the remaining memory in the message content memory block.
S24: and updating the message index in the message index memory block and releasing the write permission of the preemptive message index.
S25: and updating the head information in the memory block at the head of the queue, releasing the write permission of the queue, and finishing the write operation, so that the write process can not send a delivery message and a message body to the lock-free message queue any more, thereby ensuring that the write process can only write one message body to the lock-free message queue.
S26: and issuing a message queue entry notification to the reading processes so that all the reading processes can read newly enqueued message bodies in the lock-free message queue in time, thereby greatly improving the concurrency performance of the lock-free message queue, reducing the time for storing the message bodies in the lock-free message queue, greatly reducing the time for the message bodies to enter and exit the queue, realizing high concurrency and low delay of the lock-free message queue, and being very suitable for a scene of large-amount data interactive communication among multiple processes applied by a server.
S3: and receiving the read messages sent by at least one read process, and sending the message bodies to each read process one by one from the head of the queue of the lock-free message queue according to the sequence of the received read messages. In this embodiment, the lock-free message queue may receive a read message sent by at least one read process, and complete a concurrent operation of reading a message body by the at least one read process, and a message body in the lock-free message queue read by the read process strictly follows a first-in first-out principle, and only one message body can be read by one read process from the lock-free message queue, and each message body in the lock-free message queue can only be read by one read process. As shown in fig. 4, step S3 specifically includes:
S31: at least one read process obtains queue read permission and sends read messages to the lock-free message queue. The queue read permission refers to permission for a read process to read a message sent to the lock-free message queue and receive a message body sent by the lock-free message queue. In the embodiment, the read process adopts the atomic function to replace the lock function of the system critical area to acquire the queue read permission, so that the data security is ensured, the system call is reduced, and the concurrency performance of the lock-free message queue is improved.
S32: and traversing all the message indexes in the message index memory block based on the queue reading permission to acquire unread message contents. Specifically, when the lock-free message queue receives a read message of each read process, it needs to traverse all message indexes in the message index memory block to determine unread message indexes, and thus determine unread message contents corresponding to the unread message indexes. It can be understood that, in the process of traversing all the message indexes in the message index memory block by the read message of each read process, the read message index is not acquired, so that it is ensured that any message body read in the lock-free message queue is not read by another read process.
S33: and each reading process acquires the preemptive message index reading permission and determines the sequence of the received read messages according to the preemptive message index reading permission. The preemption message index reading permission refers to a permission of a reading process for preempting the priority of a message index in a shared memory. It can be understood that, if the read messages of the two read processes do not arrive at the lock-free message queue at the same time, the sequence of the read messages can be determined according to the time when the read messages arrive at the lock-free message queue. When two or more read messages arrive at the lock-free message queue at the same time, the sequence of the received read messages can be determined according to the priority of the preemptive message index read permission of each read process.
S34: and sequentially sending the unread message contents to a reading process corresponding to the preemptive message index reading right one by one from the head of the queue of the lock-free message queue. Specifically, according to the sequence of the read messages determined in step S33, sending the unread message content from the head of the lock-free message queue one by one to the read process corresponding to the preemptive message index read right, and completing the message read (GET) operation. This step strictly follows the first-in first-out principle, and any message body in the lock-free message queue can only be read by one read process.
S35: and updating the message index in the message index memory block and releasing the read permission of the message index.
S36: and updating the head information in the memory block at the head of the queue, releasing the queue reading permission, finishing the reading operation, so that the reading process can not send the reading message to the lock-free message queue any more, and receiving the unread message content (namely the unread message body) sent by the lock-free message queue.
s37: and issuing a message out-queue notification to all the writing processes so that all the writing processes write new message bodies into the lock-free message queue in time, thereby greatly improving the concurrency performance of the lock-free message queue, reducing the time for storing the message bodies in the lock-free message queue, greatly reducing the time for the message bodies to enter and exit the queue, realizing high concurrency and low time delay of the lock-free message queue, and being applied to a scene of large-amount data interactive communication among multiple processes applied to a server.
s4: the main process destroys the lock-free message queue in the shared memory, so that the memory block at the head of the queue releases the head information, the memory block of the message index releases the message index, and the memory block of the message index releases the message content.
in the communication method of the lock-free message queue provided by this embodiment, the lock-free message queue is created in the shared memory, so that the lock-free message queue can receive the delivery message sent by at least one writing process, and complete the writing of the message body; and can receive the read message that at least one read process sent, finish the reading of the message body. All message bodies in the lock-free message queue communication method are stored in a shared memory according to a first-in first-out principle, and concurrent safe communication of a large amount of data interaction among a plurality of processes in the same equipment can be realized through a writing process and a reading process.
Example 2
Fig. 5 shows a lock-free message queue communication system in the present embodiment. As shown in fig. 5, the lock-free message queue communication system includes a lock-free message queue creation module 1, a write process communication module 2, a read process communication module 3, and a lock-free message queue destruction module 4.
the lock-free message queue creating module 1 is configured to create a lock-free message queue in a shared memory by a host process, where the lock-free message queue includes at least one message body arranged according to a first-in first-out principle. Specifically, the main process creates an instance of a lock-free message queue class (CMsgQueueShm), calls an OPEN operation to complete the creation of the lock-free message queue, and completes the initialization operation of the queue header information. In this embodiment, all the message bodies are stored in the shared memory, and data interaction can be completed through read-write operations on the message bodies in the shared memory. Specifically, the lock-free message queue includes a queue head memory block for storing head information, a message index memory block for storing a message index, and a message content memory block for storing message content.
The memory block at the head of the queue reserves basic information of the queue, including queue capacity (i.e. the maximum number of messages to be saved), current usage number (i.e. the number of write processes and the number of read processes), number of messages to be queued, number of messages to be dequeued, number of writable messages, number of readable messages, data write position (i.e. write position offset of message content area), and data read position (i.e. read position offset of message content area). The message index memory block retains index information of each message, including whether the current message is being used (i.e. being read by a reading process or being written by a writing process normally), the number of current writing processes, the number of current reading processes, the length of message content, the number of message indexes, and the starting position of the message in the message content area. The message content memory block is used for reserving the specific content of the message body and consists of a whole large memory block.
And the writing process communication module 2 is used for receiving the delivery messages sent by at least one writing process and writing the message bodies corresponding to the delivery messages of each writing process into the tail part of the queue of the lock-free message queue in sequence according to the sequence of the received delivery messages. In this embodiment, the lock-free message queue may receive a delivery message sent by at least one write process, complete a concurrent operation of the at least one write process, and a message body written into the lock-free message queue by the write process strictly follows a first-in first-out principle, and only one message body may be written into the lock-free message queue by one write process. As shown in fig. 6, the write process communication module 2 specifically includes:
And the write permission obtaining submodule 21 is configured to enable at least one write process to successively obtain the queue write permission and the preemption message index write permission, and send a delivery message to the lock-free message queue. Wherein, the queue write permission is the permission that the write process can write the message body into the lock-free message queue. Preempting the message index write permission refers to permission of a write process to preempt the priority of the message index in the shared memory. In the embodiment, the writing process adopts the atomic function to replace the lock function of the system critical area to acquire the queue writing authority, so that the data security is ensured, the system call can be reduced, and the concurrency performance of the lock-free message queue is improved.
And the writing process sequence determining submodule 22 is configured to determine the sequence of the received delivery messages according to the preemption message index writing permission of each writing process. It can be understood that, if the delivery messages of the two writing processes do not reach the lock-free message queue at the same time, the sequence of the delivery messages can be directly determined according to the time of the delivery messages reaching the lock-free message queue. When two or more delivery messages arrive at the lock-free message queue at the same time, the sequence of the received delivery messages can be determined according to the priority of the writing authority of the preemptive message index.
the message body writing submodule 23 is configured to sequentially write the message bodies corresponding to the delivery messages of each writing process into the message content memory block at the tail of the queue of the lock-free message queue. Specifically, the message body corresponding to the delivery message of each write process is written into the lock-free message queue at the tail of the lock-free message queue according to the delivery message sequence determined in the write process sequence determining submodule 22, so as to complete the message write (PUT) operation. The message body writing submodule 23 writes the message bodies of the writing processes into the lock-free message queue strictly according to the first-in first-out principle, and one message body can only be written into the lock-free message queue by one writing process. It can be understood that the length of the message body corresponding to the delivery message of each write process is not fixed, as long as the length of the message body is not greater than the length of the remaining memory in the message content memory block.
And a message index write permission releasing module 24, configured to update the message index in the message index memory block, and release the write permission of the preemptive message index.
and the queue write permission release module 25 is configured to update the head information in the memory block at the head of the queue, release the queue write permission, and end the write operation, so that the write process cannot send the delivery message and the message body to the lock-free message queue any more, thereby ensuring that the write process can only write one message body to the lock-free message queue.
The message in-queue notification issuing sub-module 26 is configured to issue a message in-queue notification to the read processes, so that all the read processes can read a newly enqueued message body in the lock-free message queue in time, thereby greatly improving the concurrency performance of the lock-free message queue, reducing the time for storing the message body in the lock-free message queue, greatly reducing the time for the message body to enter or exit the queue, and achieving high concurrency and low delay of the lock-free message queue.
And the read process communication module 3 is used for receiving the read messages sent by at least one read process and sending the message bodies to each read process one by one from the head of the queue of the lock-free message queue according to the sequence of the received read messages. In this embodiment, the lock-free message queue may receive a read message sent by at least one read process, and complete a concurrent operation of reading a message body by the at least one read process, and a message body in the lock-free message queue read by the read process strictly follows a first-in first-out principle, and only one message body can be read by one read process from the lock-free message queue, and each message body in the lock-free message queue can only be read by one read process. As shown in fig. 7, the process reading communication module 3 specifically includes:
The queue read permission obtaining submodule 31 is configured to enable at least one read process to obtain a queue read permission and send a read message to the lock-free message queue. The queue read permission refers to permission for a read process to read a message sent to the lock-free message queue and receive a message body sent by the lock-free message queue. In the embodiment, the read process adopts the atomic function to replace the lock function of the system critical area to acquire the queue read permission, so that the data security is ensured, the system call is reduced, and the concurrency performance of the lock-free message queue is improved.
The message content obtaining sub-module 32 is configured to traverse all the message indexes in the message index memory block based on the queue read permission to obtain unread message content. Specifically, when the lock-free message queue receives a read message of each read process, it needs to traverse all message indexes in the message index memory block to determine unread message indexes, and thus determine unread message contents corresponding to the unread message indexes. It can be understood that, in the process of traversing all the message indexes in the message index memory block by the read message of each read process, the read message index is not acquired, so that it is ensured that any message body read in the lock-free message queue is not read by another read process.
And a reading process sequence determining submodule 33, configured to enable each reading process to obtain the preemption message index reading permission, and determine the sequence of the received reading message according to the preemption message index reading permission. The preemption message index reading permission refers to a permission of a reading process for preempting the priority of a message index in a shared memory. It can be understood that, if the read messages of the two read processes do not arrive at the lock-free message queue at the same time, the sequence of the read messages can be determined according to the time when the read messages arrive at the lock-free message queue. When two or more read messages arrive at the lock-free message queue at the same time, the sequence of the received read messages can be determined according to the priority of the preemptive message index read permission of each read process.
and the message body reading submodule 34 is used for sequentially sending the unread message contents to the reading process corresponding to the preemptive message index reading permission one by one from the head of the lock-free message queue. Specifically, the reading sequence determined by the reading sequence determining submodule 33 is followed, and the unread message content is sent to the reading process corresponding to the preemptive message index reading right one by one from the queue head of the lock-free message queue, so as to complete the message reading (GET) operation. This step strictly follows the first-in first-out principle, and any message body in the lock-free message queue can only be read by one read process.
And the message index read permission releasing submodule 35 is configured to update the message index in the message index memory block, and release the preemptive message index read permission.
The queue read permission releasing submodule 36 is configured to update the head information in the memory block at the head of the queue, release the queue read permission, and end the read operation, so that the read process can no longer send a read message to the lock-free message queue, and receive unread message content (i.e., unread message body) sent by the lock-free message queue.
The message dequeue notification issuing sub-module 37 is configured to issue a message dequeue notification to all writing processes, so that all writing processes write a new message body into the lock-free message queue in time, thereby greatly improving concurrency performance of the lock-free message queue, reducing time for storing the message body in the lock-free message queue, greatly reducing time for the message body to enter or exit the queue, and implementing high concurrency and low delay of the lock-free message queue.
And a lock-free message queue destroying module 4, configured to destroy the lock-free message queue in the shared memory by the master process, so that the head information is released by the memory block at the head of the queue, the message index is released by the message index memory block, and the message content is released by the message memory block.
In the communication system of the lock-free message queue provided in this embodiment, the lock-free message queue is created in the shared memory by the lock-free message queue creating module 1; the write process communication module 2 enables the lock-free message queue to receive the delivery message sent by at least one write process, and completes the writing of the message body; and the read progress communication module 3 enables the lock-free message queue to receive the read message sent by at least one read progress so as to complete the reading of the message body. All message bodies in the lock-free message queue communication system are stored in a shared memory according to a first-in first-out principle, and concurrent safe communication of a large amount of data interaction among a plurality of processes in the same equipment can be realized through a writing process and a reading process.
While the invention has been described with reference to the specific embodiments thereof, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from its scope. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.
Claims (8)
1. A lock-free message queue communication method, comprising:
Creating a lock-free message queue in a shared memory, wherein the lock-free message queue comprises at least one message body arranged according to a first-in first-out principle;
Receiving delivery messages sent by at least one writing process, and writing message bodies corresponding to the delivery messages of each writing process into the tail part of the queue of the non-lock message queue in sequence according to the sequence of the received delivery messages, wherein the method comprises the following steps: at least one writing process acquires queue writing authority and seizes message index writing authority in sequence, and sends delivery messages to the lock-free message queue; determining the sequence of the received delivery messages according to the preemption message index writing permission of each writing process; writing the message body corresponding to the delivery message of each write process into the message content memory block at the tail part of the queue of the lock-free message queue in sequence; updating the message index in the message index memory block, and releasing the writing permission of the preemptive message index; updating the head information in the memory block at the head of the queue, releasing the writing authority of the queue, and when the writing operation is finished, the writing process does not send the delivery message and the message body to the lock-free message queue any more;
receiving a read message sent by at least one read process, and sending message bodies to each read process one by one from the head of the queue of the lock-free message queue according to the sequence of the received read messages, including: at least one reading process acquires queue reading permission and sends a reading message to the lock-free message queue; traversing all message indexes in the message index memory block based on the queue reading permission to acquire unread message contents; each reading process acquires the preemptive message index reading authority and determines the sequence of the received read messages according to the preemptive message index reading authority; sending the unread message contents to a reading process corresponding to the preemptive message index reading right one by one from the head of the queue of the lock-free message queue; updating the message index in the message index memory block, and releasing the read permission of the preemptive message index; and updating the head information in the memory block at the head of the queue, and releasing the queue reading permission.
2. The lock-free message queue communication method according to claim 1, wherein the lock-free message queue comprises a queue head memory block for storing head information, a message index memory block for storing a message index, and a message content memory block for storing message content.
3. The lock-free message queue communication method of claim 2, wherein the write process employs an atomic function to obtain queue write permission; and the reading process acquires the queue reading permission by adopting an atomic function.
4. The method according to claim 1, wherein the writing the message body corresponding to the delivery message of each of the writing processes into the queue tail of the lock-free message queue in sequence further comprises: issuing a message queue-entering notification to all read processes;
Said sequentially sending message bodies from the head of said lock-free message queue to each of said read processes one by one, further comprising: a message dequeue notification is sent to all write processes.
5. A lock-free message queue communication system, comprising:
The device comprises a lock-free message queue creating module, a lock-free message queue creating module and a lock-free message queue processing module, wherein the lock-free message queue creating module is used for creating a lock-free message queue in a shared memory, and the lock-free message queue comprises at least one message body which is arranged according to a first-in first-out principle;
The write process communication module is used for receiving delivery messages sent by at least one write process and writing message bodies corresponding to the delivery messages of each write process into the tail part of the lock-free message queue in sequence according to the sequence of the received delivery messages; the write process communication module comprises: the write permission acquisition submodule is used for enabling at least one write process to successively acquire queue write permission and seize message index write permission and sending a delivery message to the lock-free message queue; the writing process sequence determining submodule is used for determining the sequence of the received delivery messages according to the preemption message index writing authority of each writing process; a message body writing sub-module, configured to sequentially write a message body corresponding to the delivery message of each write process into the message content memory block at the tail of the lock-free message queue; a message index write permission release module, configured to update the message index in the message index memory block, and release the write permission of the preemption message index; a queue write permission releasing module, configured to update header information in the memory block at the head of the queue, and release the queue write permission, where when writing operation is finished, the writing process does not send the delivery message and the message body to the lock-free message queue any more;
the read process communication module is used for receiving read messages sent by at least one read process and sending message bodies to each read process one by one from the head of the queue of the lock-free message queue according to the sequence of the received read messages; the process reading communication module comprises: a queue read permission obtaining submodule, configured to enable at least one read process to obtain a queue read permission, and send a read message to the lock-free message queue; a message content obtaining sub-module, configured to traverse all message indexes in the message index memory block based on the queue read permission to obtain unread message content; a reading process sequence determining submodule, configured to enable each reading process to acquire a preemption message index reading permission, and determine a sequence of received reading messages according to the preemption message index reading permission; the message body reading submodule is used for sequentially sending the unread message contents to a reading process corresponding to the message index reading permission from the head of the lock-free message queue one by one; a message index read permission release submodule, configured to update the message index in the message index memory block, and release the preemption message index read permission; and the queue reading permission releasing submodule is used for updating the head information in the memory block at the head of the queue and releasing the queue reading permission.
6. the lock-free message queue communication system of claim 5, wherein the lock-free message queue comprises a queue head memory block for storing head information, a message index memory block for storing a message index, and a message content memory block for storing message content.
7. The lock-free message queue communication system of claim 6, wherein the write process employs an atomic function to obtain queue write permission; and the reading process acquires the queue reading permission by adopting an atomic function.
8. The lock-free message queue communication system of claim 5, wherein the write process communication module further comprises a message in-queue notification issuing sub-module for issuing message in-queue notifications to all read processes;
the read process communication module also comprises a message dequeue notification issuing submodule used for sending message dequeue notification to all the write processes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610681120.1A CN106293973B (en) | 2016-08-17 | 2016-08-17 | lock-free message queue communication method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610681120.1A CN106293973B (en) | 2016-08-17 | 2016-08-17 | lock-free message queue communication method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106293973A CN106293973A (en) | 2017-01-04 |
| CN106293973B true CN106293973B (en) | 2019-12-13 |
Family
ID=57678405
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610681120.1A Active CN106293973B (en) | 2016-08-17 | 2016-08-17 | lock-free message queue communication method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106293973B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108153596A (en) * | 2017-12-15 | 2018-06-12 | 珠海金山网络游戏科技有限公司 | A kind of message treatment method and device based on lock-free queue |
| CN110362348A (en) * | 2018-04-09 | 2019-10-22 | 武汉斗鱼网络科技有限公司 | A kind of method, apparatus and electronic equipment of queue access data |
| CN108959032A (en) * | 2018-06-29 | 2018-12-07 | 北京金山安全软件有限公司 | Real-time monitoring method and device for server service index and electronic equipment |
| CN109376020B (en) * | 2018-09-18 | 2021-02-12 | 中国银行股份有限公司 | Data processing method, device and storage medium under multi-block chain interaction concurrence |
| CN110046047A (en) * | 2019-04-15 | 2019-07-23 | Oppo广东移动通信有限公司 | A kind of inter-process communication methods, device and computer readable storage medium |
| CN112559196B (en) * | 2019-09-10 | 2024-03-26 | 航天科工惯性技术有限公司 | Transmission method for sharing communication data between processes |
| CN111143079B (en) * | 2019-12-24 | 2024-04-16 | 浪潮软件股份有限公司 | Multi-read multi-write lock-free queue implementation method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100337205C (en) * | 2004-06-22 | 2007-09-12 | 国际商业机器公司 | Method and system for facilitating communication within shared memory environments using lock-free queues |
| US7533221B1 (en) * | 2004-12-30 | 2009-05-12 | Sun Microsystems, Inc. | Space-adaptive lock-free free-list using pointer-sized single-target synchronization |
| CN103164267A (en) * | 2013-03-29 | 2013-06-19 | 汉柏科技有限公司 | Lock-free message queue implementation method |
| CN103514053A (en) * | 2013-09-22 | 2014-01-15 | 中国科学院信息工程研究所 | Shared-memory-based method for conducting communication among multiple processes |
| CN105487933A (en) * | 2015-11-24 | 2016-04-13 | 无锡江南计算技术研究所 | Inter-kernel asynchronous message queue and shared memory based inter-domain communication method |
-
2016
- 2016-08-17 CN CN201610681120.1A patent/CN106293973B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100337205C (en) * | 2004-06-22 | 2007-09-12 | 国际商业机器公司 | Method and system for facilitating communication within shared memory environments using lock-free queues |
| US7533221B1 (en) * | 2004-12-30 | 2009-05-12 | Sun Microsystems, Inc. | Space-adaptive lock-free free-list using pointer-sized single-target synchronization |
| CN103164267A (en) * | 2013-03-29 | 2013-06-19 | 汉柏科技有限公司 | Lock-free message queue implementation method |
| CN103514053A (en) * | 2013-09-22 | 2014-01-15 | 中国科学院信息工程研究所 | Shared-memory-based method for conducting communication among multiple processes |
| CN105487933A (en) * | 2015-11-24 | 2016-04-13 | 无锡江南计算技术研究所 | Inter-kernel asynchronous message queue and shared memory based inter-domain communication method |
Non-Patent Citations (1)
| Title |
|---|
| Design and Evaluation of Nemesis, a Scalable, Low-Latency, Message-Passing Communication Subsystem;Buntinas D,et al.,;《IEEE International Symposium on Cluster Computing & the Grid》;20061231;1-9 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106293973A (en) | 2017-01-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106293973B (en) | lock-free message queue communication method and system | |
| CN110691062B (en) | Data writing method, device and equipment | |
| JP6238898B2 (en) | System and method for providing and managing message queues for multi-node applications in a middleware machine environment | |
| CN108647104B (en) | Request processing method, server and computer readable storage medium | |
| US8756613B2 (en) | Scalable, parallel processing of messages while enforcing custom sequencing criteria | |
| KR101372978B1 (en) | Managing message queues | |
| US20060182137A1 (en) | Fast and memory protected asynchronous message scheme in a multi-process and multi-thread environment | |
| US6862608B2 (en) | System and method for a distributed shared memory | |
| US20150288624A1 (en) | Low-latency processing in a network node | |
| JPH03126158A (en) | Method and apparatus for scheduling | |
| US11226852B2 (en) | System for inter-process communication | |
| US20030110322A1 (en) | Command ordering | |
| CN105426258B (en) | Mechanism and method for communication between a client and a server | |
| CN107783842B (en) | Distributed lock implementation method, device and storage device | |
| CN114928579B (en) | Data processing method, device, computer equipment and storage medium | |
| CN105068817B (en) | Write data into the method and storage equipment of storage equipment | |
| US10101963B2 (en) | Sending and receiving data between processing units | |
| CN111225063B (en) | Data exchange system and method for static distributed computing architecture | |
| US9990240B2 (en) | Event handling in a cloud data center | |
| WO2018094509A9 (en) | System for inter-process communication | |
| CN113826081A (en) | Method for transmitting message in computing system and computing system | |
| CN104769553A (en) | System and method for supporting work sharing muxing in a cluster | |
| WO2020220225A1 (en) | Resource acquisition method, related apparatus, and computer storage medium | |
| CN107203339B (en) | Data storage method and device | |
| US20100306483A1 (en) | Data Processor with Efficient Scalable Queuing and Method Therefor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |