DE112021005942T5 - Methods and systems for providing lock-free access to a shared memory area in a publish-and-subscribe system - Google Patents

Abstract

Systems and methods for providing lock-free access to a ring buffer are disclosed. The systems include a shared memory that includes the ring buffer. The ring buffer includes a plurality of sequentially arranged fixed-size buffers configured to store messages and a global header that includes a counter and a lock. The systems further include a publisher configured to write a plurality of messages to the plurality of fixed-size buffers in the ring buffer, and a subscriber configured to write one or more of the plurality of messages sent by the publisher to be written, to be read. The counter provides information related to a fixed-size buffer in the ring buffer to which the publisher will next write a message.

Description

CROSS REFERENCE AND PRIORITY CLAIM

This patent claims the benefit of U.S. Patent Application No. 17/097,590 , filed November 13, 2020, U.S. Patent Application No. 17/097,671 , filed November 13, 2020, and U.S. Patent Application No. 17/097,699 , filed November 13, 2020, all of which are incorporated herein by reference in their entirety.

BACKGROUND

Robots may require processing information from various sources to interact with environments. Certain robotic applications impose high bandwidth requirements as well as strict synchronization, latency and reliability requirements for communications. For example, robotic systems such as autonomous vehicles (AVs) require low latency to transmit control and feedback signals in real time, as ideally there should be as little delay as possible between, for example, receiving sensor inputs and corresponding vehicle operation. AVs are sophisticated systems that contain many different control systems or modules to control the various functions of the vehicle. For synchronized, low latency and reliable operation of an AV, information from various modules must be collected and analyzed in real time with very low latency to navigate the vehicle and/or identify and respond to faults. In particular, information must be exchanged between different modules efficiently and in a coordinated manner.

As the number of modules increases and tasks become more complicated, the amount of information exchanged between modules also increases. One way to manage the increased number of processing modules and the increased amount of information exchanged is a system based on a so-called publish/subscribe architecture. In a publish/subscribe system, information from a source is not sent directly to a receiving module. Instead, the information is stored by a source and made accessible (i.e. published or published) on a shared repository (central repository). The receiving module subscribes to the published information from the shared repository. The receiving module can also publish information to the shared repository, which other modules can subscribe to. Often a publisher is a producer (or writer) of data and a subscriber is a consumer (or reader) of that data and/or operating the data in these real-time systems. The subscriber does not necessarily need all the data generated - it only needs to receive the last or most current data. However, to meet performance goals, neither the publisher nor the subscriber should have to wait or be blocked for the other (or any other) task before it can gain access to data.

Access to data written in such publish/subscribe systems may include access to shared storage that stores the shared repository in a computing environment. Known solutions include using a shared ring buffer type repository. A ring buffer is a contiguous array of data cells that can contain any data. A ring buffer is limited in size to a fixed number of data cells. The disadvantage of this approach is that it requires the user to read and, if necessary, discard several entries in order to obtain the latest, i.e. the most relevant, data. This approach therefore causes a non-trivial performance hit. Further, the buffer must be sufficiently deep to allow the writer to always have sufficient space to write, and this assumes that the reader and writer are running at the same fixed rate, and not one significantly faster (or slower) than that others is. Constructing scalable applications that can do this is increasingly challenging as the number of nodes grows, and this is compounded by other issues such as the increasing latency gap between the local resources and the remote resources in a distributed system. Transferring data between network nodes can add latency, network congestion and other problems.

This document describes methods and systems aimed at addressing the problems described above and/or other problems.

SUMMARY OF THE INVENTION

In various scenarios, systems and methods for providing lock-free or lockerless access to a ring buffer are proposed. The system may execute the methods and may include a shared memory containing the ring buffer. The ring buffer can do a variety of sequential fixed-size arrayed buffers configured to store messages, and a global header that includes a counter and a lock. The system may also include a publisher configured to write a plurality of messages to the plurality of fixed-size buffers in the ring buffer, and a subscriber configured to write one or more of the plurality of messages through the Publishers are written to be read. The counter can provide information regarding a fixed size buffer in the ring buffer to which the publisher will next write a message. Optionally, each of the plurality of messages may have a finite size.

In various implementations, a negative value of the counter can indicate a state of the system in which the ring buffer contains no messages. Optionally, the publisher can assign a value of 0 to the counter after completing an initial write to the ring buffer. The publisher can then increment the value of the counter after each subsequent operation, so that an index of the fixed-size buffer in the ring buffer accessed by the publisher that last has a message is determined by dividing a value of the counter by 2

In certain implementations, the lock may be acquired by the publisher before a write is performed to one of multiple fixed-size buffers and released after the write is executed. Optionally, the system may include a plurality of publishers, each of which may be configured to write one or more messages to the plurality of fixed-size buffers in the ring buffer, such that the locking is held by a single one of the plurality of publishers at any given time can be captured.

In certain implementations, the multiple fixed-size buffers may not share common cache lines in shared memory.

In various implementations, each buffer may contain a buffer header configured to store at least one of the following: a buffer index number, synchronization information, or buffer state information. Optionally, the buffer header may occupy a unique cache line in shared memory. Additionally and/or alternatively, each buffer may also contain a data area beginning in a cache line of shared memory immediately after the unique cache line of a corresponding buffer header.

Optionally, a size of the plurality of fixed-size zones may be determined based on, for example, maximum message size, maximum number of fixed-size zones, or orientation.

In one or more implementations, the shared memory may also include a synchronization data storage area corresponding to each of the plurality of fixed-size buffers that include a buffer counter that indicates an accuracy or trustworthiness of a message transmitted from the fixed-size buffer through the Subscriber is read. A memory address of the synchronization data storage area may be contained in a buffer header of this fixed-size buffer. An initial value of the buffer counter can be set to 0, and the publisher can increment a value of the buffer counter after executing a write operation to the corresponding buffer. This allows the subscriber to compare a first value of the buffer counter before executing a read from the fixed-size buffer with a second value of the buffer counter after executing the read from that fixed-size buffer and to discard a message that has been read from the buffer is read with a fixed size if the first value does not match the second value. In such cases, a value of the buffer counter is odd during the publisher's execution of a write to the fixed-size buffer. The subscriber may be configured to wait for a write completion notification if the buffer counter value is odd before performing a read on the fixed-size buffer. Additionally and/or alternatively, the data synchronization area may also contain a futex system call (“Fast Userspace Mutual Exclusion”) configured to send a new message notification to the Subscriber when a value of the buffer counter incremented by 2 and is even. Optionally, the data synchronization area may contain a futex system call configured to send a wait notification to the subscriber when a value of the buffer counter is either odd or not incremented. The Futex system call can also send a wait notification to a variety of publishers in the system when a value of the buffer counter is odd.

Some other scenarios include systems (including processors) and methods for enabling message transfer between a publisher and a subscriber. The methods may include creating a data structure that includes a ring buffer in shared memory. The ring buffer can include a plurality of sequentially arranged, fixed-size buffers configured to store messages, and a global header, which may include a seqlock (“sequence lock”). The seqlock may contain a seqlock counter that provides information regarding a fixed size buffer in the ring buffer to which the publisher will next write a message. The methods may also include creating a synchronization data storage area corresponding to each of the plurality of fixed-size buffers, which may include a buffer counter indicating the accuracy of a message read from the fixed-size buffer by the subscriber.

In various implementations, a negative value of the counter can indicate a state of the system in which the ring buffer contains no messages. Optionally, the publisher can assign a value of 0 to the counter after completing an initial write to the ring buffer. The publisher can then increment the value of the counter after each subsequent operation, so that an index of the fixed-size buffer in the ring buffer to which the publisher last has a message is determined by dividing a value of the counter by 2 becomes

In certain implementations, the lock may be acquired by the publisher before a write is performed to one of multiple fixed-size buffers and released after the write is executed.

Optionally, the multiple fixed size buffers are created such that they do not share cache lines with each other in shared memory.

An initial value of the buffer counter can be set to 0, and the publisher can increment a value of the buffer counter after executing a write operation to the corresponding buffer. This allows the subscriber to compare a first value of the buffer counter before executing a read from the fixed-size buffer with a second value of the buffer counter after executing the read from that fixed-size buffer and to discard a message that has been read from the buffer is read with a fixed size if the first value does not match the second value. In such cases, a value of the buffer counter is odd during the publisher's execution of a write to the fixed-size buffer.

In various other scenarios, systems and methods for sharing information between a publisher and a subscriber are disclosed. The system includes shared memory and a memory broker configured to carry out the methods of this disclosure. The memory broker may include a processor and a non-transitory computer-readable medium that includes one or more programming instructions executed by the processor. The methods include, by the storage broker: receiving a request to write a message related to a topic from a publisher, determining whether a communication channel corresponding to the topic exists in shared storage. If the communication channel corresponding to the topic exists, the broker may assign a ring buffer on the communication channel to the publisher, transmitting information relating to the ring buffer to the publisher, and transmitting information relating to the ring buffer to one or multiple subscribers of the communication channel. Optionally, the storage broker may update a record associated with the allocated ring buffer in a data store.

Additionally and/or alternatively, in response to determining that the communication channel corresponding to the topic does not exist, the storage broker may create a new communication channel corresponding to the topic, the new communication channel comprising at least one ring buffer and information regarding the new one Communication channel and the at least one ring buffer to be transmitted to the publisher. The storage broker can then create a record of the new communication channel in a data store. Optionally, the storage broker can also receive a message from a subscriber to subscribe to the topic. The new communication channel may be based on, for example, a maximum size of each message on the new communication channel, a maximum number of messages to a queue, a number of publishers that are allowed to publish on a ring buffer of the new communication channel, a maximum number of ring buffers on the channel , whether ring buffer sharing is allowed, the allowed latency in the new communication channel, the types of messages, a channel naming protocol or an allowed list of publishers that are allowed to publish to the new communication channel.

In various implementations, allocating the ring buffer on the communications channel to the publisher may involve creating a new ring buffer on the communications channel to allocate to the publisher if, for example, the storage broker does not allow multiple publishers hern allows messages to be published to a single ring buffer, and all suitably configured ring buffers on the communication channel are already assigned to a live publisher. The storage broker allows multiple publishers to publish messages to a single ring buffer, but the ring buffers on the channel are not configured appropriately for the publisher's messages, existing communication channel ring buffers are not configured appropriately for the publisher's messages, channel ring buffers are located in an error state; and/or the publisher may not publish on existing ring buffers of the communication channel.

In certain implementations, allocating the ring buffer on the communication channel to the publisher may include allocating an already existing ring buffer on the communication channel to the publisher, for example, if the storage broker allows multiple publishers to publish messages to a single ring buffer and sets up a ring buffer the channel is appropriately configured for the publisher's messages, and/or the storage broker allows multiple publishers to publish messages to a single ring buffer, and there is at least one appropriately configured ring buffer on the channel that is not already assigned to a live publisher assigned.

In one or more implementations, the memory broker may also analyze a state of a plurality of ring buffers in the shared memory and determine an action with respect to one or more of the plurality of ring buffers. The action may include temporary removal from use by setting this ring buffer to “probation,” revising probation, and/or permanent removal from use. The state of the multiple ring buffers can be analyzed periodically and/or when a trigger event occurs. Examples of the triggering event may be an unexpected loss of connection between the storage broker and the publisher, the receipt of a message from a peer of the ring buffer indicating a problem with the ring buffer, the occurrence of a bad read operation on a ring buffer, the occurrence of a bad write operation on a ring buffer and/or a memory broker state reconstruction. In some examples, a ring buffer may be permanently removed from use if the ring buffer is determined to be corrupted. Additionally and/or alternatively, a ring buffer may be permanently removed from use if at least one of the following is determined: no live and active publisher associated with the ring buffer and/or during broker state reconstruction. In other examples, a ring buffer may be temporarily removed from use and the broker may place the ring buffer on a probation list. The record may include, for example, the ring buffer ID, the current write position in the ring buffer, whether or not there is a write lock on the ring buffer, a timestamp of the last write operation, a timestamp of when the ring is temporarily removed from use, and/or a Includes timestamp of the most recent record update. The record can then be used to determine whether the ring buffer should be permanently removed from use or revised from probation.

In certain implementations, the memory broker may also perform a reconstruction of the shared memory by the memory broker after a failure by temporarily removing a plurality of ring buffers in the shared memory, establishing a connection from a peer, receiving a request, a read -/write operation in the shared memory from the peer, determining whether the peer is performing the read/write operation to a ring buffer of the plurality of memory buffers before the failure, and revising the ring buffer for use by the peer if the peer has the read/write operation. /write operation to the ring buffer before the failure.

Optionally, the broker can determine that the peer is a publisher and create a new ring buffer in the shared memory for the peer if the peer does not perform the read/write operation to the ring buffer before the failure.

In various other scenarios, systems and methods for routing messages between network nodes of a distributed computing system are disclosed. The systems can contain a variety of network nodes. Each network node may include shared storage configured to store messages, a publisher, and a first bridge module. The first bridge module can determine for a shared memory area of this network node whether a subscriber exists on a remote network node. If such a subscriber exists, the first bridge module can read a plurality of messages from the shared memory area and transmit the plurality of messages to a second bridge module of the remote network node. The remote network node can be one of a variety of networks node that does not contain the publisher. The bridge module may write the plurality of messages to a remote storage area on the remote network node. Optionally, the remote memory area may be identical to the shared memory area. The subscriber can then read the large number of messages from the remote storage area.

In various implementations, the first bridge module may determine whether the subscriber exists on the remote network node by receiving information via the distributed computing system and use the received information to make the determination. The information may include, for example, one or more communication channels, one or more ring buffers, network locations of one or more publishers or a network location of one or more subscribers. The first bridge module may receive the information from the second bridge module on the remote network node and/or a data store on the publisher's network node.

In some implementations, the first bridge module may transmit the plurality of messages to the remote network node using a zero-copy network protocol and/or a stream control transfer protocol (SCTP).

In certain scenarios, the shared memory area may include a ring buffer, the ring buffer. The ring buffer may include a plurality of sequentially arranged fixed size buffers configured to store messages and a global header containing a seqlock counter indicating a precision of the plurality of messages written to the remote storage area . In such implementations, the second bridge module may replicate the ring buffer in the remote memory area.

Optionally, a plurality of subscribers on the remote network node can read the plurality of messages from the remote storage area.

BRIEF DESCRIPTION OF THE DRAWINGS

• 1 illustrates a schematic representation of a publish/subscribe data communications system.
• 2 is a schematic representation of an exemplary ring buffer.
• 3 is a schematic representation of an exemplary communication channel.
• 4 is a flowchart illustrating an example method for allocating a ring buffer by a storage broker.
• 5 is a flowchart illustrating an example procedure for state reconstruction of a storage broker upon reboot.
• 6 illustrates a schematic representation of a publish/subscribe data communications system in a distributed computing system.
• 7 is a flowchart illustrating an example method for transmitting messages between network nodes of a distributed computer system.
• 8th is a schematic representation of an example communication channel having remote subscribers.
• 9 is a block diagram of elements of a computing device on which the various systems and methods in this document could be implemented.

DETAILED DESCRIPTION

As used in this document, the singular forms “a” and “the” are intended to include the plural form, unless the context clearly dictates otherwise. Unless otherwise defined, all technical and scientific terms used herein have the same meanings as generally understood by one of ordinary skill in the art. As used herein, the term “comprehensive” shall be construed to mean, but not limited to, “including.” Additional terms relevant to this document are included at the end of this detailed description.

Together, the shared memory and communications network embodiments of this disclosure can be used to implement a multi-processing and distributed computing environment in a robotic system, such as an autonomous vehicle. Such a robotic system may support a portable operating system interface (POSIX) thread standard or other shared memory multi-threading programming models and may be implemented on an operating system (“OS”) (e.g., Linux, Unix, etc.). An embodiment of the present disclosure may enable a storage broker to boot upon an unexpected crash or abnormal termination.

1 shows a schematic representation of a publish/subscribe data communication system 150 for use in a robotic system according to an embodiment of the present disclosure. In certain embodiments, the robotic system may be an autonomous vehicle.

As in 1 As shown, one or more threads or computer processes 102a-n, each including one or more publishers 104a-n, may have communicative access to a shared memory 110 contained in a non-transitory computer-readable storage medium and/or as a virtual memory instance . Additionally, one or more threads or computer processes 106a-n, each including one or more subscribers 108a-n, may also have access to shared memory 110. In this sense, each process 102, 106, the publisher 104 and the subscriber 108 can have access to the shared memory 110.

Although some processes are illustrated that only show respective publishers or subscribers, embodiments of the disclosure may include processes that include a combination of publishers and/or subscribers within a single process. As used herein, each publisher 104a-n and/or each publisher process 102a-n may be considered a “writer” for accessing data in the shared memory 110, and each subscriber 108a-n and/or each subscriber process 106a-n may be “readers” for accessing data in shared memory 110. Publishers and Subscribers are collectively referred to as “Peers” in this disclosure.

As used herein, "programs" and/or "processes" may include all or part of a computer program that includes an executable instruction set for controlling the management and/or operation of at least one of the respective Publisher 104a-n, Subscriber 108a- n or functions of the robot system using the data communication system 150. The program and/or processes may include a computer program product, which may include machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Such machine-readable media may be any available media that can be accessed by a general-purpose or special-purpose computer or other machine with a processor. In general, such a computer program may include routines, programs, objects, components, data structures, algorithms, etc. that have the technical effect of performing certain tasks or implementing certain abstract data types. Machine-executable instructions, associated data structures, and programs represent examples of program code for carrying out the information exchange as disclosed herein. Machine-executable instructions may include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, a controller, or a special-purpose processing machine to perform a particular function or group of functions.

The data communications system 150 may include a system bus or other computer system communications components (not shown here) to facilitate communications between the publishers 104a-n, shared storage, subscribers 108a-n, or other components of the system. For example, publishers 104a-n and subscribers 108a-n may communicate with each other by passing data over a channel through shared memory 110. A channel can e.g. B. include a related set of logical multicast connections implemented in a distributed manner, and this may include a collection of network resources used to serve a community of publishers and subscribers exchanging messages. The messages can be classified according to the channel subject namespace. Various channel resources can be managed, controlled and provisioned using known methods, such as through channel services provided by channel managers. Multiple channels can share the same resources. Channels can provide a highly scalable directory service, such as, but not limited to, the following examples: publisher and subscriber information, authentication and authorization information, message types, etc. The channels facilitate content-based routing by providing logical configurations for specific content types and thus a context is provided for attributes that are transmitted over the channels.

The publishers 104a-n and/or the subscribers 108a-n can be operatively coupled to the memory 110 so that the publishers 104a-n and/or the subscribers 108a-n or any computer programs or processes can access at least part of the memory 110 .

Each process 102a-n, 106a-n that accesses the shared memory 110 may be treated equally, with each process having equal rights to read, write, expand and shrink the shared memory 110 has. Alternatively, the processes 102a-n and 106a-n that access the shared memory may also operate in a client-server/mode, where one process (server) typically has more privileges than the client. Typically, in this mode, the server process has rights to read, write, expand, and shrink memory 110, while client processes can only read and write memory 110. For example, in a POSIX environment where processes use system calls to request services from the POSIX operating system (OS), the process that requests the OS to designate an area of memory as shared memory may be called a server. All other processes that use the allocated shared memory are clients. Thus, the designation of a process as a client or server depends on the process's relationship to the allocation and use of shared memory.

The data communication network 150, which is in 1 shown is merely a schematic representation of an embodiment in which a plurality of publishers 104a-n and subscribers 108a-n may or may not be located on the same computer system of the robot system. The exact location of publishers 104a-n and subscribers 108a-n are not subject to the implementations of this disclosure. Additionally, more or fewer publishers 104a-n and/or subscribers 108a-n may be included in various implementations of this disclosure.

As one of ordinary skill in the art will understand, the storage medium may be a non-transitory computer-readable medium, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), or any other suitable medium included. Shared memory 110 may include a number of addressable memory locations for storing data. Data may be written to shared memory 110 by publishers 104a-n and data may be read from shared memory 110 by subscribers 108a-n

For a process to access shared memory, the shared memory must be mapped into the process address space. In certain embodiments, shared memory 110 may be mapped through a memory management call (“mmap”) provided by a kernel. A “mmap” system call maps the specified memory into the address space of the calling process. In a POSIX environment, the server process uses the mmap() system call to create a mapping between a range of addresses in the process's address space and a portion of a specific memory object. Before mapping a data structure into memory, the process first uses an open() system call to obtain a file descriptor for the data structure. The mmap() system call requires several parameters, including the address in the address space to which the file is to be mapped, a guard argument to specify whether the data structure is shared or private, and the file descriptor obtained with the open() system call becomes. The mmap system call returns a pointer to the beginning of the mapped data.

The memory 110 is further divided into memory areas 112a-n, each memory area comprising an association of data into at least one grouping or topic positioned at a predetermined addressable memory location. As used herein, a memory area 112a-n may include a subset of memory 110 associated with a particular usage of data storage for the robotic system that relates to a topic. Message topics typically provide the key to delivering messages between publishers and subscribers. For example, a topic may include a single allocation of data, such as speed data, pose data, sensor data (e.g., LiDAR sensor data, camera sensor data, etc.), location data, or it may include a variety of related or unrelated data items, such as waypoints or the current Trajectory of the robot system. As shown, memory areas 112a-n may be arranged sequentially; however, additional organizational structures of memory areas 112a-n may be configured to include matrices, variable allocations for each memory area 112a-n, etc. in certain embodiments. Further, each storage area 112a-n may be associated with a name, and such name may be stored in at least one temporary data store 172 that may be preserved through reboots, crashes, etc. (e.g., a /dev/shm virtual file storage in a Linux -Operating system). For example, a storage broker 170 (discussed in detail below) may create the storage regions 112a-n and associate each region with a name. A process and/or a user can access a memory area using the assigned names.

Each of the processes 102a-n, 106a-n or the publishers 104a-n and the subscribers 108a-n can get address(es) of the memory areas 112a-n from one Storage broker 170 receive (as discussed below). In this sense, each of the processes 102a-n, 106a-n, the publisher 104a-n, and/or the subscriber 108a-n may identify one or more storage areas 112a-n that store data related to the corresponding topics.

In one embodiment, the number of storage areas 112a-n in the shared memory 110 may be increased during initialization of the storage 110 based on a known number of topics accessible to the publishers and/or subscribers, with at least one storage area for each topic be specified. In another embodiment, the number of storage areas 112a-n may be defined at or during runtime (e.g., by storage broker 170) by the number of topics accessible by the publishers and/or subscribers with at least one storage area for each topic. In this sense, the number of topics can be dynamic, increasing and decreasing as needed, or only behaving additively when additional topics need to be accessed.

In certain embodiments, a memory area 112a-n may include a ring buffer 120a-n that contains a plurality of fixed-size discrete buffers. Each buffer can be configured to store a predetermined amount of data. As used herein, the term "message" refers to a data item that is written into a buffer of a memory area and that has a size that is less than or equal to the fixed size of the buffer. Optionally, the size of a buffer can be determined to be equal to the maximum allowed size of a message, and only one message can be stored in a buffer at a given time. In various implementations of this disclosure, publishers write write messages to a ring buffer by sequentially inserting data into successive buffers of the ring buffer. The messages written in the buffers of a ring buffer are read by subscribers, who can read the buffers in or out of sequence, i.e. the subscribers can skip buffers and read them out of sequence while a read operation is in progress. The key defining property that makes a collection of buffers a ring buffer is the fact that when the publisher reaches the last element of the array, it independently loops back to the beginning of the array to overwrite the oldest buffer. Thus, a ring buffer can be viewed as an infinite loop with subscriber tracking behind the publisher. This arrangement creates the effect that the publisher is always looped around the buffer, although in terms of memory address locations the buffers are actually discontinuous between the last and first buffers are. This fact is largely hidden from users of the ring buffer in a memory area. The ring buffer of the present disclosure is "lockerless" or "lock-light" in the sense that a publisher is never blocked by a subscriber from writing to a buffer of the ring buffer and the subscribers detect corrupted reads, as discussed below . Corrupted reads can be detected but not prevented because low latency is favored over reliability.

Each memory area can only contain one ring buffer. It should be noted that there may be a unique name or ID associated with each ring buffer, and there is a defined relationship between the memory area name and the corresponding name/ID of the ring buffer contained in that memory area. Since each memory area can only contain one ring buffer, the ring buffer ID can be derived from the corresponding memory area name (or vice versa).

Other memory configurations included in memory areas 112a-n are within the scope of this disclosure, such as, without limitation, a memory area that includes a plurality of buffers arranged contiguously and sequentially in memory.

In 2 1, a ring buffer 120 is shown included in a storage area 112 that includes a plurality of buffers 121a-n, each of the buffers 121a-n being configured to store a predetermined amount of data (ie, a message). In such a ring buffer 120, the number of buffers and the size of each buffer can be determined at initialization; these cannot be changed subsequently (discussed below). For example, a ring buffer 120 for storing data related to the pose of the robot system may include a first predetermined number of buffers, each configured to store a message containing a first number of bytes. In another example, a ring buffer 120 for storing data related to trajectory data may include a second predetermined number of buffers, each configured to store a message containing a second number of bytes. The fixed size and number of allocation of buffers in a memory area allows for predictable and stable memory usage, which is important in robotic systems because they contain finite messages (data stored in each buffer) required for memory read/write operation. In particular, the allocation and number of allocation of fixed sizes of buffers in a memory area creates a finite queue of messages available to subscribers to read at any given time, stored in successive buffers of a ring buffer. The subscriber can run sequentially to read the message or can randomly select a message in the queue without the need for pointers (as discussed below). It should be noted that, as in 2 shown, the ring buffer does not necessarily contain a ring configuration of fixed size buffers.

Because a ring buffer is divided into fixed-size buffers, each buffer that holds a message that allows fast random access to the ring can also efficiently examine each message in the ring buffer and decide which message is relevant. Existing storage configurations store messages as variable-sized units within a continuous sequence of bytes with message frames to distinguish messages, which in turn contain messages, which on the other hand requires parsing at least the message frame to skip messages and generally does not support any kind of random access .

It should be noted that each memory area 112a-n can only contain one ring buffer 120. Multiple publishers can write messages to a ring buffer, and multiple subscribers can read data from a ring buffer. Optionally, a single publisher can write messages to a ring buffer, and one or more subscribers can read data from a ring buffer. Channels discussed above have a one-to-one relationship with ring buffers. In particular, each ring buffer belongs to one channel, but each channel for a topic can contain many ring buffers. For example, if multiple publishers publish data related to a topic (e.g. topic = shape data, publisher = camera, LiDAR or other sensors in a vehicle), the channel can include multiple ring buffers, each corresponding to a publisher.

In another example, as in 3 shown, a channel 301 can be created in an autonomous vehicle corresponding to the topic of "pose estimation", where the pose of a vehicle is the position of the vehicle in either two or three dimensions along with its orientation. The channel 301 may include ring buffers 302 (a) and 302 (b) that store raw data collected from a LiDAR sensor, a ring buffer 303 that stores data related to processed LiDAR data, a ring buffer 304 that stores data relating to observed poses, a ring buffer 305 storing data relating to filtered poses, and a ring buffer 306 for storing debugging logs. LiDAR sensors (not shown here) are publishers to buffer the rings 302 (a) and 302 (b), and a LiDAR data processing system 311 (e.g., a sweep memory system) can subscribe to data written into the ring buffers 302 (a) and 302 (b). The LiDAR data processing system 311 processes data read from the ring buffers 302 (a) and 302 (b) and publishes them to the ring buffer 303.

LiDAR data subscribers, such as a scan matcher 312, may correspond to the data in the ring buffer 303 to extract observed pose data from the LiDAR data. The observed pose data may be published from the scan matcher 312 to the ring buffer 304 for storage. A pose filter 313 may subscribe to the ring buffer 304 and publish filtered pose data to the ring buffer 305. Multiple subscribers (such as a tracker 314, a scan matcher 312 and/or other systems) can subscribe to the filtered pose. Even though ring buffers 302 (a), 302 (b), 303, 304, and 305 are associated with a single publisher, some ring buffers 306 may receive data from multiple publishers. For example, a ring buffer for storing debugging logs can receive data from different systems.

Referring again to 2 Each buffer 121a-n of a ring buffer 120 can include a header 122 and a data section 123. The header 122 may include a sequence or index numbers up to KN, synchronization information, and/or status information (ie, empty, filled, containing the most recent data, or the like) for the corresponding buffer. Synchronization information may store information identifying a memory location (e.g., a memory address, a memory pointer, a memory area name, the location of the buffer, etc.) in the shared memory 110 that stores synchronization data related to one or more read and write operations for that buffer (discussed below). The data section 123 may be configured to store a message. It is worth noting that the data storage capacity of a buffer should be larger than the size of a message due to the size of the header.

In certain embodiments, the header occupies its own central processing unit (CPU) cache line in memory area 112a-n. This eliminates undue cache sharing between different buffers, improving performance and concurrency on multi-core CPUs and multi-CPU systems with a minimum of memory bandwidth usage and cache coherence overhead. The data portion 123 of a buffer may begin in a cache line immediately following the cache line for the header 122 (or within a predefined number of cache lines). This allows the CPU to efficiently retrieve main memory into cache memory due to the sequential memory access pattern of accessing the header 122 of the buffer first followed by the data portion 123. Since the number and size of buffers in a memory area 112a-n is fixed, such alignment of buffers in memory area 112a-n (with buffers that do not share cache lines) results in alignment of messages to CPU cache lines for efficient memory access.

Optionally, since predetermining the number and size of buffers in a ring buffer 120 allows each message to be aligned in memory as desired, this can also be used to align message data as required to perform CPU vector operations (e.g., SIMD, Single Instruction Multiple dispatch, etc.). Such instructions enable highly efficient parallel computations but place precise memory address alignment constraints on the data to be imposed in robotic system operations such as autonomous vehicles.

In certain embodiments, a ring buffer 120 may also include a global header 124. When embodied in systems with appropriate support (such as LINUX), the global header 124 may contain a sequential lock (“seqlock”) associated with memory to store a counter in addition to the seqlock. Other types of locking mechanisms are within the scope of this disclosure. The seqlock supports synchronization between two publishers (atomic operation), and the counter is used to display the current write position of a publisher in the ring buffer 120 (also referred to as the “HEAD” of the ring buffer). The seqlock is acquired by a publisher before writing data to the ring buffer 120 and released when the write operation is complete. The counter is stored as a signed integer and is initialized to have a value of -2 when the ring buffer 120 is empty. When a publisher acquires the seqlock for a first write to the ring buffer 120 (i.e. the value of the counter is -2), it increments the value to -1. After completing this first write operation, the publisher sets the value of the counter to 0 before releasing the seqlock. A publisher during each subsequent write operation increments the counter by one when the seqlock is acquired. When the seqlock is released, the publisher increments the counter by the value of the counter (before acquiring the lock) by 2 modulo the number of buffers in the ring buffer. For example, if a ring buffer contains 10 buffers, the value of the counter will be 0, 2, 4, 6,8,10, 12,14,16, 18 and back to 0 and so on. This increment-and-modulo is performed atomically to enable mutual exclusion with respect to access to the counter for the duration of an atomic operation. Such a configuration of the global header 124 allows the counter of the seqlock to be used to determine the current write position of a publisher in the ring buffer 120. Specifically, any time a value of the counter in the seqlock is negative (-2 or -1), the ring buffer 120 is empty, with no previous writes (-2) or a first write in progress (-1). However, if a value of the counter is zero or positive, the value of the counter divided by two is equal to the index of the buffer last written to and hence the current write position in the ring buffer 120. Such computation is efficient in terms of both CPU time and memory usage.

Typically, a single publisher can write data to the ring buffer 120 and can sequentially write data to the buffers of the ring buffer 120. However, the seqlock and associated counter of the global header 124 can be used so that multiple publishers can write to the ring buffer 120. Each publisher in a multi-publisher scenario can use the counter to determine the index of the buffer that it should write without overwriting a message written by a previous publisher and that has not been read by one or more subscribers.

Subscribers may use the current write position in the ring buffer 120, determined using the global header counter in combination with their own local read position (which each subscriber maintains), to determine when they have read all available messages and respond to notification should wait for a new message. For example, if the buffer count (described below) of a subscriber's current read position is the same as the current write position in the ring buffer 120, the subscriber may determine that it has read all of the messages in the ring buffer.

Referring again to 1 Shared memory 110 also includes a synchronization data area 130a-n corresponding to each buffer in a memory area present at a memory address determinable based on synchronization information contained in buffer header 122. Each synchronization data area 130 is configured to store a buffer counter that is accessible using appropriate atomic instructions by processes (publishers and/or subscriptions) that perform a read or write operation to the corresponding buffer. The use of atomic instructions allows multiple threads or processes to access the buffer without requiring additional locking of the buffer or synchronization of processes. The buffer counter for each buffer is initialized with a “0” value.

When a publisher performs a write to a buffer, it increments the value of the buffer counter by 1 at the beginning of the write operation using an atomic compare and swap instruction on the synchronization data area 130. After the write is completed, the value of the buffer counter is incremented by 1 again. The value of the buffer counter therefore increases by two for each write and is odd during a write operation and is even when no write operation is being performed (or, if the buffer counter were initialized to 1, the value of the buffer counter would be during a write operation to the buffer even and this would be odd if no writing is in progress).

During a read operation into the buffer, a subscriber begins a read by reading (and storing a local copy) of the value of the buffer counter at the corresponding data synchronization area. If the value of the buffer counter is odd, the subscriber determines that a write operation is in progress and waits for notification that the write operation is complete (notification mechanism described below) before reading data from the buffer. If the value of the buffer counter is even, the subscriber continues reading the data stored in the buffer. After the read operation is completed, the subscriber reads the value of the buffer counter at the corresponding data synchronization area a second time and compares the value with the locally stored value of the buffer counter obtained at the beginning of the read operation. If the buffer counter value is unchanged, the subscriber determines that the data in the buffer is not corrupted (i.e., no concurrent write operation occurred while the subscriber is performing the read operation). However, if the buffer counter value changes between the beginning and end of the read operation, the subscriber determines that there was a concurrent write operation, and the read operation may have inconsistent data or the read may be corrupted. If the read is determined to be corrupted, the subscriber may do at least one of the following: discard the read operation and any data derived therefrom, log an error, skip to a new buffer location, or a combination thereof. The new buffer position may be a buffer that is a predefined number of buffers ahead of the current subscriber position in the ring buffer, regardless of the publisher's current write position. Alternatively and/or additionally, the subscriber may determine the index of the buffer that will next be written to the ring buffer using the seqlock contained in the global header of the buffer and skip to that buffer and/or a buffer that this buffer (e.g. buffer immediately before the current write position, a buffer two positions behind the current write position or the like). A subscriber can also check how far behind a publisher it is and preemptively skip ahead to a new buffer location closer to the publisher to reduce the risk of corrupted reads

A subscriber can also set its delay behind a publisher in a ring buffer based on the two-increment-per-write property of the buffer counter in combination with the write pattern of advancing through the ring buffer sequentially depending on a change in the value of the buffer counter between a previous read operation and one determine the current read operation carried out by the subscriber. In some embodiments, a change in the value of the buffer counter that is greater than a threshold may indicate that the subscriber may be lapped by the publisher (the term "rounded" as used herein refers to a publisher that is lapping data in a buffer of a ring buffer before a subscriber reads the overwritten data (since the subscriber is behind the publisher) and must take an action to catch up. For example, the subscriber can skip reading data from a predefined number of buffers in the ring buffer regardless of the publisher's current write position. Alternatively and/or additionally, the subscriber may determine the index of the buffer that will next be written to the ring buffer using the counters/seqlock contained in the buffer's global header and skip to that buffer and/or a buffer , which follows this buffer (e.g., buffer immediately before the current write position, a buffer two positions behind the current writing position or the like).

As discussed above, the fixed number of buffers in the ring buffer, each of a fixed size, means that each subscriber maintains its own current read position in the ring buffer, independent of other subscribers and independent of the publisher. In other words, using different fixed-size buffers allows each buffer to be synchronized independently, with synchronization data for each buffer at predictable, easily computable memory addresses. This allows the publisher to write to one buffer while subscribers write to other buffers without interference to be read. In addition, a Subscriber that has fallen behind a Publisher and/or other Subscribers and needs the most recent message instead of historical messages can access the buffer containing the most recent message without all intermediate data (between the last read message and the most recent message). Such a configuration favors low latency over reliability, as is often required in robotic systems such as autonomous vehicles.

As discussed above, multiple publishers can write to a ring buffer. The atomic compare-and-exchange used to increment the buffer counter ensures that only a single publisher can successfully transition the counter from even to odd and continue writing to the buffer. A concurrent publisher will fail this operation and will be blocked from writing to the same buffer and will have to wait for a notification that the write is complete before proceeding with a write itself.

The memory configuration of the present disclosure also eliminates back pressure from subscribers to the publisher of a ring buffer. The publisher simply continues sequentially and sequentially in the ring buffer to perform write operations regardless of the number of subscribers and/or read positions of subscribers of the ring buffer, since each subscriber is responsible for maintaining its own read position in the ring buffer and for independently detecting when he fell behind.

Notifications

In the publish/subscribe data communication system described above, a subscriber of interest can register or subscribe to events related to a given topic through a storage broker described below; thereafter, the subscriber receives notifications of such events generated by publishers.

In such a publish/subscribe system, publishers of data do not communicate directly with subscribers; instead, publishers and subscribers interact through an intermediate entity, referred to herein as storage broker 170, but also known in the art by various other names, such as notification server or event service. Publishers can publish notifications about events occurring on a channel by first tagging these notifications with a specific topic. Subscribers receive notifications in the form of messages from the storage broker 170 for events corresponding to the topic(s) in which they are of interest. Since message producers (publishers) and subscribers (subscribers) communicate indirectly with one another via a storage broker 170, message transmission is decoupled from message reception. Consequently, neither producers nor consumers have to maintain the state with each other, and dependencies between the interacting subscribers are reduced or eliminated. The publish/subscribe scheme is therefore more flexible than other communication paradigms, such as point-to-point messaging, because publishers and subscribers can be started and stopped asynchronously. Furthermore, the event notifications are sent to the subscribers or publishers instead of polling them for updates. This leads to a reduction in network traffic.

Such a system is well suited for robotic systems (e.g. autonomous vehicles) where many sensors monitor the dynamically changing state of some underlying systems and forward measurement data to a central server location. For example, vehicle sensors in an autonomous vehicle can generate notifications of changes in the state of the underlying system and can be modeled as publishers of certain types of events (change of state in the vehicle system), or they can be subject and vehicle systems, such as perception system, prediction system, route planning system, etc. who monitor the events are modeled as subscribers.

In embodiments of the present disclosure, the counter, including the global header 124 of a ring buffer and/or a buffer counter stored in the corresponding synchronization data area 130, may be in novel notification mechanisms are used, as described below.

In certain embodiments, a notification mechanism may be based on futex (fast user space mutex) mechanisms provided by certain operating systems, such as those using a Linux kernel. In this implementation, a thread waiting for a state variable (e.g. via a pthread_cond_wait) makes a futex system call with a FUTEX_WAIT argument, causing the thread to be suspended and descheduled. When the worker notifies the blocked thread (e.g. via pthread_cond_signal), a Futex system call is made with a FUTEX WAKE argument, causing the waiting thread to be woken up and rescheduled. Such futex mechanisms require that there be a memory address accessible by all participating processes or threads, and that the data stored at that memory address be modified when a process accesses the processes or threads currently on the futex want to wait, notify/wake up.

If the synchronization data area 130 for each buffer is arranged in a ring buffer in the shared memory 110, making it accessible to both publishers and subscribers, and the value of the buffer counter stored therein is incremented each time a write operation to the corresponding one occurs, can a futex (a 32-bit integer stored in a memory location and futex system calls operating from that location) are stored in the synchronization data area 130. Such a futex system call may include a queue of processes or threads attached to the buffer counter in the same synchronization data area 130 to request operations on the queue. The queue of processes can include subscribers waiting for notification of a new write to a buffer, or publishers waiting for a write to a buffer to complete (e.g., when multiple publishers can write to a ring buffer). For example, the Futex system call can be used to send a notification to subscribers of a ring buffer when a value of a buffer counter corresponding to a buffer in the ring buffer is about to be and incremented by 2 to prompt the subscribers to post the newly written messages read out from the corresponding buffer. Similarly, the Futex system call can be used to queue Subscriber when the value of the buffer counter becomes either odd or not incremented since the last write. Since the synchronization data area 130 is unique for each buffer of a ring buffer, such a futex system call notification can also indicate the identity of the buffer on which the write operation is performed. As mentioned above, a concurrent publisher will be blocked from writing to the buffer if a write to that buffer is already in progress by another publisher and must wait for a notification that the write is complete before proceeding with a write itself. In such scenarios, the Futex system call can be used to send a notification to publishers of a ring buffer when a value of a buffer counter corresponding to a buffer changes from odd to even (or vice versa).

In some embodiments, the futex system call described above may be stored in the global header 124 and attached to the counter stored therein. Such a futex system call may include a queue of processes or threads attached to the counter in the global header 124 to request operations on the queue. For example, the Futex system call can be used to send a notification to subscribers of a ring buffer when a value of the counter is incremented by 2 to request the subscribers to read the newly written messages from the ring buffer. Similarly, the Futex system call can be used to put subscribers on a queue if the value of the counter has not been incremented since the last write to the ring buffer. As discussed above, a count can be used to identify a publisher's current write position in the ring buffer, and subscribers can read data written to previous buffers relative to the write position.

While the above description describes the use of Futex in combination with the buffer counter and/or the seqlock counter (i.e. Futex is shared among subscribers), a Futex can optionally be created for each subscriber separately. In such scenarios, each subscriber can be allocated a futex space that can store a bitmap containing 1 bit allocated to each channel (or ring buffer) for which the subscriber wants to receive notifications. When a subscriber is ready to wait for a channel or ring buffer, it can change the bit value for that channel or ring buffer to 1 (or 0) in the bitmap and then wait for the futex system call. After completing a write operation to a channel, the publisher can set the bit for that channel to 0 (or 1). change atomically using a bitwise atom AND operation. If the atomic operation is specified that the bit changed state from 1 to 0 or 0 to 1 (that is, a subscriber is waiting for the channel), a notification can be sent to the waiting subscriber using the Futex system call. This allows the subscriber to wait on multiple channels and efficiently know which channel (or channels) were responsible for waking him up

Notification mechanisms that do not use Futex system calls are also within the scope of this disclosure. Examples may include, without limitation, file-like mechanisms provided by the operating system (e.g. eventfd object, a first-in-first-out pipe, a network socket connection) with one such file-like entity per publisher/subscriber pair, UNIX socket (i.e. non-network or local) or the like can be provided.

In an eventfd-based notification mechanism, objects known as eventfd objects may be associated with file descriptors that are created by the OS upon request of a process. Such file descriptors can then be shared with other processes. Subscriber processes can efficiently wait for a file descriptor that is readable using various mechanisms such as, without limitation, select(), poll(), epoll(), etc. A publisher process can wait for the eventfd object file descriptor after performing a write operation, write to the shared memory, which efficiently wakes up the subscribing processes. Such a notification mechanism requires at least one eventfd object per ring buffer and can be created when the ring buffer is created. The eventfd object may be shared via any appropriate operating system supported means, such as, without limitation, inheritance from parent process to child process, sharing over local network sockets, or the like. This mechanism allows subscribers to efficiently wait for one of multiple channels or ring buffers to acquire new data, as operating systems provide mechanisms for waiting for notification via multiple file descriptors.

Optionally, the eventfd object can be used in a “semaphore mode” that maintains an internal counter. A publisher can increment this counter by a certain value, and when each waiting subscriber is woken up, the counter can be decremented by 1. This allows the publisher to control how many subscribers can be woken up using the eventfd notification. A separate counter (“subscriber counter”) can also be used for the number of subscribers waiting on a channel or ring buffer. When a subscriber wants to wait for a channel, it first increments the subscriber counter and then checks whether it still has to wait (by comparing it with the previous counter value). If the subscriber still has to wait, it waits for the eventfd object. However, if the subscriber does not need to wait, it immediately decrements the subscriber counter and does not wait for the eventfd object. When the publisher is ready to notify subscribers of a new write operation, it reads the waiting subscriber's count via an atomic swap instruction on the subscriber counter, which returns the subscriber counter value to the publisher and resets the subscriber counter to zero. The publisher then writes this value to the eventfd object, resulting in the correct number of subscribers being notified/awakened

In certain embodiments, use of eventfd notification (at each node) may require an eventfd broker to execute the notification mechanism. For example, the eventfd broker can create a domain socket and listen for connection requests. A peer can open a connection to the broker and open that connection as long as the broker is executing a task. Publishing peers can also create or allocate their storage region in shared storage and the corresponding eventfd object, and submit the region name and eventfd object to the eventfd broker. The eventfd broker can broadcast the scope names and eventfd objects for channels requested by a subscribing peer, including updates as publishers come and go (or restart). In some embodiments, the Eventfd broker may be the storage broker described below.

Storage broker

Referring again to 1 The shared memory 110 may also be in communication with a memory broker 170 configured to manage the memory areas 112a-n and the ring buffers 120a-n. To do this, a broker 170 may include a single process configured to execute all tasks or multiple processes assigned to perform one or more of the storage broker's tasks. The storage broker 170 may dynamically establish an inter-process communication mechanism (e.g., a UNIX domain nen socket, an Internet socket, local sockets, a network socket, a first-in-first-out pipe or the like) according to various operating system requirements for communicating with subscribers, publishers and/or shared memory 110. It is Note that the storage broker 170 only communicates metadata and is only involved during startup, state reconstruction, task restart and/or shutdown. The messages published in shared storage 110 never flow through storage broker 170.

A memory broker 170 may be initialized by creating an inter-process communication (IPC) mechanism using any method now or hereinafter known. If the IPC mechanism is a domain socket, initialization may include binding the socket to a globally configured broker address to produce a unique file path for a given network node. If a storage broker cannot generate the unique file path using the globally configured broker address, it may be an indication that another storage broker already exists/is running for the network node. Upon initialization, the storage broker may identify all existing named storage areas in the shared memory 110, for example, by accessing the temporary file storage containing named file areas (e.g., dev/shm in Linux operating systems). The storage broker can then use the expected ring buffer naming pattern to identify ring buffer IDs for each named storage area. The memory broker can also map any named memory region (e.g. using mmap()) and attempt to attach to that memory region's ring buffer. If the append is not successful, the memory broker may determine that such a ring buffer is corrupted and delete the corresponding entry in the temporary database 172 (e.g., /dev/shm entry). If a process uses such a corrupted ring buffer, Deleting the corresponding entry will not release the area until all references to the ring buffer have been removed.

It should be noted that if a notification mechanism based on shared file descriptors (e.g. Match Object described above) is selected, UNIX domain sockets must be used as the IPC mechanism between storage brokers and their peers, as this is the is the only IPC mechanism that allows file descriptors to be shared between processes.

If the append is successful, the storage broker 170 may create a record of the ring buffer in a data store 175a-n and periodically update information or metadata related to the ring buffer, such as, without limitation, ring ID, buffer size, buffer orientation, ring size (i.e. the number of buffers), the associated channel if the associated channel is known, a list of publishers known to be published to the ring, or the like. In some embodiments, if the append is successful, the storage broker 170 may place the ring buffer into the probation or probation state and create a corresponding record in a probation list (discussed below).

In certain embodiments, upon initialization, memory broker 170 may perform various functions to manage shared memory 110, such as, without limitation, accepting connections from processes (subscribers and/or publishers) on its inter-process communication sockets; validate or authenticate the connections; Exchange protocol messages with the processes, initialize, and/or create new memory areas/ring buffers/channels/etc. to assign; Review and update metadata or information related to existing ring buffers/channels/peers (periodically and/or upon occurrence of a trigger event); performing state reconstruction; Performing shared memory diagnostics and repairs; Communicating information via ring buffers to publishers, subscribers and diagnostic/test tools, or the like.

In one or more embodiments, the memory broker 170 may allocate, initialize, and/or release the various memory areas 112a-n and/or ring buffers 120a-n, for example, in response to a request from a publisher of a topic by sending an instruction to the operating system. As discussed above, a memory area can contain only one ring buffer.

4 is a flowchart illustrating an example method for allocating a ring buffer by a storage broker. At 402, the storage broker may receive a request from a publisher to publish messages related to a topic "X", and the request may include identification and authentication information corresponding to the publisher. Such identification information For example, mations may include a publisher ID, such as a security identifier, a device identifier, or the like. The identification information may also include information such as, without limitation, a globally unique identifier (GUID), a task instance name, etc. For example, the storage broker may determine the publisher identification (pose filter), the task name (pose data), and the task GUID. The term "task" as used in this description represents a unit of work for executing parallel processing of a process and a thread on a UNIX operating system, a task on a real-time operating system, and the like. Each task may publish one or more topics .

The storage broker can determine the topic X based on the identification information. Optionally, the topic "X" can be explicitly included in the request. The request may also include parameters of messages to be published by the publisher, such as, without limitation, metadata about the message, deadline, buffer sizes, default values, QoS reliability criteria, types of configuration data, or the like. Authentication information may include information used by any or hereinafter known authentication protocols, such as, without limitation, usernames, passwords, security certificates, authentication tokens, or the like.

It should be noted that the storage broker may include and/or access a data store 175a-n that contains a database of peer IDs, task GUIDs, task instance names for performing identification and authentication. Each peer ID may also be associated with a list of subscriber channels, a list of ring buffers that the peer publishes, a connection status (e.g. connected or disconnected, socket descriptor, etc.), or the like. The database may be periodically updated by the broker 170.

In certain embodiments, the publisher may initiate a connection with the storage broker (e.g., in a HELLO packet) before sending a PUBLISH MESSAGE request to the storage broker. One or more of the identification and/or authentication information may be included in the HELLO packet. Parameters for publishing the message and identifying the topic may be included in the HELLO packet and/or the PUBLISH-MESSAGE request.

At 404, the storage broker may determine whether a channel corresponding to topic X already exists in the communications network. If such a channel exists (404: YES), the storage broker may assign a ring buffer on the channel to the publisher (step 406) by creating a new ring buffer on the channel for the publisher or assigning an existing ring buffer on the channel to the publisher becomes. The storage broker may assign to the channel a newly created ring buffer on the channel in at least the following situations: (i) the system allows multiple publishers to publish messages to a single ring buffer and all appropriately configured ring buffers on the channel are already live to a publisher assigned; (ii) the system allows multiple publishers to publish messages to a single ring buffer, but the ring buffers on the channel are not suitably configured for the publisher's messages (e.g. inappropriate ring buffer size, number of buffers, message size, difference in tasks in relation to the other publishers of this ring buffer or similar); (iii) the pre-existing ring buffers are not suitably configured for the publisher's messages (e.g. inappropriate ring buffer size, number of buffers, message size, or the like); (iv) channel ring buffers are in an error state; (v) the publisher may not publish to the pre-existing ring buffers, etc. The storage broker may use the parameters included in the publisher's request to create the ring buffer in light of various channel constraints. Examples of such channel limitations may include, without limitation, maximum message size, maximum number of buffers, message type, channel priority (in terms of allowable latency), or the like.

The storage broker may assign a pre-existing ring buffer on the channel to the publisher in at least the following situations: (i) the system allows multiple publishers to publish messages to a single ring buffer, and there are one or more ring buffers on the channel that are suitable are configured for the publisher's messages; (ii) the system does not allow multiple publishers to publish messages to a single ring buffer and there is at least one appropriately configured ring buffer on the channel that is not already associated with a live publisher; etc.

In some embodiments, if a channel already exists, the storage broker may allocate a ring buffer from a trial list on the channel (discussed in more detail below).

If a channel corresponding to topic X does not already exist (404: NO), the Storage broker 408 creates 408 a new channel and at least one ring buffer for the publisher. Various channel parameters for creating the channel and the ring buffer parameters may be provided by the publisher, the storage broker and/or a subscriber. Examples of channel parameters may include, without limitation, the maximum size of each message on the channel (which is used to determine the size of each buffer in the channel's ring buffers), the maximum number of messages to the queue (which is used to determine the number of buffers in the ring buffers of the channel), the number of publishers that can be published on a ring buffer of the channel, a maximum number of ring buffers on the channel, ring buffer sharing allowed or not, channel priority in terms of allowed latency, types of Messages, channel naming protocol, permitted list of publishers who may publish to the channel, or the like. In certain embodiments, the storage broker may use a stable integer ID as a channel identifier independent of the channel character name. Such a naming protocol can simplify backend protocol processing, give the header a compact and fixed size, and reduce the risk of collisions between names or the like. For example, a 64-bit hash of the channel name can be used as the stable integer ID

In some embodiments, a channel may be associated with an approved list of publishers that can publish messages to ring buffers of the channel. In such embodiments, even if a channel corresponds to topic X, the publisher may not be allowed to send the request to publish messages to its ring buffers. In such scenarios, the storage broker may create a new channel with a new ring buffer, as discussed above with respect to step 408.

At 410 and 412, the storage broker may send a message (e.g., PUBLISH_OK) to the publisher that includes information regarding the channel and/or ring buffer to which the publisher can publish.

At 414, when a new ring buffer is allocated to the publisher in an already existing channel, the storage broker may send a message (e.g., SUBSCRIBE_TO) to all subscribers of the channel or topic "X" that includes information related to the channel and/or Obtain the new ring buffer.

When a new channel is created, the storage broker may receive a message (416) from a subscriber wishing to subscribe to the channel and/or topic "X" (e.g., a SUBSCRIBE message). Optionally, the storage broker can receive a message from a subscriber to request subscriptions to a publisher and any ring buffers/channels associated with the publisher. In response to receiving the message, the storage broker may transmit a message (e.g., a SUBSCRIBE_TO) requesting the subscriber to subscribe to the channel, topic "X", and/or publisher, and information relating to the channel and refer to each ring buffer on the new channel. The storage broker can repeat the subscription and send additional messages to the subscriber as new ring buffers are created on the channel.

Peers may terminate a channel relationship and/or relationship with a channel's ring buffer by sending a message to terminate the relationship to the storage broker, including the channel ID and/or the ring buffer ID (e.g., STOP_PUBLISH or STOP_SUBSCRIBE, respectively after). Alternatively and/or additionally, all open relationships within a storage system terminate by sending a message to the storage broker (e.g. a GOODBYE message) followed by a disconnect. For example, a CLOSE_RING message can be sent to all publishers and subscribers of a ring buffer when the ring is shut down by the broker in a clean manner.

In some embodiments, the publisher may stop publishing to a ring buffer and send a notification to the storage broker regarding the same (e.g., a GOODBYE or a STOP_PUBLISH message). The storage broker can then notify subscribers of the ring buffer that the publisher is no longer publishing data to the ring buffer (e.g. a STOP_SUBSCRIBE message). Such a notification may optionally include information regarding the last write position (e.g. the seqlock counter) so that the subscriber can continue to read messages from the ring buffer until it has read the last published message. Alternatively, a subscriber can stop reading messages from a ring buffer as soon as it becomes aware that a publisher has stopped publishing messages, even if there are unread messages in the ring buffer. A subscriber can also autonomously decide to stop reading messages from a ring buffer and send an appropriate message to the storage broker.

In one or more embodiments, the storage broker may assist in generating a channel, create a record of the channel in a data store 175a-n, and periodically (and/or when any changes are made to the channel) update information relating to the channel such as, without limitation, channel name and/or ID, a list of peers known to be subscribed to, a list of peers known to publish to the channel, a list of ring buffers known to be associated with that channel , or similar.

Similarly, when creating a ring buffer, the storage broker may create a record of the ring buffer in a data store 175a-n and periodically (and/or when a peer reports that the ring buffer is not functioning properly) update information related to the ring buffer, as without limitation, Ring ID, buffer size, buffer alignment, ring size (i.e. the number of buffers), associated channel if the associated channel is known, a list of publishers known to publish to the ring, or the like.

The storage broker may also periodically analyze a state of existing ring buffers in the system to determine whether one or more of the ring buffers should be temporarily removed from use (i.e., when the broker determines whether there are errors and the system is functioning as before), is revised from probation and/or permanently removed from use (i.e., abandoned). Optionally, the storage broker may analyze a state of a ring buffer in the system to determine whether that ring buffer is placed on probation, revised upon the occurrence of a trigger event and/or should be permanently abandoned. Examples of such trigger events may include, without limitation, an unexpected loss of connection (e.g., network connection) between a publisher and a storage broker, receiving a message from a peer of the ring buffer indicating a problem with the ring buffer (e.g., a publisher waiting for a write lock release , a subscriber waiting for new messages, or any peer-detected potential corruption in the ring buffer), occurrence of a bad read operation and/or a bad write operation to the ring buffer, memory broker reconstruction (described below), ring buffer metadata or headers an error or are invalid (e.g. number of buffers, seqlock value, size, etc.), or the like. If there is an unexpected loss of connection between a publisher and the storage broker (e.g., loss of connection without a GOODBYE message from the publisher), storage can then analyze the state of all ring buffers associated with such publisher.

When a trigger event occurs, the storage broker may analyze the state of more than one ring buffer (e.g. unexpected loss of connection between a publisher and the storage broker) or a specific ring buffer (e.g. message from a peer around a ring buffer) to determine whether the ring buffer is falsified. A ring buffer is corrupted when ring buffer metadata or headers contain an error or are invalid (e.g. number of buffers, seqlock value, size, etc.) and/or when a publisher flags a ring buffer error condition. If a ring buffer is determined to be corrupted, the storage broker may bypass the probation and exit the ring buffer, as described below. If the ring buffer(s) are not corrupted, the storage broker can capture the ring buffer state and place it on rehearsal. In some embodiments, if a connection to the publisher is reestablished before any decommitment conditions occur (described below), the storage broker may remove the ring buffer from probation and instruct the publisher to resume publishing to the ring buffer.

A ring buffer can be placed on probation if, for example, there are no live and active publisher publication data for the ring buffer. This can happen if, for example, a publisher stops publishing messages to the ring buffer; a publisher hangs, crashes, restarts or is otherwise disabled, or the like. The storage broker may determine that there is no live and active publisher publishing data to the ring buffer by analyzing at least one of the following: the value of the seqlock counter, the value of the buffer counter, or the write lock state. For example, if the value of the buffer counter does not change (i.e., not incremented) and/or the seqlock counter does not change for a certain period of time (i.e., static value), the storage broker may determine that a publisher did not publish data to the ring buffer. Similarly, if the write lock is not acquired by a publisher for a certain period of time, the storage broker can determine that a publisher did not publish data to the ring buffer. If the write lock acquired by a publisher before a write operation is not released for a certain period of time, the storage broker may determine that the publisher has crashed, hung, rebooted, or is otherwise disabled.

Finally, a ring buffer may be placed on probation during state reconstruction of a storage broker after the storage broker is restarted (described below). Such ring buffers remain on probation until the storage broker has collected enough state information to determine that there is at least one live publisher associated with a ring buffer.

After determining that a ring buffer should be placed on probation, the storage broker 170 cannot remove the ring buffer from use and read/write operations may continue while the broker analyzes the ring buffer state. Once a ring buffer is placed on probation, it remains in probation until either it is known to have a live and active publisher and there are no write locks held too long by each publisher; or until it is abandoned, based on the analysis of the state of the ring buffer during the trial.

The storage broker 170 may also generate a record of the ring buffer that is stored in a data store 175a-n (i.e. added to a trial list), the record containing at least one of the following information: ring buffer ID, current write position or HEAD in the ring buffer, whether a Write lock is present on the ring buffer (i.e. seqlock not released), timestamp of the most recent write operation, timestamp when the ring is put on probation, and timestamp of the most recent record update. In some embodiments, storage broker 170 may periodically analyze the state of each ring buffer on the probe list at a configurable frequency (e.g., about 1 Hz, about 2 Hz, about 3 Hz, or the like). Alternatively and/or additionally, the storage broker 170 may analyze the state of one or more ring buffers on the trial list upon the occurrence of a trigger event.

The storage broker 170 may analyze the state of a ring buffer by checking for state updates in the ring buffer itself, e.g., by comparing the current state with the corresponding values/information of the data set contained in the trial list. The storage broker 170 may update the record of the ring buffer in the trial list to include the current state information or value and the corresponding timestamps. For example, state information corresponding to the write lock status may be captured/released, change in the HEAD or current write position of the ring buffer, or the like.

The storage broker 170 may remove a buffer list from probation when the state of a ring buffer changes in response to probate changes to include one of the following: at least one known publisher, a change in HEAD position, and/or a release of any write locks , which are seen during the final test. After removing a ring buffer from probation, the storage broker 170 may send a notification (e.g., SUBSCRIBE_TO messages) to subscribers of that ring buffer, topic, or channel.

However, if the state of a ring buffer changes such that it cannot be removed from probation, the memory broker 170 may use the timestamps (e.g., the total time in probation, total time of write lock held, the time since the HEAD position changed, and/or the time since the last published message, etc.) in the ring buffer's held record to determine whether the ring buffer should be abandoned. For example, the storage broker 170 may examine the timestamps to determine the total time that the ring buffer has been on probation and if it is greater than a threshold time (e.g., about 8-12 seconds, about 5-15 seconds, about 2-18 seconds , about 9 seconds, about 10 seconds, about 11 seconds, or the like), the memory broker 170 may abandon the ring buffer. Similarly, the storage broker 170 may examine the timestamps to determine the total time associated with a ring buffer write lock being held or captured, and if it is greater than a threshold time (e.g., about 0.1-0.2 seconds , about 0.2-0.5 seconds or so), the memory broker 170 may exit the ring buffer. Additionally and/or alternatively, if the HEAD or write position of the ring buffer does not change for a threshold period associated with the channel of the ring buffer (e.g., about 2-8 seconds, about 3-7 seconds, about 4-6 seconds, about 5 seconds, about 4 seconds, about 7 seconds or the like), the memory broker 170 gives up the ring buffer. In yet another embodiment, if the timestamp associated with the most recent message published to the ring buffer is older than a threshold time period associated with the channel of the ring buffer (e.g., about 2-8 seconds, about 3-7 seconds, about 4-6 seconds, about 5 seconds, about 4 seconds, about 7 seconds or the like), the memory broker 170 gives up the ring buffer.

If memory broker 170 determines that the ring buffer should be abandoned, it may execute an abandon procedure. The exit procedure may involve performing a or more of the following: logging an error and performing diagnostics on the ring buffer; marking the ring buffer as invalid/abandoned in shared memory 110 (eg, in a header space of shared memory 110); Sending a notification to the ring buffer's peers (subscribers subscribing to the ring buffer and publishers publishing to the ring buffer) that it has been abandoned;

deleting the corresponding memory area from the shared memory; deleting recording associated with the ring buffer in temporary data storage 172; and discarding all metadata and information stored in connection with the ring buffer. Once a ring buffer is exited, the system or storage broker 170 can still read messages for logging purposes (even if the messages are corrupted).

In some embodiments, storage broker 170 may instruct subscribers that they can continue to consume pending messages from the ring buffer being exited, but should then release the ring once they have read all pending messages (e.g., a CLOSE_RING message). Alternatively, the storage broker 170 may instruct subscribers to immediately release the ring buffer, leaving pending messages unconsumed (e.g. ABAN-DON_RING message). It should be noted that publishers can never publish new messages to a ring buffer that is abandoned.

The storage broker may analyze and update information regarding the peers periodically and/or upon the occurrence of a triggering event. Such information may include, for example, whether a peer is connected to the storage broker 170, channels subscribed to or published, topics, latency, or the like.

As discussed above, memory broker 170 has access to metadata and/or information about the state of all shared ring buffers, memory areas, and channels associated with shared memory 120 from system startup until shutdown. Therefore, if a publisher or subscriber becomes unavailable (e.g., intentionally due to a failure, crash, or loss of connection), the storage broker 170 may recover any lost information about the shared ring buffers, memory areas, and/or channels associated with the shared memory 120, to the publisher or the subscriber upon reboot from an error/crash and/or upon reconnection to the storage broker 170 (as long as the storage broker remains alive).

The memory broker 170 may also be configured to recover from failures or crashes of the memory broker itself using the state reconstruction techniques described below. Such state reconstruction does not disrupt any existing and established publisher/subscriber relationship, so publishers continue to publish messages to existing ring buffers and subscribers continue to receive these messages.

5 is a flowchart illustrating an example procedure for state reconstruction of a memory broker upon restart after a failure or crash. It should be noted that during state reconstruction, any existing and active publisher/subscriber relationships established before the storage broker restart may continue to operate normally. The storage broker cannot break these relationships unless it determines that there is an associated problem or error (discussed below).

At 502, a storage broker can restart from a crash or failure. For example, the storage broker can create a new UNIX socket. The memory broker can determine whether the reboot results from a crash or a failure by checking for leftover or abandoned memory areas in the shared temporary memory 172 (e.g., /dev/shm/), since such abandoned memory areas only exist during a failure reboot ( and not after a clean shutdown event). Upon detection of such remaining memory areas, the memory broker may then place all existing and active ring buffers in the shared memory on probation (504) until the ring is either abandoned or assigned to a live and active publisher.

Upon reboot, the storage broker can establish a connection (506) with a peer upon receipt of a connection request (e.g. in a HELLO packet). The peer can establish the connection by sending a connection request containing identification information and/or authentication information to the storage broker. As discussed above, the identification information may include, for example, a peer ID, a task GUID, a task instance name, or the like. The storage broker may verify the identification and/or authentication information by consulting a data store configured to store peer information and the connection at the to establish verification. The storage broker may also update metadata associated with the peer in the data store 175a-n. When the broker receives a connection request from a peer, it can check the peer ID for duplicates/collisions and can deny the connection request if a connection already exists with the requesting peer.

Once connected to a peer, the storage broker may receive a publish and/or subscribe request from the peer (508). The storage broker may determine whether the peer was already published or subscribed to a ring buffer associated with the storage broker prior to the storage broker restart (510). If the peer has already been published or subscribed to a ring buffer associated with the storage broker before the storage broker restarts, the peer will have information or metadata about the ring buffer (previously provided by the storage broker to the peer, as discussed above). Examples of such information may include, without limitation, ring buffer ID/name, memory area name, channel ID/name, channel parameters, subscriber IDs (for a publisher), publisher IDs (for a subscriber), or the like.

Therefore, if the request to publish and/or subscribe contains information or metadata about the corresponding ring buffer(s), the storage broker may determine that the peer has already been published or subscribed to a ring buffer associated with the storage broker prior to the storage broker restart (510: YES). . However, if the publish and/or subscribe request does not contain information about a ring buffer, the storage broker may determine that the peer was not published or subscribed to a ring buffer that was associated with the storage broker before the access before the storage broker restart (510 : NO) and can create a new ring buffer for the peer (as referenced above 4 discussed).

If the storage broker determines that the peer has already been published or subscribed to a ring buffer associated with the storage broker before the storage broker restart (510: YES), it may use the received information about the ring buffer to determine whether to place a corresponding ring buffer in the buffer 172 (e.g. /dev/shm/), which is preserved in the event of such failures or crashes (512) (i.e. one of the ring buffers was set to probation). If the memory broker cannot identify a corresponding ring buffer in the temporary memory 172 (512: NO), it may determine that the ring buffer device was either not present in the shared memory or was present but was determined to be invalid/corrupted and abandoned, so that the corresponding entry has been deleted to form temporary storage (as discussed above). The memory broker can then transmit instructions to exit the ring buffer to the peer (514). The peer can then close the ring buffer information from its memory and not map the associated memory area.

If the storage broker can identify a corresponding ring buffer in the temporary memory 172 and/or the data stores 175a-n (512: YES), the storage broker can add the peer to a channel associated with the ring buffer (516). For example, the storage broker can add the peer to the list of publishers/subscribers for this channel. In certain embodiments, if the peer is a subscriber, the storage broker may send subscribe notifications/information regarding all known ring buffers for the channel to the peer (e.g., SUBSCRIBE_TO messages). In some embodiments, if the storage broker can identify a corresponding ring buffer but no associated channel is present, the storage broker may identify a channel using information provided by the peer and associate the channel with the peer.

In certain embodiments, if the identified ring buffer is on probation and the peer is a publisher, the storage broker may remove the ring buffer from probation. The storage broker can then send a notification to all known subscribers for that channel/ring buffer to inform them of the new ring buffer (e.g. SUBSCRIBE_TO messages). Certain subscribers may ignore this notification if they already know the existence of the ring buffer, for example if the storage broker is under state reconstruction and does not know that the subscribers already know about the new ring buffer.

In certain embodiments, if the identified ring buffer is on probation and the peer is a subscriber, the storage broker may not remove the ring buffer from probation. The storage broker can store subscriber information for ring buffers upon probation and can respond to ring commands if the ring buffer is later determined to be blocked/corrupted.

The storage broker may also update (518) the ring buffer information/metadata in the corresponding record to include information about the peer (e.g. list of publishers/subscribers), a new connection, a new channel and/or to include any other information contained in the request received from the peer.

If the storage broker determines that the peer has not already published or subscribed before the storage broker restart (510: NO), the storage broker may send a message to the peer (520). If the peer is a publisher, the storage broker may allocate a new ring buffer to the peer and the transmitted message may contain information related to the allocated ring buffer (as referred to above 4 ). If the peer is a subscriber, the message sent may contain information relating to one or more ring buffers/channels to which the subscriber may subscribe (as referred to above 4 ).

The above process allows the memory broker to reconstruct to a state that existed before a collision or error based on information from the operating system (e.g. temporary memory listing /dev/shm) and information provided by the peer when reconnect to the storage broker. During state reconstruction, ring buffers can remain in probation until they are known to have at least one live publisher. The broker can also perform diagnostic tests, for example by analyzing whether or not it has access to the shared memory, whether the header information is correct, etc.

Network bridge

The above description relates to shared storage usage and management in a publish/subscribe messaging system with respect to a single network of a communication system. In particular, the above description describes sharing messages between publishers and subscribers on the same network. Often such publish/subscribe messaging systems are used in distributed computer systems that include large numbers of networks or nodes on different networks. Such distributed computing systems should be able to handle large amounts of data and communications needed to keep different systems operating and sharing information. The following disclosure describes systems and methods for sharing data between networks of a distributed computing system using a publish/subscribe messaging system and preventing network congestion, redundancy and latency commonly associated with such distributed systems.

With reference to 6 An exemplary architecture of the distributed computer system 600 is shown. As in 6 As shown, the distributed computer system 600 may include network nodes 601a-n (or networks). Each network node 601 may include shared memory 610a-n, which includes memory areas 612a-n and ring buffers 620a-n. Each network node 601 may also include a storage broker 670a-n, temporary storage 672a-n, and data storage 675a-n. For each network node there are one or more threads or computer processes 651a-n, 652a-n,..., 653a-n, each containing one or more publishers 661a-n, 662a-n,..., 663n-a, which have communicative access to the shared storage of this network node. In addition, one or more threads or computer processes 671 at, 672a-n, ..., 673a-n, each comprising one or more subscribers 681 at, 682a-n, ..., 683a-n, may also access the shared used memory of this network node. Each of these components in a network node has a functionality/configuration as referred to above 1-5 described.

Each network node 601a-n may also include a bridge module 690a-n (i.e., software running on each node) configured to provide bridging between network nodes of the distributed computing system 600 to facilitate the sharing of messages between such network nodes, such as discussed below. The bridging or bridge connects two different networks or network nodes of the distributed computer system, allowing them to communicate with each other as if they were a single network. These may use various data transport protocols such as, without limitation, an Internet Protocol (IP), a Transport Control Protocol (TCP), a Universal Datagram Protocol (UDP), a Stream Control Transmission Protocol (SCTP), remote memory synchronization protocols such as Remote Direct Memory Access (RDMA) protocols, a datagram congestion control protocol (DCCP), a switch, a router or the like.

In certain embodiments, a bridge module may be included in the storage broker of a network node. Alternatively and/or additionally, a bridge module can run as a separate process on a network node. A bridge module may include a single process configured to perform all tasks or multiple processes assigned to perform one or more of the bridge application's tasks.

With reference to 7 Shown is a flowchart illustrating an example method for sharing information between network nodes of a distributed system. At 702, a bridge module may receive information regarding the distributed system. In particular, the bridge module can receive information about existing channels and ring buffers in the distributed system as well as the network locations of publishers and subscribers of existing channels. A bridge module may receive information about channels, ring buffers, publishers, and subscribers that exist on the same network node as the bridge module (ie, local data) from, for example, a storage broker on the bridge module's network (as discussed above, the storage brokers maintain and update channel information in the data store 675a -n). Alternatively, a bridge module can subscribe to various notifications from shared storage, publisher, subscriber, data store, etc. to receive the local information directly.

A bridge module can receive information about channels, ring buffers, publishers, and subscribers that exist on remote networks (i.e., “remote data”) from other bridge modules (on other network nodes) in the distributed system. A bridge module may interact with other bridge modules (e.g., to share information about which channels are published and subscribed to their local nodes) in the distributed system through any suitable communication protocols such as, without limitation, TCP/IP, UDP/IP, Ethernet , SCTP or the like. The system may use various synchronization methods to synchronize data received by a bridge module from other bridge modules in the distributed system. Examples may include, without limitation, eventually consistent cross-node replication mechanisms, distributed consensus algorithms (e.g., ^Paxos® , Raft® ^, etc.), non-consensus-based master-slave replication algorithms, or the like. For example, in one embodiment, each bridge module may be configured to be the master portion of its own subset of data (ie, local data), and each other bridge module may be a slave to that data. Thus, all bridge modules are both masters and slaves over different subsets of the synchronized data. In certain embodiments, the information about the existing channels, buffers, rings, publishers, subscribers, and network locations of publishers and subscribers in the distributed system may be statistically configured (e.g., at startup), and provided to the bridge modules, for example, when the information is during do not change during the term.

At 704, the bridge module may use the received information to determine, for each existing channel/ring buffer on the bridge module's network (“local channel/local buffer ring”), whether one or more subscribers exist on another network node (“remote- Subscribers”). If such remote subscribers exist (704: YES), a bridge module can identify the network node or nodes (706) on which the remote subscribers exist (“remote node”). For each such remote node, the bridge module may perform bridging 708 (to connect to a bridge module of such remote node) to transfer data from the local ring buffer(s) to the remote subscribers on that remote node. If such remote subscribers do not exist (704: NO), a bridge module can continue to monitor the distributed system by receiving information about it (702).

As in 7 As shown, to perform bridging (706) with a remote node, the bridge module may first identify the ring buffer(s)/channel(s) that have remote subscribers on the remote node (781). The bridge module can then subscribe to messages published on the identified ring buffers (782). The bridge module may transmit the messages read from the identified ring buffers to the remote node (783) and publish the messages to corresponding ring buffers in a shared memory of the remote node (784) for use by the remote subscribers on the remote node. In this way, bridge modules act as “subscribers” on the local node where the publisher is located, and as “publishers” on the remote node(s) where the remote subscribers are located. The bridge module can subscribe to ring buffers on the local node and publish to ring buffers on the remote node using the methods and systems described above.

It should be noted that for each ring buffer on the local node that the bridge module has subscribed to, a corresponding and identical ring buffer is created in the shared memory of the remote node. Specifically, using the methods of the present disclosure, a copy of the ring buffer in the local node is created on the remote node by the bridge module. Publishers and subscribers remain unattended by any other network node. Furthermore, data published to a ring buffer on a local node is transmitted to the remote node only once to create the ring buffer copy gen, regardless of the number of remote subscribers on the remote node.

The bridge module can use any of the network protocols described above (e.g. TCP, USP, SCTP, etc.) to transmit data to the remote network (via the remote network node bridge module).

Alternatively and/or additionally, other network protocols may be used to perform the bridging in a "zero-copy" manner, wherein a bridge module that includes a kernel network stack and/or network hardware (e.g. bridge, switch, router, etc.), can directly read messages from the ring buffers on the local node and write them to ring buffers on the remote node. In such scenarios, the system can preserve the integrity of the seqlock-based synchronization and notification mechanisms discussed above by performing additional checks on seqlock counters associated with the ring buffer on the local node and the corresponding ring buffer copy on the remote node are associated. At the local node, the bridge module can read the ring buffer's seqlock counter both "before" and "after" a read operation and transmit the values to the remote node's bridge module. At the remote node, the bridge module can obtain a write lock on the seqlock of the ring buffer copy, read the value "before" the seqlock value into the local memory, read the message data directly into the shared memory, the read value "after" the seqlock value read and compare the message data with the values; if there is a mismatch, the read was inconsistent and the target message must be marked as invalid. This process may be implemented using, for example, vectorized scatter/gather instructions, I/O instructions, or the like (similar to the lock-free mechanism described above).

In some embodiments, the bridge module may prioritize the transmission and publication of messages associated with particular channels or ring buffers to the remote node over other channels or ring buffers. The bridge module can therefore configure and control the latency associated with the channels. The system may determine latency based on, for example, the importance of the ring buffer/channel topic, the number of subscribers, the number of publishers, or the like.

8th illustrates the use of a network bridge to distribute messages published from publishers on one node 801 to subscribers on another node 802. For example, node 801, as in 8th shown include a ring buffer 811 that stores data related to processed LiDAR data, a ring buffer 812 that stores data related to observed poses, and a ring buffer 813 that stores data related to filtered poses. A LiDAR data processing system (not shown, but on node 801) may send data to ring buffer 811. LiDAR data subscribers, such as a scan matcher 815 (on node 801) and a tracker 821 (on node 802), may publish data in the ring buffer 811. The scan matcher 815 may publish data to the ring buffer 812, and a pose filter 816 may subscribe to the published data in the ring buffer 812. The pose filter 816 may publish data to the ring buffer 813, and multiple subscribers, such as the scan matcher 815 (at node 801), the tracker 821 (at node 802), or other systems 822 (at node 802) may receive data Subscribe to the ring buffer 813.

Subscribers to a channel/ring buffer that exist on the same network node as a publisher of that channel can read messages from a ring buffer in that network node's shared memory, as described above. For example, scan matcher 815 on node 801 subscribes to ring buffers 811 and 813, and pose filter 816 subscribes to ring buffer 812, which reads messages from the respective ring buffers using the methods discussed above. However, if the subscriber of a channel exists on a different network node from that channel's publisher, the bridge module may create a bridge on the publisher's network node to create a copy of the ring buffer on the subscriber's network node 802 (as described above), and all subscribers on network node 802 can read data published to the ring buffer by accessing a ring buffer copy. This bridging is transparent to the peers. For example, the bridge module (not shown here) on the network node 801 may create a copy 811 (a) of the ring buffer 811 on the network node 802, and the tracker 821 on the node 802 that subscribes to the ring buffer 811 may receive data from the ring buffer copy 811 ( a) read. Similarly, the bridge module on the network node 801 may create a copy 813 (a) of the ring buffer 813 on the network node 802, and the tracker 821 and/or other subscribers 822 on the node 802 that subscribe to the ring buffer 813 may receive data from the Read ring buffer copy 813 (a). In this way, the distributed system containing multiple network nodes can present a unified publish/subscribe messaging system in which subscribers attached to different nodes can receive publish messages representing the node the publisher is connected to.

9 shows an example of internal hardware that may be included in any of the electronic components of the system, such as internal processing systems, external monitoring and reporting systems, or remote servers. An electrical bus 900 serves as an information highway connecting the other hardware components shown. Processor 905 is a central processing device of the system configured to perform calculations and logic operations required to execute programming instructions. As used herein and in the claims, the terms "processor" and "processing device" may refer to a single processor or any number of processors in a set of processors that collectively perform a set of operations, such as a central one processing unit (CPU), a graphics processing unit (GPU), a remote server, or a combination thereof. Read-only memory (ROM), random access memory (RAM), flash memory, hard drives, and other devices capable of storing electronic data represent examples of storage devices 925.

A storage device may include a single device or a collection of devices through which data and/or instructions are stored. Various embodiments of the invention may include a computer-readable medium containing programming instructions configured to cause one or more processors to perform the functions described in the context of the previous figures.

An optional display interface 930 may allow information from the bus 900 to be displayed on a display device 935 in a visual, graphical, or alphanumeric format. An audio interface and an audio output (such as a speaker) may also be provided. Communication with external devices may occur using various communication devices 940, such as a wireless antenna, an RFID tag, and/or a short-range or near-field communication transceiver, each of which is selectively communicable with other components of the device via one or more communication systems can be connected. The communication device(s) 940 may be configured to be communicatively connected to a communication network, such as the Internet, a local area network, or a cellular telephone data network.

The hardware may also include a user interface sensor 945 that enables receiving data from input devices 950 such as a keyboard, a mouse, a joystick, a touch screen, a touchpad, a remote control, a pointing device, and/or a microphone. Digital frames may also be received by a camera 920 that can capture video and/or still images.

The features and functions and alternatives disclosed above may be combined into many other different systems or applications. Various components may be implemented in hardware or software or embedded software. Various currently unforeseen or unexpected alternatives, modifications, variations, or improvements may be made by those skilled in the art, each of which is also intended to be encompassed by the disclosed embodiments.

Terminology relevant to the disclosure provided above includes:

An “automated device” or “robotic device” refers to an electronic device that includes a processor, programming instructions, and one or more physical hardware components that can move in response to commands from the processor with minimal or no human intervention. Such movement allows a robotic device to perform one or more automatic functions or sets of functions. Examples of such operations, functions or tasks may include, without limitation, operating wheels or propellers to effect propulsion, flight or other transportation actions, to operate robotic lifts for loading, unloading, medical processes, engineering processes and/or the like. Example automated devices may include, without limitation, autonomous vehicles, drones, and other autonomous robotic devices.

The term “vehicle” refers to any movable form of transportation capable of carrying either one or more human occupants and/or cargo and is powered by any form of energy. The term “vehicle” includes, but is not limited to, cars, trucks, vans, trains, autonomous vehicles, airplanes, aerial drones and the like. An "autonomous vehicle is a vehicle with a processor, programming instructions and powertrain components that can be controlled by the processor without requiring a human operator. An autonomous vehicle may be fully autonomous, in that it does not require a human operator for most or all driving conditions and functions, or it may be semi-autonomous, in that a human operator may be required under certain conditions or for certain operations, or that a human vehicle operator may require the autonomous vehicle Can override the vehicle's system and take over control of the vehicle. Autonomous vehicles also include vehicles in which autonomous systems assist human operation of the vehicle, such as vehicles with driver-assisted steering, cruise control, braking, parking, and other systems.

An “electronic device” or a “computing device” refers to a device that includes a processor and a memory. Each device may have its own processor and/or memory, or the processor and/or memory may be shared with other devices such as in a virtual machine or container array. The memory contains or receives programming instructions that, when executed by the processor, cause the electronic device to perform one or more operations in accordance with the programming instructions.

The terms "memory", "storage device", "data storage", "data storage" and the like each refer to a non-transitory device on which computer-readable data, programming instructions, or both are stored. Except where expressly stated otherwise, the terms "memory", "storage device", "data storage", "data storage device" and the like are intended to include individual device embodiments, embodiments in which multiple storage devices store a set of data or instructions together or in common, as well as individual sectors within such devices.

The terms “processor” and “processing device” refer to a hardware component of an electronic device that is configured to execute programming instructions. Except where specifically stated otherwise, the singular term “processor” or “processing device” is intended to include both single processing device embodiments and embodiments in which multiple processing devices perform a process together or in concert.

As used in this document, the terms “communication link” and “communication path” mean a wired or wireless path over which a first device sends communication signals to one or more other devices and/or receives communication signals. Devices are connected or communicatively coupled to one another via a communication link if the devices can send and/or receive data over a communication link. Electronic communication refers to the transmission of data via one or more signals between two or more electronic devices, whether over a wired or wireless network, and whether directly or indirectly through one or more intermediate devices.

Although portions of the discussion herein refer to wired connections and/or wired communications for exemplary purposes, embodiments of the invention are not limited in this regard and may include one or more wired or wireless connections, may use one or more wireless communication components, may include a or use multiple methods or protocols of wireless communication or the like. Some embodiments of the invention may use wired communications and/or wireless communications.

The term “subscriber” as used herein includes, for example: B. a remote subscriber; a local subscriber, a remote client, a local client, a subscriber implemented using a virtual machine (VM), a subscriber implemented using hardware components and/or software components, a subscriber implemented using a client -Device or a client computer or the like is implemented

The term "publisher" as used herein includes, for example, a remote publisher, a local publisher, a remote server, a local server, a publisher implemented using a virtual machine (VM), a publisher implemented under using hardware components and/or software components, a publisher implemented using a publisher device or a publisher computer or the like.

In this document, when relative terms of an order, such as "first", "second", are used to modify a noun, such use is intended only to do so be to distinguish one element from another, and this is not intended to imply sequential order unless explicitly stated.

An “automated device monitoring system” is a set of hardware that is communicatively and/or electrically connected to various components (such as sensors) of a vehicle for collecting status or operating parameter values from those components. An automated device monitoring system may include or be connected to a data logging device that includes a data input (such as a wireless receiver) configured to receive device operating data directly or indirectly from the components of the device. The monitoring system may also include a processor, a transmitter, and a memory with programming instructions. A monitoring system may include a transmitter for sending commands and/or data to external electronic devices and/or remote servers. In various embodiments, a monitoring system may be embedded or integral with the other computer system components of the automated device, or may be a separate device in communication with one or more other local systems, such as in the context of an autonomous vehicle On-board diagnostic system.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 17/097590 [0001]
• US 17/097671 [0001]
• US 17/097699 [0001]

Claims (91)

A system for providing lock-free access to a ring buffer, the system comprising: a shared memory comprising the ring buffer, the ring buffer comprising: - a plurality of sequentially arranged fixed-size buffers configured to store messages, and - a global header comprising a counter and a lock; a publisher configured to write a plurality of messages to the plurality of fixed-size buffers in the ring buffer; and a subscriber configured to read one or more of the plurality of messages written by the publisher, wherein the counter provides information regarding a fixed size buffer in the ring buffer to which the publisher will next write a message. System after Claim 1 , where a negative value of the counter indicates a state of the system in which the ring buffer contains no messages. System after Claim 2 , wherein the publisher is further configured to assign a value of 0 to the counter upon completion of a first write to the ring buffer. System after Claim 3 , wherein the publisher is further configured to increment the value of the counter after each subsequent operation such that an index of the fixed-size buffer in the ring buffer to which the publisher last delivered a message is determined by a value of the The numerator is divided by 2. System after Claim 1 , where: the lock is acquired by the publisher before a write is performed to one of the plurality of fixed-size buffers; and the lock is released by the publisher after the write is executed. System after Claim 5 , further comprising: a plurality of publishers, each of the plurality of publishers configured to write one or more messages to the plurality of fixed-size buffers in the ring buffer; and wherein the suspension may be acquired by any one of the plurality of publishers at any given time. System after Claim 1 , where each of the plurality of messages has a finite size. System after Claim 1 , where the multiple fixed size buffers do not share cache lines with each other in the shared memory. System after Claim 1 , wherein each of the plurality of buffers includes a buffer header configured to store at least one of: a buffer index number, synchronization information, or buffer state information. System after Claim 9 , where the buffer header occupies a unique cache line in shared memory. System after Claim 10 , wherein each of the plurality of buffers further includes a data area beginning in a cache line of the shared memory immediately after the unique cache line of a corresponding buffer header. System after Claim 1 , wherein the shared memory further comprises a synchronization data storage area corresponding to each of the plurality of fixed size buffers, the synchronization data storage area corresponding to a fixed size buffer comprising a buffer counter indicating the accuracy of a message sent from the buffer fixed size is read by the subscriber. System after Claim 12 , wherein a memory address of the synchronization data storage area is contained in a buffer header of this fixed-size buffer. System after Claim 12 , where: an initial value of the buffer counter is set to 0; and the publisher is configured to increment a value of the buffer counter before executing the write and after executing the write to perform a write to said fixed-size buffer. System after Claim 14 , wherein the subscriber is configured to: compare a first value of the buffer counter before performing a read from that fixed size buffer with a second value of the buffer counter after performing the read from the fixed size buffer; and discarding a message read from the fixed-size buffer if the first value does not match the second value. System after Claim 12 , where a value of the buffer counter during the execution of a The number of times the publisher writes to the fixed-size buffer is odd. System after Claim 16 , where the Subscriber is configured to wait for a write completion notification if the value of the buffer counter is odd before performing a read on the fixed-size buffer. System after Claim 12 , wherein the data synchronization section further comprises a futex system call configured to send a new message notification to the subscriber when a value of the buffer counter is incremented by 2 and is even. System after Claim 12 , wherein the data synchronization section further comprises a futex system call configured to send a wait notification to the subscriber when a value of the buffer counter is either odd or has not been incremented. System after Claim 12 , wherein the data synchronization section further comprises a futex system call configured to send a wait notification to a plurality of publishers in the system when a value of the buffer counter is odd. System after Claim 1 , wherein a size of the plurality of fixed-size zones is determined based on at least one of: maximum message size, maximum number of fixed-size zones, or orientation. A method for enabling message transmission between a publisher and a subscriber, the method comprising, by a processor: Generate in shared storage: - a data structure comprising a ring buffer, the ring buffer comprising: - a plurality of sequentially arranged fixed size buffers configured to store messages, and - a global header comprising a seqlock, the seqlock comprising a seqlock counter providing information relating to a fixed size buffer in the ring buffer to which the publisher will next write a message; and - a synchronization data storage area corresponding to each of the plurality of fixed size buffers, the synchronization data storage area including a buffer counter indicating the accuracy of a message read from the fixed size buffer by the subscriber. Procedure according to Claim 22 , where a negative value of the seqlock counter indicates a state of the system in which the ring buffer contains no messages. Procedure according to Claim 23 , further comprising, by the publisher, assigning a value of 0 to the counter upon completion of a first write to the ring buffer. Procedure according to Claim 24 , further comprising, by the publisher, incrementing the value of the counter after each subsequent operation so that an index of the fixed size buffer in the ring buffer to which the publisher that is most recently a message is determined by dividing a value of the Counter is determined by 2. Procedure according to Claim 22 , where: the lock is acquired by the publisher before a write is performed to one of the plurality of fixed-size buffers; and the lock is released by the publisher after the write is executed. Procedure according to Claim 22 , where the multiple fixed size buffers do not share cache lines with each other in the shared memory. Procedure according to Claim 22 , wherein each of the plurality of buffers includes a buffer header configured to store at least one of: a buffer index number, synchronization information, or buffer state information. Procedure according to Claim 28 , where the buffer header occupies a unique cache line in shared memory. Procedure according to Claim 29 , wherein each of the plurality of buffers further includes a data area beginning in a cache line of the shared memory immediately after the unique cache line of a corresponding buffer header. Procedure according to Claim 22 , wherein a memory address of the synchronization data storage area is contained in a buffer header of this fixed-size buffer. Procedure according to Claim 22 , where: an initial value of the buffer counter is set to 0; and the publisher is configured to perform a write to this fixed-size buffer by specifying a value of the buffer counter before execution of writing and incrementing after executing the write. Procedure according to Claim 32 , further comprising, by the subscriber: comparing a first value of the buffer counter before performing a read from that fixed size buffer with a second value of the buffer counter after performing the read from the fixed size buffer; and discarding a message read from the fixed-size buffer if the first value does not match the second value. Procedure according to Claim 22 , where a value of the buffer counter is odd during the publisher's execution of a write to the fixed-size buffer. Procedure according to Claim 34 , further comprising, by the subscriber, waiting for notification of completion of the write performed when the value of the buffer counter is odd before performing a read on the fixed size buffer. System for sharing information between a publisher and a subscriber, the system comprising: a shared memory; and a storage broker comprising: a processor and a non-transitory computer-readable medium that includes one or more programming instructions that, when executed by the processor, cause the processor to: - Receive, from a Publisher, a request to write a message related to a topic; - determining whether a communication channel corresponding to the topic exists in the shared memory; and - in response to determining that the communication channel corresponding to the topic is present: - Assigning a ring buffer on the communication channel to the publisher, - Sending, to the publisher, information related to the ring buffer, and - Sending, to one or more subscribers of the communication channel, information relating to the ring buffer. System after Claim 36 , further comprising programming instructions that, when executed by the processor, cause the processor, in response to determining that the communication channel corresponding to the topic does not exist, to: generate a new communication channel corresponding to the topic, wherein the new communication channel comprises at least one ring buffer; and transmitting information regarding the new communication channel and the at least one ring buffer to the publisher. System after Claim 37 , further comprising programming instructions that, when executed by the processor, cause the processor to receive a message from a subscriber to subscribe to the topic. System after Claim 37 , where the new communication channel is created based on at least one of the following channel parameters: a maximum size of each message on the new communication channel, a maximum number of messages to a queue, a number of publishers that are allowed to be published on a ring buffer of the new communication channel, a maximum number of ring buffers on the channel, whether ring buffer sharing is allowed, an allowed latency in the new communication channel, types of messages, a channel naming protocol, or an allowed list of publishers that are allowed to publish to the new communication channel. System after Claim 37 , further comprising programming instructions that cause the processor to create a record of the new communication channel in a data memory. System after Claim 36 , wherein the programming instructions that, when executed by the processor, cause the processor to allocate the ring buffer on the communication channel to the publisher include programming instructions for causing the processor to allocate a new ring buffer on the communication channel for allocation to the publisher generate when at least one of the following is determined: - the storage broker does not allow multiple publishers to publish messages to a single ring buffer and all appropriately configured ring buffers on the communication channel are already associated with a live publisher; - the storage broker allows multiple publishers to publish messages to a single ring buffer, but the ring buffers on the channel are not configured appropriately for the publisher's messages; - existing ring buffers of the communication channel are not suitably configured for the publisher's messages; - The channel's ring buffers are in an error state; or - the publisher is not allowed to publish to existing ring buffers of the communication channel. System after Claim 36 , wherein the programming instructions that, when executed by the processor, cause the processor to allocate the ring buffer on the communication channel to the publisher, comprise programming instructions for causing the processor to allocate a pre-existing ring buffer on the communication channel to the publisher, if at least one of the following is determined: the storage broker allows multiple publishers to publish messages to a single ring buffer, and there is a ring buffer on the channel appropriately configured for the publisher's messages; or the storage broker does not allow multiple publishers to publish messages to a single ring buffer and there is at least one appropriately configured ring buffer on the channel that is not already associated with a live publisher. System after Claim 36 , further comprising programming instructions that cause the processor to update a data set associated with the assigned ring buffer in a data memory. System after Claim 36 , further comprising programming instructions that cause the processor to: analyze a state of a plurality of ring buffers in the shared memory; and determining, based on the state, an action with respect to one or more of the plurality of ring buffers, the action comprising at least one of the following: - temporarily removing from use by placing that ring buffer on probation; # - Revising from probation; or - permanently remove it from use. System after Claim 44 , wherein the programming instructions that cause the processor to analyze the state of the plurality of ring buffers include programming instructions for causing the processor to periodically analyze the state. System after Claim 44 , wherein the programming instructions for causing the processor to analyze the state of the plurality of ring buffers include programming instructions for causing the processor to analyze the state upon the occurrence of a triggering event, the triggering event comprising at least one of the following: an unexpected loss of the Connection between the storage broker and the publisher, receiving a message from a peer of the ring buffer indicating a problem with the ring buffer, occurrence of a bad read operation to a ring buffer, occurrence of a bad write operation to a ring buffer, or a storage broker state reconstruction. System after Claim 44 , where a ring buffer is permanently removed from use if the ring buffer is determined to be corrupted. System after Claim 44 , wherein a ring buffer is permanently removed from use if at least one of the following is determined: no live and active publisher associated with the ring buffer; or during memory broker state reconstruction. System after Claim 44 , where: the particular action is temporarily removed from use; and the memory broker further comprises programming instructions that cause the processor to create a record of the ring buffer in a trial list, the record comprising at least one of the following: ring buffer ID, current write position in the ring buffer, whether a write lock is present on the ring buffer, or not, a timestamp of the last write operation, a timestamp when the ring is temporarily removed from use, or a timestamp of the most recent update of the record. System after Claim 49 , further comprising programming instructions to cause the processor to examine the data set to determine whether the ring buffer should be permanently removed from use or revised from probation. System after Claim 36 , further comprising programming instructions for causing the processor to perform a reconstruction of the shared memory after a failure by: temporarily removing a plurality of ring buffers in the shared memory; Connecting from a peer; receiving, from the peer, a request to perform a read/write operation on shared memory; determining whether the peer performs the read/write operation to a ring buffer of the plurality of memory buffers before the failure; and after determining that the peer has the reading /write operation to the ring buffer before the failure to revise the ring buffer for use by the peer. System after Claim 51 , further comprising programming instructions to cause the processor to: determine that the peer is a publisher; and after determining that the peer does not perform the read/write operation prior to the failure to the ring buffer, create a new ring buffer in the shared memory for the peer. A method for sharing information between a publisher and a subscriber, the method comprising, through a storage broker: Receiving, from a Publisher, a request to write a message related to a topic; determining whether a communication channel corresponding to the topic exists in shared memory; and in response to determining that the communication channel corresponding to the topic is in place: - Allocating a ring buffer on the communication channel to the publisher, transmitting to the publisher information relating to the ring buffer, and - Sending, to one or more subscribers of the communication channel, information relating to the ring buffer. Procedure according to Claim 53 , further comprising, in response to determining that the communication channel corresponding to the topic does not exist: creating a new communication channel corresponding to the topic, the new communication channel comprising at least one ring buffer; and transmitting information regarding the new communication channel and the at least one ring buffer to the publisher. Procedure according to Claim 54 , further comprising receiving a message from a subscriber to subscribe to the topic. Procedure according to Claim 54 , where the new communication channel is created based on at least one of the following channel parameters: a maximum size of each message on the new communication channel, a maximum number of messages to a queue, a number of publishers that are allowed to be published on a ring buffer of the new communication channel, a maximum number of ring buffers on the channel, whether ring buffer sharing is allowed, an allowed latency in the new communication channel, types of messages, a channel naming protocol, or an allowed list of publishers that are allowed to publish to the new communication channel. Procedure according to Claim 54 , further comprising creating a record of the new communication channel in a data storage. Procedure according to Claim 53 , wherein allocating the ring buffer on the communications channel to the publisher includes creating a new ring buffer on the communications channel to allocate to the publisher when at least one of the following is determined: the storage broker does not allow multiple publishers to publish messages to a single ring buffer, and all appropriately configured ring buffers on the communication channel are already associated with a live publisher; the storage broker allows multiple publishers to publish messages to a single ring buffer, but the ring buffers on the channel are not configured appropriately for the publisher's messages; existing ring buffers of the communication channel are not suitably configured for the publisher's messages; The channel's ring buffers are in an error state; or the publisher is not allowed to publish to existing ring buffers of the communication channel. Procedure according to Claim 53 , wherein allocating the ring buffer on the communication channel to the publisher includes allocating a pre-existing ring buffer on the communication channel to the publisher in determining at least one of the following: the storage broker allows multiple publishers to publish messages to a single ring buffer, and there are a ring buffer on the channel appropriately configured for the publisher's messages; or the storage broker does not allow multiple publishers to publish messages to a single ring buffer and there is at least one appropriately configured ring buffer on the channel that is not already associated with a live publisher. Procedure according to Claim 53 , further comprising updating a data set associated with the allocated ring buffer in a data store. Procedure according to Claim 53 , further comprising: analyzing a state of a plurality of ring buffers in the shared memory; and Determine, based on the state, an action with respect to one or more of the plurality of ring buffers, the action comprising at least one of the following: - temporarily removing from use by placing that ring buffer on probation; revision from probation; or - permanently remove it from use. Procedure according to Claim 61 , wherein analyzing the state of the plurality of ring buffers includes analyzing the state periodically. Procedure according to Claim 61 , wherein analyzing the state of the plurality of ring buffers includes analyzing the state upon the occurrence of a trigger event, the trigger event comprising at least one of the following: an unexpected loss of connection between the storage broker and the publisher, receiving a message from a peer of the ring buffer, indicating a problem with the ring buffer, the occurrence of a bad read operation to a ring buffer, the occurrence of a bad write operation to a ring buffer, or a memory broker state reconstruction. Procedure according to Claim 61 , where a ring buffer is permanently removed from use if the ring buffer is determined to be corrupted. Procedure according to Claim 61 , wherein a ring buffer is permanently removed from use if at least one of the following is determined: no live and active publisher associated with the ring buffer; or during memory broker state reconstruction. Procedure according to Claim 61 , where: the particular action is temporarily removed from use; and the method further comprises generating a record of the ring buffer in a trial list, the record comprising at least one of the following: ring buffer ID, current write position in the ring buffer, whether or not a write lock is present on the ring buffer, a timestamp of the last write operation , a timestamp when the ring is temporarily removed from use, or a timestamp of the most recent update to the record. Procedure according to Claim 66 , which further includes reviewing the data set to determine whether the ring buffer should be permanently removed from use or revised from probation. Procedure according to Claim 53 , further comprising performing a reconstruction of the shared memory by the memory broker after a failure by: temporarily removing a plurality of ring buffers in the shared memory; Connecting from a peer; receiving, from the peer, a request to perform a read/write operation on shared memory; determining whether the peer performs the read/write operation to a ring buffer of the plurality of memory buffers before the failure; and after determining that the peer completes the read/write operation to the ring buffer before the failure, revise the ring buffer for use by the peer. Procedure according to Claim 53 , further comprising: determining that the peer is a publisher; and after determining that the peer does not perform the read/write operation prior to the failure to the ring buffer, create a new ring buffer in the shared memory for the peer. Messaging system for routing messages between network nodes of a distributed computer system, the messaging system comprising: a plurality of network nodes, each of the network nodes comprising: a shared memory comprising a shared memory area configured to store messages; a publisher; and a first bridge module configured to: Determining, for a shared storage area of that network node, whether a subscriber exists on a remote network node, the remote network node being one of the plurality of network nodes that does not include the publisher, and in response to determining that the Subscriber exists on the remote network node: Reading a plurality of messages from the shared memory area and transmitting the plurality of messages to a second bridge module of the remote network node, the second bridge module being configured to write the plurality of messages to a remote memory area on the remote network node. messaging system Claim 70 , where the remote memory area is the same as the shared memory area. messaging system Claim 70 , where the subscriber is configured to read the plurality of messages from the remote storage area. messaging system Claim 70 , wherein the first bridge module is configured to determine for the shared memory area whether the subscriber exists on the remote network node by: receiving information about at least one of the following in the distributed robot system: one or more communication channels, one or more ring buffers, Network locations of one or more publishers or a network location of one or more subscribers; and using the received information to determine for the shared storage area whether the subscriber exists on the remote network node. messaging system Claim 73 , wherein the first bridge module is configured to receive the information from the second bridge module on the remote network node. messaging system Claim 73 , wherein the first bridge module is configured to receive the information from a data store on the publisher's network node. messaging system Claim 70 , wherein the first bridge module is configured to transmit the plurality of messages to the remote network node using a “zero-copy” network protocol. messaging system Claim 70 , wherein the first bridge module is configured to transmit the plurality of messages to the remote network node using a stream control transmission protocol (SCTP). messaging system Claim 70 , wherein the shared memory area comprises a ring buffer, the ring buffer comprising: a plurality of sequentially arranged fixed size buffers configured to store messages; and a global header that includes a seqlock counter indicating an accuracy of the plurality of messages written to the remote storage area. messaging system Claim 78 , wherein the second bridge module is further configured to replicate the ring buffer in the remote memory area. messaging system Claim 70 , further comprising a plurality of subscribers on the remote network node configured to read the plurality of messages from the remote storage area. A method for routing messages between network nodes of a distributed computer system, the method comprising: through a first bridge module: - Determine, for a shared storage area of a first network node, whether a subscriber exists on a remote network node, the remote network node being one of the plurality of network nodes that does not include a publisher configured to write messages to the shared storage area , and - in response to determining that the Subscriber exists on the remote network node: - Reading a variety of messages from the shared memory area and - transmitting the plurality of messages through a second bridge module of the remote network node; and by the second bridge module, writing the plurality of messages to a remote storage area on the remote network node. Procedure according to Claim 81 , where the remote memory area is the same as the shared memory area. Procedure according to Claim 81 , where the subscriber is configured to read the plurality of messages from the remote storage area. Procedure according to Claim 81 , further comprising, by the first bridge module, determining for the shared memory area whether the subscriber exists on the remote network node by: receiving information about at least one of the following in the distributed robot system: one or more communication channels, one or more ring buffers, network locations from one or more publishers or a network location of one or more subscribers; and using the received information to determine for the shared storage area whether the subscriber exists on the remote network node. Procedure according to Claim 84 , further comprising, received by the first bridge module the information from the second bridge module on the remote network node. Procedure according to Claim 84 , further comprising, by the first bridge module, receiving the information from a data store on the publisher's network node. Procedure according to Claim 81 , further comprising, by the first bridge module, sending the plurality of messages to the remote network node using a “zero-copy” network protocol. Procedure according to Claim 81 , further comprising, by the first bridge module, sending the plurality of messages to the remote network node using a stream control transmission protocol (SCTP). Procedure according to Claim 81 , wherein the shared memory area comprises a ring buffer, the ring buffer comprising: a plurality of sequentially arranged fixed size buffers configured to store messages; and a global header that includes a seqlock counter indicating an accuracy of the plurality of messages written to the remote storage area. Procedure according to Claim 89 , further comprising, by the second bridge module, replicating the ring buffer in the remote memory area. Procedure according to Claim 81 , further comprising, by a plurality of subscribers on the remote network node, reading the plurality of messages from the remote storage area.

Images