GB2345162A - Publish and subscribe data processing with distribution agents configured to support certain streams - Google Patents
Publish and subscribe data processing with distribution agents configured to support certain streams Download PDFInfo
- Publication number
- GB2345162A GB2345162A GB9828683A GB9828683A GB2345162A GB 2345162 A GB2345162 A GB 2345162A GB 9828683 A GB9828683 A GB 9828683A GB 9828683 A GB9828683 A GB 9828683A GB 2345162 A GB2345162 A GB 2345162A
- Authority
- GB
- United Kingdom
- Prior art keywords
- topic
- data processing
- distribution
- distribution agent
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
A publish/subscribe data processing broker apparatus, having: a plurality of distribution agents (brokers) each of which has an input for receiving published messages directly from a publisher application and/or receiving subscription data from a subscriber application; a software unit for receiving at a first distribution agent a data message published on a first topic by a first publisher application; and a software unit for distributing the received published data message to a subscriber application which has requested, by entering subscription data, to receive a message on the first topic; wherein a subset of said plurality of distribution agents which include those that lie in a direct path between the publisher application and the subscriber application are configured to be the only distribution agents of said plurality of distribution agents which support said first topic.
Description
PUBLISH & SUBSCRIBE DATA PROCESSING WITH DISTRIBUTION AGENTS
CONFIGURE TO SUPPORT CERTAIN STREAMS
Field of the Invention
The present invention relates to the field of data processing and more specifically to data processing which distributes messages from suppliers (called, hereinafter,"publishers") of data messages to consumers (called, hereinafter"subscribersn) of such messages.
Background of the invention
Publish/subscribe data processing systems have become very popular in recent years as a way of distributing data messages from publishing computers to subscribing computers. The increasing popularity of the
Internet, which has connected a wide variety of computers all over the world, has helped to make such publish/subscribe systems even more popular. Using the Internet, a World wide web browser application (the term"application"or"process"refers to a software program, or portion thereof, running on a computer) can be used in conjunction with the publisher or subscriber in order to graphically display messages. Such systems are especially useful where data supplied by a publisher is constantly changing and a large number of subscribers needs to be quickly updated with the latest data. Perhaps the best example of where this is useful is in the distribution of stock market data.
In such systems, publisher applications of data messages do not need to know the identity or location of the subscriber applications which will receive the messages. The publishers need only connect to a publish/subscribe distribution agent process, which is included in a group of such processes making up a broker network, and send messages to the distribution agent process, specifying the subject of the message to the distribution agent process. The distribution agent process then distributes the published messages to subscriber applications which have previously indicated to the broker network that they would like to receive data messages on particular subjects. Thus, the subscribers also do not need to know the identity or location of the publishers. The subscribers need only connect to a distribution agent process.
One such publish/subscribe broker network system which is currently in use, and which has been developed by the Transarc Corp. (a wholly owned subsidiary of the assignee of the present patent application, IBM
Corp.) is shown in Fig. 1. Publishers 11 and 12 connect to the publish/subscribe broker network 2 and send published messages to broker 2 which distributes the messages to subscribers 31,32,33,34.
Publishers 11 and 12, which are data processing applications which output data messages, connect to broker network 2 using the well known interapplication data connection protocol known as remote procedure call (or RPC). Each publisher application could be running on a separate machine, alternatively, a single machine could be running a plurality of publisher applications. The broker network 2 is made up of a plurality of distribution agents (21 through 27) which are connected in a hierarchial fashion which will be described below as a"tree structure". These distribution agents, each of which could be running on a separate machine, are data processing applications which distribute data messages through the broker network 2 from publishers to subscribers. Subscriber applications 31,32,33 and 34 connect to the broker network 2 via RPC in order to receive published messages.
Publishers 11 and 12 first connect via RPC directly to a root distribution agent 21 which in turn connects via RPC to second level distribution agents 22 and 23 which in turn connect via RPC to third level distribution agents 24,25,26 and 27 (also known as leaf distribution agents since they are the final distribution agents in the tree structure). Each distribution agent could be running on its own machine, or alternatively, groups of distribution agents could be running on the same machine. The leaf distribution agents connect via RPC to subscriber applications 31 through 34, each of which could be running on its own machine.
In order to allow the broker network 2 to determine which published messages should be sent to which subscribers, publishers provide the root distribution agent 21 with the name of a distribution stream for each published message. A distribution stream (called hereinafter a stream|) is an ordered sequence of messages having a name (e. g., stock"for a stream of stock market quotes) to distinguish the stream from other streams. Likewise, subscribers provide the leaf distribution agents 31 through 34 with the name of the streams to which they would like to subscribe. In this way, the broker network 2 keeps track of which subscribers are interested in which streams so that when publishers publish messages to such streams, the messages can be distributed to the corresponding subscribers. Subscribers are also allowed to provide filter expressions to the broker in order to limit the messages which will be received on a particular stream (e. g., a subscriber 31 interested in only IBM stock quotes could subscribe to the stream nstock by making an RPC call to leaf distribution agent 24 and include a filter expression stating that only messages on the"stock"stream relating to IBM stock should be sent to subscriber 31).
The above-described publish/subscribe broker network architecture provides the advantage of central coordination of all published messages, since all publishers must connect to the same distribution agent (the root) in order to publish a message to the broker hierarchy. For example, total ordering of published messages throughout the broker network hierarchy is greatly facilitated, since the root can easily assign sequence numbers to each published message on a stream. However, this architecture also has the disadvantage of publisher inflexibility, since each publisher is constrained to publishing from the single root distribution agent, even when it would be much easier for a publisher to connect to a closer distribution agent.
Accordingly, publish/subscribe software designers are beginning to consider broker network architectures where publishers are allowed to publish messages directly to any distribution agent in the broker network structure. This clearly has the advantage of removing the abovementioned constraint on publishers. However, as with any tradeoff, it presents other problems. One of the major problems is that since a publisher can publish from any distribution agent, subscription data (data indicating which subscribers have subscribed to which streams/topics) must be propagated throughout the broker network hierarchy, as it cannot be determined from where a publisher on a particular topic/stream will publish from. Alternatively, published message data must be propagated throughout the broker hierarchy, so that any distribution agent that contains subscription data for a subscriber can be guaranteed to be satisfied. Some mixture of these two alternative techniques can also be used. Propagating subscription data throughout the broker (or, alternatively, propagating published message data to each distribution agent in the broker) is the only way to guarantee that published messages, from wherever they may be published, will make their way to the subscribers who have requested the messages. This requirement imposes a great strain on the broker network as it presents a very high data traffic level of either subscription data or published message data throughout the network. Further, in the case where subscription data is propagated throughout the network, the subscription data must also be locally stored and maintained with respect to each distribution agent in the broker network.
Summarv of the Invention
According to one aspect, the present invention provides a publish/subscribe data processing broker apparatus, comprising: a plurality of distribution agents each of which has an input for receiving published messages directly from a publisher application and/or receiving subscription data from a subscriber application; means for receiving at a first distribution agent a data message published on a first topic by a first publisher application; and means for distributing the received published data message to a subscriber application which has requested, by entering subscription data, to receive a message on the first topic ; wherein a subset of said plurality of distribution agents which include those that lie in a direct path between the publisher application and the subscriber application are configured to be the only distribution agents of said plurality of distribution agents which support said first topic.
According to a second aspect, the present invention provides a distribution agent data processing apparatus for use in a publish/subscribe data processing broker network, comprising: means for receiving publication and/or subscription data on a topic from a publisher and/or subscriber application; means for determining whether a neighbouring distribution agent data processing apparatus is configured to support said topic; and means for forwarding the received publication and/or subscription data on the topic to the neighbouring distribution agent data processing apparatus if the means for determining determines that the neighbouring distribution agent data processing apparatus is configured to support said topic.
According to a third aspect, the present invention provides a data processing method having method steps corresponding to each element of the data processing apparatus of the second aspect of the invention.
According to a fourth aspect, the present invention provides a computer readable storage medium having a computer program stored on it which, when executed on a computer, carries out the functionality of data processing method of the second aspect of the invention.
Therefore, the present invention recognizes that there are many inefficiencies associated with a publish/subscribe broker network where either subscription data or published message data must be propagated to each distribution agent in the network. For example, it may be known that certain topics/streams will only ever be subscribed to and published from certain distribution agents, and in these cases it is highly inefficient to propagate information concerning such publications throughout the broker network. A selected subset of the distribution agents in the network is configured (e. g., by a systems administrator setting an appropriate flag within each distribution agent in the subset) to be the only distribution agents which support a topic/stream (or a subset of the total number of defined topics/streams) and publications/subscriptions with respect to this topic will only take place within this selected subset of distribution agents. Accordingly, information (e. g., subscription data or published message data) concerning this topic (or subset of topics) need only be propagated amongst this subset of distribution agents. It thus becomes unnecessary to propagate such information to other distribution agents outside of the subset of distribution agents, greatly reducing the amount of unnecessary network traffic. In the case of propagating subscription data, a great reduction in the amount of data that need be locally maintained is also realized (since subscription data need not be propagated to distribution agents which are outside of the subset of distribution agents).
Brief Description of the Drawings
The invention will be better understood by referring to the detailed description of the preferred embodiments which will now be described in conjunction with the following drawing figures:
Figure 1 shows the architecture of a prior art publish/subscribe broker network which was referred to above; and
Figure 2 shows the architecture of a publish/subscribe broker network according to which the preferred embodiment of the present invention will be explained below.
Detailed Description of the Preferred Embodiments
In the prior art Fig. 1 discussed above, a publisher application 11, running on one computer, is, for example, a supplier of live stock market data quotes. That is, publisher application 11 provides frequent messages stating the present value of share prices. in this example, publisher application 11 is publishing messages on a stream called "stock"which has already been configured in the broker network 2. As is well known, when publisher 11 wishes to publish a stock quote message to stream"stock", publisher 11 makes an RPC call to the root distribution agent 11 which is at the top level of the broker network tree structure.
In this example, subscriber application 32, running on another computer, has sent a subscription request via an RPC call to leaf distribution agent 24, which is at the bottom level of the tree structure, indicating that subscriber 32 would like to subscribe to stream nstockn.
Thus, whenever publisher 11 publishes a data message to stream "stock"the distribution tree structure of broker network 2 channels the message down through the root distribution agent 21, through any intermediary distribution agents (e. g., 22 in the example of Fig. 1) and through the leaf distribution agent 24 to the subscriber 32. This involves a series of RPC calls being made between each successive circle in the diagram of Fig. 1 connecting publisher 11 and subscriber 32 (i. e., 11 to 21,21 to 22,22 to 24 and 24 to 32).
Figure 2 shows a different publish/subscribe broker network architecture where publisher applications can publish messages to the broker network by directly communicating with any one of a plurality of distribution agents (also referred to herein as"brokersn). For example, publisher application 201 is shown communicating directly with Broker 12.
There is no requirement in this architecture that all publisher applications communicate directly with a top (or root) distribution agent. Publisher application 201 can potentially communicate directly with any of the distribution agents shown in Fig 2, in the described examples below, it will be shown communicating directly with Broker 12.
Subscriber applications 202 and 203 would like to receive messages on the stream/topic that publisher application 201 is publishing on.
Thus, subscriber applications 202 and 203 communicate directly with
Brokers 1112 and 1221, respectively, to provide subscription data thereto informing the broker network hierarchy of their desire to receive such published messages. Since the publisher application 201 is allowed to communicate directly with any of a plurality of distribution agents, the subscription data entered by the subscriber applications must be propagated throughout the broker hierarchy to each broker shown in Fig.
2. Alternatively, a message published by publisher 201 via broker 12 would need to be sent to each and every broker in the network. This way, no matter which distribution agent the publisher application 201 happens to communicate directly with, the published messages will be able to be routed to the subscriber applications 202 and 203. As stated above, however, this creates a high performance overhead due to the excessive amounts of subscription/publication data propagation traffic throughout the broker network and due to the need to have to maintain and store such subscription data locally at each broker.
we assume in this example that publisher application 201 is publishing on a stream called"stock stream and the messages being published concern the current share prices being currently traded on a particular stock exchange. we also assume that subscriber applications 202 and 203 are computer programs being run by stock brokers who are interested in receiving the published messages on"stock stream". In this example, according to a preferred embodiment of the present invention, brokers 12,122 and 1221 are defined as members of a subset of brokers which support the stream called"stock stream", since these are the brokers that are known in advance to lie between publisher 201 and subscriber 203. Likewise, brokers 12,1,11,111 and 1112 are also defined as members of this same subset of brokers which support the stream called"stock stream, since these are the brokers that are known in advance to lie between publisher 201 and subscriber 202. The other brokers (e. g., broker 1111) shown in Fig. 2 are not included in this subset of brokers. The brokers that are included in the subset of brokers are, for example, marked as being members of the subset by setting a flag in local memory for each broker of the subset.
Accordingly, subscription/publication data concerning"stock stream"need only be propagated within this subset of brokers. That is, if the method of matching subscriptions to publications is to propagate subscription data around the broker network, a subscription is only propagated to brokers that are known to support the relevant stream.
Similarly, if the method is to propagate all publications, a publication is only propagated to brokers that support the relevant stream. For example, should subscriber 22 decide to cancel its subscription to stock stream", the cancellation data need only be propagated to brokers which are members of the defined subset.
Information about which streams a broker supports is made available to that broker's neighbours, so that the neighbours will know whether or not to forward propagating publication/subscription data to that broker.
For example, broker 111 has previously informed its neighbours (brokers 1111,1112 and 11) that broker 111 supports stream"stock stream".
Accordingly, when publication/subscription data on this stream arrives at one of the neighbours, the neighbour will know that it should further propagate the publication/subscription data to broker 111. One method for performing such previous informing operation is for the broker 111 (via, for example, a systems administrator) to publish a message on a special administration stream to the neighbours of broker 111 to inform the neighbours that broker 111 supports the stream stock stream".
Alternatively, upon receiving publication/subscription data, a neighbour could examine the flags of its neighbours in order to determine whether to forward on the publication/subscription data for a stream/topic.
Further, a combination of these two techniques could also be used.
In addition to the preferred embodiment where brokers are preconfigured to state that they support a particular stream (s), it is also possible for a broker which is meant to be part of a subset of brokers to configure itself to support a particular stream dynamically (or"on the fly"). Specifically, a broker receives publication or subscription data for a stream that the broker does not currently support and then configures itself dynamically to support that stream. Such a broker effectively supports all possible streams. of course, brokers which are not meant to support a particular topic/stream should not be provided with this dynamic configuration capability.
While the preferred embodiment of the invention has been discussed in the context of a broker network made up of a hierarchy (e. g., designed from the top down) of distribution agents, the broker network need not be hierarchial. For example, the network could also be configured as a totally connected network, with each broker connected to every other broker (or some other combination of brokers less than every other broker).
Claims (7)
- CLAIMS 1. A publish/subscribe data processing broker apparatus, comprising: a plurality of distribution agents each of which has an input for receiving published messages directly from a publisher application and/or receiving subscription data from a subscriber application; means for receiving at a first distribution agent a data message published on a first topic by a first publisher application; and means for distributing the received published data message to a subscriber application which has requested, by entering subscription data, to receive a message on the first topic; wherein a subset of said plurality of distribution agents which include those that lie in a direct path between the publisher application and the subscriber application are configured to be the only distribution agents of said plurality of distribution agents which support said first topic.
- 2. The apparatus of claim 1 wherein the distribution agents of said subset configure themselves dynamically upon receiving data concerning the topic.
- 3. The apparatus of claim 1 wherein a first distribution agent of said subset includes means for informing neighbouring distribution agents of the topics which the first distribution agent supports.
- 4. The apparatus of claim 1 wherein when a new subscription is entered subscription data is propagated between distribution agents.
- 5. A distribution agent data processing apparatus for use in a publish/subscribe data processing broker network, comprising: means for receiving publication and/or subscription data on a topic from a publisher and/or subscriber application; means for determining whether a neighbouring distribution agent data processing apparatus is configured to support said topic; and means for forwarding the received publication and/or subscription data on the topic to the neighbouring distribution agent data processing apparatus if the means for determining determines that the neighbouring distribution agent data processing apparatus is configured to support said topic.
- 6. A distribution agent data processing method for use in a publish/subscribe data processing broker network, comprising steps of: receiving publication and/or subscription data on a topic from a publisher and/or subscriber application; determining whether a neighbouring distribution agent data processing apparatus is configured to support said topic; and forwarding the received publication and/or subscription data on the topic to the neighbouring distribution agent data processing apparatus if the step of determining determines that the neighbouring distribution agent data processing apparatus is configured to support said topic.
- 7. A computer program product stored on a computer readable storage medium for, when run on a computer, carrying out a distribution agent data processing method for use in a publish/subscribe data processing broker network, comprising steps of: receiving publication and/or subscription data on a topic from a publisher and/or subscriber application; determining whether a neighbouring distribution agent data processing apparatus is configured to support said topic; and forwarding the received publication and/or subscription data on the topic to the neighbouring distribution agent data processing apparatus if the step of determining determines that the neighbouring distribution agent data processing apparatus is configured to support said topic.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| GB9828683A GB2345162A (en) | 1998-12-24 | 1998-12-24 | Publish and subscribe data processing with distribution agents configured to support certain streams |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| GB9828683A GB2345162A (en) | 1998-12-24 | 1998-12-24 | Publish and subscribe data processing with distribution agents configured to support certain streams |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| GB9828683D0 GB9828683D0 (en) | 1999-02-17 |
| GB2345162A true GB2345162A (en) | 2000-06-28 |
Family
ID=10845062
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| GB9828683A Withdrawn GB2345162A (en) | 1998-12-24 | 1998-12-24 | Publish and subscribe data processing with distribution agents configured to support certain streams |
Country Status (1)
| Country | Link |
|---|---|
| GB (1) | GB2345162A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7349980B1 (en) * | 2003-01-24 | 2008-03-25 | Blue Titan Software, Inc. | Network publish/subscribe system incorporating Web services network routing architecture |
| US7529805B2 (en) | 2001-11-21 | 2009-05-05 | Blue Titan Software, Inc. | Distributed web services network architecture |
| US7853643B1 (en) | 2001-11-21 | 2010-12-14 | Blue Titan Software, Inc. | Web services-based computing resource lifecycle management |
| CN105260479A (en) * | 2015-11-10 | 2016-01-20 | 南京国电南自美卓控制系统有限公司 | observer pattern-based real-time database real time data publishing method |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW202404318A (en) * | 2018-08-23 | 2024-01-16 | 美商阿爾克斯股份有限公司 | System for first hop gateway redundancy in a network computing environment using arp |
-
1998
- 1998-12-24 GB GB9828683A patent/GB2345162A/en not_active Withdrawn
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7529805B2 (en) | 2001-11-21 | 2009-05-05 | Blue Titan Software, Inc. | Distributed web services network architecture |
| US7853643B1 (en) | 2001-11-21 | 2010-12-14 | Blue Titan Software, Inc. | Web services-based computing resource lifecycle management |
| US8255485B2 (en) | 2001-11-21 | 2012-08-28 | Blue Titan Software, Inc. | Web services-based computing resource lifecycle management |
| US7349980B1 (en) * | 2003-01-24 | 2008-03-25 | Blue Titan Software, Inc. | Network publish/subscribe system incorporating Web services network routing architecture |
| US7788403B2 (en) * | 2003-01-24 | 2010-08-31 | Soa Software, Inc. | Network publish/subscribe incorporating web services network routing architecture |
| CN105260479A (en) * | 2015-11-10 | 2016-01-20 | 南京国电南自美卓控制系统有限公司 | observer pattern-based real-time database real time data publishing method |
| CN105260479B (en) * | 2015-11-10 | 2019-03-19 | 南京国电南自维美德自动化有限公司 | A kind of real-time data base real-time data warehouse method based on Observer Pattern |
Also Published As
| Publication number | Publication date |
|---|---|
| GB9828683D0 (en) | 1999-02-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6334151B1 (en) | Publish and subscribe data processing apparatus, method and computer program product with declaration of a unique publisher broker | |
| US6202093B1 (en) | Publish and subscribe data processing with ability to specify a local publication/subscription | |
| US6154781A (en) | Publish and subscribe data processing with subscriber option to request subscription propagation prior to acknowledgement | |
| US6298455B1 (en) | Publish and subscribe data processing with failover using cascaded sequence numbers | |
| US6182143B1 (en) | Publish and subscribe data processing apparatus, method and computer program product with use of a stream to distribute local information between neighbors in a broker structure | |
| US6643682B1 (en) | Publish/subscribe data processing with subscription points for customized message processing | |
| US7103680B1 (en) | Publish/subscribe data processing with publication points for customized message processing | |
| CA2642823C (en) | Managing rich presence collections | |
| US8849754B2 (en) | Managing topical overlap during publication and subscription | |
| US6510429B1 (en) | Message broker apparatus, method and computer program product | |
| US20080307436A1 (en) | Distributed publish-subscribe event system with routing of published events according to routing tables updated during a subscription process | |
| CA2644124C (en) | Managing rich presence collections | |
| US8495127B2 (en) | Improving scalability and throughput of a publish/subscribe network | |
| JP4588704B2 (en) | Self-management mediation information flow | |
| US20070027702A1 (en) | Organizing presence information into collections of publications | |
| EP0961452A2 (en) | Publish & subscribe data processing apparatus, method and computer program product with use of a stream to distribute administrative and configuration information | |
| GB2345162A (en) | Publish and subscribe data processing with distribution agents configured to support certain streams | |
| JP2004506272A (en) | A system that processes raw financial data and generates validated product guidance information for subscribers | |
| GB2354349A (en) | Event notification data processing with command and command notification combined into a single event | |
| KR19980036951A (en) | Multi Server / Client Device | |
| Tambe | State-of-the-art in publish/subscribe middleware for supporting mobility | |
| KR20040039288A (en) | Packet routing via payload inspection and subscription processing in a publish-subscribe network | |
| Spetka et al. | Information management for high performance autonomous intelligent systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| WAP | Application withdrawn, taken to be withdrawn or refused ** after publication under section 16(1) |