CN113992741B - Method and device for indexing release data - Google Patents
Method and device for indexing release data Download PDFInfo
- Publication number
- CN113992741B CN113992741B CN202010663711.2A CN202010663711A CN113992741B CN 113992741 B CN113992741 B CN 113992741B CN 202010663711 A CN202010663711 A CN 202010663711A CN 113992741 B CN113992741 B CN 113992741B
- Authority
- CN
- China
- Prior art keywords
- subscription
- node
- target
- proxy
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The embodiment of the application provides a method for issuing data index, which comprises the following steps: receiving release data; acquiring a subscription index, wherein the subscription index comprises a target subscription rule and identifications of M proxy nodes corresponding to the target subscription rule, and M is a positive integer; wherein the target subscription rule is contained in a plurality of subscription requests and only one of the target subscription rules is included in the subscription index; and sending the release data to M proxy nodes corresponding to the identifications of the M proxy nodes based on the release data meeting the target subscription rule, so that the M proxy nodes generate subscription messages based on the release data. According to the method and the device, the subscription index comprising the subscription rule is constructed, the proxy node which needs to send the published data can be determined only by one-step search, the query times of the subscription rule are greatly reduced, and the query cost is reduced.
Description
Technical Field
The present disclosure relates to the field of computers, and in particular, to a method and apparatus for indexing published data.
Background
Publish/subscribe is a paradigm of messages that a sender of a published message (called a publisher) does not send the published message directly to a particular recipient (called a subscriber). Rather, published messages are divided into different categories without knowing which subscribers (if any) may be present. Likewise, a subscriber may express interest in one or more categories, receive only messages of interest, and need not know which publishers (if any) are present.
In a publish-subscribe model, subscribers typically receive only a subset of all published messages that have been published. The process of selecting messages for receipt and processing is known as filtering. Publishers are responsible for defining topics to which subscribers can subscribe. In content-based systems, a message is delivered to a subscriber only if the attributes or content of the message matches a subscriber-defined constraint. Subscribers are responsible for categorizing messages.
In the prior art, a subscriber sends subscription rules to a server corresponding to a publisher through a client, and the publisher can trigger query operation on all subscription rules of the subscriber every time the publisher publishes a message, wherein the query times are proportional to the number of subscribers and the number of published messages. Thus, when the number of subscribers and the number of published messages is large, a huge query overhead is introduced.
Disclosure of Invention
In a first aspect, an embodiment of the present application provides a method for publishing data index, where the method includes: receiving release data; acquiring a subscription index, wherein the subscription index comprises a target subscription rule and identifications of M proxy nodes corresponding to the target subscription rule, and M is a positive integer; wherein the target subscription rule is contained in a plurality of subscription requests and only one of the target subscription rules is included in the subscription index; and sending the release data to M proxy nodes corresponding to the identifications of the M proxy nodes based on the release data meeting the target subscription rule, so that the M proxy nodes generate subscription messages based on the release data.
According to the method and the device, the subscription index comprising the subscription rule is constructed, the subscription rule and the proxy node are associated, the proxy node for publishing the data index is selected by using the published data retrieval rule index, the same subscription rules included in a plurality of subscription requests from the client are combined, the proxy node for transmitting the published data can be determined by only one-step retrieval, the query times of the subscription rule are greatly reduced, and the query cost is reduced.
In one possible implementation, the sending the published data to the M proxy nodes includes: and sending the release data to the M proxy nodes in a multicast mode.
In one possible implementation, the M proxy nodes include at least one first node and at least one second node, and the sending the published data to the M proxy nodes includes: transmitting the release data to the at least one first node in a multicast mode; and sending the release data to the at least one second node in a point-to-point mode.
In one possible implementation, the at least one first node pushes a plurality of subscription messages to the client in a preset time period, and the subscription messages pushed to the client in a first subscription state in the plurality of subscription messages are larger than a preset proportion or a preset number of times, and the first subscription state indicates that the client is in a subscription state based on the target subscription rule.
In one possible implementation, the at least one second node pushes a plurality of subscription messages to the client in a preset time period, and the subscription messages pushed to the client in a second subscription state in the plurality of subscription messages are smaller than or equal to a preset proportion or a preset number of times, and the second subscription state indicates that the client cancels the subscription state based on the target subscription rule.
The embodiment of the application provides a multicast group election method with simplified calculation based on a multicast group election mechanism with stable proxy node combination weight, comprising proxy node stability estimation, stable proxy node combination calculation and weight estimation, and a message distribution mechanism with multicast and point-to-point combination by selecting through weight aggregation sequencing, thereby enabling multicast capability to reduce message sending times.
In one possible implementation, the target subscription rule includes a key value interval, and the published data includes a target key value, where the target key value belongs to the key value interval.
In one possible implementation, the target subscription rule further comprises a subscription message format; the subscription message is generated by the M proxy nodes based on the subscription message format.
In one possible implementation, the plurality of subscription message formats includes a first subscription message format and a second subscription message format, and at least one proxy node corresponding to the first subscription message format and at least one proxy node corresponding to the second subscription message format each include a target node; the sending the published data to the M proxy nodes includes: sending the release data to at least one proxy node corresponding to the first subscription message format; sending the release data to at least one proxy node corresponding to the second subscription message format; wherein the published data is sent only once to the target node.
In the embodiment of the application, when different subscription rules of the same proxy node include different subscription message formats, the proxy node is combined and pushed to release data. And the messages which need to be sent for multiple times are combined into one transmission, so that the number of times of sending the release data among the proxy nodes is reduced.
In one possible implementation, the sending the published data to the M proxy nodes includes: and determining that each of the M proxy nodes continuously receives release data of preset times within preset time, and sending the release data to the M proxy nodes.
In one possible implementation, the sending the published data to the M proxy nodes includes: and sending the release data to the M proxy nodes based on the time interval between the last data sending to the M proxy nodes is larger than a preset value.
In the embodiment of the application, the M proxy nodes may also acquire the frequency of the latest push, and if the push is too fast, increase the cooling delay before pushing. And combining the trigger events in the time delay interval, and triggering the inquiry only once. Frequent pushing of subscription rules for hotspots; the embodiment can reduce the pushing times by controlling the time delay.
In a second aspect, the present application provides a published data indexing device, the device comprising: the receiving module is used for receiving the release data; the acquisition module is used for acquiring a subscription index, wherein the subscription index comprises a target subscription rule and identifications of M proxy nodes corresponding to the target subscription rule, and M is a positive integer; wherein the target subscription rule is contained in a plurality of subscription requests and only one of the target subscription rules is included in the subscription index; and the sending module is used for sending the release data to M proxy nodes corresponding to the identifications of the M proxy nodes based on the release data meeting the target subscription rule, so that the M proxy nodes generate subscription information based on the release data.
In one possible implementation, only one of the target subscription rules is included in the subscription index.
In one possible implementation, the sending module is configured to send the published data to the M proxy nodes by multicast.
In a possible implementation, the M proxy nodes include at least one first node and at least one second node, and the sending module is configured to send the published data to the at least one first node by multicast;
and sending the release data to the at least one second node in a point-to-point mode.
In one possible implementation, the at least one first node pushes a plurality of subscription messages to the client in a preset time period, and the subscription messages pushed to the client in a first subscription state in the plurality of subscription messages are larger than a preset proportion or a preset number of times, and the first subscription state indicates that the client is in a subscription state based on the target subscription rule.
In one possible implementation, the at least one second node pushes a plurality of subscription messages to the client in a preset time period, and the subscription messages pushed to the client in a second subscription state in the plurality of subscription messages are smaller than or equal to a preset proportion or a preset number of times, and the second subscription state indicates that the client cancels the subscription state based on the target subscription rule.
In one possible implementation, the target subscription rule includes a key value interval, and the published data includes a target key value, where the target key value belongs to the key value interval.
In one possible implementation, the target subscription rule further comprises a subscription message format; the subscription message is generated by the M proxy nodes based on the subscription message format.
In one possible implementation, the plurality of subscription message formats includes a first subscription message format and a second subscription message format, and at least one proxy node corresponding to the first subscription message format and at least one proxy node corresponding to the second subscription message format each include a target node; the sending module is configured to send the published data to at least one proxy node corresponding to the first subscription message format;
sending the release data to at least one proxy node corresponding to the second subscription message format; wherein the published data is sent only once to the target node.
In one possible implementation, the sending the published data to the M proxy nodes includes: and determining that each of the M proxy nodes continuously receives release data of preset times within preset time, and sending the release data to the M proxy nodes.
In one possible implementation, the sending the published data to the M proxy nodes includes: and sending the release data to the M proxy nodes based on the time interval between the last data sending to the M proxy nodes is larger than a preset value.
In a third aspect, the present application provides an issue data indexing apparatus, which may comprise a processor, the processor being coupled to a memory, the memory storing program instructions which, when executed by the processor, implement the method of any of the first aspects and above. For the steps executed by the processor in each possible implementation manner of the first aspect, reference may be specifically made to the first aspect, which is not described herein.
In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored therein, which when run on a computer causes the computer to perform the method of any of the first aspects described above.
In a fifth aspect, the present application provides circuitry comprising processing circuitry configured to perform the method of the first aspect and any one of the above.
In a sixth aspect, the present application provides a computer program product which, when run on a computer, causes the computer to perform the method of any one of the first aspects described above.
Drawings
FIG. 1 is a schematic architecture diagram of a system provided in an embodiment of the present application;
FIG. 2 is a schematic architecture of a system according to an embodiment of the present application;
FIG. 3 is a flowchart of a method for publishing data index according to an embodiment of the present disclosure;
FIG. 4a is a flowchart illustrating a subscription index construction according to an embodiment of the present application;
FIG. 4b is a flowchart illustrating subscription index distribution between proxy nodes according to an embodiment of the present application;
FIG. 4c is a flowchart illustrating a subscription index query provided in an embodiment of the present application;
fig. 4d is a flowchart illustrating data distribution between proxy nodes according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a device for indexing published data according to an embodiment of the present application;
FIG. 6 is a schematic representation of a computer readable storage medium provided herein;
fig. 7 is a schematic structural diagram of a server provided in the present application.
Detailed Description
Embodiments of the present application will now be described with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some, but not all embodiments of the present application. As one of ordinary skill in the art can appreciate, with the development of technology and the appearance of new scenes, the technical solutions provided in the embodiments of the present application are applicable to similar technical problems.
The terms first, second and the like in the description and in the claims of the present application and in the above-described figures, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or modules is not necessarily limited to those steps or modules that are expressly listed or inherent to such process, method, article, or apparatus. The naming or numbering of the steps in the present application does not mean that the steps in the method flow must be executed according to the time or logic sequence indicated by the naming or numbering, and the execution sequence of the steps in the flow that are named or numbered may be changed according to the technical purpose to be achieved, so long as the same or similar technical effects can be achieved.
FIG. 1 is a schematic diagram of a subscription-publishing system provided by an embodiment of the present application, including client clients (100 a, 100b, 100c, 100d, 100e, and 100 f), proxy node proxy nodes (200 a, 200b, and 200 c), and database, as shown in FIG. 1; the client is a publisher subscribing the request or publishing the data; the proxy node is a service node of a message publish/subscribe service. Multiple proxy nodes may form a service cluster, each proxy node being connected to a portion of clients, as shown in FIG. 1, client100a, client100b being connected to proxy node 200a, client100c, client100d being connected to proxy node 200b, client100e, client100f being connected to proxy node 200 c; database is used to store published data. Publishers can write published data to databases and subscribers can subscribe to published data using conditional queries.
Referring to fig. 2, fig. 2 is an architecture schematic of a system provided in an embodiment of the present application, and as shown in fig. 2, a server 2000 may include a proxy node 2100, and the proxy node (or simply "broker") 2100 may include a publish-subscribe service 2120, a data storage proxy 2130 and a communication proxy 2140. The publish-subscribe service 2120 includes a subscribe service 2121 and a publish service 1021; the subscription service 2121 may provide a subscription service for a client where a subscriber is located (receive a subscription request of the client and push a subscription message to the client based on the subscription request), and the publishing service may provide a publishing service for a client where a publisher is located (receive publishing data sent by the client and push appropriate publishing data to the client where the subscriber is located based on the subscription request sent by the client where the subscriber is located); the specific process may be implemented based on rule replication 2121, rule index 2122, rule execution 2123, and message distribution 2124. Proxy node 2100 may also include a data storage proxy 2130 and a communication 2140, where communication 2140 may communicate with database 1001 in server cluster 1000 to obtain data from database 1001 or write data to database 1001, and communication 2140 may also communicate with server/terminal device 3000 to receive published data sent by clients, push appropriate published data to clients where subscribers are located, or receive subscription requests of clients and push subscription messages to clients based on subscription requests. The server/terminal device 3000 may include a client3100 of a subscriber or a client3100 of a publisher, wherein the client3100 may include a subscription proxy 3111 and a publication proxy 3111, the client3100 may further include a communication 3120, and the communication 3120 may be in communication with the proxy node 2100.
The embodiment of the application provides a published data indexing method which is used for solving the problem of overlarge query cost when published data is pushed. Referring to fig. 3, fig. 3 is a flowchart of a method for indexing published data according to an embodiment of the present application, and as shown in fig. 3, the method for indexing published data according to the embodiment of the present application includes:
301. the proxy node receives the published data.
In this embodiment of the present application, an agent node in the server may receive release data uploaded by a publisher, where the release data may include a target key value, for example, the publisher may be a merchant, the release data may be product preference information, the target key value may be a product category, and a price of the product, etc.
302. The proxy node acquires a subscription index, wherein the subscription index comprises a target subscription rule and identifications of M proxy nodes corresponding to the target subscription rule, and M is a positive integer; wherein the target subscription rule is contained in a plurality of subscription requests and only a single one of the target subscription rules is included in the subscription index.
In this embodiment, only one target subscription rule is included in the subscription index.
In the embodiment of the present application, after receiving the published data, the proxy node may acquire a subscription index from the database, and determine the target subscription rule of the published data from the subscription index.
In the embodiment of the present application, a subscription client as a consumer may issue a subscription request (including a subscription rule) to a proxy node in a proxy node cluster, where the subscription rule may include, for example, a push data interval of interest (abbreviated as "key value interval") and a subscription message format.
After receiving the subscription request, the proxy nodes in the proxy node cluster can acquire the subscription rules contained in the subscription request.
In one implementation, the obtained subscription rules may be shared among the proxy nodes in the proxy node cluster, the proxy nodes in the proxy node cluster may analyze the subscription rules, merge the same subscription rules into one, and record only one subscription rule (for example, may be a target subscription rule) in the subscription index for the same subscription rule when the subscription index is established, since each subscription rule may correspond to one proxy node (the proxy node receives a subscription request of a client having a connection relationship with the proxy node, and there is a correspondence relationship between the proxy node and the subscription rule obtained from the subscription request), the target subscription rule may correspond to the identities of M proxy nodes.
In one implementation, the proxy node cluster may include a proxy node dedicated to constructing the subscription index, where the proxy node may obtain the subscription rule sent by the proxy node in the proxy node cluster (or the subscription request, where the proxy node dedicated to constructing the subscription index may itself complete the step of obtaining the subscription rule from the subscription request), where the proxy node in the proxy node cluster dedicated to constructing the subscription index may analyze the subscription rule, merge the same subscription rule into one, and record only one subscription rule in the subscription index for the same subscription rule when the subscription index is constructed. The proxy node dedicated to building the subscription index may then distribute the built subscription index to other nodes in the proxy node cluster.
The above method of constructing a subscription index is merely illustrative, and may be performed by any one of the proxy nodes in the proxy node cluster, which is not limited herein.
Referring to fig. 4a, fig. 4a is a schematic flow chart of subscription index construction according to an embodiment of the present application. For example, a certain online shopping platform may provide for collecting product offers of various products of various merchants, and provide the consumer with product offers of interest to the consumer via various dimensions on a conditional basis via a client of the terminal device. The online shopping platform software is designed with a product preferential information table (table name is discover), and records in the table comprise a plurality of key values, such as: product name, product model, product category, brand, merchant, discount, price, etc. The merchant issues preferential information to the shopping platform and inserts data into the table; the consumer may subscribe to the offer information through the subscription client, and specifically, the subscription client may send a subscription request (i.e., the subscription offer information described above) to the proxy node, and correspondingly, the proxy node may receive the subscription request.
For example, a consumer a may wish to purchase a HUAWEI Mate series phone, subscribe to Mate related preference information from a shopping platform, and may describe the filtering criteria (alternatively referred to as subscription rules) of the subscription using the following subscription request (referred to as query statement in fig. 4 a).
SELECT product name, discount, price, merchant FROM discount WHERE product name= "HUAWEI Mate".
For example, a consumer B wants to buy a mobile phone with a price of 5000-6000 RMB and subscribes relevant preferential information to a shopping platform, and the following subscription request can be used for describing the subscription rule of subscription.
SELECT product name, discount, price, merchant FROM discount WHERE product category= "Phone" AND price bean heat 5000AND 6000.
In the embodiment of the application, the subscription rule can be subjected to key value interval analysis, and further the key value interval is extracted from the subscription rule. In one implementation, a query statement is converted into multiple ANDs clause format connected in OR. Illustrating:
before conversion: SELECT product name, discount, price, merchant FROM discount WHERE (product name= "HUAWEI Mate 30" or product name= "HUAWEI P40") AND price <5000;
after conversion: SELECT product name, discount, price, merchant FROM discount WHERE (product name= "HUAWEI Mate 30" AND price < 5000) OR (product name= "HUAWEI P40" AND price < 5000).
In this embodiment of the present application, one ANDS clause in the OR linked list may be obtained iteratively, and whether there is an index key in the clause is determined, if yes, the { key: value interval } add to the output list. Wherein { key: the method of value interval } can be referred to as an example as follows:
1) Assuming that the subscription index has been created with the key "product name", the { key is obtained from "product name=" HUAWEI Mate "AND price < 5000: value interval } is { product name: [ "HUAWEI Mate", "HUAWEI Mate" ] }.
2) Assuming that the subscription index has been created by the key "price", { key: value interval } is { price: [4000,5000]}.
In the embodiment of the present application, the mapping relationship between the key value interval and the proxy node may be pushed to all other proxy nodes. Thereby enabling the proxy nodes to generate subscription indexes;
for example, referring to fig. 4b, fig. 4b is a flowchart illustrating subscription index distribution between proxy nodes provided in the embodiment of the present application, where, for a new proxy node to start joining a cluster, subscription queue change information sent by other proxy nodes may be received, a snapshot is obtained from the other proxy nodes, and the change information and the snapshot are combined; and then, on one hand, the subscription team change information sent by other proxy nodes can be received, the subscription rule change information is applied, and on the other hand, the subscription request of the client connected with the client can be received, the key value interval is analyzed, and the subscription change information (comprising the key value interval) is sent to the other proxy nodes.
By way of example, the subscription index in embodiments of the present application may include subscription rules and subscription message formats, e.g., the subscription rules may be described by the following message formats:
index name: mapping table names, key names and subscription uses, for example: idx_sub_discover_price.
Index key value: key value intervals such as: [4000,4000],[4000,5000].
Index data: the main information includes: a plurality of query sentences (queries) corresponding to key-value intervals, one query sentence including at least one filter and a plurality of output formats (output formats), each output format corresponding to a class of subscriptions, one class of subscriptions involving a plurality of proxy nodes, wherein a part of proxy nodes (stable proxy nodes) stably focus on such subscriptions, a multicast group (multicast IP) may be used to distribute push messages to improve efficiency, and other proxy nodes (unstable un-stable proxy nodes do not normally focus on such subscriptions, and point-to-point distribute push messages, wherein the stable proxy nodes (hereinafter referred to as first nodes in embodiments) may be understood as: in a preset period before the at least one first node sends the publishing data in a multicast manner, the at least one first node pushes multiple subscription messages to clients, wherein the subscription messages with the proportion larger than the preset proportion in the multiple subscription messages are pushed to the clients maintaining a first subscription state by the at least one first node, and the first subscription state is the subscription state of the target subscription rule. And subscription messages with the proportion smaller than or equal to a preset proportion among the subscription messages are pushed to the client side canceling the first subscription state by the at least one second node. At least one first node may constitute a stable push combination, which refers to a class of clients subscribed to a connection with multiple proxy nodes that often pay attention to using multicast components for a period of time to send push messages more efficiently than point-to-point distribution messages.
In one implementation, each of the M proxy nodes continuously receives the published data a preset number of times within a preset time before the published data is transmitted to the M proxy nodes. Taking the example that the subscription rule includes a key value interval, the key value interval maps a group of proxy nodes, and with the issuing and cancellation of the subscription rule, the proxy nodes may be newly added into the proxy nodes and/or old proxy nodes may be deleted.
303. Based on the publishing data meeting the target subscription rule, sending the publishing data to M proxy nodes corresponding to the identifications of the M proxy nodes, so that the M proxy nodes generate subscription messages based on the publishing data;
in this embodiment of the present application, the subscription rule may include a key value interval, and the published data includes a target key value, where the target key value belongs to the key value interval included in the target subscription rule. That is, after the proxy node receives the published data, the proxy node can search in which key value interval the subscription index is seen in based on the target key value in the published data, and if the target key value is determined to be in the key value interval included in the target subscription rule, the M proxy nodes corresponding to the target subscription rule can be determined.
In this embodiment of the present application, the identifier of M proxy nodes represents an identifier of each proxy node in M proxy nodes, where the identifier may indicate an address where the proxy node is located, and the proxy node may send the published data based on the identifiers of other proxy nodes.
In this embodiment of the present application, the publishing data may be sent to the M proxy nodes, so that the M proxy nodes push a subscription message to a client, where the subscription message is generated based on the publishing data.
In this embodiment of the present application, the published data may be sent to the M proxy nodes by multicast.
Referring to fig. 4c, fig. 4c is a schematic flow chart of a subscription index query according to an embodiment of the present application.
The proxy node may receive the publishing information, obtain a subscription list (subscription index), iteratively obtain a key of the list and its subscription information index, search a key interval index using a key value of the publishing information to obtain a key value interval list, iteratively obtain an interval in the key value interval list, iteratively obtain a query statement query related to the key value interval, trigger overload based on the query, add a delay trigger scheduling queue, trigger not overload based on the query, perform query, convert a query result according to a formatting list, and distribute the formatting result to the client.
In this embodiment of the present application, the M proxy nodes include at least one first node and at least one second node, and the server may send the published data to the at least one first node by using a multicast manner; and sending the release data to the at least one second node in a point-to-point mode. In this embodiment of the present application, the weights may be summed and sequenced according to the combined packet for all push combinations and weights, and a combination with a high weight is selected to create a multicast group (including the at least one first node), and the proxy node outside the multicast group is the at least one second node.
In this embodiment of the present application, a stable combining weight is calculated, where the stable combining weight is used to measure the benefit generated after using the combination as multicast, and may be calculated by the following method: the amount of resource overhead savings or reduced amount of operation of the multicast distribution relative to the point-to-point distribution is calculated per unit time.
Referring to fig. 4d, fig. 4d is a schematic flow chart of data distribution between proxy nodes according to an embodiment of the present application. The proxy node may acquire a multicast group, use multicast to send a message based on the multicast group, acquire a peer-to-peer proxy node list, acquire related information of one proxy node in the proxy node list, and use peer-to-peer proxy nodes to send a message.
In the embodiment of the application, when the message is distributed among the proxy nodes, the published data can be used for replacing the push content to combine the push message. The embodiment of the application provides a multicast group election method with simplified calculation based on a multicast group election mechanism with stable proxy node combination weight, comprising proxy node stability evaluation, stable proxy node combination calculation and weight evaluation, and a message distribution mechanism with multicast and point-to-point combination by selecting through weight aggregation sequencing, thereby enabling multicast capability to reduce message sending times.
The result comparison graph/table (number of data transmission times between proxy nodes) of the embodiment of the present application and the prior art may be as follows:
in this embodiment of the present application, the time point of sending the published data to the M proxy nodes is based on that a time interval between the time point of sending the published data to the M proxy nodes and the time point of last sending the published data to the M proxy nodes is greater than a preset value. That is, the M proxy nodes may also obtain the frequency of the latest push, and if the push is too fast, increase the cooling delay before pushing. And combining the trigger events in the time delay interval, and triggering the inquiry only once. Frequent pushing of subscription rules for hotspots; the embodiment can reduce the pushing times by controlling the time delay.
The comparison chart/table (the number of queries to the database) of the results of the embodiment of the present application and the prior art may be as follows:
in this embodiment of the present application, the plurality of subscription message formats includes a first subscription message format and a second subscription message format, where at least one proxy node pair corresponding to the first subscription message format includes a target node, and at least one proxy node pair corresponding to the second subscription message format includes the target node; the published data can be sent to at least one proxy node corresponding to the first subscription message format; sending the release data to at least one proxy node corresponding to the second subscription message format; wherein the published data is sent only once to the target node.
In the embodiment of the application, when the same proxy node uses different subscription rules to relate to the same key value, the key value is used to replace the content merging push message quantity. And the messages which need to be sent for multiple times are combined into one transmission, so that the number of times of sending the release data among the proxy nodes is reduced. The comparison graph/table (the number of data transmission between proxy nodes) of the embodiment of the present application and the result of the prior art may be as follows:
In the embodiment of the application, the subscription index comprising the subscription rule is constructed, the subscription rule and the proxy node are associated, the client selection for publishing the data index is performed through judgment of the subscription rule by using the publishing data retrieval rule index, and the query times of the subscription rule are greatly reduced.
The comparison chart/table (the number of queries to the database) of the results of the embodiment of the present application and the prior art may be as follows:
the embodiment of the application provides a method for issuing data index, which comprises the following steps: receiving release data; acquiring a subscription index, wherein the subscription index comprises a target subscription rule and identifications of M proxy nodes corresponding to the target subscription rule, and M is a positive integer; wherein the target subscription rule is contained in a plurality of subscription requests and only one of the target subscription rules is included in the subscription index; and sending the release data to M proxy nodes corresponding to the identifications of the M proxy nodes based on the release data meeting the target subscription rule, so that the M proxy nodes generate subscription messages based on the release data. By constructing the subscription index comprising the subscription rules, associating the subscription rules and the proxy nodes, selecting the proxy nodes for publishing the data index by using the published data retrieval rule index, merging the same subscription rules contained in a plurality of subscription requests from the client, and determining the proxy nodes for transmitting the published data only by one-step retrieval, so that the query times of the subscription rules are greatly reduced, and the query cost is reduced.
Referring to fig. 5, fig. 5 is a schematic structural diagram of an apparatus for indexing published data according to an embodiment of the application, and as shown in fig. 5, the apparatus 500 includes:
a receiving module 501, configured to receive published data; an obtaining module 502, configured to obtain a subscription index, where the subscription index includes a target subscription rule and identifiers of M proxy nodes corresponding to the target subscription rule, and M is a positive integer; wherein the target subscription rule is contained in a plurality of subscription requests and only one of the target subscription rules is included in the subscription index; and a sending module 503, configured to send the publish data to M proxy nodes corresponding to the identities of the M proxy nodes based on the publish data meeting the target subscription rule, so that the M proxy nodes generate a subscription message based on the publish data.
In one possible implementation, the sending module is configured to send the published data to the M proxy nodes by multicast.
In a possible implementation, the M proxy nodes include at least one first node and at least one second node, and the sending module is configured to send the published data to the at least one first node by multicast;
And sending the release data to the at least one second node in a point-to-point mode.
In one possible implementation, the at least one first node pushes a plurality of subscription messages to the client in a preset time period, and the subscription messages pushed to the client in a first subscription state in the plurality of subscription messages are larger than a preset proportion or a preset number of times, and the first subscription state indicates that the client is in a subscription state based on the target subscription rule.
In one possible implementation, the at least one second node pushes a plurality of subscription messages to the client in a preset time period, and the subscription messages pushed to the client in a second subscription state in the plurality of subscription messages are smaller than or equal to a preset proportion or a preset number of times, and the second subscription state indicates that the client cancels the subscription state based on the target subscription rule.
In one possible implementation, the target subscription rule includes a key value interval, and the published data includes a target key value, where the target key value belongs to the key value interval.
In one possible implementation, the target subscription rule further comprises a subscription message format; the subscription message is generated by the M proxy nodes based on the subscription message format.
In one possible implementation, the plurality of subscription message formats includes a first subscription message format and a second subscription message format, and at least one proxy node corresponding to the first subscription message format and at least one proxy node corresponding to the second subscription message format each include a target node; the sending module is configured to send the published data to at least one proxy node corresponding to the first subscription message format;
sending the release data to at least one proxy node corresponding to the second subscription message format; wherein the published data is sent only once to the target node.
In one possible implementation, the sending the published data to the M proxy nodes includes: and determining that each of the M proxy nodes continuously receives release data of preset times within preset time, and sending the release data to the M proxy nodes.
In one possible implementation, the sending the published data to the M proxy nodes includes: and sending the release data to the M proxy nodes based on the time interval between the last data sending to the M proxy nodes is larger than a preset value.
The embodiment of the application provides a release data indexing device, which comprises: the receiving module is used for receiving the release data; the acquisition module is used for acquiring a subscription index, wherein the subscription index comprises a target subscription rule and identifications of M proxy nodes corresponding to the target subscription rule, and M is a positive integer; wherein the target subscription rule is contained in a plurality of subscription requests and only one of the target subscription rules is included in the subscription index; and the sending module is used for sending the release data to M proxy nodes corresponding to the identifications of the M proxy nodes based on the release data meeting the target subscription rule, so that the M proxy nodes generate subscription information based on the release data. According to the method and the device, the subscription index comprising the subscription rules is constructed, the subscription rules and the proxy nodes are associated, the proxy nodes for publishing the data index are selected by using the published data retrieval rule index, the same subscription rules included in a plurality of subscription requests from the client are combined, the proxy nodes for transmitting the published data can be determined only by one-step retrieval, the query times of the subscription rules are greatly reduced, and the query cost is reduced.
Referring to fig. 6, the present application also provides a computer readable storage medium, in some embodiments, the method disclosed in fig. 3 above may be implemented as computer program instructions encoded on a computer readable storage medium in a machine readable format or encoded on other non-transitory media or articles of manufacture. Fig. 6 schematically illustrates a conceptual partial view of an example computer program product comprising a computer program for executing a computer process on a computing device, arranged in accordance with at least some embodiments presented herein. In one embodiment, the example computer program product 600 is provided using a signal bearing medium 601. The signal bearing medium 601 may include one or more program instructions 602 that when executed by one or more processors may provide the functionality or portions of the functionality described above with respect to fig. 3. Thus, for example, referring to the embodiment shown in FIG. 3, one or more features of blocks 301-304 may be carried by one or more instructions associated with signal bearing medium 601. Further, the program instructions 602 in fig. 6 also describe example instructions.
In some examples, signal bearing medium 601 may comprise a computer readable medium 603 such as, but not limited to, a hard disk drive, compact Disk (CD), digital Video Disk (DVD), digital tape, memory, read-Only Memory (ROM), or random access Memory (Random Access Memory, RAM), among others. In some implementations, the signal bearing medium 601 may contain a computer recordable medium 604 such as, but not limited to, memory, read/write (R/W) CD, R/W DVD, and the like. In some implementations, the signal bearing medium 601 may include a communication medium 605 such as, but not limited to, a digital and/or analog communication medium (e.g., fiber optic cable, waveguide, wired communications link, wireless communications link, etc.). Thus, for example, the signal bearing medium 601 may be conveyed by a communication medium 605 in wireless form (e.g., a wireless communication medium that complies with the IEEE 802.11 standard or other transmission protocol). The one or more program instructions 602 may be, for example, computer-executable instructions or logic-implemented instructions. In some examples, a computing device of a computing device may be configured to provide various operations, functions, or actions in response to program instructions 602 communicated to the computing device through one or more of computer readable medium 603, computer recordable medium 604, and/or communication medium 605. It should be understood that the arrangement described herein is for illustrative purposes only. Thus, those skilled in the art will appreciate that other arrangements and other elements (e.g., machines, interfaces, functions, orders, and groupings of functions, etc.) can be used instead, and some elements may be omitted altogether depending on the desired results. In addition, many of the elements described are functional entities that may be implemented as discrete or distributed components, or in any suitable combination and location in conjunction with other components.
Based on the same inventive concept, an embodiment of the present invention provides a published data indexing device, which may be a proxy node in the above embodiment or a server where the proxy node is located, referring to fig. 7, a published data indexing device 700 includes: a transmitter 703, a receiver 704, a memory 702 and a processor 701 coupled to the memory 702. The transmitter 703, receiver 704, memory 702, and processor 701 may be connected by a bus or otherwise (in fig. 7, a connection by a bus is an example). Wherein:
the processor 701 may be one or more central processing units (Central Processing Unit, CPU), and in fig. 7, a processor is taken as an example, where the processor 701 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.
The transmitter 703 is used for transmitting data to the outside;
the receiver 704 is configured to receive data from outside.
It should be noted that, in the embodiment of the present invention, the host may be a separate device in the database system, for example, the host is a separate physical server. The host may also be a non-independent device, in which case, in an application scenario, the host may be built into the host or exist as a functional module of the host (e.g., the host and the host are connected by an I/O interface and coexist on the same physical server as different virtual machines); in another application scenario, the host may be built in a certain standby machine, or exist as a functional module of a certain standby machine (for example, a certain standby machine and the host are connected through an I/O interface as different virtual machines and coexist in the same physical server).
It should be further noted that the execution steps of the processor 701 and other technical features related to the processor 701 may refer to the relevant content of the proxy node in the method embodiment of fig. 3, which is not described herein again.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.
In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of elements is merely a logical functional division, and there may be additional divisions of actual implementation, e.g., multiple elements or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or all or part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or other network device, etc.) to perform all or part of the steps of the method described in the embodiment of fig. 3 of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The above embodiments are merely for illustrating the technical solution of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.
Claims (19)
1. A method of publishing a data index, the method comprising:
receiving release data;
acquiring a subscription index, wherein the subscription index comprises a target subscription rule and identifications of M proxy nodes corresponding to the target subscription rule, and M is a positive integer; wherein the target subscription rule is contained in a plurality of subscription requests and only one of the target subscription rules is included in the subscription index;
based on the publishing data meeting the target subscription rule, sending the publishing data to M proxy nodes corresponding to the identifications of the M proxy nodes, so that the M proxy nodes generate subscription messages based on the publishing data;
the M proxy nodes comprise at least one first node, and the at least one first node is a node which stably pays attention to the target subscription rule;
the target subscription rule further includes a subscription message format; the plurality of subscription message formats comprise a first subscription message format and a second subscription message format, and at least one proxy node corresponding to the first subscription message format and at least one proxy node corresponding to the second subscription message format both comprise target nodes;
The sending the published data to the M proxy nodes includes: sending the release data to at least one proxy node corresponding to the first subscription message format; sending the release data to at least one proxy node corresponding to the second subscription message format; wherein the published data is sent only once to the target node.
2. The method of claim 1, wherein the sending the published data to the M proxy nodes comprises:
and sending the release data to the M proxy nodes in a multicast mode.
3. The method of claim 1, wherein the M proxy nodes include at least one first node and at least one second node, and wherein the sending the published data to the M proxy nodes comprises:
transmitting the release data to the at least one first node in a multicast mode;
and sending the release data to the at least one second node in a point-to-point mode.
4. A method according to claim 3, wherein the at least one first node pushes a plurality of subscription messages to clients within a preset time period, wherein the subscription messages pushed to clients in a first subscription state among the plurality of subscription messages are greater than a preset proportion or a preset number of times, and wherein the first subscription state indicates that the clients are in a subscription state based on the target subscription rule.
5. The method according to claim 3 or 4, wherein the at least one second node pushes a plurality of subscription messages to the client within a preset time period, wherein the subscription messages pushed to the client in a second subscription state among the plurality of subscription messages are smaller than or equal to a preset proportion or a preset number of times, and the second subscription state indicates that the client cancels the subscription state based on the target subscription rule.
6. The method of any one of claims 1 to 5, wherein the target subscription rule includes a key value interval, and the published data includes a target key value, and the target key value belongs to the key value interval.
7. The method of claim 6, wherein the subscription message is generated by the M proxy nodes based on the subscription message format.
8. The method according to any one of claims 1 to 7, wherein said sending the published data to the M proxy nodes comprises: and determining that each of the M proxy nodes continuously receives release data of preset times within preset time, and sending the release data to the M proxy nodes.
9. The method according to any one of claims 1 to 8, wherein said sending said published data to said M proxy nodes comprises: and sending the release data to the M proxy nodes based on the time interval between the last data sending to the M proxy nodes is larger than a preset value.
10. A published data indexing device, the device comprising:
the receiving module is used for receiving the release data;
the acquisition module is used for acquiring a subscription index, wherein the subscription index comprises a target subscription rule and identifications of M proxy nodes corresponding to the target subscription rule, and M is a positive integer; wherein the target subscription rule is contained in a plurality of subscription requests and only one of the target subscription rules is included in the subscription index;
the sending module is used for sending the release data to M proxy nodes corresponding to the identifications of the M proxy nodes based on the release data meeting the target subscription rule, so that the M proxy nodes generate subscription information based on the release data;
the M proxy nodes comprise at least one first node, and the at least one first node is a node which stably pays attention to the target subscription rule;
The target subscription rule further includes a subscription message format; the plurality of subscription message formats comprise a first subscription message format and a second subscription message format, and at least one proxy node corresponding to the first subscription message format and at least one proxy node corresponding to the second subscription message format both comprise target nodes;
the sending the published data to the M proxy nodes includes: sending the release data to at least one proxy node corresponding to the first subscription message format; sending the release data to at least one proxy node corresponding to the second subscription message format; wherein the published data is sent only once to the target node.
11. The apparatus of claim 10, wherein the sending module is configured to send the published data to the M proxy nodes by multicast.
12. The apparatus according to claim 10 or 11, wherein the M proxy nodes comprise at least one first node and at least one second node, and the sending module is configured to send the published data to the at least one first node by multicast;
And sending the release data to the at least one second node in a point-to-point mode.
13. The apparatus of claim 12, wherein the at least one first node pushes a plurality of subscription messages to clients within a preset time period, wherein subscription messages pushed to clients in a first subscription state among the plurality of subscription messages are greater than a preset proportion or a preset number of times, and wherein the first subscription state indicates that the clients are in a subscription state based on the target subscription rule.
14. The apparatus according to claim 12 or 13, wherein the at least one second node pushes a plurality of subscription messages to the client within a preset time period, wherein subscription messages pushed to the client in a second subscription state among the plurality of subscription messages are less than or equal to a preset proportion or a preset number of times, and wherein the second subscription state indicates that the client cancels the subscription state based on the target subscription rule.
15. The apparatus according to any one of claims 10 to 14, wherein the target subscription rule comprises a key value interval, and the published data comprises a target key value, the target key value belonging to the key value interval.
16. The apparatus of claim 15, wherein the subscription message is generated by the M proxy nodes based on the subscription message format.
17. The apparatus according to any one of claims 10 to 16, wherein said sending said published data to said M proxy nodes comprises: and determining that each of the M proxy nodes continuously receives release data of preset times within preset time, and sending the release data to the M proxy nodes.
18. The apparatus according to any one of claims 10 to 17, wherein said sending said published data to said M proxy nodes comprises: and sending the release data to the M proxy nodes based on the time interval between the last data sending to the M proxy nodes is larger than a preset value.
19. A computer storage medium storing one or more instructions which, when executed by one or more computers, cause the one or more computers to implement the method of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010663711.2A CN113992741B (en) | 2020-07-10 | 2020-07-10 | Method and device for indexing release data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010663711.2A CN113992741B (en) | 2020-07-10 | 2020-07-10 | Method and device for indexing release data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113992741A CN113992741A (en) | 2022-01-28 |
| CN113992741B true CN113992741B (en) | 2023-06-20 |
Family
ID=79731268
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010663711.2A Active CN113992741B (en) | 2020-07-10 | 2020-07-10 | Method and device for indexing release data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113992741B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101420391A (en) * | 2007-08-30 | 2009-04-29 | 软件Ag公司 | Systems and/or methods for providing feature-rich proprietary and standards-based triggers via a trigger subsystem |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB0228941D0 (en) * | 2002-12-12 | 2003-01-15 | Ibm | Methods, apparatus and computer programs for processing alerts and auditing in a publish/subscribe system |
| EP1849093A2 (en) * | 2005-01-06 | 2007-10-31 | Tervela Inc. | Hardware-based messaging appliance |
| CN103324520B (en) * | 2013-06-19 | 2016-04-06 | 西北工业大学 | A kind of method for subscribing merging based on condition and share |
| CN104283838A (en) * | 2013-07-02 | 2015-01-14 | 南京理工大学常熟研究院有限公司 | Efficient similar subscription merging method for rejecting service attack |
| CN103458033B (en) * | 2013-09-04 | 2016-05-11 | 北京邮电大学 | Event-driven, service-oriented Internet of Things service provider system and method for work thereof |
| CN104702638B (en) * | 2013-12-06 | 2019-04-12 | 华为技术有限公司 | The subscription distribution method and device of event |
| CN104092767B (en) * | 2014-07-21 | 2017-06-13 | 北京邮电大学 | A kind of publish/subscribe system and its method of work for increasing message queue model |
| CN105260479B (en) * | 2015-11-10 | 2019-03-19 | 南京国电南自维美德自动化有限公司 | A kind of real-time data base real-time data warehouse method based on Observer Pattern |
| US20190052717A1 (en) * | 2016-02-18 | 2019-02-14 | Nec Corporation | Communication system, edge server, method, and storage medium |
| CN110557423B (en) * | 2018-06-04 | 2022-02-11 | 珠海全志科技股份有限公司 | Message pushing method and system |
| CN110263282A (en) * | 2019-06-19 | 2019-09-20 | 南京邮电大学 | A kind of design method of the content distribution subscription communication pattern based on Kafka |
-
2020
- 2020-07-10 CN CN202010663711.2A patent/CN113992741B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101420391A (en) * | 2007-08-30 | 2009-04-29 | 软件Ag公司 | Systems and/or methods for providing feature-rich proprietary and standards-based triggers via a trigger subsystem |
Non-Patent Citations (1)
| Title |
|---|
| 胡昔祥 ; .面向大规模分布式计算的发布/订阅系统.浙江大学学报(工学版).2008,(第05期),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113992741A (en) | 2022-01-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7437373B2 (en) | Method and system for correlating information | |
| CN105740337B (en) | A kind of event fast matching method in distribution subscription system based on content | |
| US8266590B2 (en) | Managing dynamic configuration data for a set of components | |
| US20080243789A1 (en) | Groupware system with improved contact data handling | |
| CN110019240A (en) | A kind of service data interaction method, apparatus and system | |
| US20080261516A1 (en) | Playing digital content from satellite radio media based on taste profiles | |
| JP2014528126A (en) | Distributing multi-source push notifications to multiple targets | |
| JP2014531092A (en) | Distributed data stream processing method and system | |
| CN111258978B (en) | Data storage method | |
| CN101160580A (en) | Virtual network linking computers whose users share similar interests | |
| WO2004068355A1 (en) | Information processing device, content management method, content information management method, and computer program | |
| CN106470150B (en) | Relation chain storage method and device | |
| US7734587B2 (en) | Syndication of content based upon email user groupings | |
| JP2009193250A (en) | Distributed directory server, distributed directory system, distributed directory method, and program | |
| US20180270605A1 (en) | System and method for providing location data over a messaging system | |
| US20020062268A1 (en) | Scheme for presenting recommended items through network based on access log and user preference | |
| CN109033404A (en) | Daily record data processing method, device and system | |
| JP4506387B2 (en) | Information communication system, node device, overlay network forming method, etc. | |
| JP2023058423A (en) | Digital object access method and system in human cyber physical fusion environment | |
| CN114971714A (en) | Accurate customer operation method based on big data label and computer equipment | |
| CN113992741B (en) | Method and device for indexing release data | |
| CN103428231B (en) | Offline download method and system | |
| US9727655B2 (en) | Searching system, method and P2P device for P2P device community | |
| CN111324674B (en) | Service processing method, device, server and storage medium | |
| US20090049055A1 (en) | Method, apparatus, and system to provide content using content preference information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |