WO2016197858A1

WO2016197858A1 - Method and device for message notification

Info

Publication number: WO2016197858A1
Application number: PCT/CN2016/084595
Authority: WO
Inventors: 喻奎; 段培乐; 李闪; 孙敬
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2015-06-12
Filing date: 2016-06-03
Publication date: 2016-12-15
Also published as: CN106302621A; CN106302621B

Abstract

Disclosed in the present application are a method and device for message notification. The method comprises: adding, by a logical node, a single ID to each item of information to be sent; comparing, by the logical node and according to a preset rule, an ID of the information and an ID of a received information; sending, by the logical node, the information, the information to be sent including a comparison result, to one of child logical nodes; and if the received information is processed by the logical node without generating new information, comparing, by the logical node, the IDs of the received information, and including a comparison result in information sent to a server, so that the server can notify a user after determining, according to the preset rule, that data information sent by the user is processed. The above method in the present application can successfully notify the user after the user data information is processed, and further reduces network and memory costs.

Description

Message notification method and device

The present application claims priority to Chinese Patent Application Serial No. No. No. No. No. No. No. No. No. No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No

Technical field

The present application relates to the field of network technologies, and in particular, to a message notification method and device.

Background technique

At present, the distributed computing service model can be mainly divided into offline computing and real-time computing. Off-line computing mainly uses batch processing to process data, which is suitable for big data processing services with high throughput, low cost, and insensitive data processing delay. Real-time computing, including streaming computing, quasi-real-time services, etc., refers to the use of distributed ideas and methods for real-time processing of massive "streaming" data, suitable for high-concurrency, low-latency data real-time processing services. Such as open source architecture storm (a stream data processing architecture) provides streaming computing services, spark streaming (a quasi-real-time distributed data processing architecture) provides quasi-real-time services. Regardless of the computing model, in a distributed environment, the data processing engine part can be abstracted into a DAG (Directed Acyclic Graph) framework to process data. DAG can be understood as a directed acyclic graph composed of some logical processing nodes with sequential dependencies. After the processing of the parent node is completed, the data is transmitted to the child nodes for processing until all the leaf nodes are processed, and the data input by the client is completed. It is considered to be processed by the data processing engine. For example, Hadoop (a software framework capable of distributed processing of large amounts of data), the data warehouse tool Hive also converts user-submitted sql (Structured Query Language) into MR (Map-Reduce, map reduction) The DAG is composed of DAG to process batch data. Spark (a batch data processing architecture) is also designed as a DAG processing flow composed of operators of RRD (Resilient Distributed Datasets) data units. The topology of Storm The topology is more typical of the DAG framework.

In the above system, it can be considered that the data flow direction generally undergoes three stages of data input, data processing, and result output, wherein the DAG processing model belongs to the data processing part. In some business scenarios, when the data unit is processed, some users want to know the status in time to proceed to the next process. This requires the data processing engine to provide a message notification framework or other means such as the client API to enable the user to know the processing in time. As a result, the data granularity, latency, correctness, and fault tolerance of the message notification determine the difficulty of the data processing engine to implement the message notification mechanism. For offline computing based on DAG architecture, users can call the client's API (client's operating system) A call interface that is reserved for the application) obtains whether the input batch of data is processed or not, and achieves the purpose of notifying the user. Most offline computing services have low requirements for data granularity and timeliness of message notification, and therefore generally do not It is necessary to adopt a message notification mechanism that pushes the PUSH mode in a timely manner in a strict sense. For many streaming computing services based on DAG architecture, the granularity and latency of data notification may become very high, and even the data granularity needs to be reduced to each data, that is, when a piece of data flows into the streaming computing engine, this After the strip data and all new data generated by this data are processed by all DAGs, the data processing engine needs to notify the user immediately, and the user makes corresponding decisions according to the processing result.

In the process of implementing the present application, the applicant found that the prior art has at least the following problems:

Based on the current distributed computing system, there is no mature and efficient message notification mechanism, and the message notification mechanism cannot achieve the expected effect due to the defects of some distributed computing system frameworks.

Summary of the invention

The purpose of the application is to provide a message notification method and device, which compares the received information with the ID in the information to be sent according to a preset rule, and sends the comparison result to one of the child logical nodes to continue comparison. When the logical node does not generate the information information after receiving the information, the last information carrying the comparison result of the ID is sent to the server, and the server determines whether the data information sent by the user is processed according to the preset rule. The ID in the comparison information reaches the judgment whether the data information sent by the user is processed, and reaches the purpose of notifying the user. Such processing is performed by each logical node and the server in the system, and the ID in the information is compared, and the Occupy too many resources of a device, so that the message notification mechanism can achieve higher performance.

The technical solution of the present application is as follows:

A message notification method, the method comprising:

The logical node adds a unique ID for each information to be sent, wherein the logical node sends each information carrying a unique ID can only be obtained by one of the following logical nodes;

The logical node compares the information to be sent with the ID in the received information according to a preset rule;

The logical node sends the information to be sent carrying the comparison result to one of the child logical nodes, so that the child logical node uses the ID included in the received information and the ID and information in the information to be sent. Compare the results of the comparison;

When the logical node does not generate new information after processing the received information, the logical node compares the IDs in the received information, and carries the comparison result in the information sent to the server, so that the Service The device notifies the user after determining that the data information sent by the user is processed according to the preset rule.

When the logical node is a root logical node, the root logical node receives and forwards data information input by the user, so that the root logical node forwards data information input by the user to the sub-logic of the root logical node. The node processes the data information input by the user;

The ID information is not added to the data information input by the user.

The method further includes:

After the logical node receives the data information sent by the user, the logical node receives the same data information that is sent by the server and sent by the user, so that the server pairs the logic according to the preset rule. The information returned by the node carrying the comparison result is judged, and according to the judgment result, it is judged whether the data information sent by the user is processed.

The preset rule is that when there are even numbers of identical IDs, the even number of identical IDs are deleted;

When there are an odd number of identical IDs, only one of the odd number of identical IDs is retained.

A message notification method, the method comprising:

The server receives information that is sent by the logical node and carries the comparison result;

The server determines, according to a preset rule, the received information carrying the comparison result according to a preset rule, and notifies the user after determining that the data information sent by the user is processed;

Wherein the logical node is a logical node that does not generate new information according to the received information;

The result of the comparison is that the logical node is generated by comparing the IDs contained in the received information.

The server determines the received comparison result according to the preset rule in a preset time, and notifies the user after determining that the data information sent by the user is processed, specifically:

The server determines the number of the same ID in the received comparison result within a preset time. If the number of the same ID is an even number, the server deletes the even number of the same ID, and the ID does not exist in the server. When the data information sent by the user is processed, and the user is notified;

If the number of the same ID is an odd number, the server retains one of the odd number of identical IDs, and continues to receive the information carrying the comparison result, and when there is an ID in the server within a predetermined time, it indicates The data sent by the user is not processed, and the data information sent by the user is notified to be completed when the ID does not exist in the server before the end of the preset time.

The method further includes:

After the user sends the data information to the logical node, the server sends the same data information that is sent by the user to the logical node;

The server receives information returned by the logical node and carrying the comparison result;

The server determines the comparison result in the information according to a preset rule in a preset time, so that the server determines whether the data information sent by the user is processed according to the determination result.

A logical node device, the device comprising:

Adding a module, configured to add a unique ID for each information to be sent, wherein the logical node sends each information carrying a unique ID can only be obtained by one of the following logical nodes;

a comparison module, configured to compare the information to be sent and the ID in the received information according to a preset rule;

a first sending module, configured to send, to the one of the child logical nodes, the information to be sent that carries the comparison result, so that the child logical node uses the ID included in the received information and the information to be sent The ID is compared with the comparison result;

a second sending module, when the logical node does not generate new information after processing the received information, the logical node compares the IDs in the received information, and carries the comparison result in the information sent to the server So that the server notifies the user after the processing of the data information sent by the user is determined according to the preset rule.

The ID information is not added to the data information input by the user.

The device further includes:

Receiving, after receiving the data information sent by the user, the receiving module is configured to receive the same data information that is sent by the server and sent by the user, so that the server is configured according to the preset rule. The information returned by the logical node carrying the comparison result is judged, and according to the judgment result, it is judged whether the data information sent by the user is processed.

A server device, the device comprising:

a first receiving module, configured to receive, by the logical node, the received information and the information to be sent that carries the comparison result;

a first determining module, configured to determine, according to a preset rule, the received information carrying the comparison result according to a preset rule;

a notification module, configured to notify the user after determining that the data information sent by the user is processed;

The first determining module is specifically configured to:

Determining, in a preset time, the number of the same IDs in the received comparison result. If the number of the same IDs is an even number, the server deletes the even number of identical IDs, and when there is no ID in the server, it indicates The data information sent by the user is processed;

The device further includes:

a sending module, after the user sends the data information to the logical node, to send the same data information that is sent by the user to the logical node;

a second receiving module, configured to receive information that is returned by the logical node and that carries a comparison result;

The second determining module is configured to determine, according to the preset rule, the comparison result in the information according to the preset rule, so that the server determines, according to the determination result, whether the data information sent by the user is processed.

The application compares the received information with the ID in the information to be sent according to a preset rule, and sends the information carrying the comparison result to one of the next group of logical nodes to continue comparison, when the logical node receives When the new information is no longer generated, the comparison result of the last ID is carried in the information sent to the server, and the server determines whether the data information sent by the user is processed according to the preset rule. This application achieves the ID in the comparison information. Determine whether the data information sent by the user is processed and reach the purpose of notifying the user. Such processing is performed by each logical node and the server in the system, and the ID in the information is compared, and the device is not occupied too much. Resources, so that the message notification mechanism can achieve higher performance.

DRAWINGS

In order to more clearly illustrate the present application or the technical solutions in the prior art, the following drawings will be briefly introduced in the present application or the description of the prior art. It is obvious that the drawings in the following description are only the present application. Some of the embodiments can be obtained by those of ordinary skill in the art in view of the drawings without any inventive effort.

FIG. 1 is a schematic diagram of a message notification method in an embodiment of the present application;

2 is a schematic flowchart of information processing in an embodiment of the present application;

3 is a second schematic diagram of a message notification method in an embodiment of the present application;

4 is a schematic structural diagram of a stream framework of a distributed streaming computing system according to an embodiment of the present application;

FIG. 5 is a schematic diagram of a record data processing topology based on a stream system processing according to an embodiment of the present application;

6 is a schematic diagram of a working principle based on a Batch mechanism in an embodiment of the present application;

FIG. 7 is a schematic structural diagram of a logical node in an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a server in an embodiment of the present application.

detailed description

The technical solutions in the present application are clearly and completely described in the following with reference to the drawings in the present application. It is obvious that the described embodiments are a part of the embodiments of the present application, and not all of the embodiments. Other embodiments obtained by those skilled in the art based on the embodiments in the present application are all within the scope of the present application.

The logical node exists in the server, and the server includes a root logical node and a multi-level logical node under the root logical node, and the root logical node and other multi-level logical nodes may be collectively referred to as a logical node, wherein if a logic A node has a subordinate logical node, and the subordinate logical node is called a sub logical node of the logical node. As shown in FIG. 2, the server includes: a root logical node PN (Processer Node) and a multi-level logical node PN1-PN6, and PN1 and PN2 are called subordinate logical nodes of the root logical node PN, that is, sub-logic of the PN. At the same time, the root logical node PN is called the upper logical node of the logical nodes PN1 and PN2, PN1 and PN2 can be called the first logical node, and the lower logical nodes of PN1 are PN3, PN4, PN5, PN3, PN4, PN5 are called The child logical node of PN1, PN6 is the lower logical node of PN2, that is, the child logical node of PN2 is PN6, PN2 is the superior logical node of PN6, and PN3, PN4, PN5 and PN6 can be called secondary logical nodes.

The data information sent by the user is forwarded by the server to the root logical node, and the server records the data information sent by the user.

A message notification method based on a distributed computing system, as shown in FIG. 1, the method includes the following steps:

Step 101, the logical node adds a unique ID (Identity, identity number) for each information to be sent. code).

The information that the logical node sends each carrying a unique ID can only be obtained by one of the child logical nodes of the logical node.

Specifically, as shown in FIG. 2, the IDs added by PN1 in the three information to be sent are r3, r4, and r5, respectively, wherein each information in PN1 can only be sent to the child logical nodes PN3, PN4 of PN1, In one of PN5, for example, r3 information is added to PN3, r4 information is added to PN4, and r5 information is added to PN5.

The logical node processes the received information according to a preset logical calculation manner to generate information to be sent. Specifically, the logical calculation method used by the logical node to process the received information is preset. Also, the logical calculations used by different logical nodes may be different.

Step 102: The logical node compares the information to be sent with the ID in the received information according to a preset rule.

When the logical node is a root logical node, the root logical node receives and forwards data information input by the user, so that the root logical node forwards data information input by the user to the child logical node of the root logical node. Processing data information input by the user;

The data information input by the user is generated according to a preset fixed format edit.

The ID information is not added to the user input data, and after the root logical node receives the data information input by the user, when the data information is sent to the child logical node of the root logical node, An ID is added to the data information.

When there are an odd number of identical IDs, only one of the odd number of identical IDs is reserved. Of course, the preset rule may also be an exclusive OR algorithm, and may also be other methods or algorithms, but any technology that can achieve the present application can be achieved. The method or algorithm of the effect belongs to the protection scope of the present application.

Specifically, as shown in FIG. 2, the PN is a root logical node, and the PN receives data information input by the user, and the ID does not exist in the data information input by the user, but when the PN is sent to the sub-logical nodes PN1 and PN2, An ID is added to the data information. If the ID of the information transmitted to PN1 is r1 and the ID of the information transmitted to PN2 is r2, the comparison result of the IDs of the information received and transmitted by the root logical node is r1 and r2.

If the ID included in the information received by PN1 is r1, the PN1 logically calculates the information with the ID r1 according to a preset logical calculation manner, and adds an ID to the information obtained through the logical calculation, wherein the logic is added. The calculated information is the information to be sent. If the ID in the information to be sent is r3, r4, r5, the pass After comparison, the information received by PN1 and the ID in the information to be transmitted are not the same, and the comparison result obtained by PN1 is r1, r3, r4, r5.

Step 103: The logical node sends the information to be sent carrying the comparison result to one of the child logical nodes, so that the child logical node uses the ID included in the received information and the information to be sent. The ID is compared to the comparison result.

The information received by the logical node includes the ID assigned by the superior logical node for the information and the result of comparing the ID included in the information received by the superior logical node with the ID in the information to be sent, and the comparison result is an ID. set.

Specifically, as shown in FIG. 2, the PN sends the information carrying r1 to PN1, and also sends the comparison result (r1, r2) to PN1, and the ID in the information to be transmitted by PN1 is r3, r4, r5, so PN1 There is an even number of r1, so it is deleted, and the comparison result is r2, r3, r4, r5. At PN1, a message carrying the ID and the comparison result can be sent to one of the logical nodes PN3, PN4, PN5.

Step 104: When the logical node does not generate new information after processing the received information, the logical node compares the IDs in the received information, and carries the comparison result in the information sent to the server, to And causing the server to notify the user after determining that the data information sent by the user is processed according to the preset rule.

Specifically, as shown in FIG. 2, PN3 does not generate new information after processing the received information, and the ID of the information received by PN3 is r3, and the received comparison result is r2, r3, r4, r5, and an even number. R3, so r3 is deleted, the comparison result is r2, r4, r5, and PN3 carries the result in the information of the sending server;

PN4 does not generate new information after processing the received information, and PN4 does not receive the comparison result of PN1 transmission. Only the received information has an ID in the PN4, and therefore, the PN4 will receive the information. The ID (r4) is sent to the server.

The method further includes:

Specifically, in order to ensure that the server loses data due to special reasons, the server cannot notify the user in time, and the server receives the data information sent by the user after the PN receives the data information according to a preset mechanism, and the PN receives the One or more data information sent by the server and sent by the user, and performing the same operation as the data information sent by the user, after the server receives all the comparison results, according to a preset rule Whether the same data information sent by the root logical node and the user is processed is completed. If the processing is completed, the data information sent by the user is also completed, but the user is not notified in time due to the data loss of the server. The user is notified that the data information sent by the user is processed.

Of course, the logical node directly sends the ID of the information to be sent and the ID of the received information to the server, and the server compares all the received IDs according to a preset rule to determine the data information sent by the user. Whether the processing is completed.

A message notification method, as shown in FIG. 3, the method includes the following steps:

Step 301: The server receives information that is sent by the logical node and carries the comparison result.

Wherein, the logical node is a logical node that does not generate new information according to the received information.

Step 302: The server determines, according to a preset rule, the received information carrying the comparison result according to a preset rule, and notifies the user after determining that the data information sent by the user is processed.

Specifically, as shown in FIG. 2, the logical nodes PN3, PN4, PN5, and PN6 are logical nodes that no longer generate information, indicating that the data information sent by the user is processed, then the logical nodes PN3, PN4, PN5 and The PN 6 compares the received ID in the received information with the received comparison result according to a preset rule, and transmits the comparison result to the server.

The server determines, according to a preset rule, the received information carrying the comparison result according to the preset rule, and notifies the user after determining that the data information sent by the user is processed, specifically:

The server determines, in a preset time, the number of the same IDs in the received information carrying the comparison result, and if the number of the same IDs is an even number, the server deletes the even number of the same IDs, when the server If the ID does not exist, it indicates that the data information sent by the user is processed, and the user is notified;

Specifically, as shown in FIG. 2, the logical nodes PN3, PN4, PN5, and PN6 are logical nodes that no longer generate information, and the PN3 compares the ID in the received information with the received comparison result: r2, r4, R5, PN4 is: r4, PN5 is: r5, point PN6 is: r2, and the logical nodes PN3, PN4, PN5, PN6 will be compared. The server sends the server. At this time, the IDs appearing in the server are all evenly deleted, indicating that the data information sent by the user is processed, so the server can notify the user according to the comparison result.

At the same time, since the processing cycle in which the user sends a data message is limited, if the processing is not completed within one processing cycle, it indicates that the processing has timed out or the system has a problem.

After the user sends the data information to the logical node, the method further includes:

Sending, by the server, the same data information that is sent by the user to the logical node;

In order to further illustrate the technical idea of the present application, the technical solution of the present application will be described in conjunction with a specific application scenario in which the preset rule is an exclusive OR operation.

Specifically, as shown in FIG. 4, a distributed streaming computing system framework includes three functional components: a control node controller, a data source shard (which can be understood as a root logical node), and a data processing logical topology topology, and A message synchronization node Acker Node for message synchronization and a batch data synchronization node GlobalBatch Syner, wherein Topology is a user-built DAG logic flow chart for processing data, which is composed of one or more logical nodes PN having sequential dependencies. Each PN can have multiple physical running instances. The data flowing into the topology flows to each PN for processing according to the path of the DAG. The shard is used to receive data information sent by the user. It first writes the data to the distributed file system. To ensure that the data is properly placed, and then sent to the N topologies connected to the latter for processing. The Controller is configured to accept a create topology request sent by the user, and then perform necessary conversion processing to be scheduled to run on a specific one or more machines, wherein the distributed streaming computing system framework exists in the server.

The specific data information record processing process is as follows: the user inputs a record from the client to the server, first selects a shard node in the server to land, and then forwards it to the N topologies corresponding to the shard. Process it. Each topology PN may generate N (N>=0) new records after processing the received record, and continue to send to its sub-PN node for processing, so that the calculation path of the topology DAG is continuously generated. The process of the new record can be understood as generating a data information tree record tree triggered by the source record. The shape of the tree generated by each source record is not necessarily the same, but their DAG shape is definitely a subtree of topology DAG, and the root node is the same. If there are multiple topologies connected to the shard, the new record generated by the source record triggers a forest. When the record forest is not growing, that is, no new record is generated, the source record can be considered to be finished by the stream engine, and the user can be notified of the end state of the processing. At the same time, different record forests may be generated due to different data information carried by different records.

When a piece of data information is processed, the information in it always appears in pairs, that is, one logical node sends a message, and the information is received by another logical node. Therefore, each newly generated record can be assigned one. Randomly generated unique ID. When a record is created by a PN, the record will be received by another PN. The same ID corresponding to the two PNs will be 0 when performing an exclusive OR operation, indicating that the record is The PN was successfully processed. Further, when a record forest is completed, all records correspond to a pair of identical IDs, so the result obtained by the XOR operation is definitely 0.

FIG. 5 is a flowchart of a record processing based on a stream system according to an embodiment of the present application, wherein FIG. 5 shows a record forest generated by two records processed by a stream engine, and the first image shows the topology. The shape of the forest, the latter two figures are the record forest shape generated by two records: r1 and r2.

Each record forest has only leaf nodes (logical nodes that do not generate new information) to communicate with the Acker Node to reduce network overhead. All leaf nodes of a record forest will send the results of the local XOR partial ID calculations to The same Acker Node, since these records are derived from the same source record, they will have a message notification field with the same value as the source record called Mesg ID, and get the ID number to be sent to the acker Node through the Mesg ID. After the acker node obtains the partial exclusive value AckVal of a Mesg ID, it will XOR the value with the existing ackval corresponding to the Mesg ID saved in the current memory. If the value is 0, the Mesg ID is The corresponding source record is processed by the stream engine. The Mesg ID can be used to notify the user of the status result. If the value is not 0, it indicates that it has not been processed yet, and can continue to wait.

Specifically, the user submits two requests to create a topology: top1 and top2, and specifies to process the data on shard1. After the user request is sent to the Controller, the Controller generates the necessary floor plans and then schedules the two topologies in the computing cluster in order. The PN of each topology can be performed by thread granularity or process granularity, depending on the implementation mechanism of the resource scheduling system used by the cluster. The Controller randomly generates an integer value (ID) for each topology, and the first PN of each topology also carries the integer. The initial ackval of the source record received from the shard, for example: when top1 is created, the integer t1 is randomly assigned, and the initial ackval=t1 of all source records on the acker Node. When top2 is created, the system randomly assigns the integer t2 to the acker Node. The ackval=t1^t2 (^ indicates the XOR operator), and the ackval value is the result of the XOR operation.

After top1 and top2 are scheduled, and the message notification field value of r1 is mesgId1, after landing, r1 is forwarded to the first PN1 of top1 and the first PN2 of top2, and PN2 gives it an initial value of ackval=t1 after receiving r1. Since top2 has only one logical node PN2, PN2 is the leaf node, that is, r1 of the top2 record tree is the leaf node, so ackval=t1 of r1 on PN2 is sent to the acker Node, because the initial ackval= of all records on the acker Node T1^t2, XOR it with the sent t1 to get ackval=t1^t2^t1=t2, the result is not 0, so the acker Node continues to wait for other ackval of mesgID1 to send it; PN1 gives it after receiving r1 The initial value ackval=t2, then r1 is processed by PN2 will generate three records (r3, r4, r5), the system will randomly assign the initial ackval (t3, t4, 5), PN2 will select the last one by default. The record(r5) carries the octval(t3,t4) of the newly generated record(r3,r4) and the octval(t2) of the received record(r1) as its ackval, so ackval= of r5 T2^t3^t4, r3 ackval=t3, r4 ackval=t4, r5 not You need XOR's own ackval(t5), because after it is sent to the next PN, this t5 will eventually be XORed, so this extra XOR is not needed. R3, r4, r5 are then sent to PN3, PN4, PN5, respectively.

All PNs that have no direct or indirect dependencies are run in parallel, in no order, so PN3, PN4, and PN5 will process the sent records in parallel. PN3 receives r3, no new record is generated, so it belongs to a leaf node of record tree. PN3 will send ackval=t3 carried by r3 to acker node. At this time, ackval=t2^t3 of mesgId1; PN4 receives r4. A new record(r6) is generated. At this time, r6 is the last newly generated record. According to the above steps, ackval=t4 that r6 needs to carry and r6 to the next PN6; PN5 will generate two after receiving r5. A new record (r7, r8), the system allocates a new initial ackval=t7 for the first new record r7, and finally r7ackval=t7, according to the ackval(t7) of the above step r7 and the ackval of r5 (t2^t3^ T4) will be sent to r8 to carry it down, so r8ackval=t2^t3^t4^t7, r7 and r8 are sent to PN7 and PN8 respectively.

After receiving the r6, the PN6 does not generate a new node and belongs to a leaf node of the record tree. Therefore, the ackval=t4 carried by the r6 is sent to the acker node. At this time, the ackval=t2^t3^t4 of the mesgId1; the PN7 receives After r7, no new node is generated, and ackval=t7 carried by r7 is sent to acker Node. At this time, ackval=t2^t3^t4^t7 of mesgId1; after PN8 receives r8, no new one is generated. The node will send ackval=t2^t3^t4^t7 carried by r8 to the acker Node. At this time, ackval=t2^t3^t4^t7^t2^t3^t4^t7=0 of mesgId1.

Finally, the ackval=0 of the mesgID1 corresponding to the source record(r1) on the acker node, the source r1 is considered to have been The completion is completed, and the record forest corresponding to r1 does not grow anymore. At this time, mesgID1 can be notified to the user, and the user can make the next decision after obtaining the result.

The above embodiment tracks the structure of the record tree based on the XOR principle. Each record only needs to add 8 bytes to save the ackval value (the stream system uses 8 bytes by default, and can of course be extended to 16 bytes to reduce The same record probability that different records in the same record forest are randomly generated, and the ackval is carried to the leaf node to send the final local ackval to the Acker Node for final calculation, only need to consume the necessary network resources.

Figure 6 shows a flow diagram of an auxiliary fault-tolerant mechanism to handle exceptions. Based on this mechanism, the Acker Node does not need to persist any state, and the record forest leaf node does not need to re-transmit the ackval to the Acker Node. At the same time, it can ensure that the mesgID corresponding to all source records will not be lost in any case, and the user can be notified in time. This fault-tolerant mechanism solves these problems by using the concept of Batch. One batch will contain several records, all shards. The incremental Batch ID will be distributed periodically, and all the incoming records belong to a Batch until the next Batch ID arrives. The Batch ID will flow into all the PNs of all theology in a sequential order as a special record. Synchronize all the parent PN's Batch IDs have arrived, if it is, then the Batch is over, the PN will no longer have a record belonging to the Batch ID, and then generate a Batch IDEnd message to all sub-PNs. Each PN is the same process flow until the leaf PN.

The Topology PN can be subdivided into a root node agent PN and a non-root node inner PN, the agentPN is the root node of the topology DAG, and the innerPN is the other node of the topology DAG. All the topologies connected to the same shard are needed for optimization purposes. The agentPNs are all placed in the same process. Each data source instance shard instance corresponds to an agent PN process. As shown in Figure 6, the agent PNs of top1 and top2 are placed in the same process, and the agent PN of each topology is located. The process is an agent node. There is also a Mesg Notifier in the agentNode process. It is a component of the message notification framework that connects to the user. When it is notified by the Acker Node that a record has been processed, the Mesg ID of the record is immediately uploaded. Notifying the user, or when it receives a Global Batch ID End message, all the records belonging to the Batch ID and less than the Batch ID that have not been notified to the user are sent to notify the user that the records have been processed. InnerPN refers to all other nodes of topology, not a certain PN node. Blobal Batch Syner is a single point (BBS), which is used to globally synchronize the end of the Batch ID message. When the leaf PN node of the topology generates a Batch IDEnd message, it will send it to the BBS. When the BBS receives all the leaves from all theology. When the PN's Batch IDEnd message is generated, a Global Batch ID End message is generated for the Batch ID and sent to all Acker Node and Mesg Notifier for all Agent PNs. The Global Batch ID End indicates that the stream engine has processed all the records belonging to the Batch ID and less than the Batch ID. The Mesg Notifier can naturally notify the user of these records. At the same time, when the Acker Node receives the Global Batch ID End, it will clear all the records belonging to the Batch ID and less than the Batch ID. It is not necessary to maintain their status, because these records have been processed, and MesgNotifier will notify the user of these records. .

As indicated by reference numeral 1 in Figure 6, an instance of shard will send batch1 in order, source record(r1, r2, r3), batch2, source record(r4, r5) to its connected agent node process; process The agent PN and Mesg Notifier of top1 and top2 will receive all of this data, Mesg Notifier will save the data into memory until notification to the user, the other agent PN will process the data and then generate the new record and The original batch ID is sent to the child node.

As shown by the number 2 in Figure 6, each topology will process the record tree corresponding to r1, r2, and r3 at a certain time. In this process, the ackval in the record tree leaf node of r1, r2, and r3 will be used. Send to the Acker Node in sequence to perform an exclusive OR operation. When all the records in batch1 are processed, all the leaf nodes of topology will send the batch1End message to the BBS single point.

As shown by the number 3 in Fig. 6, all ackvals of the record tree of r1, r2, and r3 of top1 are sent to the Acker Node, r2 of top2, and all ackvals of the record tree of r3 are sent to the Acker Node, and top2 The ackval of all the leaf nodes of the record tree of r1 may be lost in the process of sending to the Acker Node due to network and other reasons. Therefore, after the exclusive operation of the Acker Node, only the ackval of r2 and r3 is 0. The value of r1 is not 0, so AckerNode will only notify the user of the Mesg Notifier by r2, r3. As for r1, another mechanism, the Batch mechanism, is required to ensure the notification to the user.

When the BBS accepts the message of the completion of the Batch1End of all the leaves of all the topologies of the PN, the message of Global Batch1End is generated. At this point, all the records of batch1 have been processed, so the Global Batch1End will be sent to the mesg Notifier and Acker. Node.

As shown by the reference numeral 4 in FIG. 6, the Mesg Notifier first receives the r2 and r3 sent by the Acker Node, and informs the user that the two records have been processed, as shown by the number 5 in FIG. 6, and the batch1 has the r1. The user has not been notified yet. When Mesg Notifier receives the Global Batch1End message sent by the BBS, it notifies the user of r1. Until now, all the records in batch1 have been sent to the user without losing any record. When Acker Node receives the message of Global Batch1End, it will clear the ackval state of r1, no need to maintain it, and save memory resources.

The present application compares the received information with the ID in the information to be sent according to a preset rule by the logical node. Comparing, and sending the comparison result to one of the next group of logical nodes to continue comparison, when the logical node no longer generates information information after receiving the information, the comparison result of the last ID is sent to the server, and the server according to the preset rule Determining whether the data information sent by the user is processed is completed. The present application determines whether the data information sent by the user is processed by comparing the ID in the information, and achieves the purpose of notifying the user. The processing manner is shared by each logical node and the server in the system. Completed, and compares the ID in the information, does not occupy too many resources of a certain device, so that the message notification mechanism can achieve relatively high performance, and does not need to know the state of information processing on each logical node to save a lot of Internet resources.

Based on the same application concept as the above method, the present application also proposes a logical node device. As shown in FIG. 7, the device includes:

An adding module 71, configured to add a unique ID for each information sent, where the logical node sends each information carrying a unique ID can only be obtained by one of the following logical nodes;

The comparing module 72 is configured to compare the information to be sent and the ID in the received information according to a preset rule;

The first sending module 73 is configured to send the comparison result to one of the next group of logical nodes, so that the logical node that receives the comparison result compares the received information with the ID in the information to be sent with the comparison result. ;

a second sending module 74, when the logical node does not generate new information after processing the received information, sending a result of comparing the IDs in the received information to the server, so that the server Determining, according to the preset rule, whether the data information sent by the user is processed.

When the logical node is a root logical node, the information received by the root logical node is data information sent by the user;

The ID information is not added to the data information sent by the user.

The device further includes:

Receiving, after receiving the data information sent by the user, the receiving module is configured to receive the same data information that is sent by the server and sent by the user, so that the server is configured according to the preset rule. The comparison result returned by the logical node is judged, and according to the judgment result, it is judged whether the data information sent by the user is processed.

Odd to the same application concept as the above method, the present application also proposes a server device, as shown in FIG. 8, the device includes:

The first receiving module 81 is configured to receive, by the logical node, the received information and the ID in the information to be sent. The result of the comparison;

The first determining module 82 is configured to determine, according to a preset rule, the received comparison result in a preset time;

The notification module 83 is configured to notify the user after determining that the processing of the data information sent by the user is completed;

The first determining module 81 is specifically configured to:

If the number of the same ID is an odd number, the server retains one of the odd number of identical IDs, and continues to receive the comparison result, indicating that the user sends the ID when there is an ID in the server within a predetermined time The data is not processed, and the user is notified that the outgoing information processing sent by the user is completed until the ID is not present in the server before the end of the preset time.

The device further includes:

a second receiving module, configured to receive a comparison result returned by the logical node;

The second determining module is configured to determine the comparison result according to the preset rule in a preset time, so that the server determines, according to the determination result, whether the data information sent by the user is processed.

Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, and of course, can also be through hardware, but in many cases, the former is a better implementation. the way. Based on this understanding, the technical solution of the present application contributes essentially or to the prior art. The portion may be embodied in the form of a software product stored in a storage medium, including instructions for causing a terminal device (which may be a cell phone, a personal computer, a server, or a network device, etc.) to execute the application. The methods described in the various examples.

The above description is only a preferred embodiment of the present application, and it should be noted that those skilled in the art can also make several improvements and retouchings without departing from the principles of the present application. The scope of protection of this application shall be considered.

Those skilled in the art can understand that the modules in the apparatus in the embodiments may be distributed in the apparatus of the embodiment according to the description of the embodiments, or the corresponding changes may be located in one or more apparatuses different from the embodiment. The modules of the foregoing embodiments may be integrated into one or may be deployed separately; may be combined into one module, or may be further split into multiple sub-modules. The serial numbers of the embodiments of the present application are merely for the description, and do not represent the advantages and disadvantages of the embodiments.

The above disclosure is only a few specific embodiments of the present application, but the present application is not limited thereto, and any changes that can be made by those skilled in the art should fall within the protection scope of the present application.

Claims

A message notification method, characterized in that the method comprises:

The logical node adds a unique ID for each information to be sent, wherein the logical node sends each information carrying a unique ID can only be obtained by one of the following logical nodes;

The logical node compares the information to be sent with the ID in the received information according to a preset rule;

The logical node sends the information to be sent carrying the comparison result to one of the child logical nodes, so that the child logical node uses the ID included in the received information and the ID and information in the information to be sent. Compare the results of the comparison;

When the logical node does not generate new information after processing the received information, the logical node compares the IDs in the received information, and carries the comparison result in the information sent to the server, so that the The server notifies the user after determining that the data information sent by the user is processed according to the preset rule.
The method of claim 1, wherein when the logical node is a root logical node, the root logical node receives and forwards data information input by the user, so that the root logical node inputs the user Data information is forwarded to a child logical node of the root logical node to process data information input by the user;

The ID information is not added to the data information input by the user.
The method of claim 1 wherein the method further comprises:

After the logical node receives the data information sent by the user, the logical node receives the same data information that is sent by the server and sent by the user, so that the server pairs the logic according to the preset rule. The information returned by the node carrying the comparison result is judged, and according to the judgment result, it is judged whether the data information sent by the user is processed.
The method according to claim 1, wherein the preset rule is that when there are even numbers of identical IDs, the even number of identical IDs are deleted;

When there are an odd number of identical IDs, only one of the odd number of identical IDs is retained.
A message notification method, characterized in that the method comprises:

The server receives information that is sent by the logical node and carries the comparison result;

The server determines, according to a preset rule, the received information carrying the comparison result according to a preset rule, and notifies the user after determining that the data information sent by the user is processed;

Wherein the logical node is a logical node that does not generate new information according to the received information;

The result of the comparison is that the logical node is generated by comparing the IDs contained in the received information.
The method of claim 5, wherein the predetermined rule is when there are even numbers of identical IDs, Then deleting the even number of identical IDs;

When there are an odd number of identical IDs, only one of the odd number of identical IDs is retained.
The method according to claim 6, wherein the server determines the received comparison result according to a preset rule within a preset time, and notifies the user after determining that the data information sent by the user is processed, specifically :

The server determines the number of the same ID in the received comparison result within a preset time. If the number of the same ID is an even number, the server deletes the even number of the same ID, and the ID does not exist in the server. When the data information sent by the user is processed, and the user is notified;

If the number of the same ID is an odd number, the server retains one of the odd number of identical IDs, and continues to receive the information carrying the comparison result, and when there is an ID in the server within a predetermined time, it indicates The data sent by the user is not processed, and the data information sent by the user is notified to be completed when the ID does not exist in the server before the end of the preset time.
The method of claim 5, wherein the method further comprises:

After the user sends the data information to the logical node, the server sends the same data information that is sent by the user to the logical node;

The server receives information returned by the logical node and carrying the comparison result;

The server determines the comparison result in the information according to a preset rule in a preset time, so that the server determines whether the data information sent by the user is processed according to the determination result.
A logical node device, the device comprising:

Adding a module, configured to add a unique ID for each information to be sent, wherein the logical node sends each information carrying a unique ID can only be obtained by one of the following logical nodes;

a comparison module, configured to compare the information to be sent and the ID in the received information according to a preset rule;

a first sending module, configured to send, to the one of the child logical nodes, the information to be sent that carries the comparison result, so that the child logical node uses the ID included in the received information and the information to be sent The ID is compared with the comparison result;

a second sending module, when the logical node does not generate new information after processing the received information, the logical node compares the IDs in the received information, and carries the comparison result in the information sent to the server So that the server notifies the user after the processing of the data information sent by the user is determined according to the preset rule.
The device according to claim 9, wherein when said logical node is a root logical node, said The root logical node receives and forwards the data information input by the user, so that the root logical node forwards the data information input by the user to the child logical node of the root logical node to process the data information input by the user;

The ID information is not added to the data information input by the user.
The device of claim 9, wherein the device further comprises:

Receiving, after receiving the data information sent by the user, the receiving module is configured to receive the same data information that is sent by the server and sent by the user, so that the server is configured according to the preset rule. The information returned by the logical node carrying the comparison result is judged, and according to the judgment result, it is judged whether the data information sent by the user is processed.
The device according to claim 9, wherein the preset rule is that when there are even numbers of identical IDs, the even number of identical IDs are deleted;

When there are an odd number of identical IDs, only one of the odd number of identical IDs is retained.
A server device, the device comprising:

a first receiving module, configured to receive, by the logical node, the received information and the information to be sent that carries the comparison result;

a first determining module, configured to determine, according to a preset rule, the received information carrying the comparison result according to a preset rule;

a notification module, configured to notify the user after determining that the data information sent by the user is processed;

Wherein the logical node is a logical node that does not generate new information according to the received information;

The result of the comparison is that the logical node is generated by comparing the IDs contained in the received information.
The device according to claim 13, wherein the preset rule is that when there are even numbers of identical IDs, the even number of identical IDs are deleted;

When there are an odd number of identical IDs, only one of the odd number of identical IDs is retained.
The device of claim 14, wherein the first determining module is specifically configured to:

Determining, in a preset time, the number of the same IDs in the received comparison result. If the number of the same IDs is an even number, the server deletes the even number of identical IDs, and when there is no ID in the server, it indicates The data information sent by the user is processed;

If the number of the same ID is an odd number, the server retains one of the odd number of identical IDs, and continues to receive the information carrying the comparison result, and when there is an ID in the server within a predetermined time, it indicates The data sent by the user is not processed, and the data information sent by the user is notified to be completed when the ID does not exist in the server before the end of the preset time.
The device of claim 13, wherein the device further comprises:

a sending module, after the user sends the data information to the logical node, to send the same data information that is sent by the user to the logical node;

a second receiving module, configured to receive information that is returned by the logical node and that carries a comparison result;

The second determining module is configured to determine, according to the preset rule, the comparison result in the information according to the preset rule, so that the server determines, according to the determination result, whether the data information sent by the user is processed.