WO2012162969A1

WO2012162969A1 - Distributed queue message reading method, device and system

Info

Publication number: WO2012162969A1
Application number: PCT/CN2011/079241
Authority: WO
Inventors: 胡斐然; 刘先宁; 晏鹏
Original assignee: 华为技术有限公司
Priority date: 2011-09-01
Filing date: 2011-09-01
Publication date: 2012-12-06
Also published as: CN103503388A; CN103503388B

Abstract

A distributed queue message reading method, device and system. The method includes: receiving an acquisition message request sent from a client; acquiring a message corresponding to the acquisition message request from a distributed queue, with the message carrying an internal identifier ID; extracting the internal identifier ID carried in the message, and judging whether or not a distributed database has stored the internal identifier ID; and if it is yes, then the internal identifier ID stored in the distributed database carried in the message is deleted, and the internal identifier ID is deleted from the acquired message, and the message with the internal identifier ID deleted is sent to the client. The embodiments of the present invention can avoid a client reading the same message from a distributed queue, reducing the risk of message re-processing.

Description

Distributed queue message reading method, device and system

The present invention relates to the field of data processing, and in particular, to a distributed queue message reading method, device, and system.

Background technique

With the rise of cloud computing, distributed has also become a trend of software design and development, and one of the core of distributed software development is distributed queues. In distributed queues, in order to improve the reliability of distributed queues, distributed queues often store multiple copies of a message when storing a message; and when a message of a distributed queue is read by a client, distributed The queue needs to delete all copies of the message, so that the same message is not read and processed by the client, which affects the business process.

In a practical application, a distributed queue may have a large number of concurrent accesses. After a message of a distributed queue is read, it takes a period of time to delete all copies of the message. When the number of copies of the message is large, all copies of the message are deleted. The required period of time is also longer, during which time the client may read the same message, which increases the risk of message duplication.

Summary of the invention

In the above-mentioned deficiencies, the distributed queue message reading method, device, and system provided by the embodiments of the present invention can prevent the client from reading the same message from the distributed queue, and reduce the risk of repeated message processing.

A method for reading a distributed queue message, comprising:

Receiving a request for obtaining a message sent by the client;

Obtaining, by the distributed queue, a message corresponding to the obtained message request, where the message carries an internal identifier ID;

Extracting an internal identifier ID carried by the message, and determining whether the internal identifier ID is stored in the distributed database; if yes, deleting the internal identifier ID carried by the message stored in the distributed database, and obtaining the obtained The internal ID is deleted from the message, and a message that the internal ID is deleted is sent to the client.

Another method for reading distributed queue messages includes:

Receiving a request for obtaining a message sent by the client; Obtaining, by the distributed queue, a message corresponding to the obtained message request, where the message carries an internal identifier ID;

Extracting an internal identification ID of the message, and determining a corresponding distributed row-level lock according to the internal identification ID, and if the Value value stored in the distributed row-level lock indicates readable, the Value value is The indication is unreadable, and the internal identification ID is deleted from the obtained message, and a message that the internal identification ID is deleted is sent to the client.

A distributed queue message reading device, comprising:

a receiving unit, configured to receive a request for obtaining a message sent by the client;

An obtaining unit, configured to obtain, from the distributed queue, a message corresponding to the obtained message request, where the message carries an internal identifier ID;

a determining unit, configured to extract an internal identifier ID of the message, and determine whether the internal identifier ID is stored in the distributed database;

a control unit, configured to delete an internal identifier ID carried by the message stored in the distributed database, and delete the internal identifier ID from the obtained message, when the judgment result of the determining unit is YES, And sending a message that deletes the internal identifier ID to the client;

The distributed database is configured to store an internal identification ID of the message.

Another distributed queue message reading device includes:

a second receiving unit, configured to receive a request for acquiring a message sent by the client;

a second obtaining unit, configured to obtain, from the distributed queue, a message corresponding to the obtained message request, where the message carries an internal identifier ID;

An extracting unit, configured to extract an internal identifier ID of the message, and determine a corresponding distributed row level lock according to the internal identifier ID;

a identifying unit, configured to identify whether the Value value stored in the distributed row level lock is readable; and a second control unit, configured to: when the value value stored in the distributed row level lock is readable, The Value value is set to indicate that it is unreadable, and the internal identification ID is deleted from the obtained message, and a message that the internal identification ID is deleted is sent to the client.

A distributed queue message reading system, comprising a client and any of the above distributed queue message reading devices;

The client is configured to send a get message request to the distributed queue message reading device, to And receiving a message corresponding to the acquired message request sent by the distributed queue message reading device for performing service processing.

In the embodiment of the present invention, after receiving the request for obtaining the message sent by the client, the message corresponding to the request for obtaining the message may be obtained from the distributed queue and the internal identifier ID carried by the message may be extracted, if the distributed database stores The internal identifier ID deletes the internal identifier ID carried in the message stored in the distributed database, deletes the internal identifier ID from the message, and sends a message that deletes the internal identifier ID to the client. Based on this technical solution, when any client repeatedly reads a message that has already been read, even if the same message (ie, a copy of the message) exists in the distributed queue, the message has been deleted in the distributed database. The internal ID of the ID is not sent to the client for processing. This prevents the client from reading the same message from the distributed queue and reduces the risk of repeated message processing.

In the embodiment of the present invention, after receiving the request for obtaining a message sent by the client, the message corresponding to the request for obtaining the message may be obtained from the distributed queue, and the internal identifier ID carried by the message is extracted, and the internal identifier carried by the message is received. After the identifier ID determines the distributed row level lock, it identifies whether the Value value stored in the distributed row level lock indicates readability, and if the representation is readable, the Value value is set to indicate unreadable, and the content is deleted from the message. The internal ID is identified and a message with the internal ID removed is sent to the client. Based on this technical solution, when any client repeatedly reads a message that has already been read, even if the same message (ie, a copy of the message) exists in the distributed queue, the value of the value in the distributed row-level lock is The representation is unreadable and will not be sent to the client for processing, which prevents the client from reading the same message from the distributed queue, reducing the risk of message duplication.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present invention. Other drawings may also be obtained from those of ordinary skill in the art in view of the drawings.

1 is a schematic flowchart of a method for reading a distributed queue message according to an embodiment of the present invention; FIG. 2 is a schematic flowchart of a method for storing a distributed queue message included in the method shown in FIG. A modular schematic diagram of a distributed queue message storage method;

FIG. 4 is a schematic flowchart of another method for reading a distributed queue message according to an embodiment of the present invention; Figure

FIG. 5 is a schematic flow chart of still another method for reading a distributed queue message according to an embodiment of the present disclosure;

6 is a modular schematic diagram of another method for reading a distributed queue message according to an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of a distributed queue message reading device according to an embodiment of the present invention; FIG. 9 is a schematic structural diagram of a distributed queue message reading system according to an embodiment of the present invention; A flow diagram of another distributed queue message reading method provided;

11 is a schematic flowchart of a distributed queue message storage method included in the method shown in FIG. 10; FIG. 12 is a schematic diagram of a modular diagram of the distributed queue message storage method shown in FIG.

FIG. 13 is a schematic flow chart of another method for reading a distributed queue message according to an embodiment of the present disclosure;

FIG. 14 is a schematic flow chart of still another method for reading a distributed queue message according to an embodiment of the present disclosure;

FIG. 15 is a schematic diagram of a DHT according to an embodiment of the present invention;

16 and FIG. 17 are schematic structural diagrams of a distributed queue message reading device according to an embodiment of the present invention;

FIG. 18 is a schematic structural diagram of another distributed queue message reading system according to an embodiment of the present invention.

detailed description

BRIEF DESCRIPTION OF THE DRAWINGS The technical solutions in the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without creative work are within the scope of the present invention.

The method, device and system for reading distributed queue messages provided by the embodiments of the present invention can prevent the client from reading the same message from the distributed queue, and reduce the risk of repeated processing of the message. The following is a detailed description.

Referring to FIG. 1, FIG. 1 is a schematic diagram of a method for reading a distributed queue message according to an embodiment of the present invention. Schematic diagram of the process. As shown in FIG. 1, the method may include the following steps:

101. Receive a request for obtaining a message sent by a client.

In one embodiment, the get message request sent by the client may be received by the deduplication proxy server. Generally, the obtaining message request sent by the client carries the indication information (for example, the message self identifier), and the indication information is used to indicate the message that the client wants to obtain, which is common knowledge known to those skilled in the art, and is implemented by the present invention. The case is not described in detail.

102. Obtain a message corresponding to the foregoing obtaining a message request from a distributed queue, where the message carries an internal identifier ID;

In an embodiment, the obtaining the message corresponding to the foregoing obtaining the message request from the distributed queue may be:

The deduplication proxy server forwards the get message request sent by the client to the distributed queue, and receives the message corresponding to the above acquired message request sent by the distributed queue.

The distributed queue may send the message corresponding to the foregoing acquisition message request to the deduplication proxy server according to the indication information (for example, the message self identifier) carried in the acquisition message request sent by the client.

103. Extract an internal identifier ID of the message, and determine whether the internal identifier ID is stored in the distributed database; if yes, execute step 104; if no, execute step 105.

In an embodiment, after the deduplication proxy server extracts the internal identifier ID of the message, the extracted internal identifier ID may be compared with the internal identifier ID stored in the distributed database, and if the message is found to be stored in the distributed database, The internal identification ID (that is, the extracted internal identification ID is the same as one of the internal identification IDs stored in the distributed database), indicating that the message currently obtained from the distributed queue corresponding to the above-mentioned acquisition message request has not been used by the client. After reading, step 104 can be performed; otherwise, if the internal identifier ID carried by the message is not stored in the distributed database (that is, the extracted internal identifier ID is different from the internal identifier ID stored in the distributed database), then The message corresponding to the above-mentioned acquisition message request that has been obtained from the distributed queue has been read by the client, and step 105 can be performed.

104. Delete the internal identifier ID carried in the message stored in the distributed database, delete the internal identifier ID from the obtained message, and send the message that deletes the internal identifier ID to the client. In an embodiment, if it is found that the internal identifier ID extracted from the foregoing message is the same as the one of the internal identifier IDs stored in the distributed database, the message currently obtained from the distributed queue corresponding to the foregoing acquisition message request is not yet obtained. It has been read by the client. At this time, the proxy server can be deleted to delete the internal identification ID carried in the above message stored in the distributed database. In this way, even if the same message (that is, a copy of the message) exists in the distributed queue, the internal identification ID carried in the message has been deleted in the distributed database, and is not sent to the client for processing, thereby preventing The client reads the same message from the distributed queue, reducing the risk of message duplication.

In addition, after the proxy server deletes the internal identifier ID carried in the message stored in the distributed database, the internal identifier ID carried in the message may be deleted, and the message that the internal identifier ID has been deleted is sent to the client, so that the message is sent to the client. The client can perform business processing on the message.

105. Send an error response to the client.

In an embodiment, if it is found that the internal identifier ID extracted from the foregoing message is different from the internal identifier ID stored in the distributed database, the message corresponding to the foregoing acquisition message request currently obtained from the distributed queue has been After the client reads it, the deduplication proxy server can send an error response null to the client, preventing the client from reading the same message from the distributed queue for repeated processing. After the distributed queue sends the message corresponding to the above-mentioned acquisition message request to the deduplication proxy server, the distributed queue can delete all the copies of the message corresponding to the above-mentioned acquisition message request, thereby preventing the client from reading from the distributed queue as much as possible. Repeated processing of the same message also keeps the distributed queue as free as possible for storing other messages and their copies.

In an embodiment, corresponding to the above-mentioned reading process, the foregoing method provided by the embodiment of the present invention may further include a message storing process before the step 101, and the storing process may include the following steps:

A. The deduplication proxy server receives the input message from the client.

B. The deduplication proxy server generates an internal identification ID for the client input message, and adds the internal identification ID to the client input message.

In an embodiment, the deduplication proxy server may generate an internal identification ID for the client input message by using a random generation manner or a hash HASH algorithm, where the deduplication proxy server generates different internal identification IDs for each client input message. . C. The deduplication proxy server stores the message after joining the internal identification ID to the distributed queue; wherein the distributed queue also stores at least one copy of the message after joining the internal identification ID.

D. The deduplication proxy server stores the internal ID ID generated for the client input message to the distributed database.

Through the above steps A~B, the deduplication proxy server can generate an internal identification ID for the client input message, join the client input message, and store the message after adding the internal identification ID to the distributed queue, and will be the client. The internal identification ID generated by the end input message is stored in the distributed database, so that the deduplication proxy server can perform the distributed queue message reading according to the method shown in FIG. 1 to prevent the client from reading the same message from the distributed queue. Reduce the risk of message duplication.

The foregoing steps (the sequence between the steps D and the step D may be reversed, or the steps C and D may be performed at the same time, which are not limited in the embodiment of the present invention.

In the embodiment of the present invention, the message stored in the distributed queue carries the internal identifier ID. After receiving the request for obtaining the message sent by the client, the message corresponding to the request for obtaining the message may be obtained from the distributed queue, and the distribution is determined. Whether the internal identifier ID is stored in the database, if the internal identifier ID is the same as the internal identifier ID stored in the distributed database, the internal identifier carried in the message stored in the distributed data may be deleted. The ID, and the internal identification ID carried in the above message are deleted, and the message that the internal identification ID is deleted is sent to the client. Based on the above technical solution, even if the same message (ie, a copy of the message) exists in the distributed queue, the internal identification ID carried in the message is deleted in the distributed database, and is not sent to the client for processing. This prevents the client from reading the same message from the distributed queue, reducing the risk of message duplication.

Referring to FIG. 2, FIG. 2 is a schematic flowchart of a distributed queue message storage method included in the method shown in FIG. 1. As shown in FIG. 2, the distributed queue message storage method includes the following steps:

201. The deduplication proxy server receives the message M input by the client.

202. The deduplication proxy server uses the HASH algorithm to generate an internal identification ID for the message M, and adds the generated internal identification ID to the message M.

203. The deduplication proxy server stores the internal identification ID generated for the message M to the distributed database. 204. The deduplication proxy server receives the storage confirmation message returned by the distributed database.

205, was added to duplicate proxy server stores the message M to the distributed internal identification ID team bad ¹ J; wherein the distributed queue also stores at least one message M is added after the replica internal identification ID.

In the method shown in FIG. 2, the foregoing step 205 may be performed first, and then the step 203 and the step 204 are performed, which are not limited in the embodiment of the present invention.

On the basis of the method shown in FIG. 2, the deduplication proxy server can perform distributed queue message reading according to the method shown in FIG. 1 to prevent the client from reading the same message from the distributed queue, and reducing the repeated processing of the message. risk.

Please refer to FIG. 3. FIG. 3 is a modular schematic diagram of the distributed queue message storage method shown in FIG. 2. As shown in FIG. 3, the process of the distributed queue message storage method is as follows:

(1), client A sends a message to the deduplication proxy server;

(2), the deduplication proxy server generates an internal identification ID for the message sent by the client A, and stores the internal identification ID in the distributed database;

(3) The deduplication proxy server sends the message sent by the client A after joining the internal identifier ID to the distributed queue storage; wherein, the distributed queue further stores at least one message sent by the client A after joining the internal identifier ID. A copy.

On the basis of the module diagram shown in FIG. 3, the deduplication proxy server can perform distributed queue message reading according to the method shown in FIG. 1 to prevent the client from reading the same message from the distributed queue, and reducing the message repetition processing. risks of.

Referring to FIG. 4, FIG. 4 is a schematic flowchart diagram of another method for reading a distributed queue message according to an embodiment of the present invention. As shown in FIG. 4, the distributed queue message reading method includes the following steps:

401. The deduplication proxy server receives the Get message request Get sent by the client.

402. The deduplication proxy server forwards the Get message request Get sent by the client to the distributed queue.

403. The de-duplication proxy server receives the message M corresponding to the foregoing get message request Get sent by the distributed queue, where the message M carries an internal identifier ID.

404. The deduplication proxy server extracts the internal identifier ID of the message M.

405. The deduplication proxy server determines, according to the extracted internal identifier ID, that the internal identifier ID is not stored in the distributed database (ie, the extracted internal identifier ID and the internal storage stored in the distributed data) The IDs are not the same).

406. The deduplication proxy server sends an error response null to the client.

In the method shown in FIG. 4, the deduplication proxy server determines that the internal identification ID is not stored in the distributed database (ie, the extracted internal identification ID is different from the internal identification ID stored in the distributed data), indicating that the current a message sent by the distributed queue corresponding to the above obtained message request Get

M has been read by the client, and the deduplication proxy server can send an error response null to the client, preventing the client from reading the same message from the distributed queue for repeated processing.

Referring to FIG. 5, FIG. 5 is a schematic flowchart diagram of still another method for reading a distributed queue message according to an embodiment of the present invention. As shown in FIG. 5, the distributed queue message reading method includes the following steps:

501. The deduplication proxy server receives the Get message request Get sent by the client.

502. The deduplication proxy server forwards the Get message request Get sent by the client to the distributed queue.

503. The de-duplication proxy server receives the message M corresponding to the foregoing get message request Get sent by the distributed queue, where the message M carries an internal identifier ID.

The distributed queue may delete all the copies of the message M stored in the distributed queue after the message M corresponding to the above-mentioned acquisition message request Get is sent to the de-duplication proxy server.

504. The deduplication proxy server extracts the internal identifier ID of the message M.

505. The deduplication proxy server determines, according to the extracted internal identifier ID, that the internal identifier ID is stored in the distributed database (that is, the extracted internal identifier ID is the same as one of the internal identifier IDs stored in the distributed data).

506. The deduplication proxy server deletes the internal identifier ID carried by the foregoing message stored in the distributed database.

507. The deduplication proxy server deletes the internal identifier ID carried in the message, and sends a message that deletes the internal identifier ID to the client.

In the method shown in Figure 5, even if the same message (that is, a copy of the message) exists in the distributed queue, the internal identification ID carried in the message is deleted in the distributed database, and is not sent to the client. The processing is performed to prevent the client from reading the same message from the distributed queue and reducing the risk of message duplication.

In one embodiment, a distributed queue may have a large number of concurrent client accesses, in which case By applying the foregoing method provided by the embodiment of the present invention, the client can be prevented from reading the same message from the distributed queue, and the risk of repeated processing of the message is reduced. The details are described below by way of specific examples.

Referring to FIG. 6, FIG. 6 is a schematic diagram of a modularization of another method for reading a distributed queue message according to an embodiment of the present invention. As shown in FIG. 6, the clients A, B, and C respectively access the distributed queue through their respective deduplication proxy servers, that is, the client A sends a message request to the distributed queue through its corresponding deduplication proxy server; End B sends a get message request to the distributed queue through its corresponding deduplication proxy server; client C sends a get message request to the distributed queue through its corresponding deduplication proxy server, thereby constituting concurrent access to the distributed queue.

Suppose that clients A, B, and C need to read the message M stored in the distributed queue, after the distributed queue receives the request for obtaining a message sent by the client A, B, and C through the corresponding deduplication proxy server, The get message requests sent by the clients A, B, and C are processed one by one. For example, the information request is processed, that is, the distributed queue may send the message M corresponding to the acquisition message request sent by the client A to the client according to the indication information (for example, the message self identifier) carried in the acquisition message request sent by the client A. The de-duplication proxy server corresponding to the terminal A; after receiving the message M corresponding to the request for obtaining the message sent by the client A, the de-duplication proxy server corresponding to the client A may extract the internal identifier ID of the message M, And determining whether the internal identifier ID is stored in the distributed database, and if yes, deleting the internal identifier ID carried by the message M stored in the distributed database, deleting the internal identifier ID carried by the message M, and deleting the The message of the internal identification ID is sent to client A.

After the distributed queue sends the message M corresponding to the get message request sent by the client A to the deduplication proxy server corresponding to the client A, the distributed queue can start all the copy operations of deleting the message M. Since the distributed queue deletes all the copies of the message M, it takes a period of time, and during this time, the distributed queue will continue to process the request messages of the client B and C, so the message M is deleted in the distributed queue. Before all copies of the distributed queue, it is still possible for the distributed queue to send a copy of the message M to the client B or the deduplication proxy server corresponding to the client C. Further, assuming that the distributed queue sends a copy of the message M to the deduplication proxy server corresponding to the client B, the deduplication proxy server corresponding to the client B receives the acquisition message request sent by the distributed queue and sent by the client B. After the corresponding M message (ie, copy), the message M can be extracted. The internal ID is determined, and it is determined that the internal identification ID is not stored in the distributed database (deleted by the deduplication proxy server corresponding to the client A), and the deduplication proxy server corresponding to the client B sends an error response null to the client B. Therefore, it can be avoided that client B reads the same message M from the distributed queue for repeated processing. In the case of concurrent access, it is also possible to prevent the client from reading the same message from the distributed queue, reducing the risk of repeated processing of the message.

In the embodiment of the present invention, the deduplication proxy server is transparent to the client without losing the distributed advantage, and the client still uses the distributed queue as in the prior art, and the distributed queue does not need to process the client to read. The repetitive situation reduces the difficulty of the client using the distributed queue, and the CPU is shown in FIG. 7. FIG. 7 is a schematic structural diagram of a distributed queue message reading device according to an embodiment of the present invention. As shown in FIG. 7, the device may include:

The receiving unit 701 is configured to receive a request for acquiring a message sent by the client.

The obtaining unit 702 is configured to obtain, from the distributed queue, a message corresponding to the foregoing get message request, where the message carries an internal identifier ID;

The determining unit 703 is configured to determine, according to the extracted internal identifier ID, whether the internal identifier ID is stored in the distributed row database 705;

The control unit 704 is configured to: when the determination result of the determining unit 703 is YES, delete the internal identification ID carried in the message stored in the distributed database 705, and delete the internal identification ID from the obtained message, and delete the The message of the internal identifier ID is sent to the client; or when the judgment result of the determining unit 703 is negative, the error response is sent to the client;

a distributed database 705, configured to store an internal identifier ID of the foregoing message;

Referring to FIG. 8, FIG. 8 is a schematic structural diagram of another distributed queue message reading device according to an embodiment of the present invention. As shown in FIG. 8, the device further includes a generating unit 706 and a storage unit 707 on the basis of the device shown in FIG. 7. In the device shown in FIG. 8, the receiving unit 701 is further configured to receive a client input message, and correspondingly, the generating unit 706 is configured to generate an internal identifier ID for the client input message, and join the client input message; The client input message after the internal ID is added to the distributed queue; wherein the distributed queue further stores at least one join A copy of the client input message after the ID is identified; and an internal identification ID generated for the client input message is stored to the distributed database 705.

As an optional implementation manner, the generating unit 706 in the device shown in FIG. 8 is specifically configured to generate an internal identifier ID for the client input message by using a random generation manner or a hash HASH algorithm, and join the client input message; The internal identification ID generated by each client input message is different from each other.

As an optional implementation, in the device shown in FIG. 7 and FIG. 8 , the obtaining unit 702 is specifically configured to forward the acquisition message request sent by the client to the distributed queue, and receive the acquired message sent by the distributed queue. The corresponding message is requested, and the message carries an internal identification ID.

In one embodiment, after the distributed queue sends a message corresponding to the above-mentioned acquisition message request to the acquisition unit 702, the distributed queue is also used for all copies of the message.

In the embodiment of the present invention, the message stored in the distributed queue carries the internal identifier ID. After the receiving unit 701 receives the request for obtaining the message sent by the client, the obtaining unit 702 may obtain the request corresponding to the obtained message from the distributed queue. The message determining unit 703 can determine whether the internal identifier ID is stored in the distributed database based on the extracted internal identifier ID. If yes, the control unit 704 deletes the internal identifier ID carried in the message stored in the distributed database 705. And deleting the internal identification ID carried by the message, and sending a message that deletes the internal identification ID to the client. Based on the foregoing technical solution, even if the same message (ie, a copy of the message) exists in the distributed queue, the internal identification ID carried by the message is deleted in the distributed database, and is not sent to the client for processing. It can prevent the client from reading the same message from the distributed queue, reducing the risk of message duplication.

Referring to FIG. 9, FIG. 9 is a schematic structural diagram of a distributed queue message reading system according to an embodiment of the present invention. As shown in FIG. 9, the system may include a client 901 and a distributed queue message reading device 902; wherein the structure and functions of the distributed queue message reading device 902 are the same as those of the distributed queue shown in FIG. 7 or 8. The message reading device is the same, and is not described in the embodiment of the present invention.

The client 901 is configured to send a get message request to the distributed queue message reading device 902, and receive a message corresponding to the get message request sent by the distributed queue message reading device 902 for service processing.

The distributed queue message reading system provided by the embodiment of the present invention can avoid the client 901 from dividing The same message is read in the layout queue, reducing the risk of message duplication.

In the method, device, and system described in the foregoing embodiments, the internal identifier ID generated for the client input message is stored in the distributed database, thereby preventing the client from reading the same message from the distributed queue, thereby reducing The risk of repeated processing of messages. As another optional implementation manner, the internal identification ID generated for the client input message may also be stored in the distributed row level lock, which also prevents the client from reading the same message from the distributed queue and reducing the message. The risk of repeated processing. The details are described below.

Referring to FIG. 10, FIG. 10 is a schematic flowchart diagram of another method for reading a distributed queue message according to an embodiment of the present invention. As shown in FIG. 10, the method may include the following steps:

The steps 1001 and 1002 are the same as the steps 101 and 102 in the foregoing embodiment, and the embodiments of the present invention are not repeated herein.

1003. Extract an internal identifier ID of the message, and determine a corresponding distributed row level lock according to the internal identifier ID.

In the embodiment of the present invention, a key value key-Value value pair is stored in each distributed row level lock. If the key value key stored in a distributed row level lock is the internal ID, the corresponding correspondence may be determined. Distributed row-level locks. The distributed row level lock will be described in detail later in the embodiment of the present invention.

1004. Determine whether the value of the value stored in the determined distributed row-level lock indicates that the value is readable. If yes, go to step 1005. If no, go to step 1006.

In one embodiment, the value of Value = the internal ID of the above may be readable, and the value of Value is unreadable when other values are used; or, when the value is non-zero, the value is readable, and the value of Value is zero. The embodiment of the present invention is not limited.

In an embodiment, after the de-duplication proxy server extracts the internal identifier ID of the message, the corresponding distributed row-level lock may be determined according to the internal identifier ID, and if the value of the value stored in the distributed row-level lock is found to be readable, It is indicated that the message currently obtained from the distributed queue corresponding to the above obtained message request has not been read by the client, and then step 1005 may be performed; otherwise, if the value of the value stored in the distributed row level lock is found to be unreadable The message corresponding to the above-mentioned acquisition message request that has been obtained from the distributed queue has been read by the client, and then step 1006 can be performed. 1005. Set the Value value to indicate that it is unreadable, and delete the internal identifier ID from the obtained message, and send a message that deletes the internal identifier ID to the client.

In an embodiment, if the Value value stored in the distributed row-level lock is found to be readable, it indicates that the message currently obtained from the distributed queue corresponding to the above-mentioned acquisition message request has not been read by the client. The deduplication proxy server can set the Value value stored in the distributed row-level lock to indicate that it is unreadable, so that even if the same message (ie, a copy of the message) exists in the distributed queue, it will be stored in the distributed row-level lock. The Value value indicates that it is unreadable and will not be sent to the client for processing, thus avoiding the client reading the same message from the distributed queue and reducing the risk of message duplication.

In addition, after the deduplication proxy server sets the value of the value stored in the distributed row-level lock to indicate that it is unreadable, the internal identifier ID carried by the message may be deleted, and the message of the deleted internal identifier ID is sent to the client, so that the client The terminal can perform business processing on the message.

1006. Send an error response to the client.

In an embodiment, if the Value value stored in the distributed row-level lock is found to be unreadable, it indicates that the message currently obtained from the distributed queue corresponding to the above-mentioned acquisition message request has been read by the client, and the proxy is deduplicated. The server can send an error response null to the client, preventing the client from reading the same message from the distributed queue for repeated processing.

In one embodiment, the method shown in FIG. 10 may further include the following steps:

After the distributed queue sends the message corresponding to the above-mentioned acquisition message request to the deduplication proxy server, the distributed queue can delete all the copies of the message corresponding to the above-mentioned acquisition message request, thereby preventing the client from reading from the distributed queue as much as possible. Repeated processing of the same message also keeps the distributed queue as free as possible for storing other messages and their copies.

In one embodiment, corresponding to the above reading process, the method shown in FIG. 10 may further include a storage process of the message before the step 1001, and the storing process may include the following steps:

A. The deduplication proxy server receives the input message from the client.

In one embodiment, the deduplication proxy server may generate an internal identification ID for the client input message by using a random generation method or a hash HASH algorithm, where the deduplication proxy server is each The internal IDs generated by the client input messages are different from each other.

C. The deduplication proxy server stores the message after joining the internal ID to the distributed queue; wherein the distributed queue also stores at least one copy of the message after joining the internal ID.

D. The internal identifier ID generated by the client input message is stored as a key value key in the distributed row level lock, and the key value key matches a Value value (ie, a key-value value pair is stored in the distributed row level lock) ), the Value value indicates readability.

Through the above steps A~B, the deduplication proxy server can generate an internal identification ID for the client input message, join the client input message, and store the message after adding the internal identification ID to the distributed queue, and will be the client. The internal identification ID generated by the end input message is stored to the distributed row level lock, so that the deduplication proxy server can perform the distributed queue message reading according to the method shown in FIG. 10, so as to prevent the client from reading the same from the distributed queue. Messages, reducing the risk of message duplication.

In the method shown in FIG. 10, after receiving the request for obtaining a message sent by the client, the deduplication proxy server may obtain a message corresponding to the acquired message request from the distributed queue and extract an internal identification ID carried by the message, and After determining the distributed row level lock according to the internal identifier ID carried by the message, identifying whether the Value value stored in the distributed row level lock indicates readability, if readable, setting the Value value to indicate unreadable, and The internal ID ID carried in the message is deleted, and a message for deleting the internal ID is sent to the client. Based on this technical solution, even if the same message (ie, a copy of the message) exists in the distributed queue, the Value value in the distributed row-level lock is not readable, and is not sent to the client for processing, thereby avoiding The client reads the same message from the distributed queue, reducing the risk of repeated processing of the message.

Referring to FIG. 11, FIG. 11 is a schematic flowchart of a distributed queue message storage method included in the method shown in FIG. As shown in FIG. 11, the distributed queue message storage method includes the following steps:

1101. The deduplication proxy server receives the message M input by the client.

1102. The deduplication proxy server uses the HASH algorithm to generate an internal identification ID for the message M, and adds the generated internal identification ID to the message M.

1103. Deduplicating the internal identification ID generated by the proxy server for the message M, the internal identifier The ID is stored as a key value key in a distributed row-level lock. The key value key matches a Value value (ie, a key-value value pair is stored in the distributed row-level lock), and the Value value indicates readability.

1104. The deduplication proxy server receives the storage confirmation message returned by the distributed row level lock.

1105. The deduplication proxy server stores the message M after adding the internal identifier ID to the distributed team ;l; wherein the distributed queue further stores at least one copy of the message M after joining the internal identifier ID.

In the method shown in FIG. 11, the foregoing step 1105 may be performed first, and then the step 1103 and the step 1104 are performed, which are not limited in the embodiment of the present invention.

On the basis of the method shown in FIG. 11, the deduplication proxy server can perform distributed queue message reading according to the method shown in FIG. 10, so as to prevent the client from reading the same message from the distributed queue, and reducing the message repetitive processing. risk.

Referring to FIG. 12, FIG. 12 is a modular schematic diagram of the distributed queue message storage method shown in FIG. As shown in FIG. 12, the process of the distributed queue message storage method is as follows:

(1), client A sends a message to the deduplication proxy server;

(2) The deduplication proxy server generates an internal identification ID for the message sent by the client A, and stores the internal identification ID as a key value key in the distributed row level lock, and the key value key matches a Value value ( That is, a key-value pair is stored in the distributed row-level lock, and the Value value is readable;

Based on the block diagram shown in FIG. 12, the deduplication proxy server can perform distributed queue message reading according to the method shown in FIG. 10, so as to prevent the client from reading the same message from the distributed queue, and the message is repeated. The risk of processing.

Referring to FIG. 13, FIG. 13 is a schematic flowchart diagram of another method for reading a distributed queue message according to an embodiment of the present invention. As shown in FIG. 13, the distributed queue message reading method includes the following steps: 1301. The deduplication proxy server receives the Get message request Get sent by the client.

1302. The deduplication proxy server forwards the Get message request Get sent by the client to the distributed queue.

1303. The deduplication proxy server receives the request for obtaining the message from the distributed queue. The message M corresponding to Get, wherein the message Μ carries an internal identification ID.

1304. The deduplication proxy server extracts the internal identifier ID of the message M.

1305. The deduplication proxy server determines a corresponding distributed row level lock according to the internal identifier ID, and determines that the value value stored in the distributed row level lock indicates that the value is unreadable;

1306. The deduplication proxy server sends an error response null to the client.

In the method shown in FIG. 13, when the deduplication proxy server determines that the Value value stored in the distributed row-level lock indicates that the value is unreadable, the message M corresponding to the acquired message request Get currently obtained from the distributed queue is indicated. It has been read by the client. The deduplication proxy server can send an error response null to the client, preventing the client from reading the same message from the distributed queue for repeated processing.

Referring to FIG. 14, FIG. 14 is a schematic flowchart diagram of still another method for reading a distributed queue message according to an embodiment of the present invention. As shown in FIG. 14, the distributed queue message reading method includes the following steps: 1401. The deduplication proxy server receives the Get message request Get sent by the client.

1402. The deduplication proxy server forwards the Get message request Get sent by the client to the distributed queue.

1403. The de-duplication proxy server receives the message M corresponding to the foregoing get message request Get sent by the distributed queue, where the message M carries an internal identifier ID.

1404. The deduplication proxy server extracts an internal identifier ID of the message M.

1405. The deduplication proxy server determines a corresponding distributed row level lock according to the internal identifier ID, and determines that the value value stored in the distributed row level lock is readable.

1406. The deduplication proxy server sets the value of the value stored in the distributed row level lock to indicate that it is unreadable.

1407. The deduplication proxy server deletes the internal identifier ID carried in the message, and sends a message that deletes the internal identifier ID to the client.

In the method shown in Figure 14, even if the same message (ie, a copy of the message) exists in the distributed queue, the Value value stored in the distributed row-level lock is unreadable and will not be sent to the client. Processing, thus avoiding the client reading the same message from the distributed queue, reducing The risk of repeated processing of messages. In the case of the request, the client can also prevent the client from reading the same message from the distributed queue, reducing the risk of message duplication.

In the embodiment of the present invention, the design idea of the above distributed row level lock may be as follows:

1. Since the message sent by the client carries its own identifier (belonging to the prior art), it can be based on the self-identity carried by the message, and generates an internal identifier ID through the hash algorithm (because of the principle of the hash algorithm, different messages) It will correspond to a different internal ID).

2. The generated internal ID is stored as a Key in the distributed row-level lock to form a key-Value value pair. The Value value can be any non-null value (for example, Key=internal ID, value=internal ID) , means readable.

3. The distributed row-level lock storing the key-value pair may be regarded as a KV storage engine implemented by a distributed hash table (DHT) algorithm. The principle of storing data by the KV storage engine is to first fix the space for storing data and form a 0~2 ^Λ 32 ring (as shown in Figure 15), and then generate a value through the Hash algorithm. The value forms a host node in the data space, and then when the Key-Value value pair is stored, the KV storage engine obtains a hash value through the Hash algorithm, and the hash value is represented as a value node on the storage space ring. If the value node does not fall on a host node, the KV storage engine will find the first (or N if N backups are needed) along the storage space ring. The host node stores the key-value pair host.

4. When the deduplication proxy server determines the corresponding distributed row level lock (ie, the KV storage engine) according to the extracted internal identifier ID, and determines the internal identifier stored in the determined distributed row level lock (ie, the KV storage engine). When the value corresponding to the ID (ie Key) indicates that it is readable, this process is also called the acquisition lock process; if the Value value indicates readable, it indicates that the current acquisition from the distributed queue corresponds to the above-mentioned acquisition message request. The message has not been read by the client and can be read by the client. Among them, the client can read the message to indicate that the lock is acquired.

5. When the proxy server deletes and sends the internal ID ID carried by the above message to the client, the value of the value stored in the distributed row-level lock (that is, the KV storage engine) can be set to indicate that it is unreadable.

(eg value value is set to null, or zero), so that other clients can no longer read this message, this The process is also known as the locking process.

Step 3 above describes the DHT implementation principle of the KV storage engine. The Key-Value value is uniformly distributed on a certain host (or a few hosts), and distributed throughout the distributed external service. The access of the row-level locks will also be distributed to different hosts, which will not generate massive access to one host, and avoid distributed row-level locks as a new performance bottleneck.

Referring to FIG. 16, FIG. 16 is a schematic structural diagram of another distributed queue message reading device according to an embodiment of the present invention. As shown in FIG. 16, the device may include:

The second receiving unit 1601 is configured to receive a request for acquiring a message sent by the client.

The second obtaining unit 1602 is configured to obtain, from the distributed queue, a message corresponding to the foregoing acquiring message request, where the message carries an internal identifier ID;

The extracting unit 1603 is configured to extract an internal identifier ID of the message, and determine a corresponding distributed row level lock 1606 according to the internal identifier ID;

The identifying unit 1604 is configured to identify whether the value stored in the distributed row level lock 1606 is readable;

a second control unit 1605, configured to: when the Value value representation stored in the distributed row level lock 1606 is readable, set the Value value to indicate unreadable, and delete the internal identification ID from the obtained message, and A message with the above internal ID is deleted and sent to the client.

Referring to FIG. 17, FIG. 17 is a schematic structural diagram of another distributed queue message reading device according to an embodiment of the present invention. As shown in FIG. 17, the device further includes a second generating unit 1607 and a second storage unit 1608 on the basis of the device shown in FIG. 16. In the device shown in FIG. 17, the second receiving unit 1601 is further configured to receive a client input message; correspondingly, the second generating unit 1607 is configured to generate an internal identifier ID for the client input message, and join the client input message; The second storage unit 1608 is configured to store the client input message after joining the internal identifier ID to the distributed queue; wherein the distributed queue further stores at least one copy of the client input message after joining the internal identifier ID; and the second The storage unit 1608 stores the internal identification ID generated for the client input message as a key value key into the distributed row level lock 1606, the key value key matching a Value value, the Value value being readable.

In one embodiment, when the key value key of the distributed row level lock stored by the extracting unit 1603 is the internal ID, the corresponding distributed row level lock 1606 can be determined. Referring to FIG. 18, FIG. 18 is a schematic structural diagram of a distributed queue message reading system according to an embodiment of the present invention. As shown in FIG. 18, the system may include a client 1801 and a distributed queue message reading device 1802; wherein the structure and functions of the distributed queue message reading device 1802 are the same as the distributed queue shown in FIG. 16 or FIG. The message reading device is the same, and is not described in the embodiment of the present invention.

The client 1801 is configured to send a get message request to the distributed queue message reading device.

1802. Receive a message corresponding to the request for obtaining a message sent by the distributed queue message reading device 1802 for performing service processing.

Based on the above technical solution, even if the same message (ie, a copy of the message) exists in the distributed queue, the value of the value in the distributed row-level lock is unreadable, and is not sent to the client for processing, thereby avoiding The client reads the same message from the distributed queue, reducing the risk of message duplication.

A person skilled in the art can understand that all or part of the steps of implementing the above method embodiments may be completed by using hardware related to the program instructions. The foregoing program may be stored in a computer readable storage medium, and the program is executed when executed. The foregoing storage medium includes: a Read-Only Memory (ROM), a Random Access Memory (RAM), a disk or an optical disk, and the like, which can store program codes. Medium.

The foregoing description of the method, apparatus, and system for reading a distributed queue message provided by the embodiments of the present invention are merely used to help understand the method and core idea of the present invention. Meanwhile, general techniques in the art are provided. In view of the above, the description of the present invention is not limited to the scope of the present invention.

Claims

Rights request

A method for reading a distributed queue message, comprising:

Receiving a request for obtaining a message sent by the client;

2. The method according to claim 1, wherein the method further comprises:

After the message corresponding to the obtained message request is obtained from the distributed queue, all copies of the message corresponding to the acquired message request are deleted in the distributed queue.

The method according to claim 1 or 2, wherein the method further comprises: receiving an input message of the client;

Generating an internal identification ID for the client input message, and adding the internal identification ID to the client input message;

And storing the client input message after adding the internal identifier ID to the distributed queue; the distributed queue further storing at least one copy of the client input message after adding the internal identifier ID;

The internal identification ID generated for the client input message is stored to the distributed database.

The method according to claim 3, wherein the generating an internal identification ID for the client input message comprises:

The internal identification ID is generated by using the random generation method or the hash HASH algorithm for the client input message, wherein the internal identification ID generated for each client input message is different from each other.

The method according to any one of claims 1 to 4, wherein the obtaining, by the distributed queue, a message corresponding to the request for obtaining a message comprises:

Forwarding the get message request sent by the client to the distributed queue;

Receiving a message corresponding to the acquired message request sent by the distributed queue.

A method for reading a distributed queue message, comprising: Receiving a request for obtaining a message sent by the client;

The method according to claim 6, wherein the method further comprises:

Receiving input messages from the client;

Generating an internal identification ID for the client input message, and joining the client input message; storing the client input message after joining the internal identification ID to the distributed queue; the distributed queue further storing at least a copy of the client input message after the internal ID is added;

The internal identification ID generated for the client input message is stored as a key value key in the distributed row level lock, the key value key is matched with a Value value, and the Value value indicates readable.

The method according to claim 7, wherein the determining the corresponding distributed row level lock according to the internal identifier ID comprises:

And determining, according to the internal identifier ID, a corresponding distributed row level lock, where the corresponding distributed row level lock stores a key value key=the internal identifier ID.

A distributed queue message reading device, comprising:

a control unit, configured to delete an internal identifier ID carried by the message stored in the distributed database, and delete the internal identifier ID from the obtained message, when the judgment result of the determining unit is YES, And sending a message that deletes the internal identifier ID to the client; The distributed database is configured to store an internal identification ID of the message.

The device according to claim 9, wherein after the obtaining unit acquires a message corresponding to the acquiring message request from the distributed queue, the distributed queue deletes the message corresponding to the acquiring message request All copies of it.

11. Apparatus according to claim 9 or 10, characterized in that

The receiving unit is further configured to receive a client input message;

The device further includes:

a generating unit, configured to generate an internal identification ID for the client input message, and join the client input message;

a storage unit, configured to store the client input message after adding the internal identifier ID to the distributed queue; the distributed queue further storing at least one of the client input messages after adding the internal identifier ID a copy; and an internal identification ID generated for the client input message is stored to the distributed database.

The device according to claim 11, wherein the generating unit is specifically configured to generate an internal identifier for the client input message by using a random generation manner or a hash HASH algorithm.

ID, and join the client input message; the internal ID ID generated for each client input message is different from each other.

The device according to any one of claims 90 to 12, wherein the acquiring unit is specifically configured to forward an acquisition message request sent by the client to the distributed queue, and receive the distributed A message sent by the queue corresponding to the obtained message request, where the message carries an internal identifier ID.

14. A distributed queue message reading device, comprising:

An identification unit, configured to identify whether the Value value stored in the distributed row level lock is readable; and a second control unit, configured to: when the Value value stored in the distributed row level lock is readable, Setting the Value value to indicate unreadable, and deleting the internal identification ID from the obtained message, and transmitting a message deleting the internal identification ID to the client.

15. Apparatus according to claim 14 wherein:

The second receiving unit is further configured to receive a client input message;

The device further includes:

a second generating unit, configured to generate an internal identification ID for the client input message, and join the client input message;

a second storage unit, configured to store the client input message after adding the internal identifier ID to the distributed queue; the distributed queue further storing at least one of the client input after adding the internal identifier ID a copy of the message; and storing an internal identification ID generated for the client input message as a key value key into the distributed row level lock, the key value key matching a Value value, the Value value indicating read.

The device according to claim 15, wherein the key value of the distributed row-level lock stored by the extracting unit is key=the internal identifier ID.

A distributed queue message reading system, comprising: a client, and the distributed queue message reading device according to any one of claims 9 to 13, or any one of claims 14 to 16 The distributed queue message reading device;

The client is configured to send a get message request to the distributed queue message reading device, and receive a message corresponding to the get message request sent by the distributed queue message reading device for service processing.