CN116455832A - Method for consuming sequence of message queue based on lane mode - Google Patents

Abstract

The invention discloses a method for consuming sequence of message queues based on a lane mode, which comprises the following steps: s1, a producer sends service data and sets keywords at the same time, and sends the service data to a message queue cluster; s2, establishing a lane queue, and performing sequential sequencing according to a first-in first-out principle; s3, setting a message control module, acquiring data, and partitioning lane information according to the business data keywords to acquire corresponding lane information; s4, sending the data to a corresponding lane queue; s5, queue sequential consumption, wherein once sequential consumption fails, the same keywords are directly sent into a failure queue, so that complete sequential consumption of data is ensured; s6, the failure queue circularly consumes, fails for multiple times, and sends a short message and a mail to inform a producer; the invention can adaptively adjust the number of processing procedures and the processed data quantity, thereby realizing rapid data consumption while ensuring sequential consumption.

Description

Method for consuming sequence of message queue based on lane mode

Technical Field

The invention belongs to the field of message queue consumption sequences, and particularly relates to a method for consuming a message queue based on a lane mode.

Background

If the message queue is only provided with one consumer, if the machine of the consumer fails, the message queue cannot complete the complete consumption of the message, and if the message is to be consumed continuously, the message queue needs to be switched to other machines to operate, namely a plurality of consumer terminals are arranged in the system; the problem that the message is consumed by multiple consumers needs to be sequentially executed is an important problem to be solved in a scenario that multiple consumers consume the distributed message queue message in parallel, how to keep the sequential consumption among multiple consumers.

In the prior art, the consumption sequence of the message queue can not ensure the sequential consumption and simultaneously consume the data rapidly, so we propose a method for consuming the sequence of the message queue based on the lane mode.

Disclosure of Invention

The invention aims to solve the defects in the prior art, and provides a method for consuming the sequence of the message queue based on a lane mode.

In order to achieve the above purpose, the present invention provides the following technical solutions:

a method for message queue consumption order based on lane style, comprising the steps of:

s1, a producer sends service data and sets keywords at the same time, and sends the service data to a message queue cluster;

s2, establishing a lane queue, and performing sequential sequencing according to a first-in first-out principle;

s3, setting a message control module, acquiring data, and partitioning lane information according to the business data keywords to acquire corresponding lane information;

s4, sending the data to a corresponding lane queue;

s5, queue sequential consumption, once the sequential consumption fails, sending the service data into a failure queue, storing the key words, and directly sending the same key words into the failure queue, so as to ensure complete sequential consumption of the data;

s6, the failure queue circulates consumption, fails in multiple consumption, sends short messages and mails to inform the producer, and informs the producer of all failure of the service data of the same key words.

Preferably, the lane queue described in step S2 supports the following operations:

queue (): creating an empty queue, wherein parameters are not needed, and the empty queue is returned;

enqueue (item): adding an element at the tail of the queue, requiring an element as a parameter, and not returning any value;

dequeue (): removing an element from the head of the queue, requiring no parameters, and returning an element and modifying the contents of the queue;

isEmpty (): checking whether the queue is empty, does not need parameters, and returns a boolean value;

size (): the number of elements in the queue is returned, no parameters are needed, and an integer is returned.

Preferably, in the order sorting of the first-in first-out principle, the enqueuing algorithm of the circular queue is as follows:

tail＝tail+1 (1)

if tail=n+1, then tail=1;

if head=tail, i.e. the tail pointer coincides with the head pointer, then the element is filled in the queue, and overflow error processing is performed; otherwise Q (tail) =x, ending the whole process, where X represents a new in-out element.

Preferably, in step S5, the MQ framework is adopted to ensure the accuracy of the consumption sequence of the message queue, which is specifically as follows:

the service data generated by message consumption are sent to a service port of a producer in a Queue form of sending Queue-1, queue-2, queue-N, and the service data sent by a plurality of groups are processed in a partitioning mode, and the time of the service data is marked by a time stamp;

transmitting each service data generated by the service port of the processed producer to zfs-mqs for global message (MongoDB) storage and compensation;

and sending the processed service data to the MQ end in a Queue form of sending the Queue-1, the Queue-2, the Queue-N, processing the consumption information of the service data, and judging whether the Redis exists according to FastDTW calculation formula calculation.

Preferably, if the Redis exists, calling the business service to complete business processing, reaching a business service end of a consumer, then carrying out Redis consumption record according to the processing batch of the current sequence business data, finally sending the next business data message, and the like until the processing of all the sequence queue business data is completed;

and if the Redis does not exist, returning the service data sent by the producer, and resetting the key words to be sent to the message queue cluster.

Preferably, the FastDTW calculation formula takes the similarity of time sequences recorded by Redis consumption, uncertain time sequences and expected distances as judgment bases;

when the FastDTW calculation formula uses the similarity of time sequences as a judgment basis, the time sequences of the dis consumption records are set according to c= { S1, S1..sn } first, where N represents the number of time sequences, and then the FastDTW calculation formula is: RQ (Q, C, ε) = { S|S ε C|distance (Q, S) +.epsilon }, where ε is the distance spread.

Preferably, in step S1, the keyword setting is specifically:

determining keywords of the travel itinerary; setting a synonymous set of keywords; determining the priority of each keyword set; the keyword sets are ordered by priority.

Preferably, the partitioning of the lane information in step S3 is specifically:

the lane identification is acquired, the acquired lane identification is put into a message, and the message is sent to a message queue;

after the consumer end receives the message, the backbone consumer analyzes the lane identification in the message and forwards the lane identification to the corresponding lane queue, and if the corresponding lane queue is not available, a new lane queue is created;

checking whether a corresponding swim lane consumer exists, if so, consuming the swim lane identified message by the swim lane consumer, and if not, consuming the swim lane identified message by the backbone consumer.

Preferably, the checking whether the corresponding lane consumer exists or not waits for the lane consumer to be online for consumption if the corresponding lane consumer exists but the lane consumer is not online

There is a corresponding swim lane consumer, but the swim lane consumer is not online beyond the set time threshold, and is consumed by the backbone consumer.

The invention has the technical effects and advantages that: compared with the existing message queue consumption sequence, the method provided by the invention can adaptively adjust the number of processing procedures and the processed data quantity, thereby realizing rapid data consumption while ensuring sequential consumption;

secondly, the invention can also realize the partition queue division of message information data and the sequential sequencing of the first-in first-out principle, ensure that a plurality of messages in the same service port have transaction characteristics, ensure the message sequency among a plurality of pairs of columns, strictly ensure the message sequency in a cluster and single-machine deployment mode, prevent consumption failure, cause message information to be blocked, and simultaneously prevent repeated consumption.

Drawings

FIG. 1 is a flow chart of a method for consuming order of message queues based on a lane mode according to the present invention.

Detailed Description

The present invention will be described in further detail with reference to specific embodiments in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The invention provides a method for consuming sequence of message queues based on a lane mode, which comprises the following steps:

s1, a producer sends service data and sets keywords at the same time, and sends the service data to a message queue cluster; in step S1, the keyword setting specifically includes:

determining keywords of the travel itinerary; setting a synonymous set of keywords; determining the priority of each keyword set; sorting the keyword sets according to the priority;

s2, establishing a lane queue, and performing sequential sequencing according to a first-in first-out principle;

the lane queue described in step S2 supports the following operations:

queue (): creating an empty queue, wherein parameters are not needed, and the empty queue is returned;

enqueue (item): adding an element at the tail of the queue, requiring an element as a parameter, and not returning any value;

dequeue (): removing an element from the head of the queue, requiring no parameters, and returning an element and modifying the contents of the queue;

isEmpty (): checking whether the queue is empty, does not need parameters, and returns a boolean value;

size (): the number of elements in the return queue, no parameters are needed, and an integer is returned;

further, in the first-in first-out principle of sequential ordering, the enqueuing algorithm of the circular queue is as follows:

tail＝tail+1 (1)

if tail=n+1, then tail=1;

if head=tail, i.e. the tail pointer coincides with the head pointer, then the element is filled in the queue, and overflow error processing is performed; otherwise, Q (tail) =x, ending the whole process, wherein X represents a new in-out element;

s3, setting a message control module, acquiring data, and partitioning lane information according to the business data keywords to acquire corresponding lane information;

s4, sending the data to a corresponding lane queue;

s5, queue sequential consumption, once the sequential consumption fails, sending the service data into a failure queue, storing the key words, and directly sending the same key words into the failure queue, so as to ensure complete sequential consumption of the data;

in step S5, the MQ framework is adopted to ensure the accuracy of the consumption sequence of the message queue, which is specifically as follows:

the service data generated by message consumption are sent to a service port of a producer in a Queue form of sending Queue-1, queue-2, queue-N, and the service data sent by a plurality of groups are processed in a partitioning mode, and the time of the service data is marked by a time stamp;

transmitting each service data generated by the service port of the processed producer to zfs-mqs for global message (MongoDB) storage and compensation;

the processed business data are sent to an MQ end in a Queue form of sending Queue-1, queue-2, queue-N, the consumption information of the business data is processed, and then whether Redis exists is judged according to FastDTW calculation; the FastDTW calculation method takes the time sequence similarity, the uncertain time sequence and the expected distance of Redis consumption records as judgment basis;

when the FastDTW calculation formula uses the similarity of time sequences as a judgment basis, the time sequences of the dis consumption records are set according to c= { S1, S1..sn } first, where N represents the number of time sequences, and then the FastDTW calculation formula is: RQ (Q, C, ε) = { S|S ε C|distance (Q, S) +.ε }, where ε is the distance spread;

if Redis exists, calling business service to complete business processing and reach a business service end of a consumer, then carrying out Redis consumption record according to the processing batch of the current sequence business data, finally sending the next business data message, and so on until all the sequence queue business data processing is completed;

if Redis does not exist, returning the service data sent by the producer, and resetting the keywords to be sent to the message queue cluster;

it should be noted that, when the FastDTW formula uses the uncertain time sequence as a basis for judgment, each data queue element in the uncertain time sequence is set to x, that is: x=r (X) +epsilon (X), where r (X) represents the true value of the data queue element, epsilon (X) represents the error value of the data queue element, and the total time in the uncertain time series is set to T, i.e., t= (T1, T2..ti), and ti is a random variable of the true value at the timestamp i;

s6, the failure queue circulates consumption, fails in multiple consumption, sends short messages and mails to inform the producer, and informs the producer of all failure of the service data of the same key words.

As for the present embodiment, the partitioning of the lane information in step S3 is specifically:

the lane identification is acquired, the acquired lane identification is put into a message, and the message is sent to a message queue;

after the consumer end receives the message, the backbone consumer analyzes the lane identification in the message and forwards the lane identification to the corresponding lane queue, and if the corresponding lane queue is not available, a new lane queue is created;

checking whether a corresponding swim lane consumer exists, if so, consuming the swim lane identified message by the swim lane consumer, and if not, consuming the swim lane identified message by the backbone consumer.

If the corresponding lane consumer exists, but the lane consumer is not online, waiting for the lane consumer to be online for consumption;

there is a corresponding swim lane consumer, but the swim lane consumer is not online beyond the set time threshold, and is consumed by the backbone consumer.

It is stated in this implementation that the FastDTW formula performs the steps of computing the MQ framework: A. firstly, inputting an acquired sequence A and a determined time sequence B to be detected;

B. the FastDTW algorithm is executed by taking the time sequence similarity, the uncertain time sequence and the expected distance of the Redis consumption record as judgment bases, and when coarse granularity is processed to n (n > 3) th fine granularity, a matrix with fine granularity is put into an MQ framework to execute calculation;

C. performing MR calculations:

1) Because the calculation matrix of FastDTW is an n m matrix, firstly dividing the sequence X into p subsequences X0, X1, … Xp-1 with the length of [ m/p ], then dividing the sequence Y into q subsequences Y0, Y1, … Yp-1 with the length of [ n/q ], constructing p q submatrices Mf g, fε [1, p ], gε [1, q ], and the size of each submatrix is [ m X n ]/[ p ] q ];

2) The paths solved by each submatrix are used as key values, and the numbers are used as value values for sorting;

3) And transmitting the sequenced values into a Reduce part for path summarizing and screening, and reducing the values together to obtain the total dynamic reduced path of the service queue in the Redis.

Finally, it should be noted that: the foregoing description is only illustrative of the preferred embodiments of the present invention, and although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that modifications may be made to the embodiments described, or equivalents may be substituted for elements thereof, and any modifications, equivalents, improvements or changes may be made without departing from the spirit and principles of the present invention.

Claims (9)

1. A method for message queue consumption order based on lane style, comprising the steps of:

s1, a producer sends service data and sets keywords at the same time, and sends the service data to a message queue cluster;

s2, establishing a lane queue, and performing sequential sequencing according to a first-in first-out principle;

s3, setting a message control module, acquiring data, and partitioning lane information according to the business data keywords to acquire corresponding lane information;

s4, sending the data to a corresponding lane queue;

s5, queue sequential consumption, once the sequential consumption fails, sending the service data into a failure queue, storing the key words, and directly sending the same key words into the failure queue, so as to ensure complete sequential consumption of the data;

s6, the failure queue circulates consumption, fails in multiple consumption, sends short messages and mails to inform the producer, and informs the producer of all failure of the service data of the same key words.

2. The method of claim 1, wherein the order of consumption of the message queues is based on a lane approach: the lane queue described in step S2 supports the following operations:

queue (): creating an empty queue, wherein parameters are not needed, and the empty queue is returned;

enqueue (item): adding an element at the tail of the queue, requiring an element as a parameter, and not returning any value;

dequeue (): removing an element from the head of the queue, requiring no parameters, and returning an element and modifying the contents of the queue;

isEmpty (): checking whether the queue is empty, does not need parameters, and returns a boolean value;

size (): the number of elements in the queue is returned, no parameters are needed, and an integer is returned.

3. A method of message queue consumption order based on lane style as claimed in claim 2, wherein: in the sequential ordering of the first-in first-out principle, the enqueuing algorithm of the circular queue is as follows:

tail＝tail+1 (1)

if tail=n+1, then tail=1;

if head=tail, i.e. the tail pointer coincides with the head pointer, then the element is filled in the queue, and overflow error processing is performed; otherwise Q (tail) =x, ending the whole process, where X represents a new in-out element.

4. The method of claim 1, wherein the order of consumption of the message queues is based on a lane approach: in step S5, the MQ framework is adopted to ensure the accuracy of the consumption sequence of the message queue, which is specifically as follows:

the service data generated by message consumption are sent to a service port of a producer in a Queue form of sending Queue-1, queue-2, queue-N, and the service data sent by a plurality of groups are processed in a partitioning mode, and the time of the service data is marked by a time stamp;

transmitting each service data generated by the service port of the processed producer to zfs-mqs for global message (MongoDB) storage and compensation;

and sending the processed service data to the MQ end in a Queue form of sending the Queue-1, the Queue-2, the Queue-N, processing the consumption information of the service data, and judging whether the Redis exists according to FastDTW calculation formula calculation.

5. The method of claim 4, wherein the order of consumption of the message queues is based on a lane approach, wherein: if Redis exists, calling business service to complete business processing and reach a business service end of a consumer, then carrying out Redis consumption record according to the processing batch of the current sequence business data, finally sending the next business data message, and so on until all the sequence queue business data processing is completed;

and if the Redis does not exist, returning the service data sent by the producer, and resetting the key words to be sent to the message queue cluster.

6. The method of claim 4, wherein the order of consumption of the message queues is based on a lane approach, wherein: the FastDTW calculation method takes the time sequence similarity, the uncertain time sequence and the expected distance of Redis consumption records as judgment basis;

when the FastDTW calculation formula uses the similarity of time sequences as a judgment basis, the time sequences of the dis consumption records are set according to c= { S1, S1..sn } first, where N represents the number of time sequences, and then the FastDTW calculation formula is: RQ (Q, C, ε) = { S|S ε C|distance (Q, S) +.epsilon }, where ε is the distance spread.

7. The method of claim 1, wherein the order of consumption of the message queues is based on a lane approach: in step S1, the keyword setting specifically includes:

determining keywords of the travel itinerary; setting a synonymous set of keywords; determining the priority of each keyword set; the keyword sets are ordered by priority.

8. The method of claim 1, wherein the order of consumption of the message queues is based on a lane approach: the partitioning of the lane information in step S3 specifically includes:

the lane identification is acquired, the acquired lane identification is put into a message, and the message is sent to a message queue;

after the consumer end receives the message, the backbone consumer analyzes the lane identification in the message and forwards the lane identification to the corresponding lane queue, and if the corresponding lane queue is not available, a new lane queue is created;

checking whether a corresponding swim lane consumer exists, if so, consuming the swim lane identified message by the swim lane consumer, and if not, consuming the swim lane identified message by the backbone consumer.

9. The method of claim 8, wherein the order of consumption of the message queues is based on a lane approach: in the checking of whether the corresponding lane consumer exists, if the corresponding lane consumer exists but the lane consumer is not online, waiting for the lane consumer to be online for consumption

There is a corresponding swim lane consumer, but the swim lane consumer is not online beyond the set time threshold, and is consumed by the backbone consumer.