CN111552575A

CN111552575A - Message queue-based message consumption method, device and equipment

Info

Publication number: CN111552575A
Application number: CN201911417643.5A
Authority: CN
Inventors: 张阳; 崔昌栋; 钱佳林; 柴猛; 崔永超; 尹涛; 陈慧敏; 姜凯洋; 朱树强; 张朝旭; 刘文杰; 王仁斌; 张宏振
Original assignee: Shanghai Envision Innovation Intelligent Technology Co Ltd; Envision Digital International Pte Ltd
Current assignee: Shanghai Envision Innovation Intelligent Technology Co Ltd; Envision Digital International Pte Ltd
Priority date: 2019-12-31
Filing date: 2019-12-31
Publication date: 2020-08-18
Anticipated expiration: 2039-12-31
Also published as: CN111552575B; WO2021137768A1

Abstract

The application discloses a message consumption method, a message consumption device and message consumption equipment based on a message queue, and belongs to the field of computer data communication. The method comprises the following steps: writing the message into a corresponding current writing position in the message file, wherein the message is generated by a producer program, the current writing position is a writing position determined in n message files according to a circular writing sequence, and n is a positive integer; acquiring a reading position of a consumer program in the message file, wherein the reading position is determined in the n message files according to a cyclic reading sequence; in response to the coincidence of the current writing position and the current reading position, updating the current reading position to a target reading position, the target reading position being a reading position located after the current reading position in the writing direction of the circular writing sequence; and sending the message in the message file read by the target reading position to a consumer program.

Description

Message queue-based message consumption method, device and equipment

Technical Field

The present application relates to the field of computer data communications, and in particular, to a message consumption method, device and apparatus based on a message queue.

Background

Message Queue (MQ) is a communication method between applications, and an application can return messages immediately after sending the messages to the MQ, so that a Message system ensures reliable delivery of the messages. The producer (message publisher) of the message publishes the message to the message queue, and the consumer (message active acquirer) of the message acquires the message from the message queue, so that the producer and the consumer do not interfere with each other.

Disclosure of Invention

The embodiment of the application provides a message consumption method, a message consumption device and message consumption equipment based on a message queue, so that the influence of a consumer with low consumption speed on a producer or other consumers in a message transmission system when reading a message is reduced to the minimum. The technical scheme is as follows:

according to one aspect of the present application, a message queue-based message consumption method includes:

writing a message into a corresponding current writing position in a message file, wherein the message is generated by a producer program, the current writing position is a writing position determined in n message files according to a circular writing sequence, and n is a positive integer;

acquiring a reading position of a consumer program in the message file, wherein the reading position is determined in the n message files according to a cyclic reading sequence;

in response to the coincidence of the current writing position and the current reading position, updating the current reading position to a target reading position, the target reading position being a reading position located after the current reading position in the writing direction of the circular writing sequence;

and sending the message in the message file read by the target reading position to the consumer program.

According to another aspect of the present application, there is provided a message queue-based message consumption apparatus, the apparatus including:

the writing module is used for writing a message into a corresponding current writing position in the message file, wherein the message is generated by a producer program, the current writing position is a writing position determined in n message files according to a circular writing sequence, and n is a positive integer;

the acquisition module is used for acquiring the reading position of the consumer program in the message file, wherein the reading position is determined in the n message files according to a cyclic reading sequence;

a processing module, configured to update the current reading position to a target reading position in response to a coincidence of the current writing position and the current reading position, where the target reading position is a reading position located after the current reading position in a writing direction of the circular writing sequence;

and the reading module is used for sending the message in the message file read by the target reading position to the consumer program.

According to another aspect of the present application, there is provided a computer device provided with a sensor, the computer device comprising a processor and a memory, the memory having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by the processor to implement a message queue based message consumption method as described above.

The beneficial effects brought by the technical scheme provided by the embodiment of the application at least comprise:

when the current writing position is coincident with the current reading position, the current reading position is updated to the target reading position, namely the current reading position is updated along the writing direction of the circular writing sequence, so that the reading position and the writing position of the consumer program keep a distance again, and only a small part of messages are lost for the consumer program with lower consumption speed occasionally.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a schematic diagram of a computer system provided in an exemplary embodiment of the present application;

FIG. 2 is a block diagram of a server provided in an exemplary embodiment of the present application;

FIG. 3 is a block diagram of a messaging system provided in an exemplary embodiment of the present application;

FIG. 4 is a flowchart of a message queue-based message transmission method according to an exemplary embodiment of the present application;

FIG. 5 is a flowchart of a message queue-based message transmission method according to another exemplary embodiment of the present application;

FIG. 6 is a block diagram of a messaging system provided in another exemplary embodiment of the present application;

FIG. 7 is a flowchart of a method for performing a write operation to a message queue according to an example embodiment of the present application;

FIG. 8 is a diagram illustrating a reading of a message file at a time provided by an exemplary embodiment of the present application;

FIG. 9 is a flowchart of a message queue-based message consumption method provided by an exemplary embodiment of the present application;

FIG. 10 is a schematic illustration of a reading of a message file at a time as provided by another exemplary embodiment of the present application;

FIG. 11 is a flowchart of a message queue-based message consumption method provided by another exemplary embodiment of the present application;

FIG. 12 is a block diagram of a message queue based message transmission apparatus according to an exemplary embodiment of the present application;

FIG. 13 is a block diagram of a message queue based message consumption device as provided by an exemplary embodiment of the present application;

fig. 14 is a schematic device structure diagram of a computer apparatus according to an exemplary embodiment of the present application.

Detailed Description

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

First, terms referred to in the embodiments of the present application are described:

the producer: refers to an application or module that generates messages, such as a social application.

The consumer: refers to an application program or a module for reading messages, and can perform subsequent processing on the read messages. For example, the social application needs to read the verification code in the sms application.

FIG. 1 shows a block diagram of a computer system provided in an exemplary embodiment of the present application. The computer system 100 includes: an application 120 (producer) to support sending messages, an application 160 (consumer) to support obtaining messages, and a message queue 140.

The producer program 120 establishes a connection with the message queue 140. The producer program 120 includes: social-type applications, shopping-type applications, video-type applications, music-type applications, image-type applications, and the like.

The message queue 140 includes at least one of a server, a plurality of servers, a cloud computing platform, and a virtualization center. Message queue 140 includes a memory 142 and a processor 144, memory 142 for storing messages issued by producer program 120, and processor 144 for enabling consumer program 160 to retrieve producer-generated messages.

The consumer program 160 establishes a connection with the message queue 140. The consumer program 160 includes: social-type applications, shopping-type applications, video-type applications, music-type applications, image-type applications, and the like.

Alternatively, producer program 120 and consumer program 160 may be the same application, or different classes of applications, or different applications under the same category.

Alternatively, the producer program 120 and the message queue 140 are in the same terminal, or the consumer program 160 and the message queue 140 are in the same terminal, or the producer program 120 and the consumer program 160 are in the same terminal, or the producer program 120, the consumer program 160 and the message queue 140 are distributed on different terminals, respectively.

Optionally, message queue 140 undertakes primary computational work, and producer program 120 and consumer program 160 undertake secondary computational work; alternatively, message queue 140 undertakes secondary computational work, and producer program 120 and consumer program 160 undertake primary computational work; alternatively, the message queue 140, the producer program 120, and the consumer program 160 may perform cooperative computing using a distributed computing architecture.

It is understood that at least one producer program 120 and at least one consumer program 160 exist in the computer system 100, and the present embodiment is only described by taking one producer program 120 and one consumer program 160 as an example. The above-mentioned producer program 120 and consumer program 160 are of the same or different device types of the terminal, and the device types include: at least one of a smartphone, a tablet, an e-book reader, an MP3 player, an MP4 player, a laptop portable computer, and a desktop computer, a notebook computer.

Those skilled in the art will appreciate that the number of terminals described above may be greater or fewer. For example, the number of the terminals may be only one, or the number of the terminals may be several tens or several hundreds, or more, and the number of the terminals and the type of the device are not limited in the embodiment of the present application.

Fig. 2 shows a schematic structural diagram of a server provided in an exemplary embodiment of the present application. The server may be the message queue 140 shown in fig. 1. Specifically, the method comprises the following steps:

the server 200 includes a Central Processing Unit (CPU) 201, a system Memory 204 including a Random Access Memory (RAM) 202 and a Read Only Memory (ROM) 203, and a system bus 205 connecting the system Memory 204 and the Central Processing Unit 201. The server 200 also includes a basic input/output System (I/O System)206, which facilitates the transfer of information between devices within the computer, and a mass storage device 207 for storing an operating System 213, application programs 214, and other program modules 215.

The basic input/output system 206 includes a display 208 for displaying information and an input device 209, such as a mouse, keyboard, etc., for user input of information. Wherein a display 208 and an input device 209 are connected to the central processing unit 201 through an input output controller 210 connected to the system bus 205. The basic input/output system 206 may also include an input/output controller 210 for receiving and processing input from a number of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, input-output controller 210 also provides output to a display screen, a printer, or other type of output device.

The mass storage device 207 is connected to the central processing unit 201 through a mass storage controller (not shown) connected to the system bus 205. The mass storage device 207 and its associated computer-readable media provide non-volatile storage for the server 200. That is, the mass storage device 207 may include a computer-readable medium (not shown) such as a hard disk or Compact disk Read Only Memory (CD-ROM) drive.

Without loss of generality, computer readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash Memory or other solid state Memory technology, CD-ROM, Digital Versatile Disks (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media is not limited to the foregoing. The system memory 203 and mass storage device 207 described above may be collectively referred to as memory.

According to various embodiments of the present application, server 200 may also operate as a remote computer connected to a network through a network, such as the Internet. That is, the server 200 may be connected to the network 212 through the network interface unit 211 connected to the system bus 205, or the network interface unit 211 may be used to connect to other types of networks or remote computer systems (not shown).

The memory further includes one or more programs, and the one or more programs are stored in the memory and configured to be executed by the CPU.

Producer refers to the application that generates the message, consumer refers to the application that obtains (or actively reads or has access to) the message, and message queue refers to the "container" in which the message generated by the producer is stored. The producer only needs to produce the message and store the message in the message queue, and does not need to determine which consumer obtains the message from the message queue, so that one producer corresponds to one consumer, one producer corresponds to a plurality of consumers, or a plurality of producers correspond to a plurality of consumers through the message queue. For example, the producer is a communication application and the consumer is a shopping application, and the shopping application reads a message, such as a passcode, placed in a message queue by the communication application.

As shown in fig. 3, a producer 101 stores a message in the message queue 10, the message queue 10 generates a message file, writes the message in the message file, and records the read positions of the consumer 102 and the consumer 103 on the message file. Because of the difference in consumption speed between different consumers, i.e., consumer 102 and consumer 103 read the same message at different speeds. When consumer 102 and consumer 103 actively read the message from message queue 10, consumer 102 and consumer 103 may read the message file according to the respective corresponding read locations. By recording the reading position of the consumer when reading the file, the message can be independently acquired between the consumer 102 and the consumer 103 without interference. In one example, consumer 102 reads the forty-sixth byte of file 2, consumer 103 reads the first thousand hundred twelve bytes of file 3, and message queue 10 records the read locations of consumer 102 and consumer 103, from which the reading of messages may continue when consumer 102 and consumer 103 read again.

It will be appreciated that consumers 102 and 103 may retrieve messages from the message queue 10 at the same time or in a sequence from the message queue 10. The above-mentioned producers 101 may be one or more, the producer may generate one or more types of messages, the types of messages generated by a plurality of producers may be the same or different, and the message file generated in the message queue 10 may be one or more. Illustratively, five message files are generated in the message queue 10, the five message files being divided into a class a and a class B, the class a messages being generated by three producers, and the class B messages being generated by two producers, wherein the three producers generating the class a messages are each a social application, the two producers generating the class B messages are each a music application, one consumer reading the class a messages, and two consumers reading the class B messages. And the three consumers read the messages in the message file according to the respective reading positions without mutual interference.

Fig. 4 is a flowchart illustrating a message queue-based message transmission method according to an exemplary embodiment of the present application. The method may be applied to a computer system 100 as shown in fig. 1, the method comprising the steps of:

step 401, obtaining a message file, where the message file stores a message generated by a producer program in a form of a message queue.

The message file refers to a file storing a message generated by a producer program, and the producer program is any application program, such as a social application program, a game application program, a music application program, a video application program, an image application program, a system application program, and the like.

Optionally, the message queue generates a message file according to a message generated by the producer program, where the generated message file contains the message, or the message queue is default-provided with a message file, and after the message generated by the producer program is acquired, the message is stored in the message file, or the message file is acquired from the producer program by the message queue, and the message file is generated by the producer program. Illustratively, a message processing module is arranged in the message queue, and the message processing module generates a message file according to the message generated by the producer program.

The number of the message files may be one or more, the message files may be of the same type or different types, the producer programs may be one or more, one producer program may generate one or more messages of the same type or different types, and the types of the producer programs may be the same or different.

Step 402, determining the reading position of the consumer program on the message file from the consumer state data, wherein the consumer state data corresponds to the consumer program one by one.

Consumer status data refers to data that characterizes the read status of a consumer program. Optionally, the reading state comprises: at least one of an identification of the message file to be read, a reading start position, reading content, reading times, reading time and a reading end position. Optionally, the consumer status data is stored in a consumer status file, or the consumer status data may be uploaded to a cloud server.

In one example, reading the state includes: the message queue determines the identifier of the message file to be read from the consumer data, can determine the message file where the message to be read by the consumer program is located, can determine the initial position to be read by the consumer program according to the read starting position, and can determine the position of the consumer program after the message reading is completed at this time according to the read ending position, optionally, the position is the initial position of the next message reading.

The consumer state data may be one or more copies, and the consumer program may be one or more copies, each corresponding to a respective read location.

Step 403, reading the first message in the message file according to the reading position.

Optionally, the read position includes at least one of an identification (e.g., name or number) of the read file, a read start position (e.g., at byte 52), and a read end position (e.g., at byte 520). Illustratively, the reading position is the starting position of the consumer program for reading the message at this time, the reading position is at the 65 th byte in the message file B, the message queue determines that the message file B is the message file to be read by the consumer program according to the reading position, and determines that the consumer program starts to read from the 65 th byte in the message file B.

At step 404, a first message is sent to the consumer program.

Illustratively, the message queue sends the corresponding message at byte 65 in message file B to the consumer program, as depicted in step 403.

In summary, in the method provided in this embodiment, the message file storing the message is obtained through the message queue, the reading position of the consumer program is determined according to the consumer status data, and the read message is sent to the consumer program according to the reading position of the consumer. When a plurality of consumer programs read the messages in the message queue, the message queue is not needed to copy a plurality of message copies, so that a plurality of consumers can independently consume the messages, and the message transmission efficiency of the message queue is improved.

Fig. 5 is a flowchart illustrating a message queue-based message transmission method according to another exemplary embodiment of the present application. The method may be applied to a computer system 100 as shown in fig. 1, the method comprising the steps of:

step 501, at least two message files are obtained.

Optionally, the message file comprises at least two message files arranged in sequence. Optionally, the order is an order in which messages are cyclically written to the message file, or the order is an order in which messages are cyclically read, or the order is an order in which the message file is generated.

Fig. 6 shows a block diagram of a message transmission system according to an exemplary embodiment of the present application. The message transmission system 200 includes a producer 1 (i.e., a producer program), a message queue 21, and n consumers (i.e., consumer programs), where n is a positive integer (e.g., consumer 1, consumer 2 … …, consumer n).

Illustratively, the producer 1 generates two messages and sends the two messages to the message queue 21, the message queue 21 includes a message processing module 201, and the message processing module 201 generates two message files according to the two messages.

Optionally, the message file includes at least two types of message files, and each type of message file corresponds to a respective message tag. Illustratively, producer 1 has generated two types of messages, corresponding to two message tags, one for each type of message. At this time, the message queue 21 acquires the message file including the following sub-steps:

and S1, determining a first message label corresponding to the consumer program.

The message queue 21 first determines a first message tag, which corresponds to the type of the first message.

S2, determining a message file corresponding to the first message label from the at least two types of message files.

Illustratively, when the first message tag is a, the type corresponding to the first message file is class a; when the first message label is B, the type corresponding to the first message file is B type; when the first message tag is C, the type corresponding to the first message file is class C. The form of the label is not limited in the present application.

In one example, the message queue 21 determines that the first message tag corresponding to the consumer program is a, the consumer may determine that the type of the message file is type a, and the message queue 21 may further determine the read location of the consumer program from the message file belonging to type a.

Step 502, determining the reading position of the consumer program on the message file from the consumer state data, wherein the consumer state data corresponds to the consumer program one by one.

Optionally, the number of the consumer programs is at least two, each consumer program corresponds to a respective consumer state file, and consumer state data of the consumer program is stored in the consumer state file.

Illustratively, as shown in fig. 6, the consumer state file corresponding to consumer 1 is file 1, the consumer state file corresponding to consumer 2 is file 2, and the consumer state file corresponding to consumer 3 is file 3, and file 1, file 2, and file 3 are different from each other.

Step 503, according to the first message file identifier in the reading position, determining a first target message file to be read from the at least two message files.

Step 504 reads the first message in the first target message file according to the first message file offset in the read position.

Optionally, the reading position comprises: a first message file identification and a first message file offset. The first message file is used for representing which file in the message queue is read by the consumer program, such as the message in the message file A is read by the consumer program; the first message file offset is used to characterize the read location of the consumer program in the first message file, such as at byte 52 in the first message file. Optionally, the message file offset is expressed in bytes.

The following describes a relationship between the first message file identifier, the first message file offset, and the first message file tag with reference to the table.

Watch 1

Wherein the first message file identifier 2019122115330001 indicates that the first message file numbered 0001 at 12/21/2019, 15/53, and offset _ a indicates that the offset of the first message file numbered 0001 is a bytes. The form of the message file identification and the message file offset is not limited in this application.

Step 505, a first message is sent to a consumer program.

Step 505 is identical to step 404, and will not be described herein.

It will be appreciated that the above steps may be performed in a loop, i.e. the consumer program may loop through the reading of the messages in the message file. The above-described steps S1 and S2 and other steps may be performed in combination or not.

In summary, in the method provided in this embodiment, when the message queue obtains multiple message files, the read position of the first message is determined according to the first message file identifier and the first message file offset, and if the multiple message files belong to multiple types, the message queue may determine the type of the message file according to the message tag corresponding to the message file, and then read the first message according to the first message file identifier and the first message read position. The multiple consumer programs can independently read the messages from the message queue, and the one-shot multiple-receiving mode and the multiple-shot multiple-receiving mode under the mode of multiple messages or multiple types of messages are realized.

The following describes a process in which a message queue writes a message generated by a producer program into a message file.

Fig. 7 is a flowchart illustrating a message queue-based message transmission method in combination with a message tag according to an exemplary embodiment of the present application. The method may be applied to a computer system 100 as shown in fig. 1, the method comprising the steps of:

step 701, obtaining a second message generated by the producer program.

The message file includes at least two message files arranged in order, each message file having an upper file size limit. Illustratively, the upper limit of the file size of, for example, message file 1 is 100 Megabytes (MB).

Optionally, the second message generated by one producer program is a message belonging to a plurality of types, or a plurality of messages generated by a plurality of producer programs, and the message generated by each producer program is of a different type.

Optionally, the message file includes at least two types of message files, and each type of message file corresponds to a respective message tag. When the acquired second message belongs to a plurality of types, the step comprises the following sub-steps:

and S11, determining a second message label corresponding to the second message.

The message queue first determines a second message label, which corresponds to the first message type.

S22, determining a message file corresponding to the second message label from the at least two types of message files.

Illustratively, when the second message tag is a, the type of the corresponding second message file is class a; when the first message label is B, the type corresponding to the first message file is B type; when the first message tag is C, the type corresponding to the first message file is class C. The form of the label is not limited in the present application.

Step 702, obtaining a last write position of the producer program, where the last write position includes: a second message file identification and a second message file offset.

The writing position represents the position corresponding to the message queue when the message generated by the producer program is written into the message file last time. The second message file identification is used for representing which file the message queue writes the message into, for example, the message queue writes the producer program into the message file A; the second message file offset is used to characterize where in the second message file the message queue writes the message, such as at byte 52 in the second message file. Optionally, the data amount of the message is represented in bytes.

The following describes a relationship between the second message file identifier, the second message file offset, and the second message file tag with reference to the table pair.

Watch two

Wherein the second message file identifier 2019122114330001 indicates that the first message file numbered 0001 at point 33 of 21/2019, and offset _ c indicates that the offset of the second message file numbered 0001 is P bytes. The form of the message file identification and the message file offset is not limited in this application.

Step 703a, in response to that the total amount of the second messages in the kth message file does not reach the upper limit of the file size, writing the second messages in the kth message file, and updating the latest writing position.

The kth message file is a message file corresponding to the second message file identifier, the total amount of the second message is the sum of the data volume of the second message and the offset of the second message file, k is greater than or equal to 2, and k is an integer.

When the message queue writes the message into the message file, whether the size of the total amount of the second message is larger than the file size of the message file to be written is judged, the total amount of the second message is the sum of the message to be written and the current writing position, and if the sum is not larger than the file size of the message file to be written, the message queue directly writes the message to be written into the message file. The last write location of the message queue update is the byte in the kth message file.

Illustratively, k is 3, the total amount of the second message does not reach the upper limit of the file size of the 3 rd message file (the second message file is identified as 0003), the message queue writes the message into the 110 th byte in the 3 rd message file, updates the 110 th byte of the message file with the latest writing position of the number of 0003,

step 703b, in response to that the total amount of the second messages reaches the upper limit of the file size in the kth message file, generating a kth +1 message file, writing the second messages into the kth +1 message file, updating the latest writing position, wherein k is greater than or equal to 2, and k is an integer.

And if the size of the total amount of the second messages reaches the upper limit of the file size of the message file to be written, writing the message to be written into the next message file of the message file to be written. Illustratively, when a message is to be written to the message file numbered 0004 (which is the 4 th file in the message queue), and the total size of the message is greater than the message file numbered 0004, the message queue generates the 5 th file, which is numbered 0005.

Optionally, the message queue may generate a (k + 1) th file in advance, for example, when the message queue generates the k file, the (k + 1) th file is generated at the same time, or the message queue generates n files at the same time, where n is the total number of message files in the message queue.

Step 703c, in response to that the total amount of the second messages reaches the upper limit of the file size in the nth message file, deleting the messages in the 1 st message file, writing the second messages into the 1 st message file, updating the latest writing position, wherein k is less than or equal to n, and n is a positive integer greater than 2.

Optionally, consuming the file further comprises: and consuming the upper limit n of the number of the files.

Illustratively, n is 50, there are 50 message files in the message queue, the number of the message file is 0000-.

It can be understood that the above steps may be executed in a cycle, that is, the message queue may write the message generated by the producer program into the message file in a cycle, and after the message is written into the message file, the process of writing the message and the process of reading the message may be executed simultaneously or in a certain order.

Schematically, as shown in fig. 8, it shows the read-write situation of the message file at a certain time. The read-write conditions include: the read location of consumer program 311 on message file 03 (the arrow points to the read location), the read location of consumer program 312 on message file 04, the read location of consumer program 313 on message file 09, and the write location of message file 03 is location 301.

In an alternative embodiment, the message queue may delete messages in the message file.

And setting the message storage time in the kth message file to be larger than a time threshold, wherein the message file corresponding to the latest writing position is the mth message file, m is less than or equal to n, and m is a positive integer.

S10, obtaining the storage time of the message in the k-1 message file.

As shown in fig. 6, the message queue 21 is provided with a garbage collection module 203, which can delete messages with too long retention time. Illustratively, k is 10, the time threshold is 60 seconds, there are 50 message files in the message queue, and the message file written with the message at this time is the 20 th message file. And if the storage time of the messages in the 10 th message file arranged according to the circular writing sequence in the message queue is greater than the time threshold, the message queue acquires the storage time of the messages in the 9 th message file.

And S20, in response to the fact that the storage time of the messages in the (k-1) th message file is larger than the time threshold, deleting the messages in a second target message file, wherein the second target message file is a message file which is positioned between the (k) th message file and the (m) th message file in a reverse order.

The reverse sequence is the reverse sequence of the cyclic writing sequence, for example, the cyclic writing sequence is to write the messages into the message files according to the ascending sequence of the numbers of the message files, for example, the message queue starts to write from the message file with the number of 001, the message file with the number of 001 is the 1 st file, the message file with the number of 050 is written till the message file with the number of 050 (the 50 th message file), and after 50 message files are completely written, the 1 st message file is rewritten. If the message file corresponding to the last written position is the message file (20 th message file) with the number of 020, when the message file larger than the storage time is determined, the message file corresponding to the message file with the number of 020 is checked from the 20 th message file until the 1 st message file is checked, and if the storage time of the message in the 1 st message file still does not exceed the time threshold, the 50 th message file and the 49 th message file … … 21 st message file are checked until the message file corresponding to the message with the storage time larger than the time threshold is determined.

In one example, when the storage time of the message in the 9 th message file is also greater than the time threshold, the second target message file is a message file located between the 10 th message file and the 20 th message file in reverse order, excluding the 10 th message file and the 20 th message file, i.e., the message in the 1 st message file of the 9 th message file, the 8 th message file, and the 7 th message file … … is deleted, while the message in the 21 st message file of the 49 th message file, the 48 th message file, and the 47 st message file … … is deleted.

In summary, in the method provided in this embodiment, the target message file to which the second message can be written is determined by determining the upper limit of the file size reached by the total amount of the second message in the file to be written, and if there are multiple messages or multiple types of messages, the message queue determines the type of the message file to be written by determining the message tag, and then writes, so as to ensure that the messages written by the message queue are accurate and ordered.

When multiple consumers actively read messages in a message queue, there may be a case where the speed at which the consumers read the messages (consumption speed) is relatively low. In the related art, the message queue does not buffer messages, only transmits the messages to the consumers, and does not care that the consumers with low consumption speed can not read the messages in time.

Based on the above situation, when a plurality of consumers with different consumption speeds read the messages transmitted by the message queue, the consumers with lower consumption speeds lose a large amount of messages.

In order to solve the above problem, the present application further provides a message consumption method based on a message queue, which is applicable to all cases of writing messages circularly, and is not limited to the case of writing messages in the above embodiments.

Fig. 9 is a flowchart illustrating a message queue based consumption method according to an exemplary embodiment of the present application, which may be applied to the computer system 100 shown in fig. 1, and includes the following steps:

step 901, writing a message into a corresponding current writing position in the file, where the message is generated by the producer program, the current writing position is a writing position determined in n message files according to a circular writing sequence, and n is a positive integer.

Illustratively, n is 50, 50 message files are stored in the message queue, the numbers of the message files are 000 to 049 respectively, and the message queue circularly writes the received messages generated by the producer into the 50 files. Optionally, the writing order of the message files includes an increasing order by file number or a decreasing order by number. Illustratively, when the message file with the number 049 is full, the message is written into the message file with the number 000, the originally written message in the message file with the number 000 is overwritten by a new message, or the originally written message is deleted first and then written into a new message. Alternatively, the message file may also be written cyclically in the order of the initials of the file names, such as cyclically starting from the file name a.

Illustratively, the message queue writes the message into the 100 th byte in the message file with the current writing position being number 030.

And step 902, acquiring the reading position of the consumer program in the message file, wherein the reading position is determined in the n message files according to the circular reading sequence.

The cyclic read order coincides with the cyclic write order, which is determined when the cyclic write order of the messages is determined. Illustratively, the loop reading order is read in increasing order of file numbers. Illustratively, the reading position obtained by the message queue is at the 100 th byte in the message file with the number of 030.

Step 903, in response to the coincidence of the current writing position and the current reading position, updating the current reading position to a target reading position, where the target reading position is located at a reading position after the current reading position in the writing direction of the circular writing sequence.

Optionally, the target reading position is a reading position set by a default of the message queue, or a reading position automatically selected by the server according to the consumption speed of the consumer program. Optionally. The distance between the reading position and the target reading position is fixed, or the distance is larger and larger, or the distance is intelligently adjusted according to the consumption speed of a consumer program.

Illustratively, the reading position of this time is at the 100 th byte in the message file numbered 030, and the reading position of the target is at the 100 th byte in the message file numbered 048. The message queue updates the current read position from byte 100 in the 030 message file to byte 100 in the 048 message file.

Each time the read location of the consumer program and the write location of the producer program coincide, the message queue updates the corresponding read location to the target location. Optionally, when there are at least two coincidences of the read position and the write position during the reading of the consumer program and the writing of the producer program, step 903 further includes the following sub-steps:

step 9031, a first timestamp of the last time the history read position of the consumer program and the history write position of the producer program coincide is obtained.

Illustratively, the historical read location of the consumer program and the historical write location of the producer program coincide twice, and the timestamp of the first (i.e., last) coincidence is time M1.

And 9032, acquiring a second timestamp when the reading position and the writing position coincide.

Illustratively, the timestamp of the second (i.e., this) time of the coincidence is time M2.

Step 9033, updating the reading position of this time to the target reading position according to the difference value between the first time stamp and the second time stamp.

Optionally, there is a corresponding relationship between the difference between the first timestamp and the second timestamp and the target reading position, for example, y ═ kx is satisfied between the difference between the timestamps and the target reading position, where y is the target reading position, k is a constant, and x is the timestamp difference. Illustratively, the target read position is kX (M2-M1).

And 904, sending the message in the message file read by the target reading position to the consumer program.

The message queue starts reading at byte 100 in the message file of 048 and sends the read message to the consumer program.

In summary, in the method of this embodiment, when the current writing position coincides with the current reading position, the current reading position is updated to the target reading position, that is, the current reading position is updated along the writing direction of the circular writing sequence, so that the reading position and the writing position of the consumer program maintain a distance again, and for the consumer program with a low consumption speed occasionally, only a small part of messages will be lost.

Fig. 10 is a flowchart illustrating a message queue-based message consumption method according to an exemplary embodiment of the present application, which may be applied to the computer system 100 shown in fig. 1, and includes the following steps:

step 1001, writing a message into a corresponding current writing position in the file, where the message is generated by the producer program, the current writing position is a writing position determined in n message files according to a circular writing sequence, and n is a positive integer.

Step 1002, obtaining a reading position of the consumer program in the message file, where the reading position is determined in the n message files according to the circular reading sequence.

Step 1001 and step 1002 are identical to step 901 and step 902 shown in fig. 9, and are not described here again.

Step 1003, obtaining a first time stamp of the last time when the history reading position of the consumer program and the history writing position of the producer program are overlapped.

Step 1004, acquiring a second time stamp when the reading position and the writing position coincide.

Step 1003 and step 1004 are identical to step 9031 and step 9032, and are not described herein again.

Step 1005a, responding to that the difference value between the first time stamp and the second time stamp is greater than the time threshold, and updating the reading position of this time to the first target reading position.

The length between the current reading position and the first target reading position is a second length, wherein the second length is a preset value. Illustratively, the preset value is 10 megabytes in length. Optionally, the preset value is a starting value or a default value set by the computer device.

As shown in fig. 10, the message file includes at least two message files arranged in order, and a message file 03, a message file 04, and a message file 09 are stored in the message queue. Illustratively, consumer 1 corresponds to the reading position 411 (as shown by an arrow), consumer 2 corresponds to the reading position 412, consumer 3 corresponds to the reading position 413, and the current writing position of the message queue is the writing position 401. The read location 411, which has a fast read speed, will "catch up" with the write location 401; the read speed of the read location 412 is slow, and the write location 401 is close to the read location 412, and will "catch up" with the read location 412; the reading speed of the reading position 413 is normal, and the writing position 401 and the reading position 413 are kept at a certain distance.

Illustratively, when the write location 401 "catches up" with the read location 412 for the first time, the message queue updates the consumer's read location 412, moves the read location 412 by a length corresponding to 10 megabytes from this read location to the write direction of the recurring write sequence, e.g., the moved read location is the read location 413, and records the corresponding timestamp M1.

Illustratively, the time threshold is the slow start maximum decision time N. When M2-M1 > N (N > 0, N indicates the longest determination time for slow start), the message queue updates the consumer's reading position 412 and moves the reading position 412 by a length corresponding to 10 megabytes from this reading position to the writing direction of the circular writing sequence.

Step 1005b, responding to that the difference value between the first time stamp and the second time stamp is not larger than the time threshold, and updating the reading position of this time to a second target reading position.

The length between the history reading position and the history target reading position is a first length, and the length between the current reading position and the first target reading position is a second length, wherein the first length is smaller than the second length. Optionally, the second length and the first length satisfy a corresponding relationship, such as a multiple relationship, or the second length is determined by a preset manner, such as a look-up table, for example, the first length a corresponds to the second length b, or the second length is equal to the first length plus a predetermined step size.

When M2-M1 is not more than N, the message queue updates the reading position 412 of the consumer, and the reading position 412 is moved by the length corresponding to 20 megabytes from the reading position at this time to the writing direction of the circular writing sequence; on the basis, when the writing position 401 "catches up" with the reading position 412 for the third time, the time stamp M3 is recorded, and if M3-M2 is less than or equal to N, the reading position 412 is moved by a length corresponding to 40 megabytes from the current reading position to the writing direction of the circular writing sequence.

Optionally, when the length of the current reading position of the consumer program at the time of updating meets a certain condition, the updated target reading position will be changed.

The updated target read position is obtained as follows:

and in response to the total length reaching the preset length, updating the reading position to be a third target reading position, wherein the length between the reading position and the third target reading position is the preset length.

The total number of times is the number of times each time the read location of the consumer program coincides with the write location of the producer program.

The reading position is updated to the target position in the above manner until the length of movement of the reading position 412 reaches the preset maximum value (i.e., the preset length) Q. When the message queue moves the read position to the write direction of the circular write sequence by a length corresponding to Q (640 megabytes), the write position 401 can "catch up" with the read position updated by the consumer after a long time, and before the moved length reaches a preset maximum value Q, the moved length of the read position is still 10 megabytes corresponding to the time interval between any time stamp and the last time stamp being greater than N.

Alternatively, by:

and updating the reading position to a third target reading position in response to the total times reaching the preset times.

Illustratively, when the number of times of coincidence between the read position of the consumer program and the write position of the producer program reaches 10 times, the read position of this time is shifted by a length corresponding to Q bytes in the write direction of the circular write sequence.

Step 1106, sending the message in the message file read by the target reading position to the consumer program.

Step 1106 is identical to step 904 and will not be described herein.

In summary, for the consumer program which consumes slowly, each time the consumer program is "caught up", the lost message is at most and less until the preset length is reached, and the reading position is updated to the target reading position corresponding to the preset length, so that the consumer program which consumes slowly can be "caught up" again after a long time, and the resource consumption of the consumer program which consumes slowly on the message system and the influence on the producer program and other consumer programs are reduced to the minimum.

In this scenario, the producer program is an online shopping program (customer end) corresponding to the user, and the consumer program includes a management program of an online shopping platform, an online shopping program (merchant end) corresponding to the merchant, and an express program corresponding to an express company.

In the second killing shopping scene, a plurality of producer programs are accessed into the message queue, illustratively, the message generated by each producer program is a shopping order corresponding to the second killing commodity, and the shopping order comprises the commodity purchased by the customer, the quantity, the price, the order number, the receiving address, whether the coupon is used and other information. The message queue generates a corresponding message file according to the shopping order, illustratively, the message file can be provided with a label according to the type of the commodity. And the consumer program reads the messages in the message file according to the tags corresponding to the message file, the first message file identifier and the first message file offset, and the three consumers independently read the messages from the message file according to the reading positions of the consumers. If so, the management program of the online shopping platform processes the read message to obtain that the turnover of the food class under the one-second killing scene is five ten thousand yuan; processing the read message by an online shopping program corresponding to the merchant to obtain ten thousand yuan of turnover of a commodity in a one-second killing scene; and processing the message read by the express program to obtain that the delivery address of the user A is the number 187 of the XX street in the MG district in the MG city.

In summary, when the method provided by the embodiment is applied to the second killing shopping scene, a plurality of consumers can read the messages in the message queue at the same time, so that the transmission efficiency of the message queue is improved, and the smoothness of message transmission of the server in the high concurrency scene is ensured.

The following are embodiments of the apparatus of the present application, and for details that are not described in detail in the embodiments of the apparatus, reference may be made to corresponding descriptions in the above method embodiments, and details are not described herein again.

Fig. 12 is a schematic structural diagram illustrating a message queue-based message transmission apparatus according to an exemplary embodiment of the present application. The apparatus can be implemented as all or a part of a terminal by software, hardware or a combination of both, and includes:

a first obtaining module 1210, configured to obtain a message file, where the message file stores a message generated by a producer program in a form of a message queue;

the first processing module 1220 is configured to determine, from the consumer status data, a reading position of the consumer program on the message file, where the consumer status data corresponds to the consumer program one to one;

a first reading module 1230, configured to read a first message in the message file according to the reading position;

a sending module 1240 for sending the first message to the consumer program.

In an alternative embodiment, the number of the consumer programs is at least two, each consumer program corresponds to a respective consumer status file, and the consumer status files store consumer status data of the consumer programs.

In an alternative embodiment, the message file comprises at least two message files arranged in sequence, and the reading position comprises: a first message file identification and a first message file offset;

the first processing module 1220 is configured to determine, according to the first message file identifier in the reading position, a first target message file to be read from the at least two message files;

the first reading module 1230 is configured to read the first message in the first target message file according to the first message file offset in the reading position.

In an optional embodiment, the message file includes at least two types of message files, each type of message file corresponds to a respective message tag;

the first obtaining module 1210 is configured to determine a first message tag corresponding to a consumer program; and determining a message file corresponding to the first message label from at least two types of message files.

In an alternative embodiment, the apparatus includes a write module, the message file including at least two message files arranged in sequence, each message file having an upper file size limit;

the first obtaining module 1210 is configured to obtain a second message generated by a producer program; obtaining a last write location of a producer program, the last write location comprising: a second message file identification and a second message file offset;

the first writing module 1250 is configured to, in response to that the total amount of the second messages in the kth message file does not reach the upper limit of the file size, write the second messages in the kth message file, and update the latest writing position; the kth message file is a message file corresponding to the second message file identifier, the total amount of the second message is the sum of the data volume of the second message and the offset of the second message file, k is greater than or equal to 2, and k is a positive integer;

the first writing module 1250 is configured to generate a (k + 1) th message file in response to that the total amount of the second messages reaches the upper limit of the file size in the (k + 1) th message file, write the second message into the (k + 1) th message file, and update the latest writing position.

In an alternative embodiment, the consuming the file further comprises: the upper limit n of the number of the consumed files,

the first writing module 1250 is configured to, in response to that the total amount of the second messages reaches the upper limit of the file size in the nth message file, delete the message in the 1 st message file, write the second message into the 1 st message file, update the last writing position, where k is equal to or less than n, and n is a positive integer greater than 2.

the first processing module 1220 is configured to determine a second message tag corresponding to a second message; and determining a message file corresponding to the second message label from at least two types of message files.

In an optional embodiment, the apparatus includes a deleting module 1260, where the storage time of the message in the kth message file is greater than a time threshold, the message file corresponding to the last write position is the mth message file, m is less than or equal to n, and m is a positive integer;

the first obtaining module 1210 is configured to obtain a storage time of a message in a k-1 th message file;

the deleting module 1260 is configured to delete the message in the second target message file in response to that the storage time of the message in the k-1 th message file is greater than the time threshold, where the second target message file is a message file that is located between the k-th message file and the m-th message file in a reverse order.

Fig. 13 is a diagram illustrating a structure of a message queue-based message consumption apparatus according to an exemplary embodiment of the present application. The apparatus can be implemented as all or a part of a terminal by software, hardware or a combination of both, and includes:

a second writing module 1310, configured to write a message into a corresponding current writing position in the message file, where the message is generated by the producer program, the current writing position is a writing position determined in n message files according to a circular writing sequence, and n is a positive integer;

a second obtaining module 1320, configured to obtain a reading position of the consumer program in the message file, where the reading position is a reading position determined in the n message files according to the cyclic reading sequence;

a second processing module 1330, configured to update the current reading position to a target reading position in response to that the current writing position coincides with the current reading position, where the target reading position is a reading position located after the current reading position in the writing direction of the circular writing sequence;

the second reading module 1340 is configured to send the message in the message file read by the target reading position to the consumer program.

In an alternative embodiment, the second obtaining module 1320 is configured to obtain a first timestamp of a last time when the historical read location of the consumer program and the historical write location of the producer program coincide;

the second obtaining module 1320 is configured to obtain a second timestamp when the current reading position coincides with the current writing position;

the second processing module 1330 is configured to update the reading position of this time to be the target reading position according to a difference between the first timestamp and the second timestamp.

In an optional embodiment, the second processing module 1330 is configured to, in response to that a difference between the first timestamp and the second timestamp is not greater than a time threshold, update the current reading position to be the second target reading position; the length between the history reading position and the history target reading position is a first length, and the length between the current reading position and the second target reading position is a second length, wherein the first length is smaller than the second length.

In an optional embodiment, the second processing module 1330 is configured to, in response to a difference between the first timestamp and the second timestamp being greater than a time threshold, update the current reading position to be the first target reading position; the length between the current reading position and the first target reading position is a second length, wherein the second length is a preset value.

In an optional embodiment, the second obtaining module 1320 is configured to obtain a total length of the history reading position of the consumer program when the history reading position is updated in the history updating process; the second processing module 1330 is configured to, in response to that the total length reaches a preset length, update the reading position of this time to a third target reading position, where a length between the reading position of this time and the third target reading position is the preset length; or, the second obtaining module 1320 is configured to obtain the total number of times when the history reading position of the consumer program is updated in the history updating process; the second processing module 1340 is configured to update the reading position of this time to a third target reading position in response to the total number of times reaching a preset number of times.

The present application also provides a computer device, as shown in fig. 14, the computer device 1400 includes a processor 1401 and a memory 1402, where the memory 1402 stores at least one instruction, at least one program, code set, or instruction set, and the instruction, program, code set, or instruction set is loaded and executed by the processor 1401 to implement the message queue based message transmission method and the message queue based message consumption method provided by the above method embodiments.

It should be understood that reference to "a plurality" herein means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above description is only exemplary of the present application and should not be taken as limiting, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A message queue-based message consumption method, the method comprising:

2. The method according to claim 1, wherein the updating the current reading position to a target reading position in response to the current writing position coinciding with the current reading position comprises:

acquiring a first time stamp of the historical reading position of the consumer program and the historical writing position of the producer program when the historical reading position and the historical writing position of the producer program coincide last time;

acquiring a second timestamp when the reading position and the writing position coincide;

and updating the reading position of this time to the target reading position according to the difference value of the first time stamp and the second time stamp.

3. The method according to claim 2, wherein the updating the current reading position to the target reading position according to the difference between the first timestamp and the second timestamp comprises:

updating the reading position of the time to be a second target reading position in response to the fact that the difference value of the first time stamp and the second time stamp is not larger than a time threshold value;

the length between the historical reading position and the historical target reading position is a first length, the length between the current reading position and the second target reading position is a second length, and the first length is smaller than the second length.

4. The method according to claim 2, wherein the updating the current reading position to the target reading position according to the difference between the first timestamp and the second timestamp comprises:

updating the reading position to be a first target reading position in response to the difference value between the first timestamp and the second timestamp being larger than a time threshold;

the length between the reading position and the first target reading position is a second length, wherein the second length is a preset value.

5. The method of claim 4, further comprising:

and updating the reading position to be a third target reading position in response to the second length reaching a preset length, wherein the length between the reading position and the third target reading position is the preset length.

6. A message queue-based message consumption apparatus, the apparatus comprising:

7. The apparatus of claim 6,

the acquisition module is used for acquiring a first timestamp of the last time when the historical reading position of the consumer program and the historical writing position of the producer program coincide;

the obtaining module is used for obtaining a second timestamp when the reading position and the writing position coincide;

and the processing module is used for updating the reading position to the target reading position according to the difference value between the first time stamp and the second time stamp.

8. The apparatus of claim 6,

the processing module is configured to update the reading position of this time to a first target reading position in response to a difference between the first timestamp and the second timestamp being greater than a time threshold;

the length between the historical reading position and the historical target reading position is a first length, the length between the current reading position and the first target reading position is a second length, and the first length is equal to the second length.

9. The apparatus of claim 6,

the processing module is configured to update the reading position of this time to a second target reading position in response to that a difference between the first timestamp and the second timestamp is not greater than a time threshold;

10. The apparatus according to any one of claims 7 to 9,

the acquisition module is used for acquiring the total length of the historical reading position of the consumer program when the historical reading position is updated in the historical updating process; the second processing module is configured to update the reading position of this time to a third target reading position in response to that the total length reaches a preset length, where a length between the reading position of this time and the third target reading position is the preset length;

or the like, or, alternatively,

the acquisition module is used for acquiring the total times of updating the historical reading position of the consumer program in the historical updating process; and the second processing module is used for responding to the total times reaching preset times and updating the current reading position to be the third target reading position.