CN114138530A - Message processing method, device, equipment and storage medium based on cloud protogenesis - Google Patents

Abstract

The invention discloses a message processing method, a message processing device, message processing equipment and a storage medium based on cloud originality. The method comprises the following steps: receiving a message processing request, wherein the message processing request comprises a target operation type; monitoring the abstract current message state of the current message flow; when the current message state is a fault-free state, executing a target operation process corresponding to the target operation type, and processing the message processing request; and executing a message recovery process when the current message state is a fault state, and executing a target operation process corresponding to the target operation type to process the message processing request when the current message state is monitored to be a non-fault state. The method can execute the target operation process corresponding to the target operation type only when the current message state is ensured to be a fault-free state, and the message processing request is processed, so that the consistency of subsequent message processing is ensured.

Description

Message processing method, device, equipment and storage medium based on cloud protogenesis

Technical Field

The present invention relates to the field of cloud-native technologies, and in particular, to a method, an apparatus, a device, and a storage medium for processing a message based on cloud-native.

Background

In the existing message processing system, when the message processing is carried out again after the message processing has a failure, the consistency of the message reading and writing can not be ensured.

Disclosure of Invention

The embodiment of the invention provides a message processing method and device based on cloud originality, computer equipment and a storage medium, and aims to solve the problem that the existing message processing system cannot guarantee the consistency of message reading and writing.

A message processing method based on cloud-native comprises the following steps:

receiving a message processing request, wherein the message processing request comprises a target operation type;

monitoring the abstract current message state of the current message flow;

when the current message state is a fault-free state, executing a target operation process corresponding to the target operation type, and processing the message processing request;

and executing a message recovery process when the current message state is a fault state, and executing a target operation process corresponding to the target operation type to process the message processing request when the current message state is monitored to be a non-fault state.

A cloud-native based message processing apparatus, comprising:

a processing request receiving module, configured to receive a message processing request, where the message processing request includes a target operation type;

the message state monitoring module is used for monitoring the abstract current message state of the current message flow;

the first processing module is used for executing a target operation process corresponding to the target operation type when the current message state is a fault-free state, and processing the message processing request;

and the second processing module is used for executing a message recovery process when the current message state is in a fault state, and executing a target operation process corresponding to the target operation type to process the message processing request when the current message state is monitored to be in a non-fault state.

A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the cloud-native based message processing method when executing the computer program.

A computer-readable storage medium storing a computer program which, when executed by a processor, implements the cloud-native based message processing method described above.

According to the cloud-native-based message processing method, the cloud-native-based message processing device, the computer equipment and the storage medium, when the current Broker receives the message processing request, the abstract current message state of the current message flow needs to be monitored, and only when the current message state is guaranteed to be in a fault-free state, the target operation process corresponding to the target operation type is executed, the message processing request is processed, and the consistency of subsequent message processing is guaranteed.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments of the present invention will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive labor.

Fig. 1 is a schematic diagram of an application environment of a cloud-based message processing method according to an embodiment of the present invention;

FIG. 2 is a flow chart of a cloud-based native message processing method in an embodiment of the invention;

FIG. 3 is another flow chart of a cloud-based native message processing method in an embodiment of the invention;

FIG. 4 is another flow chart of a cloud-based native message processing method in an embodiment of the invention;

FIG. 5 is another flow chart of a cloud-based native message processing method in an embodiment of the invention;

FIG. 6 is another flow chart of a cloud-based native message processing method in an embodiment of the invention;

FIG. 7 is another flow chart of a cloud-based native message processing method in an embodiment of the invention;

FIG. 8 is another flow chart of a cloud-based native message processing method in an embodiment of the invention;

FIG. 9 is a diagram of a cloud-based native message processing apparatus according to an embodiment of the present invention;

FIG. 10 is a schematic diagram of a computer device according to an embodiment of the invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The cloud-based message processing method provided by the embodiment of the invention can be applied to the application environment shown in fig. 1. Specifically, the message processing method based on cloud-native is applied to a message processing middleware architecture system based on cloud-native, and the system comprises at least one pulser cluster, each pulser cluster comprises a Broker cluster, a bootkeeper cluster and a Zookeeper cluster, each Broker cluster comprises at least one Broker (message service), and each bootkeeper cluster comprises at least one bootie (message storage).

Each Broker (message service) includes a dispatch interface (Dispatcher), a Load Balancer (Load Balancer) connected to the dispatch interface, Global replication interfaces (Global Replicators), message flow abstractions (managed bridges) and a Cache (Cache), and the message flow abstractions (managed bridges) are connected to the Bookie (message store) through BK Client (BZ interface).

Each bootie (message store) is used to implement a message store.

The Zookeeper cluster is connected to each Broker and Bookie and interacts with the Zookeeper clusters (ZKs) of the other Pulsar clusters.

In an embodiment, as shown in fig. 2, a cloud-native based message processing method is provided, which is described by taking the cloud-native based message processing middleware architecture system in fig. 1 as an example, and includes the following steps of:

s201: receiving a message processing request, wherein the message processing request comprises a target operation type;

s202: monitoring the abstract current message state of the current message flow;

s203: when the current message state is a fault-free state, executing a target operation process corresponding to the target operation type, and processing the message processing request;

s204: and when the current message state is in a fault state, executing a message recovery process, and when the current message state is monitored to be in a non-fault state, executing a target operation process corresponding to the target operation type to process the message processing request.

The current Broker refers to a Broker for which the system receives a message processing request at the current time.

The message processing request refers to a request received by a system at the current moment for processing a message, and the message processing request includes but is not limited to a message write request, a message read request and a message publishing request. The target operation type refers to a type defined in the message processing request for processing the message, and includes, but is not limited to, a write operation type, a read operation type, and an issue operation type.

As an example, in step S201, the current Broker may receive a message processing request sent by a user through a client, where the message processing request may be a message write request or a message publishing request sent by a message producer through the client, a target operation type corresponding to the message write request is a write operation type, and a target operation type corresponding to the message publishing request is a publishing operation type; the method can also be used for a message reading request sent by a message consumer through a client, wherein a target operation type corresponding to the message reading request is a reading operation type.

Wherein, the current message flow abstraction refers to the message flow abstraction (managed widget) in the current Broker. The current message state is a state used for reflecting whether the current message flow abstraction can work normally at the current moment of the system. Generally, a pulser cluster includes a plurality of brokers, each of which is stateless, and a message flow abstraction (managed widget) is a module for implementing state management, that is, when a message is written, the message is continuously added at the end of the stream through a writer process; reading or consuming a certain message at the message consumer, and determining the consumption position of each message consumer so as to realize state management.

The message flow abstraction (managed ridge) is an append-only data structure, and has only one writer process, and this writer process is responsible for writing of multiple bootkeeper storage nodes (i.e. booties), and the entries of the message flow abstraction (managed ridge) will be copied to multiple booties. Ledgers itself has very simple semantics: the Broker can create a message flow abstraction (managed widget), add content to the widget and close the widget; when a hedger is closed, the hedger will only be opened in read-only mode unless explicit data is to be written or because the writer process is suspended causing the hedger to close; when an entry in the hedgehog is no longer useful, the entire legder may be deleted.

As an example, in step S202, when receiving the message processing request, the current Broker needs to monitor the current message state in the current message stream abstraction to evaluate whether normal message writing, reading, or publishing operations can be performed.

As an example, in step S203, when the current Broker monitors that the current message state of the current message flow abstraction is a no-fault state, it indicates that the current message flow abstraction in the current Broker can normally work, and at this time, a target operation process corresponding to the target operation type may be executed to process the message processing request.

As an example, in step S204, when the current Broker monitors that the current message state of the current message stream abstraction is a fault state, it indicates that the current message stream abstraction in the current Broker cannot work normally, and if a target operation process corresponding to a target operation type is directly executed, a message processing request is processed, which may possibly cause inconsistency in message processing, at this time, a message recovery process needs to be executed first to perform message recovery processing, for example, a message recovery process may be used to perform a rollback operation to roll back a system state before a fault occurs; after the message recovery processing is executed, the current message state of the current message stream abstraction is monitored again, only when the current message state is in a fault-free state, the target operation process corresponding to the target operation type is executed, the message processing request is processed, and the consistency of message processing is favorably ensured. Understandably, because the current message stream abstraction can only be written by one writer process, the writer process has no conflict when writing operation is carried out, and the writing operation is more efficient, when the current message state is in a fault state, the current message stream abstraction can start a message recovery process to carry out recovery processing, and re-determine the current message state, and only when the current message state is in a non-fault state, a target operation process corresponding to a target operation type can be executed to process a message processing request, so as to ensure the consistency of subsequent message processing.

In this example, executing a target operation process corresponding to a target operation type to process a message processing request specifically includes: (1) and if the target operation type is the write operation type, executing a write operation process, acquiring a message to be processed from the message processing request, writing the message to be processed into the current Bookie, and writing the message to be processed into the current Bookie through a message flow abstraction (managed leader), which is favorable for ensuring the write efficiency of the message to be processed. (2) And if the target operation type is the read operation type, executing a write operation process, and pulling the required message to be processed from the current Bookie. In this example, the current Broker may pull the required to-be-processed message from the current Bookie in a stream pulling manner, so as to improve the obtaining efficiency of the to-be-processed message. Flow pulling is an improved version of long polling, not only achieves zero latency between individual calls and requests, but also provides bi-directional message flow, and employs a flow pulling approach with lower end-to-end delay than all existing long polling message systems (e.g., Kafka). (3) And if the target operation type is the issuing operation type, executing an issuing operation process and issuing the to-be-processed message corresponding to the message processing request.

In the message processing method based on cloud-native provided by this embodiment, when receiving a message processing request, the current Broker needs to monitor the current message state of the current message stream abstraction first, and only when it is ensured that the current message state is a fault-free state, a target operation process corresponding to a target operation type is executed to process the message processing request, which is helpful for ensuring consistency of subsequent message processing.

In an embodiment, as shown in fig. 3, step S301, namely executing a target operation process corresponding to a target operation type, processes the message processing request, and includes:

s301: when the target operation type is the write operation type, obtaining a message to be processed and a message processing type from the message processing request;

s302: when the message processing type is a batch processing type, acquiring a current batch identifier and a message sequence identifier corresponding to the message to be processed, and judging whether a batch index state queue corresponding to the current batch identifier exists or not;

s303: if the batch index state queue exists, writing the message to be processed into the current Bookie, and updating the current index state corresponding to the message sequence identification in the batch index state queue;

s304: if the batch index state queue does not exist, writing the message to be processed into the current Bookie, creating a batch index state queue corresponding to the current batch identifier, and recording the current index state corresponding to the message sequence identifier;

s305: monitoring the current index state of the batch index state queue, and releasing the storage space corresponding to the batch index state queue when the current index state is the confirmation completion state.

The message to be processed refers to a message to be processed by the message processing request. Understandably, when the target operation type is the write operation type, that is, when the message processing request is the message write request, the message to be processed refers to a message carried in the message processing request. The message processing type refers to a type that needs to process the message to be processed at this time.

As an example, in step S301, when the current Broker monitors that the current message status is a non-failure status and the target operation type is a write operation type, the message processing request needs to be parsed, and the message to be processed and the message processing type are obtained from the message processing request, so as to perform a message write operation according to the message to be processed and the message processing type.

The batch processing type is a processing type in which a plurality of messages need to be written in a batch. The batch index status queue refers to a queue created in advance for storing progress statuses of batch writes of a plurality of messages.

As an example, in step S302, when the current Broker identifies that the message processing type in the message processing request is a batch type, it indicates that the user writes batch messages through the client, at this time, a current batch identifier and a message sequence identifier in the message processing request need to be read, where the current batch identifier is an identifier for uniquely identifying which batch of messages is specifically the current batch identifier, and the message sequence identifier is an identifier for uniquely identifying each message in the batch messages to be processed. In this example, after acquiring the current batch identifier and the message sequence identifier from the message to be processed, the current Broker may query whether a batch index status queue corresponding to the current batch identifier exists, to determine that the same batch of messages has been written into the Bookie before the current time of the system.

As an example, in step S303, when the current Broker has the batch index state queue, it indicates that the same batch of messages is already written into the current Bookie before the current time of the system, at this time, the current Broker writes a message to be processed into the current Bookie, and updates the current index state corresponding to the message sequence identifier in the batch index state queue, that is, the current index state corresponding to the message sequence identifier in the batch index state queue is updated from the to-be-transmitted state to the confirmation complete state. Understandably, the message processing request may also carry a message batch number corresponding to the current batch identifier, and the message batch number may be understood as the number of times that batch transmission is required. For example, when a batch of messages a needs to be written into the current bootie, the batch of messages a may be divided into 10 batches of writes, and then the number of the message batches is 10, each batch of messages to be processed carries a unique message sequence identifier a1 … … a10, and the like, and the current index state corresponding to each message sequence identifier is recorded in the batch index state queue, which may reflect a state of whether the message to be processed has completed a write operation.

As an example, in step S304, when the current Broker has a batch index state queue, it indicates that the same batch of messages is not written into the current Bookie before the current time of the system, at this time, the current Broker writes a message to be processed into the current Bookie, then a batch index state queue corresponding to the current batch identifier needs to be created, and in the created batch index state queue, a current index state corresponding to the message sequence identifier is recorded, that is, the current index state corresponding to the message sequence identifier in the batch index state queue is recorded as a confirmation completion state.

As an example, in step S305, the current Broker monitors the current index state in the batch index state queue, and when all the current index states are the confirmation completion state, determines that the same batch of messages corresponding to the current batch of identifiers are all written into the current Bookie, and at this time, releases the storage space corresponding to the batch index state queue.

Understandably, when the target operation type is the write operation type and the message processing type is the batch processing type, the batch index state queue is utilized to monitor the progress of writing all messages to be processed in the same batch into the current Bookie, which is beneficial to ensuring that all messages to be processed in the same batch can be accurately written into the current Bookie.

In an embodiment, as shown in fig. 4, the step S301, namely, executing the target operation process corresponding to the target operation type, and processing the message processing request includes:

s401: when the target operation type is the write operation type, obtaining a message to be processed and a message processing type from the message processing request;

s402: when the message processing type is a fragment processing type, acquiring a plurality of fragment messages corresponding to the message to be processed;

s403: storing a plurality of fragmented messages in a current cache;

s404: writing all the fragment messages in the current cache into the current Bookie, and updating the message transmission state of each fragment message in the current cache;

s405: and if all the message transmission states are the fragmentation completion state, releasing the current cache.

The message to be processed refers to a message to be processed by the message processing request. Understandably, when the target operation type is the write operation type, that is, when the message processing request is the message write request, the message to be processed refers to a message carried in the message processing request. The message processing type refers to a type that needs to process the message to be processed at this time.

As an example, in step S401, when the current Broker monitors that the current message status is a no-fault status and the target operation type is a write operation type, the message processing request needs to be parsed, and the message to be processed and the message processing type are obtained from the message processing request, so as to perform a message write operation according to the message to be processed and the message processing type.

The fragmentation processing type is that the same data is divided into a plurality of fragmentation messages, so that the storage space required by each fragmentation message is small, and the improvement of the resource utilization rate of the to-be-processed message storage is facilitated.

As an example, in step S402, when the current Broker identifies that the message processing type in the message processing request is a fragment processing type, the current Broker may perform fragment processing on the message to be processed based on a preset fragment message capacity or a user-defined fragment message capacity, so as to obtain a plurality of fragment messages.

The current Cache refers to a Cache (Cache) in the current Broker, and is used for caching messages to be processed.

As an example, in step S403, after the current Broker divides the message to be processed into multiple fragment messages, the fragment messages need to be stored in a current cache in the current Broker, which is helpful for ensuring the security of transmission of the message to be processed, and avoiding that part of the fragment messages are lost during writing the multiple fragment messages into the bootie, thereby affecting the integrity of the entire message to be processed.

As an example, in step S404, when the current Broker caches all the fragmentation messages in the current cache, the current Broker may write all the fragmentation messages in the current cache into a current bootie, where the current bootie may be one bootie or multiple booties, so as to implement write operation on the fragmentation messages, and after each fragmentation message is written into the current bootie, the message transmission state of each fragmentation message in the current cache needs to be updated, that is, the message transmission state corresponding to the fragmentation message is updated from the to-be-transmitted state to the fragmentation completed state.

As an example, in step S405, when monitoring that the message transmission states corresponding to all fragment messages are fragment complete states, the current Broker determines that all messages to be transmitted have been written into the current Bookie, and at this time, releases the current cache, so that the current cache can perform cache processing on other messages, which is beneficial to ensuring resource utilization.

In this example, when a message producer issues or writes a message to be processed, if the message size of the message to be processed is larger than the maximum allowable issue load size, the message producer needs to divide the message to be processed into a plurality of fragment messages, issue the plurality of fragment messages to the current Broker individually or sequentially, the current Broker stores all the received fragment messages in the current cache, stores the fragment messages and other messages to be processed on a message flow abstraction (managed leader) in the same manner, and writes all the fragment messages and other messages to be processed in the current Bookie.

Understandably, when the message producer divides the message to be processed into a plurality of fragment messages and stores the fragment messages in at least one current Bookie, when the message consumer reads the message to be processed, namely, when the target operation type of the message processing request sent to the current Broker is a read operation type and the message processing type is a fragment processing type, a buffer area for realizing message merging needs to be opened in the current cache, so that after all fragment messages are collected, the fragment messages are merged to form the message to be processed. Generally speaking, when a message producer fails to issue all the fragment messages corresponding to the message to be processed, a message consumer cannot receive all the fragment messages corresponding to the same message to be processed within the target expiration time, and then the message consumer determines that the issuance of the message to be processed fails, and at this time, the fragment messages in the buffer area can be cleared, so that the storage space is saved, and the resource utilization rate is improved. The target expiration time may be a system default expiration time or a user-defined expiration time, such as 1 h.

In an embodiment, as shown in fig. 5, the cloud-native based message processing method further includes:

s501: monitoring a message storage duration corresponding to a message to be processed in the current Bookie;

s502: and if the message storage duration is greater than the target duration threshold, executing a data migration process, and migrating the message to be processed in the current Bookie to a disk space.

The message storage duration refers to the duration of the message to be processed stored in the current Bookie. The target duration threshold is a preset threshold for evaluating whether the message storage duration reaches a longer criterion. The data migration process is a process for implementing data migration.

As an example, in step S501, the system monitors, in real time, a message storage duration corresponding to each message to be processed in the current Bookie, where the message storage duration is a time difference between a current time of the system and a message write time corresponding to the message to be processed. The message writing time corresponding to the message to be processed refers to the time when the message to be processed is written into the current Bookie.

As an example, in step S502, when it is monitored that the message storage duration corresponding to the to-be-processed message is greater than the target duration threshold, it is described that the to-be-processed message is already stored in the current booster for a longer time, and is most likely to be old data with lower timeliness, at this time, the to-be-processed message with the longer message storage duration may be migrated from the current booster to a disk space with lower cost, so as to release the current booster with higher cost, which is beneficial to reducing the message storage cost.

Understandably, the to-be-processed message with the message storage duration being greater than the target duration threshold is stored in the disk space with lower cost, so that the storage cost can be reduced, the new and old data are naturally isolated, the subsequent conflict and contention of IO read-write can be avoided, in addition, in the processing processes of node capacity expansion, error recovery and the like, a large amount of operations such as copying, recovery and the like can be avoided, and the high availability of the system is favorably improved.

In an embodiment, as shown in fig. 6, the cloud-native based message processing method further includes:

s601: receiving a concurrent storage request formed based on a message to be processed;

s602: calling a concurrent storage process according to the concurrent storage request, and concurrently writing the messages to be processed into the associated Bookie corresponding to the first quantity;

s603: monitoring a storage feedback message returned by the associated Bookie, and counting a second quantity of the storage feedback messages;

s604: acquiring a message feedback proportion according to the first quantity and the second quantity;

s605: and if the message feedback proportion is larger than the target proportion threshold value, controlling the concurrent storage process to stop working.

The concurrent storage request is a request for implementing concurrent storage of messages. The concurrent storage process is a pre-configured process for implementing concurrent storage. The associated Bookie is other Bookies than the current Bookie already storing messages to be processed. The target proportion threshold is a preset threshold for evaluating whether the proportion-larger criterion is reached.

As an example, in step S601, the current Broker may receive a concurrent storage request based on one item of the pending message, so as to concurrently store the pending message according to the concurrent storage request, so as to guarantee the security of the pending message.

As an example, in step S602, the current Broker may invoke a preset concurrent storage process according to the concurrent storage request, and concurrently write the to-be-processed message into the associated bookies corresponding to the first number through a Global replication interface (Global Replicators) in the current Broker, where the first number is the number of messages to be processed for concurrent processing, that is, the number of written associated bookies.

As an example, in step S603, the current Broker may listen to a storage feedback message returned by the associated Bookie in real time, where the storage feedback message may reflect that the message to be processed has been copied and stored to the associated Bookie. In this example, the current Broker may count the number of received storage feedback messages in real time, and determine the number as a second number, where the second number may reflect how many associated bookies have already stored messages to be processed at the current time of the system.

As an example, in step S604, the current Broker may determine, as the message feedback ratio, a quotient of the first number and the second number according to the first number corresponding to the transmission of the message to be processed and the second number corresponding to the reception of the stored feedback message.

As an example, in step S605, the current Broker may calculate a message feedback ratio in real time, compare the message feedback ratio with a target ratio threshold preset by the system, and when the message feedback ratio is greater than the target ratio threshold, determine to copy and store the to-be-processed message in more associated booties, which may effectively ensure the security of the to-be-processed message, and at this time, control the concurrent storage process to stop working in order to save system resources.

In this example, Pulsar employs a similar Qurom protocol, and provides an available Bookie pool (i.e., an associated Bookie), when a concurrent storage process is executed, a part of the associated Bookie in which a message to be processed can be concurrently written, and as long as a part of the associated Bookie is returned successfully, the copy storage is determined to be successful, for example, when a message feedback ratio determined by the first number and the second number is greater than a target ratio threshold (e.g., 50%), the copy storage of the message to be processed is determined to be completed. By adopting the concurrent storage mode, the data replication can be carried out more efficiently, and particularly, when the number of data replicas is large, the effect is more obvious. Understandably, the copy storage of each message to be processed is only stored in different Bookies, and the storage processing is not performed by different Broker, so that the storage processing flow can be simplified.

In an embodiment, as shown in fig. 7, step S301, namely executing a target operation process corresponding to a target operation type, processes the message processing request, and includes:

s701: when the target operation type is the issuing operation type, obtaining a message to be processed, a message subject and a current producer from the message processing request;

s702: determining a producer mode corresponding to the message theme based on the message theme;

s703: if the producer mode is not the producer exclusive mode, directly issuing a message to be processed;

s704: and if the producer mode is the producer exclusive mode, acquiring the target authority corresponding to the message theme and the current authority of the current producer, verifying the authority according to the target authority and the current authority, and issuing the message to be processed after the authority verification is passed.

Wherein the publishing operation type is a type for implementing a message publishing operation. The message theme refers to a theme corresponding to the message to be processed. The current producer refers to the producer that triggered the message processing request. Producer exclusive mode refers to a mode in which only one producer can exclusively use a certain message subject.

As an example, in step S701, when the current Broker monitors that the current message status is a no-fault status and the target operation type is an issuing operation type, the message processing request needs to be parsed, and information such as a message to be processed, a message subject, a current producer, and the like is obtained from the message processing request.

As an example, in step S702, after the current Broker obtains the message topic from the message processing request, the Broker needs to query a record in a system database according to the message topic to determine a producer mode corresponding to the message topic, so as to evaluate whether the producer mode is a producer exclusive mode, and further determine whether the current producer has an authority to issue a to-be-processed message corresponding to the message topic.

As an example, in step S703, when the producer mode corresponding to the message topic is not the producer exclusive mode, the Broker currently indicates that the publishing of the to-be-processed message corresponding to the message topic is not limited by the identity of the producer, and at this time, the to-be-processed message may be directly published.

The target permission corresponding to the message theme refers to permission of issuing the message corresponding to the message theme. The current rights of the current producer are the rights of messages that the current producer can publish.

As an example, in step S704, when the producer mode corresponding to the message topic is the producer exclusive mode, it indicates that the publishing of the to-be-processed message corresponding to the message topic is limited by the identity of the producer, and it is required to obtain the target authority corresponding to the message topic and the current authority of the current producer; and the authority is checked according to the target authority and the current authority, and the message to be processed is issued only after the authority is checked to pass, so that the message issuing process of different message themes can be managed to meet the user requirements.

In this example, in step S704, if the producer mode is the producer exclusive mode, the target right corresponding to the message topic and the current right of the current producer are obtained, right verification is performed according to the target right and the current right, and after the right verification passes, the to-be-processed message is issued, which specifically includes: (1) if the producer mode is the producer exclusive mode, whether the message theme has an existing producer is judged, and the existing producer refers to a producer which sends other messages of the same message theme before the current time of the system. (2) And if the message theme does not have the existing producer, associating the current producer with the message theme and issuing the message to be processed. That is, when there is no existing producer in the message topic, it is described that although the producer mode of a certain message topic is the producer exclusive mode, there is no existing producer issuing a message corresponding to the message topic before the current time of the system, and at this time, the current producer and the message topic may be associated and a message to be processed may be issued. (3) And if the message theme has the existing producer, performing authority verification on the current producer and the existing producer, and issuing the message to be processed after the authority verification is passed. In this example, if the current authority of the current producer is greater than the target authority corresponding to the message topic, it is indicated that the current producer has the authority to issue the to-be-processed message corresponding to the message topic, the authorization verification is passed, and the to-be-processed message can be issued; if the current authority of the current producer is not greater than the target authority corresponding to the message subject, the current producer is proved to have no authority to send the message to be processed corresponding to the message subject, the authority verification is determined not to pass, and the release failure information can be fed back.

In one embodiment, as shown in fig. 8, step S301, namely executing a target operation process corresponding to a target operation type, processes the message processing request, and includes:

s801: acquiring a message pushing request, wherein the message pushing request comprises a message theme and a pushing frequency;

s802: sending the message pushing request to a message consumer, and waiting to select to receive consumption confirmation information returned by the message consumer within target waiting time;

s803: if the message confirmation information is received within the target waiting time and is the positive consumption information, pushing the to-be-processed message corresponding to the message theme to the message consumer according to the message frequency;

s804: if the message confirmation information is received within the target waiting time and is negative consumption information, the message theme and the message consumers are recorded in an associated mode;

s805: and if the message confirmation information is not received within the target waiting time, executing a retransmission pushing process, and sending the message pushing request to the message consumer again.

Wherein the message push request is a request for realizing message push. The message theme is the theme of the message that needs to be pushed. The push frequency refers to how often a message is pushed every time. A message consumer is a user who consumes messages. The target wait time is a preset wait time. The consumption confirmation information is confirmation information of the message consumer.

As an example, in step S801, the current Broker may obtain a message push request triggered by a user, where a message topic and a push frequency that need to be pushed are displayed on the message topic.

As an example, in step S802, after acquiring the message push request, the current Broker needs to send the message push request to the message consumer, so that the message consumer can determine whether to receive the message related to the message topic pushed by the system at the push frequency according to the message topic and the push frequency in the message push request. In this example, after sending the message pushing request to the message consumer, the current Broker needs to wait for receiving the consumption confirmation information returned by the message consumer within the target waiting time, so as to determine whether the message consumer receives the message corresponding to the pushed message topic.

Wherein the positive consumption information is information of the consent to receive the push fed back by the message consumer. A pending message refers to a message that needs to be pushed to a message consumer.

As an example, in step S803, the current Broker receives the message confirmation information within the target waiting time, and the message confirmation information is positive consumption information, which indicates that the message consumer agrees to receive push feedback, at this time, the to-be-processed message corresponding to the message topic may be pushed to the message consumer according to the message frequency, so as to implement to directionally push the to-be-processed message corresponding to the specific message topic to the message consumer related to the message topic, which is beneficial to improving the pertinence of the to-be-processed message.

The negative consumption information refers to information that the message consumer feeds back and does not agree to receive the push.

As an example, in step S804, the current Broker receives the message confirmation information within the target waiting time, and the message confirmation information is negative consumption information, which indicates that the message consumer feeds back a piece of information that is not agreed to receive and push, and determines that the message consumer does not pay attention to the message corresponding to the message topic, at this time, the message topic and the message consumer are recorded in a correlated manner, so as to perform statistical analysis on the message consumer and the message topic that the message consumer does not pay attention to subsequently.

As an example, in step S805, when the message acknowledgement information is not received by the current Broker within the target waiting time, there may be a message delay, at this time, a retransmission push process is performed to resend the message push request to the message consumer. In this example, when the current Broker executes a retransmission pushing process, the message retransmission times corresponding to each message consumer need to be updated, and when the message retransmission times do not reach the retransmission times threshold, the message pushing request is sent to the message consumer again; when the message retransmission times reach the retransmission times threshold value, the default message consumer does not pay attention to the message corresponding to the message theme, and at the moment, the message theme and the message consumer are recorded in a correlated mode, so that the message consumer and the message theme which is not paid attention to the message consumer can be subjected to statistical analysis in the following process.

In one embodiment, as shown in fig. 8, step S301, namely executing a target operation process corresponding to a target operation type, processes the message processing request, and includes:

(1) currently, the Broker pushes a message to be processed to the message consumer, and waits for receiving consumption confirmation information or consumption cancellation information fed back by the message consumer within a preset waiting time.

(2) And if the consumption confirmation information can be received within the preset waiting time, deleting or reserving the message to be processed. In this example, when the message consumer successfully consumes a pending message, the message consumer sends a consumption confirmation message to the current Broker.

(3) If the consumption confirmation information or the consumption cancellation information is not received within the preset waiting time or the consumption cancellation information is received within the preset waiting time, the message to be processed is retransmitted to the consumption consumer. In this example, the pending message is not consumed successfully, the current Broker automatically re-delivers the pending message, and the unacknowledged message automatic re-delivery mechanism tracks all unacknowledged pending messages within the preset wait time, that is, the pending message for which the consumption information is not received within the preset wait time will be re-sent to the message consumer. When the message consumer does not successfully consume the pending message within the pre-waiting time, the message consumer wants to consume the pending message again, the message consumer can send a cancellation consumption message to the current Broker, and the Broker retransmits the pending message to the message consumer.

In an embodiment, the cloud-native based message processing method further includes: (1) currently, a Broker receives a pending message sent by a message producer, the pending message including a message Topic (Topic) and a sending pattern. (2) And the current Broker routes the message partition to be processed to the target Broker corresponding to the message Topic (Topic) according to the sending mode so as to update the target Broker to be the new current Broker. (3) And the target Broker (i.e. the new current Broker) determines the corresponding target subscription mode according to the message Topic (Topic). (4) And the target Broker (namely the new current Broker) sends the target message to the corresponding message consumer according to the target subscription mode. The subscription mode is any one of exclusive, shared, disaster recovery and key sharing.

It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

In an embodiment, a message processing apparatus based on cloud-origin is provided, and the message processing apparatus based on cloud-origin is in one-to-one correspondence with the message processing method based on cloud-origin in the above embodiment. As shown in fig. 9, the cloud-native based message processing apparatus includes a processing request receiving module 901, a message status listening module 902, a first processing module 903, and a second processing module 904. The functional modules are explained in detail as follows:

a processing request receiving module 901, configured to receive a message processing request, where the message processing request includes a target operation type;

a message state monitoring module 902, configured to monitor a current message state of a current message stream abstraction;

a first processing module 903, configured to execute a target operation process corresponding to a target operation type when a current message state is a no-fault state, and process a message processing request;

the second processing module 904 is configured to execute a message recovery process when the current message state is a fault state, and execute a target operation process corresponding to the target operation type to process the message processing request when it is monitored that the current message state is a non-fault state.

In one embodiment, a cloud-native based message processing apparatus includes:

the first write operation unit is used for acquiring a message to be processed and a message processing type from the message processing request when the target operation type is the write operation type;

a state queue judging unit, configured to, when the message processing type is a batch processing type, obtain a current batch identifier and a message sequence identifier corresponding to a message to be processed, and judge whether a batch index state queue corresponding to the current batch identifier exists;

the first batch processing unit is used for writing the message to be processed into the current Bookie if a batch index state queue exists, and updating the current index state corresponding to the message sequence identifier in the batch index state queue;

the first batch processing unit is used for writing the message to be processed into the current Bookie if no batch index state queue exists, creating a batch index state queue corresponding to the current batch identifier, and recording the current index state corresponding to the message sequence identifier;

and the state queue releasing unit is used for monitoring the current index state of the batch index state queue and releasing the storage space corresponding to the batch index state queue when the current index state is the confirmation completion state.

In one embodiment, a cloud-native based message processing apparatus includes:

the second write operation unit is used for acquiring the message to be processed and the message processing type from the message processing request when the target operation type is the write operation type;

the message processing device comprises a fragmentation message acquisition unit, a message processing unit and a message processing unit, wherein the fragmentation message acquisition unit is used for acquiring a plurality of fragmentation messages corresponding to messages to be processed when the message processing type is a fragmentation processing type;

the fragment message cache unit is used for storing a plurality of fragment messages in the current cache;

the fragmentation message writing unit is used for writing all fragmentation messages in the current cache into the current Bookie and updating the message transmission state of each fragmentation message in the current cache;

and the current cache releasing unit is used for releasing the current cache if all the message transmission states are the fragmentation completion states.

In one embodiment, a cloud-native based message processing apparatus includes:

the storage duration monitoring unit is used for monitoring the message storage duration corresponding to the message to be processed in the current Bookie;

and the message migration processing unit is used for executing a data migration process and migrating the message to be processed in the current Bookie to the disk space if the message storage duration is greater than the target duration threshold.

In one embodiment, a cloud-native based message processing apparatus includes:

a concurrent request acquisition unit for receiving a concurrent storage request formed based on the message to be processed;

the concurrent storage processing unit is used for calling a concurrent storage process according to the concurrent storage request, and concurrently writing the messages to be processed into the associated Bookie corresponding to the first quantity;

the feedback message counting unit is used for monitoring the storage feedback messages returned by the associated Bookie and counting the second quantity of the storage feedback messages;

a feedback proportion obtaining unit, configured to obtain a message feedback proportion according to the first number and the second number;

and the concurrent process stopping unit is used for controlling the concurrent storage process to stop working if the message feedback proportion is larger than the target proportion threshold.

In one embodiment, a cloud-native based message processing apparatus includes:

the sending operation processing unit is used for acquiring the message to be processed, the message subject and the current producer from the message processing request when the target operation type is the issuing operation type;

the producer mode determining unit is used for determining a producer mode corresponding to the message theme based on the message theme;

the first message issuing unit is used for directly issuing the message to be processed if the producer mode is not the producer exclusive mode;

and the second message sending unit is used for acquiring the target authority corresponding to the message subject and the current authority of the current producer if the producer mode is the producer exclusive mode, performing authority verification according to the target authority and the current authority, and issuing the message to be processed after the authority verification is passed.

In one embodiment, a cloud-native based message processing apparatus includes:

the push request acquisition unit is used for acquiring a message push request, and the message push request comprises a message theme and a push frequency;

the push request sending unit is used for sending the message push request to the message consumer and receiving the consumption confirmation information returned by the message consumer to be selected within the target waiting time;

the first push processing unit is used for pushing the message to be processed corresponding to the message theme to the message consumer according to the message frequency if the message confirmation information is received within the target waiting time and the message confirmation information is the positive consumption information;

the second pushing processing unit is used for associating and recording the message theme and the message consumer if the message confirmation information is received within the target waiting time and is negative consumption information;

and the third push processing unit is used for executing a retransmission push process and sending the message push request to the message consumer again if the message confirmation information is not received within the target waiting time.

Specific limitations on the cloud-based message processing apparatus may refer to the above limitations on the cloud-based message processing method, which are not described herein again. The modules in the cloud-native based message processing apparatus may be wholly or partially implemented by software, hardware, or a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 10. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing data adopted or generated in the process of executing the cloud-native-based message processing method. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a cloud-native based message processing method.

In an embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and when the processor executes the computer program, the cloud-based native message processing method in the foregoing embodiments is implemented, for example, S201 to S204 shown in fig. 2, or as shown in fig. 3 to fig. 8, which is not described herein again to avoid repetition. Alternatively, when the processor executes the computer program, the functions of each module/unit in the embodiment of the cloud-native-based message processing apparatus are implemented, for example, the functions of the processing request receiving module 901, the message status monitoring module 902, the first processing module 903, and the second processing module 904 shown in fig. 9, which are not described herein again to avoid repetition.

In an embodiment, a computer-readable storage medium is provided, where a computer program is stored on the computer-readable storage medium, and when executed by a processor, the computer program implements the cloud-based native message processing method in the foregoing embodiments, for example, S201 to S204 shown in fig. 2, or shown in fig. 3 to fig. 8, which is not described herein again to avoid repetition. Alternatively, when executed by a processor, the computer program implements functions of each module/unit in the foregoing message processing apparatus based on cloud-native, for example, functions of the processing request receiving module 901, the message status monitoring module 902, the first processing module 903, and the second processing module 904 shown in fig. 9, and in order to avoid repetition, details are not described here again.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules, so as to perform all or part of the functions described above.

The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.

Claims (10)

1. A message processing method based on cloud-based originality is characterized by comprising the following steps:

receiving a message processing request, wherein the message processing request comprises a target operation type;

monitoring the abstract current message state of the current message flow;

when the current message state is a fault-free state, executing a target operation process corresponding to the target operation type, and processing the message processing request;

and executing a message recovery process when the current message state is a fault state, and executing a target operation process corresponding to the target operation type to process the message processing request when the current message state is monitored to be a non-fault state.

2. The cloud-native based message processing method of claim 1, wherein the executing the target operation process corresponding to the target operation type processes the message processing request, comprising:

when the target operation type is a write operation type, acquiring a message to be processed and a message processing type from the message processing request;

when the message processing type is a batch processing type, acquiring a current batch identifier and a message sequence identifier corresponding to the message to be processed, and judging whether a batch index state queue corresponding to the current batch identifier exists or not;

if the batch index state queue exists, writing the message to be processed into the current Bookie, and updating the current index state corresponding to the message sequence identifier in the batch index state queue;

if the batch index state queue does not exist, writing the message to be processed into the current Bookie, creating a batch index state queue corresponding to the current batch identifier, and recording the current index state corresponding to the message sequence identifier;

monitoring the current index state of the batch index state queue, and releasing the storage space corresponding to the batch index state queue when the current index state is a confirmation completion state.

3. The cloud-native based message processing method of claim 1, wherein the executing the target operation process corresponding to the target operation type processes the message processing request, comprising:

when the target operation type is a write operation type, acquiring a message to be processed and a message processing type from the message processing request;

when the message processing type is a fragment processing type, acquiring a plurality of fragment messages corresponding to the message to be processed;

storing a plurality of the fragmented messages in a current cache;

writing all the fragment messages in the current cache into the current Bookie, and updating the message transmission state of each fragment message in the current cache;

and if all the message transmission states are the fragmentation completion state, releasing the current cache.

4. The cloud-native based message processing method of claim 1, wherein the cloud-native based message processing method further comprises:

monitoring a message storage duration corresponding to a message to be processed in the current Bookie;

and if the message storage duration is greater than the target duration threshold, executing a data migration process, and migrating the message to be processed in the current Bookie to a disk space.

5. The cloud-native based message processing method of claim 1, wherein the cloud-native based message processing method further comprises:

receiving a concurrent storage request formed based on the message to be processed;

according to the concurrent storage request, calling a concurrent storage process, and concurrently writing the message to be processed into the associated Bookie corresponding to the first quantity;

monitoring a storage feedback message returned by the associated Bookie, and counting a second quantity of the storage feedback message;

acquiring a message feedback proportion according to the first quantity and the second quantity;

and if the message feedback proportion is larger than a target proportion threshold value, controlling the concurrent storage process to stop working.

6. The cloud-native based message processing method of claim 1, wherein the executing the target operation process corresponding to the target operation type processes the message processing request, comprising:

when the target operation type is a publishing operation type, obtaining a message to be processed, a message subject and a current producer from the message processing request;

determining a producer mode corresponding to the message theme based on the message theme;

if the producer mode is not the producer exclusive mode, directly issuing the message to be processed;

and if the producer mode is a producer exclusive mode, acquiring a target authority corresponding to the message theme and the current authority of the current producer, performing authority verification according to the target authority and the current authority, and issuing the message to be processed after the authority verification is passed.

7. The cloud-native based message processing method of claim 1, wherein the executing the target operation process corresponding to the target operation type processes the message processing request, comprising:

acquiring a message pushing request, wherein the message pushing request comprises a message theme and a pushing frequency;

sending the message pushing request to a message consumer, and waiting to select to receive consumption confirmation information returned by the message consumer within target waiting time;

if message confirmation information is received within the target waiting time and is positive consumption information, pushing a message to be processed corresponding to the message theme to the message consumer according to the message frequency;

if message confirmation information is received within the target waiting time and is negative consumption information, the message theme and the message consumer are recorded in an associated mode;

and if the message confirmation information is not received within the target waiting time, executing a retransmission pushing process, and sending the message pushing request to the message consumer again.

8. A cloud-native-based message processing apparatus, comprising:

a processing request receiving module, configured to receive a message processing request, where the message processing request includes a target operation type;

the message state monitoring module is used for monitoring the abstract current message state of the current message flow;

the first processing module is used for executing a target operation process corresponding to the target operation type when the current message state is a fault-free state, and processing the message processing request;

and the second processing module is used for executing a message recovery process when the current message state is in a fault state, and executing a target operation process corresponding to the target operation type to process the message processing request when the current message state is monitored to be in a non-fault state.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the cloud-native based message processing method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the cloud-native based message processing method according to any one of claims 1 to 7.

Images