CN114253748B

CN114253748B - Message processing system and message processing method

Info

Publication number: CN114253748B
Application number: CN202111617631.4A
Authority: CN
Inventors: 叶刚
Original assignee: Beijing Yusys Technologies Group Co ltd
Current assignee: Beijing Yusys Technologies Group Co ltd
Priority date: 2021-12-27
Filing date: 2021-12-27
Publication date: 2023-01-31
Anticipated expiration: 2041-12-27
Also published as: CN114253748A

Abstract

The invention provides a message processing system and a message processing method, wherein the system comprises: the message production end sends the message in a pre-sending state to the message agent, executes local service operation and persists the local service operation to the first database, and sends a service operation result to the message agent; a message broker comprising: the receiving module is used for receiving the message in the pre-sending state; a second database storing messages in a pre-send state; a processing module, comprising: the pre-sent message processing submodule modifies the state of the message into a sending state and sends the message to the message queue when the confirmation information indicates that the service operation is successful; otherwise, deleting the message in the pre-sending state; a message queue for caching messages; the message consumption end receives the message and processes the local service; and the message state monitoring system is used for monitoring the message state and triggering the message agent to process the message with the abnormal state when the message state is abnormal. The system can ensure the consistency of the messages.

Description

Message processing system and message processing method

Technical Field

The invention relates to asynchronous message processing technology, which is mainly applied to the field of consumption finance, in particular to a message processing system and a message processing method.

Background

When providing a product based on a distributed message publish/subscribe scenario for users, some kind of message queue middleware product (ActiveMQ, rocktmq, rabbitMQ, kafka, etc., which are popular in the market) needs to be adopted as a part of the infrastructure to support the needs of services.

In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:

however, in the current situation, due to the problems that users have different types of message middleware products, and have different capabilities in technical teams, and obvious differences in constraint conditions for deployment and implementation, a general technical barrier is lacked, and huge research and development costs are additionally paid to meet the needs of different message middleware.

Disclosure of Invention

In view of this, embodiments of the present invention provide a message processing system and a message processing method to shield different message queue technology stacks, implement consistency of message distribution, and ensure that messages are not lost and not consumed repeatedly.

In a first aspect, an embodiment of the present invention provides a message processing system, where the message processing system includes:

the message production terminal is used for sending a message in a pre-sending state to the message agent, executing local business operation, persisting a business operation result to the first database, and sending the business operation result to the message agent;

a message broker, the message broker comprising: the receiving module is used for receiving the message in the pre-sending state; a second database for storing the message in a pre-send state; a processing module, comprising: the pre-sent message processing submodule is used for modifying the state of the message into a sending state when the confirmation information which is sent by the message production end and is associated with the business operation result indicates that the business operation is successful, and controlling a sending module to send the message to the message queue; if the confirmation information indicates that the service operation fails, deleting the message in the pre-sending state; the sending module is used for sending messages to the message queue under the control of the processing module;

the message queue is used for caching the messages sent by the message agent;

the message consumption end is used for receiving the message and processing the local service;

and the message state monitoring system is used for monitoring the state of the message in the full life cycle of the message and triggering the message agent to process the message with the abnormal state when the state of the message is abnormal.

In a second aspect, an embodiment of the present invention provides a message processing method, where the method includes:

the message production end sends a message in a pre-sending state to a message agent, executes local business operation, persists a business operation result to a first database, and sends the business operation result to the message agent;

the message agent receives the message in the pre-sending state, stores the message in the pre-sending state to a second database, modifies the state of the message into a sending state when the confirmation information which is sent by the message production end and is associated with the business operation result indicates that the business operation is successful, sends the message to the message queue, and deletes the message in the pre-sending state if the confirmation information indicates that the business operation is failed;

the message queue caches the messages sent by the message agent;

the message consumption end receives the message and processes the local service;

and the message state monitoring system monitors the state of the message in the full life cycle of the message, and triggers the message agent to process the message with abnormal state when the state of the message is abnormal.

In a third aspect, an embodiment of the present invention provides a message processing method, where the method includes:

monitoring the state of a message during its full life cycle;

and when the abnormal state of the message is monitored, triggering the message agent to process the message with the abnormal state.

Optionally, when the state of the message is monitored to be abnormal, the message agent is triggered to process the message with the abnormal state, which specifically includes any one or more of the following steps:

when the message in the abnormal sending state is monitored, triggering the message agent to process the message in the abnormal sending state;

when the dead message is monitored, triggering the message agent to delete or resend the dead message;

when the messages which are not confirmed are monitored, triggering the message agent to resend the messages which are not confirmed, recording the sending times, and converting the messages which are not confirmed into a death state if the sending times exceed the preset retry times;

and when the messages to be cleaned or backed up are monitored, triggering the message agent to clean the messages to be cleaned or back up the messages to be backed up.

Optionally, when the message in the abnormal sending state is monitored, the message agent is triggered to process the message in the abnormal sending state, which specifically includes:

monitoring abnormal pre-sending state information in the information agent, wherein the abnormal pre-sending state information refers to information of which the duration in a pre-sending state exceeds a preset first time threshold, and deleting the abnormal pre-sending state information; and/or the presence of a gas in the gas,

and monitoring an abnormal sending state message in the message agent, wherein the abnormal sending state message is a message of which the duration of the sending state exceeds a preset second time threshold, and triggering the message agent to resend the abnormal sending state message.

In a fourth aspect, an embodiment of the present invention provides a message status monitoring system, which includes:

the message state monitoring module is used for monitoring the state of the message in the whole life cycle of the message;

and the message exception handling module is used for triggering the message agent to handle the message with the abnormal state when the abnormal state of the message is monitored.

In some possible embodiments, the message exception handling module is specifically configured to perform any one or more of the following:

when the messages which are not confirmed are monitored, the message agent is triggered to resend the messages which are not confirmed, the sending times are recorded, and if the sending times exceed the preset retry times, the messages which are not confirmed are converted into a death state;

and when the message to be cleaned or backed up is monitored, triggering the message agent to clean the message to be cleaned or back up the message to be backed up.

In some possible embodiments, the message exception handling module is specifically configured to:

In a fifth aspect, a computer-readable storage medium is provided, in which a computer program is stored, and the computer program is executed by a processor to implement the steps of any one of the message processing methods.

In a sixth aspect, there is also provided a computer apparatus comprising:

one or more processors;

storage means for storing one or more programs;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the steps of any of the message processing methods described above.

The technical scheme has the following beneficial effects: the embodiment of the invention can solve the problems of how to rapidly release and receive the message and how to ensure the consistency of the message in the asynchronous message communication scene.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1A is an overall functional block diagram of a message processing system of an embodiment of the present invention;

FIG. 1B is a detailed functional block diagram of a message processing system according to an embodiment of the present invention;

FIG. 2 is a schematic diagram illustrating monitoring of the status of a message during its full life cycle according to an embodiment of the present invention;

FIG. 3 is a deployment diagram of a scalability design of an embodiment of the invention;

FIG. 4 is a flow chart of a message processing method according to an embodiment of the present invention;

FIG. 5 is a flow chart of another message processing method of an embodiment of the present invention;

FIG. 6 is a functional block diagram of a message condition monitoring system of an embodiment of the present invention;

FIG. 7 is a functional block diagram of a computer-readable storage medium of an embodiment of the present invention;

FIG. 8 is a functional block diagram of a computer device of an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.

The embodiment of the invention belongs to the asynchronous message processing technology, is mainly applied to the field of consumption finance, and is used for solving the problems of how to rapidly release and receive messages and how to ensure the consistency of the messages in an asynchronous message communication scene.

The inventor finds out in research that the prior art has the following problems:

(1) The maintenance workload is large. Some project parties actually maintain multiple sets of message queue middleware products for historical reasons, which results in increased operation and maintenance costs (configuring monitoring, alarming, index acquisition and the like of different products).

(2) The technical complexity is high. The method mainly embodies consideration and design of problems of message backlog, loss, compensation, retry, idempotent, current limitation, routing, sequentiality, cleaning, archiving, version upgrading, technology type selection and the like, which need a great deal of effort. The problems of consumption throughput, traffic scaling, high reliability, technology model selection, version upgrading and the like are considered from the aspect of architecture.

(3) Increasing the development burden of the service developers. Service developers need to fully pay attention to and solve the technical complexity of the message queue, and especially have higher requirements on large data volume and high concurrency scenes, limit delivery progress, disperse development efforts, damage product quality and the like.

In order to solve the above problem, embodiments of the present invention provide a general framework capable of ensuring that a service architecture is not affected by a specific message middleware, so that product development is more focused on the service field, and the speed and quality of product delivery are greatly improved.

1. Brief introduction to solution

As an independent middleware, by encapsulating a specific message queue middleware technology, the technology difference is shielded, a general protocol interface and specification are provided for service development, and service codes and technology codes are formed to evolve and manage separately, specifically as follows:

(1) Weakening the strong dependence of the mq middleware.

(2) And solving the consistency problem of the message among the micro-services by adopting a final consistency architecture idea.

(3) And realizing the ordered message consumption function.

(4) And deeply customizing to realize an enterprise solution meeting the characteristics of the accounting platform.

2. Definition of terms

RM, an acronym for Reliable Message handling system.

The Broker, the message Broker, is the service system's proxy for asynchronous message processing via RM technology. For the RM, encapsulation of the mq technology stack is done by the Broker, which is the RM's message Broker.

Data partitioning means that data is classified according to a hash algorithm, so that the data always stably belongs to a certain partition.

Instance cloning, i.e., the concept of replication, RM systems utilize the techniques of instance cloning to provide high availability for message processing.

The slicing factor, in the accounting system of this embodiment, defines the algorithm parameters participating in slicing through the debit number or the client number. In the RM system, a production end designates a fragmentation factor, and then the RM system calculates the fragmentation factor so as to obtain the index number of feign and find out an agent for calling the designated broker. Feign is a term in spring-bound open source architecture, and its role is to provide encapsulation and load balancing processing for http communication protocols.

3. Means for solving the problems

The embodiment of the invention realizes technical encapsulation through 3 aspects:

(1) And encapsulating the message queue interactive interface to achieve the purpose of shielding communication difference.

(2) And the distributed deployment strategy of the microservice is realized, and the aim of shielding the distributed deployment difference of the message queue is fulfilled.

(3) The method realizes a set of universal message distribution consistency, shields different message queue technology stacks, and ensures that the messages are not lost and not repeatedly consumed.

4. Principle of the technology

The 3 aspects mentioned in technical means are explained:

(1) A generic interaction interface is defined for the service invoker, while the technology differences are encapsulated inside the architecture.

(2) Based on the spring-bound technology stack, technical supports such as service registration discovery, central configuration management, routing distribution, load balancing, fusing and current limiting are provided for the RM which is independently deployed.

(3) Through the distributed deployment characteristics, a group of distributed transaction persistence sequences are designed, and the final consistency of message distribution is realized through technologies such as compensation and retry.

5. Description of constraints

(1) The RM system only provides the interfacing of the difference technology stack for the service system, and the definition and specification of the interface protocol are smoothed, so as to ensure the uniformity of the interfacing of the service codes, and the capability of the MQ technology per se is not rewritten or the benefit of the MQ technology per se is not reduced.

(2) The RM serving as a micro-service architecture can be independently deployed and version-evolved, and Mysql is used as a basic technology of bottom-layer data storage.

(3) The RM-supported index acquisition technology is based on a spring-boot activator module and monitoring middleware conforming to the interface specification, such as prometheus and the like. The activator is a term in spring-closed open source architecture, and its role is to provide definition and burial point for monitoring indicators.

(4) RM systems have provided external interfaces to different message queues, and currently, RM systems support MQ technology stacks such as ActiveMq, rocktMq, and the like. However, because the RM framework provides an extension specification of secondary development, the development cost of interfacing a new technology stack is very low, the secondary development can be completed within 2 days generally, and all technical output results of the RM are easily possessed.

6. Application scenario

It is applicable to the following scenarios:

(1) Focusing on platform level commonality, the use of message queues is completely encapsulated into the RM framework, and the business code is unaware of changes to the RM (such changes include what technology message queues to use, what communication protocols to use, transparently adding compute nodes, and governing functions around messages, etc.).

(2) The final consistency of the message is of concern. The management and control capability is mainly embodied in how to solve the problems of message loss, message compensation, cleaning, tracing, transparent transmission and the like, which is a very important operation and maintenance characteristic of the RM, and solves the problems of distributed system difficulties, such as data loss, inconsistency, repeated consumption and the like, faced by message management.

By compensating, it is meant that when the message states are inconsistent, the message push is reinitiated until the messages eventually agree.

The compensation algorithm of the present embodiment includes: the monitoring system is responsible for checking the sending state, retries and records the retry times when the message sending fails, and marks the message as dead if the retry times exceeds the threshold value of the retry times, and changes the message into manual processing.

The state inconsistency includes a number of possibilities, typical scenarios are as follows:

scene one: for the production side, what the pre-sent message is successful, but what the business operation fails.

The solution is as follows: the message state monitoring system is responsible for monitoring messages which are in a pre-sending state for a long time and deleting the messages after a certain time.

Scene two: for the production end, the pre-sending of the message is successful, the service operation is successful, but what the sending of the message fails.

The solution is as follows: the message state monitoring system is responsible for periodically inquiring the service execution result of the production end, if the result is found to be successful, the message broker initiates the sending of the pre-message, and the pre-message is ensured to be sent to the message queue.

Scene three: for the message broker, what happens is that the production side has received a successful request, but the sending message queue fails.

The solution is as follows: the message state monitoring system is responsible for monitoring the message which is in the sending state for a long time and then sending the message again.

Scene four: for a message broker, the message queue has been successfully sent, but the message state is always in progress or what is going to be sent.

The solution is as follows: the message state monitoring system is responsible for monitoring messages which are in a sending state or are to be sent for a long time, and when the number of times of retransmission is detected to be over limit, the messages are converted into death and sent to the client.

Scene five: for the consumer, the message is already listened to, but what happens is the local service processing failure.

The solution is as follows: the message state monitoring system is responsible for monitoring messages which are in a sending state or are to be sent for a long time and then resending the messages.

Scene six: for the consumer, the local message has been successfully processed, but what does the failure to publish the notification.

The solution is as follows: the message state monitoring system is responsible for monitoring the message which is in the sending state for a long time and then sending the message again, and the consuming end has to ensure the idempotent mechanism (namely, the idempotent mechanism is designed by the consuming end) because the consuming end has successfully processed the message.

(3) Focusing on the scalability of the RM, which is mainly embodied in instance cloning, data partitioning, monitoring, alarming, containerized operation, and the like, this is a very important operation and maintenance characteristic of the RM, and solving such problems can enable developers to release them from the scalability, and really put efforts on business design and development.

7. Description of the Critical design

7.1 Universal interface Standard design

The RM system accepts the functions of all message queue technologies, and the design goal is to provide the simplest interface protocol for the service system, which mainly includes the following contents:

receiving a message by an interface needing to be realized;

the message initiates the required configuration (e.g., thread, queue name snooped, order-related configuration, etc.).

The responsibility assumed for the business system and RM system is as follows:

for the production side service, such as order service, sending an http request to the RM agent service through the flight interface of the spring-closed technology stack, that is, the production side is unaware of the RM, which further embodies that the service code does not need to concern the design target of the MQ technology stack.

The RM agent service Broker takes over the responsibility of messaging, isolating the messaging activity from the service producer.

For consumer services, such as accounting engine services, only a self-defined class for implementing an imsgrceevecallbackhandler interface needs to be defined for receiving messages and processing own messages. In addition, the consumer side, after starting, passes through the internal configuration of the RM.

The main components of the RM system are as follows:

(1) The production end component mainly comprises a producer-starter and an mq-producer, wherein the mq-producer encapsulates each mq technology stack, for example, active mq, and the component is active-producer; and encapsulating the socket tmq to be socket-producer. Each producer is designed to encapsulate and hide the technical specification for the corresponding mq. The producer-starter is the name of the component package, the component function is to provide the reference of the package for the production message end, and the package realizes all related functions.

(2) A consumer-side component, mainly comprising a consumer-stator and an mq-consumer, wherein the mq-consumer encapsulates each mq technology stack, for example, encapsulates an active mq, and then the component is an active-consumer; and encapsulating the socket tmq to be socket-consumer. Each consumer is designed to encapsulate and hide the technical specification corresponding to mq. Wherein, consumer-starter is the name of the component package, and the component function is to provide the reference of the package for the consuming message end, and the package realizes all related function realization.

(3) The kernel component mainly comprises a kernel function (core), an automatic assembly function (autoconfig), a client interface (client), an instance factory (factory) and the like, and is responsible for operations such as interface specification definition, instance factory, dynamic configuration, message sending, message format, message notification, message result monitoring and the like.

(4) And the message agent component Broker is responsible for defining the request sent by the production end, converting the request message into a message for persistence and then distributing the message to a message queue.

7.2 message consistency design

As shown in fig. 1A, the message processing system includes:

and the message production end is used for sending the message in the pre-sending state to the message agent, executing local service operation, persisting the service operation result to the first database, and sending the service operation result to the message agent. In one example, the local business operations include, but are not limited to, a loan business operation.

A message broker, the message broker comprising: the receiving module is used for receiving the message in the pre-sending state; a second database for storing messages in a pre-send state; a processing module, comprising: the pre-sending message processing submodule is used for modifying the state of the message in the pre-sending state into a sending state when the confirmation information which is sent by the message production terminal and is associated with the service operation result indicates that the service operation is successful, and controlling the sending module to send the message to the message queue; if the confirmation information indicates that the service operation fails, deleting the message in the pre-sending state; and the sending module is used for sending the message to the message queue under the control of the processing module.

And the message queue is used for caching the messages sent by the message agent.

And the message consumption end is used for receiving the message and processing the local service.

Specifically, the pre-transmission message processing submodule is used for judging whether the confirmation information is successful or not, and controlling the transmission module to transmit the message to the message queue if the confirmation information is successful; if the acknowledgment information fails, the message in the pre-send state is deleted. The message in the pre-sending state means that before sending the message to the message queue, the message production end sends the message to a message broker (broker) first, and determines how to operate after waiting for the message production end to send confirmation information (service operation success/service operation failure/no response). If the confirmation message is successful, the message agent sends the message to the message queue; if the acknowledgment message is a failure, the message broker does not send the message to the message queue; if the result is not received for a long time, the message monitoring (monitor) polls the state of the message production terminal, and finds that the service operation is successful, the message agent sends the message to the message queue; if the service operation is found to fail, the message broker deletes the pre-sent message.

As shown in fig. 1B, in some embodiments, the processing module in the message broker further comprises: the message state monitoring and processing submodule is used for executing any one or more of the following operations:

cleaning messages at regular time or backing up messages at regular time; deleting the dead message; initiating the resending of the death message; inquiring the overtime message, and retransmitting the overtime message; the message monitoring system is responsible for inquiring an interface provided by a message production end and returning an inquiry result; and inquiring the messages which are overtime and exceed the preset sending times, and triggering manual processing. The message monitoring system can be responsible for inquiring the interface provided by the message production end and returning the inquiry result.

In some embodiments, the message consuming side is specifically configured to monitor messages in a message queue, and the message agent is responsible for delivering the messages; after the message in the message queue is successfully delivered to the message consumption end, receiving the message and processing the local service according to the received message; after the local service is processed, the processing result is stored in a service database of the message consumption end, and a notice is issued to the message agent to indicate that the message is successfully consumed.

In some embodiments, a message condition monitoring system to perform any one or more of the following;

and monitoring the message in the abnormal sending state, and triggering the message agent to process the message in the abnormal sending state.

Monitoring the dead message, and triggering a message agent to delete or resend the dead message; as an example, a message may be deleted when it cannot be processed due to a message format error. The method can also comprise the following steps: monitoring the dead message, outputting prompt or notice information to the user or client, receiving (first) processing instruction input by the user or client, and triggering the message agent to delete or resend the dead message according to the processing instruction.

Monitoring the message which is not confirmed, triggering the message agent to resend the message which is not confirmed, and recording the sending times; and if the sending times exceed the preset retry times (retransmission times), converting the unconfirmed message into a death state, and further triggering manual processing.

And monitoring the message to be cleaned or backed up, and triggering the message agent to delete the message to be cleaned or back up the message to be backed up. In one example, for a meaningless request, such as a message format that is itself problematic or defective, the message to be cleaned may be deleted; when the log audit requirement is preset or the log audit requirement is defaulted, the message to be backed up can be backed up. In another example, the steps may include: when the messages to be cleaned or backed up are monitored, query information, such as a query popup window or an interface with interactive options, is output to the user, then a (second) processing instruction input by the user is received based on the query information, and the message agent is triggered to delete the messages to be cleaned or back up the messages to be backed up correspondingly according to the processing instruction.

Specifically, one message corresponds to a data table maintained in the message broker, and the main fields include:

messageId, message ID;

status, message status, pre-send (preset)/send (sending)/consumed (consumed);

dead, death flag, true indicates death.

In some embodiments, the message status monitoring system is specifically configured to: monitoring abnormal pre-sent state information in the information agent, wherein the abnormal pre-sent state information refers to information of which the duration in a pre-sent state exceeds a preset first time threshold, and deleting the abnormal pre-sent state information; and/or monitoring an abnormal sending state message in the message agent, wherein the abnormal sending state message refers to a message of which the duration in the sending state exceeds a preset second time threshold, and triggering the message agent to resend the abnormal sending state message. Specifically, pre-sending is a behavior in which a production end sends a message to an agent before processing a local service; to be sent refers to the action of a message broker preparing to send messages meeting a predetermined condition to a message queue. The predetermined conditions include the following: (1) the message is in a pre-send state; (2) The message is in the sending state (sending state), times out, and does not reach the upper limit of the number of retransmissions.

In some embodiments, the message status monitoring system is further configured to: and monitoring the retransmission times of the status message in abnormal transmission, marking the status message in abnormal transmission as a dead message when the retransmission times exceed a preset retransmission times threshold, and sending a notice indicating manual processing. Specifically, the operation of deleting the dead message is decided by manual processing, and the message broker deletes the dead message in response to the manual processing operation instruction.

In some embodiments, the message status monitoring system is further configured to: and inquiring the service operation result of the message production end regularly, and if the service operation result is successful, triggering the message agent to initiate the sending of the pre-sent message to ensure that the message in the pre-sent state is sent to the message queue. For example, for a deposit service, a successful service operation means that: receiving a loan request of the loan applicant, creating loan data, a repayment plan and the like, and indicating that the business operation is successful.

In some embodiments, the message status monitoring system is further configured to: when the data of the message producing end and the message consuming end are consistent, the message agent is triggered to update the state of the message to be consumed.

In some embodiments, the message consuming side comprises:

the receiving and sending module is used for monitoring the messages in the message queue and receiving the messages after the messages arrive;

the idempotent checking module is used for carrying out idempotent checking on the received message;

the business operation module is used for carrying out business operation on the messages which pass the idempotent examination and storing the processing result of the business operation to a third database;

the third database is used for storing the processing result of the business operation;

and the notification module is used for issuing a notification to the message broker, wherein the notification indicates that the message is successfully consumed.

In some embodiments, the message producer is further configured to: specifying a fragmentation factor, including for example a debit number or a customer number; obtaining an index number of feign according to the fragmentation factor and the number of message agents; and determining to call the message agent matched with the index number according to the index number of the feign. Specifically, the slicing factor may be utilized to perform modulo operation on the number of message agents to obtain a digital value, or the slicing factor is first hashed and then the number of message agents is subjected to modulo operation to obtain a digital value, an index number of feign is determined based on the digital value, and a message agent matched with the index number, for example, a message agent with the index number included in the name number, is determined and called according to the index number of feign.

The following five parts of the message processing system are mainly described in more detail with reference to fig. 1B:

(1) Message producing terminal

The upstream system of the service system takes the role of a message production end and is responsible for processing the local service and sending messages. Specifically, the message producer is responsible for executing local services and sending messages. The basic process is as follows: firstly, sending a pre-message, then executing local service operation, associating the message with service data and storing the associated message in a disk.

In some embodiments, the message producer performs the following steps:

first, a pre-message is sent to the message broker (broker), which persists the request (the pre-message) to the database. The request is the "pre-message". The pre-message is a message record and the status is a "pre-send" status. When the pre-message is sent, the data is consistent between the producing side and the consuming side because the business of the producing side is not processed yet.

Then, local business operations are performed and the pre-message is persisted to a database. The local service operation refers to an operation of changing the service state, and simultaneously stores the service operation and the previously sent pre-message into the database together through the identifier management. For example, before performing a service operation, a pre-message is sent, with message id = a; and after the pre-message is successfully sent, the business operation is started to be executed. When the business operation is successful and the message is stored (generally, a business has a unique business transaction serial number), the message id is also stored. Thus, there are two types of data for the database: the service serial number is used as a data record of the unique identifier; and the service serial number + the message id are used as the associated record of the unique identifier.

Finally, the result of the execution of the service operation (success/failure) is sent to the message broker.

(2) Message broker (broker)

The message broker is responsible for managing the full lifecycle of the message. The basic processing flow is as follows: after receiving the pre-sending information, the message agent stores the pre-sending information to a local disk, and sets the message state as 'pre-sending'. Then after receiving the confirmation information of the production end, the message agent executes the following operations: operation 1: changing the message status to "send"; operation 2: the message is sent to a message queue.

In some embodiments, the message broker is responsible for receiving messages from the message producer and persisting to the local database; and by tracking the full lifecycle state of the message in order to determine how to process the message, it includes the following processes:

receiving and storing a pre-sent message;

receiving and processing messages; if the message is successful, sending a message to a message queue; if it fails, the message is deleted. Specifically, the message is pre-sent to be in a "pre-sending state", and the production end successfully performs the service processing and successfully sends the acknowledgement information to indicate that the message is successfully converted to be in the "acknowledgement state", that is, to indicate that the production end has completed the consistency operation of the message and the service processing, that is, to indicate that the message is successfully sent.

For the message broker, it receives the pre-message from the producer. If the confirmation information sent by the production end is received again (namely the production end sends the confirmation information after executing the service processing of the production end), the success is represented; if no confirmation is received from the production end, the monitoring system detects the condition and initiates an inquiry to the production end. If the production end service processing result is that the service is successfully processed, the monitoring system informs a message agent that the message is confirmed, namely, the message indicates success; if the result of the production end business processing is failure, the failure is indicated.

Managing other messages, including specifically any one or more of: periodically clearing/backing up messages; deleting the dead message at regular time; initiating a resending of the death message; inquiring the overtime message, and redelivering; inquiring the message which is overtime and exceeds the delivery times, and switching to manual processing and the like.

Specifically, clean, meaning delete; backup, meaning migration of data to other storage media. Only when the message of the final state is reached, cleaning or backup is needed, and the final state comprises:

1. has been consumed. This is a successful process state, indicating a normal production and consumption process.

2. Has died. Namely, the message agent sends the message all the time, reaches the upper limit of the sending times and is finally handed to the state after manual processing.

In summary, messages that have been successfully processed should be backed up periodically; for messages that cannot be resolved, eventually marked as system discarded, should be cleaned up.

The initiating of the resending of the death message specifically includes: and the message agent is dead, namely the message agent tries to send the message until the upper limit of the sending times is exceeded, the message agent is marked as dead and the process is changed into manual processing. Such as a downtime in the message queue middleware. When the fault is eliminated, the dead is changed into the non-dead, and the sending times are reduced to zero, the monitoring system identifies and resends the message until the message is successful or the message is dead, namely the message is failed to be sent again and exceeds the upper limit of the sending times.

(3) Message queue (real-time message service or message middleware)

A message queue is a data structure that sends messages from a producer to a consumer asynchronously through a first-in-first-out queue. The message queue refers to a message queue of a specific technology stack, for example, activemq, which may be deployed in a single point or in a cluster, and is determined by the current technical solution of the client. Generally, the IT system construction of each client adopts a set of own technical model selection scheme, and the technical model selection is not mandatory to the RM system. Namely, the message queue can adopt any technology, and the adaptive scheme is as follows:

the activemq-producer and the activemq-consumer of the RM adapt to an activemq message queue architecture;

a rocktmq-producer and a rocktmq-consumer of the RM adapt to a rocktmq message queue architecture;

the rabbitmq-producer and rabbitmq-connumer of the RM adapt the rocktmq message queue architecture.

The RM architecture is responsible for resolving technical differences, independent of the business system (producer and consumer).

(4) Message consumption terminal

The message consumption end is responsible for receiving the message and executing local business operation. The basic processing flow mainly comprises the following steps: and after the message of the message queue arrives, the consumption end executes the local service, stores the message and the service after associating, and then sends the notification to the message agent.

In some embodiments, the message consuming side takes on the role of consuming side by the downstream system of the service system, for receiving the message sent by the upstream system and processing the local service, which performs the following 3 steps:

firstly, monitoring the message in a message queue, wherein the message is delivered by a message broker;

when the message arrives, the message consumption end receives the message and processes the local service;

after the local service is processed, the processing result is stored in a disk (stored in a service database) and a notification message is issued to the message broker, wherein the notification message indicates that the acknowledgement information has been successfully consumed, so that the message broker can manage the message status.

(5) Message state monitoring system

The message state monitoring system is mainly used for operation in various abnormal states, is a key service for guaranteeing the final consistency of messages, and has at least one of the following main functions; monitoring a message sending state; monitoring for dead messages; monitoring messages that are not acknowledged; by monitoring the tracking state, an instruction is issued to trigger the message broker to perform message cleaning or backup etc.

In some embodiments, the message monitoring system mainly solves the management of the exception scenario of the operation processing, and mainly comprises the following typical scenarios:

scene one: for the message producer, the pre-sent message is successful, but the business operation fails.

The inventor of the application finds that the business operation is not successful all the time, the data of the message production end and the data of the message consumption end are consistent, and only the pre-sent message in the message agent needs to be deleted. Therefore, the technical means adopted by the embodiment is as follows: the message state monitoring system is responsible for monitoring the messages which are in the pre-sending state for a long time and deleting the messages after a certain time.

Scene two: for the message production end, the pre-sent message is successful, the service operation is successful, but the message sending is failed.

The inventor of the present application has found that, since the business operation of the message producer is successful and the message broker fails to successfully send the message to the message consumer, the data is inconsistent, so it is necessary to ensure that the message broker successfully sends the message. Therefore, the technical means adopted by the embodiment is as follows: the message state monitoring system is responsible for regularly inquiring the service execution result of the message production end, if the result is found to be successful, the message agent initiates the sending of the pre-sent message, and the message agent is ensured to be sent to the message queue. Wherein, the message has a unique status identifier, and the aforementioned pre-message, the information to be confirmed, the message being sent, the message consumed, etc. are all statuses of the message at different stages.

Scene three: for the message broker, a request that the message producer service operation succeeds has been received, but the sending message queue fails.

The inventor of the present application has found that, since the business operation of the message producer is successful and the message broker does not successfully send the message to the consumer, resulting in data inconsistency, it is necessary to ensure that the message broker successfully sends the message out. Therefore, the technical means adopted by the embodiment is as follows: the message state monitoring system is responsible for monitoring the message which is in the sending state for a long time and then sending the message again.

Scene four: for a message broker, it has been successfully sent to the message queue, but the message state is always in a sending state.

The inventor of the application finds that the phenomenon indicates that the message agent fails to retry sending for many times, and the final data of the message producing end and the final data of the message consuming end are inconsistent, which may be caused by system problems or network problems and must be processed manually as soon as possible. Therefore, the technical means adopted by the embodiment is as follows: the message state monitoring system is responsible for monitoring the message which is in the sending state for a long time, and when the number of times of retransmission is detected to be out of limit, the message is converted into death and sent to the client (for manual processing).

Scene five: for the message consuming side, the message is already listened to, but the local service processing fails.

The inventor of the application finds that the phenomenon indicates that the message producing end and the message consuming end have definite data inconsistency and need the message producing end to reinitiate the message. Therefore, the technical means adopted by the embodiment is as follows: the message state monitoring system is responsible for monitoring the message which is in the sending state for a long time and then resending the message.

Scene six: for the message consuming side, the local message has been processed successfully, but the notification fails to be issued.

The inventor of the present application has found through research that this phenomenon indicates that the data of the message producing side and the message consuming side are consistent, and the message broker is required to correctly update the state of the message to be consumed. Therefore, the technical means adopted by the embodiment is as follows: the message state monitoring system is responsible for monitoring the message which is in the sending state for a long time and then sending the message again, and the message consumption end has to ensure the idempotent mechanism (namely, the idempotent mechanism is designed by the message consumption end) because the message consumption end has successfully processed the message.

The resending here means that the consuming end has succeeded, but when the notification is sent to the message broker, the message broker cannot acquire information whether the message succeeded or not due to network interruption or down of the consuming end.

Taking this scenario as an example, after the consuming end successfully acquires the message and executes the local service, the consuming end needs to be notified to the message broker, and at this time, the consuming end is down. It is not known to the message broker why the message consumer did not send a notification. The processing mode of the embodiment of the invention is that the message agent pushes again. For the consuming side, idempotent operations are to be solved. And the data state is ensured to be consistent under the request of the same transaction serial number.

As shown in fig. 2, the so-called message consistency is that the final state of the business processing of the production side and the business processing of the consumption side are consistent.

The "consumed" state of the state diagram indicates that the message is eventually consistent. The full flow of messages to ensure their final consistency is the design intent that such state diagrams express.

S1, firstly, a production end initiates a proportional transaction to require to execute local services, and at the moment, the production end firstly sends a pre-message to a message agent to inform the agent service that a service needs to be processed in the future.

And S2, the message agent stores the message in a storage and waits for message confirmation.

And S3, after the production end finishes the service processing, sending confirmation information to inform the message agent that the message can be sent to the message queue.

And S4, after receiving the message confirmation, the message agent sends the message to the message queue, and sets the message state as 'sending in'.

And S5, if the message state is always in the sending state and reaches the time upper limit, monitoring and initiating the resending operation by the monitoring system, and simultaneously recording the sending times plus 1.

And S6, if the message state is always in the 'sending' state and reaches the upper limit of the sending times, monitoring and triggering the execution of message death (identified by another field of dead) by the monitoring system.

And S7, executing the local service after the message consumer successfully receives the message, and sending a notification to the message agent. The message broker, upon receipt of the message, sets the message status to "consumed".

And (3) scalability design:

the purpose of the scalability design is to solve the problems of processing concurrency and the number of unit time through horizontal extension under the large data high concurrency scene.

The design idea of scalability is as follows:

in order to increase the number of machines for processing messages, the message processing machines are partitioned, each partition processes a part of data, and a hashing algorithm is generally performed according to message _ id, so that the processed messages can be evenly distributed on different areas. The greater the number of partitions, the more machines that process messages, and the greater the ability to process in parallel. For a message production end, a destination address to be sent and deleted needs to obtain the number of a destination machine through a hash algorithm; similarly, the message agent sends the message to the message queue, and the consumption end pushes the consumption result to the message agent, which all obtain the target address through the same algorithm. For example, the message broker deploys 3 message brokers, the numbers are 0, 1 and 2, respectively, and the message producer sends the message to the message broker according to the debit number (loann), specifically to which broker, using the following formula: machine number = loann no modulo 3.

Since a single partition, once down, will cause all the messages shunted to that partition to be unmanaged, the cloning of instances within the partition must be designed to ensure that when one instance fails, it will be consumed by another instance, with the greater the number of instances, the higher the availability, and generally at least 2 instances will be set. Each instance proposal is deployed on a separate machine, preferably across a computer room.

If all instances of a partition are down, the RM can dynamically identify and shunt messages to other normal partitions. This process is performed completely dynamically without manual intervention. The dynamic identification and the distribution processing are managed uniformly by a service registration center, the service registration center is a uniform solution of a spring-closed micro service architecture, and the RM calls the related functions of the service registration center to realize the dynamic identification and the distribution processing.

Fig. 3 is a deployment diagram of a scalability design of an embodiment of the invention. As shown in fig. 3, the deployment mode of the message broker includes:

as shown in part (a) of fig. 3, the first deployment mode of the message broker, which has the same serverId in the first deployment mode, performs instance cloning, specifically including: the message agent comprises a plurality of machines in a single partition, the machines have a common server identifier, a clone instance is operated on each machine, and when the clone instance on one machine fails, the clone instance on another machine in the partition is switched to continue to consume; or,

as shown in part (b) of fig. 3, the second deployment mode of the message broker, which has different server ids in the second deployment mode, specifically includes: the message agent comprises a plurality of partitions, each partition is provided with a machine, the server identifications of each machine are different, and when the machine in any one partition is down, the message to be processed is distributed to the machines in other normal partitions; or,

as shown in part (c) of fig. 3, a deployment mode three of the message broker has different serverids in the deployment mode, and multiple deployments exist for a certain serverId: the message agent comprises a plurality of partitions, only one machine is configured in at least one partition, a plurality of machines are configured in at least one partition, in the partitions configured with the plurality of machines, the plurality of machines have a common server identifier and run corresponding clone instances, and when the machine in any one partition is down, the message to be processed is distributed to the machines which have not failed the clone instances in other normal partitions.

The technical scheme of the embodiment of the invention has the following beneficial technical effects:

1. the universality of the platform level is improved, and the platform level is mainly embodied in the following points:

only the interface specification of the RM needs to be learned once, and the technical characteristics of different MQs are encapsulated by the RM;

service personnel do not need to care about operation and maintenance requirements such as technology stack configuration and deployment;

business personnel can quickly access the requirements of a client research and development team on the MQ, and generally, only 1-2 days are needed for accessing a new MQ, so that the development cost is greatly reduced.

2. The consistency of the message is solved by the following points:

service developers do not need to pay attention to the problem of message loss, and the RM system automatically completes local storage and global monitoring of messages;

the business developer does not need to develop message compensation codes, and the RM system automatically completes the tracking and resending of the message state;

and message compensation failure caused by the distributed system network and the like is automatically pushed to a manual platform by the RM system, and manual intervention is notified through an alarm system. The alarm system is a universal solution, and general micro service systems are standardized, similar to system platforms such as prometheus, zabbix and the like. If no alarm system is installed, the alarm can be realized by inputting a log or writing some codes to monitor the state.

3. The scalability of the RM is mainly embodied in the following points:

under a high-flow scene, the pressure of data processing can be quickly solved by increasing the number of RM partitions, and the partitions are increased to have no perception on upstream and downstream systems of a service;

under a high-reliability scene, the high availability of the partitions can be guaranteed by increasing the number of machine instances of the single partitions.

4. Other valuable characteristics are realized, which are mainly reflected in the following points:

the operations such as backup, cleaning and the like of the message are automatically completed, and business personnel do not need to intervene;

under the scene that the requirement on high message reliability is not so high, the problem can be solved through the transparent message, and business personnel do not need to make additional development;

service personnel can conveniently trace information such as the source, the route, the original message and the like of the message;

the resource utilization rate of the message receiving node is exerted, and a service developer does not need to make any additional development and only adjusts the performance through configuration.

Fig. 4 is a flowchart of a message processing method according to an embodiment of the present invention. As shown in fig. 4, the method includes the steps of:

s110, the message production end sends a message in a pre-sending state to the message agent, executes local business operation, persists a business operation result to a first database, and sends the business operation result to the message agent;

s120, the message agent receives the message in the pre-sending state, stores the message in the pre-sending state to a second database, modifies the state of the message into a sending state when the confirmation information which is sent by the message production end and is associated with the service operation result indicates that the service operation is successful, sends the message to the message queue, and deletes the message in the pre-sending state if the confirmation information indicates that the service operation is failed;

s130, caching the message sent by the message agent by the message queue;

s140, the message consumption end receives the message and processes the local service;

s150, the message state monitoring system monitors the state of the message in the full life cycle of the message, and when the state of the message is abnormal, the message agent is triggered to process the message with the abnormal state.

In some embodiments, the method further comprises any one or more of:

the message agent regularly cleans the message or regularly backs up the message;

the message agent deletes the dead message;

the message agent initiates the resending of the death message;

the message agent inquires the overtime message and delivers the overtime message again;

the message agent queries for messages that have timed out and exceeded a preset delivery number of times, triggering manual processing.

In some embodiments, in step S150, the message status monitoring system monitors the status of the message in the full life cycle of the message, and when the status of the message is abnormal, triggers the message broker to process the message with abnormal status, specifically including performing any one or more of the following:

the message state monitoring system monitors the message in the abnormal sending state and triggers the message agent to process the message in the abnormal sending state;

the message state monitoring system monitors the dead message and triggers the message agent to delete or resend the dead message;

the message state monitoring system monitors the messages which are not confirmed, and triggers the message agent to process the messages which are not confirmed;

the message state monitoring system monitors the message to be cleaned or backed up, and triggers the message agent to clean the message to be cleaned or back up the message to be backed up.

In some embodiments, the monitoring the message in the abnormal sending state, and triggering the message broker to process the message in the abnormal sending state specifically includes:

the message state monitoring system monitors abnormal pre-sending state messages in the message agent, the abnormal pre-sending state messages refer to messages of which the duration in a pre-sending state exceeds a preset first time threshold, and the abnormal pre-sending state messages are deleted; and/or the presence of a gas in the gas,

and the message state monitoring system monitors the state message in abnormal sending in the message agent, wherein the state message in abnormal sending refers to the message of which the duration time in the sending state exceeds a preset second time threshold, and triggers the message agent to resend the state message in abnormal sending.

Specifically, the message state monitoring system monitors the number of times of resending the status message in abnormal sending, marks the status message in abnormal sending as a dead message when the number of times of resending exceeds a preset number of times of resending threshold, and sends out a notification indicating manual processing.

In some embodiments, the message state monitoring system periodically queries the service operation result of the message production end, and if the service operation result is successful, the message agent is triggered to initiate sending of the pre-sent message, so that the pre-sent message is ensured to be sent to the message queue.

In some embodiments, when the data of the message producer and the message consumer are consistent, the message condition monitoring system triggers the message broker to update the status of the message to consumed.

Fig. 5 is a flow chart of another message processing method according to the embodiment of the invention. As shown in fig. 5, the method includes the steps of:

s210, monitoring the state of the message in the whole life cycle of the message;

specifically, the full life cycle of a message refers to the overall process from generation to death of the message, and the state of the message includes but is not limited to: pre-send (preset), send (sending), consumed (consumed), dead (dead).

S220, when the abnormal state of the message is monitored, the message agent is triggered to process the message with the abnormal state.

Optionally, step S220 may specifically include any one or more of the following:

Fig. 6 is a functional block diagram of a message status monitoring system according to an embodiment of the present invention. As shown in fig. 6, it includes:

a message status monitoring module 310, configured to monitor a status of a message in a full life cycle of the message;

and the message exception handling module 320 is configured to trigger the message broker to handle the message with the abnormal state when the abnormal state of the message is monitored.

In some embodiments, the message exception handling module 320 is specifically configured to perform any one or more of the following:

In some embodiments, the message exception handling module 320 may be specifically configured to:

FIG. 7 is a functional block diagram of a computer-readable storage medium of an embodiment of the present invention. As shown in fig. 7, a computer program 410 is stored in the computer-readable storage medium 400, and when executed by a processor, the computer program 410 implements the steps of any of the message processing methods described above.

FIG. 8 is a functional block diagram of a computer device of an embodiment of the present invention. As shown in fig. 8, it includes:

one or more processors 501;

one or more communication interfaces 502;

a communication bus 304;

a memory 503 for storing one or more programs;

the processor 301, the communication interface 302 and the memory 303 complete mutual communication through the communication bus 304;

the one or more programs, when executed by the one or more processors 501, cause the one or more processors 501 to implement the steps of any of the message processing methods described above.

Processor 501 may be a general-purpose Processor including a Central Processing Unit (CPU), a Network Processor (NP), etc.; but also Digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.

Memory 503 may include mass storage for data or instructions. By way of example, and not limitation, memory 503 may include a Hard Disk Drive (HDD), a floppy Disk Drive, flash memory, an optical Disk, a magneto-optical Disk, tape, or a Universal Serial Bus (USB) Drive or a combination of two or more of these. The memory 503 may include removable or non-removable (or fixed) media, where appropriate.

The communication bus 504 includes hardware, software, or both for coupling the above-described components to each other. For example, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus or a combination of two or more of these. A bus may include one or more buses, where appropriate. Although specific buses have been described and shown in the embodiments of the invention, any suitable buses or interconnects are contemplated by the invention.

It should be clear to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional units and modules is only used for illustration, and in practical applications, the above function distribution may be performed by different functional units and modules as needed, that is, the internal structure of the apparatus may be divided into different functional units or modules to perform all or part of the above described functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

The foregoing description of specific embodiments has been presented for purposes of illustration and description. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

Although the present application provides method steps as in an embodiment or a flowchart, more or fewer steps may be included based on conventional or non-inventive labor. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. When an actual apparatus or client product executes, it may execute sequentially or in parallel (e.g., in the context of parallel processors or multi-threaded processing) according to the embodiments or methods shown in the figures.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims

1. A message processing system, characterized in that the message processing system comprises:

the message production terminal is used for sending the message in a pre-sending state to the message agent, executing local service operation, persisting a service operation result to the first database, and sending the service operation result to the message agent; the message production end is further configured to: assigning a fragmentation factor, wherein the fragmentation factor comprises a borrow number; carrying out Hash processing on the fragmentation factors, then carrying out modular operation on the number of message agents to obtain a digital value, and determining the index number of feign based on the digital value; determining to call a message agent matched with the index number according to the index number of the feign; the message production end is specifically used for sending an http request to the message agent through a flight interface of a spring-closed technology stack;

the message agent is used for isolating the message sending behavior from the message production end; the message broker includes: the receiving module is used for receiving the message in the pre-sending state; a second database for storing the message in the pre-send state; a processing module, comprising: the pre-sending message processing submodule is used for modifying the state of the message in the pre-sending state into a sending state when the confirmation information which is sent by the message production end and is associated with the service operation result indicates that the service operation is successful, and controlling the sending module to send the message to the message queue; if the confirmation information indicates that the service operation fails, deleting the message in the pre-sending state; the sending module is used for sending messages to the message queue under the control of the processing module;

the message queue is used for caching the messages sent by the message agent;

the message consumption end is used for receiving the message and processing the local service by defining a self-defined class for realizing an IMsgrceeveCallbackHandler interface;

the message state monitoring system is used for monitoring the state of the message in the full life cycle of the message and triggering the message agent to process the message with abnormal state when the state of the message is abnormal; the message state monitoring system is specifically configured to: monitoring abnormal pre-sending state information in the information agent, wherein the abnormal pre-sending state information refers to information of which the duration in a pre-sending state exceeds a preset first time threshold, and deleting the abnormal pre-sending state information; monitoring an abnormal sending state message in the message agent, wherein the abnormal sending state message is a message of which the duration of the sending state exceeds a preset second time threshold, and triggering the message agent to resend the abnormal sending state message;

wherein the deployment mode of the message broker comprises:

the message agent comprises a plurality of partitions, only one machine is configured in at least one partition, a plurality of machines are configured in at least one partition, in the partitions configured with the plurality of machines, the plurality of machines have a common server identifier and run corresponding clone instances, and when the machine in any one partition is down, the message to be processed is distributed to the machines which have not failed the clone instances in other normal partitions.

2. The system of claim 1, wherein the processing module further comprises:

the message state monitoring and processing submodule is used for executing any one or more of the following operations:

cleaning messages at regular time or backing up messages at regular time;

deleting the dead message;

initiating the resending of the death message;

inquiring the overtime message, and retransmitting the overtime message;

and inquiring the messages which are overtime and exceed the preset sending times, and triggering manual processing.

3. The system according to claim 1, wherein the message consumer is specifically configured to listen to the message in the message queue, and the message is responsible for delivery by the message broker; after the message in the message queue is successfully delivered to the message consumption end, receiving the message and processing the local service according to the received message; after the local service is processed, the processing result is stored in the service database of the message consumption end, and a notice is issued to the message broker, wherein the notice indicates that the message is successfully consumed.

4. The system of claim 1, wherein the message status monitoring system is specifically configured to:

and monitoring the retransmission times of the status message in abnormal transmission, marking the status message in abnormal transmission as a dead message when the retransmission times exceeds a preset retransmission times threshold, and sending a notice indicating manual processing.

5. The system of claim 1, wherein the message status monitoring system is further configured to: and inquiring the service operation result of the message production end periodically, and if the service execution result is that the service operation is successful, triggering the message agent to send the message to the message queue.

6. The system of claim 1, wherein the message status monitoring system is further configured to: and when the data of the message production end and the message consumption end are consistent, triggering the message agent to update the state of the message to be consumed.

7. The system of claim 3, wherein the message consumer comprises:

a notification module to issue a notification to the message broker indicating that the message has been successfully consumed.

8. A method of message processing, the method comprising:

the message production end sends a message in a pre-sending state to a message agent, executes local service operation, persists a service operation result to a first database, and sends the service operation result to the message agent; the message production end also specifies a fragmentation factor, and the fragmentation factor comprises a borrow number; carrying out Hash processing on the fragmentation factors, then carrying out modular operation on the number of message agents to obtain a digital value, and determining the index number of feign based on the digital value; determining to call a message agent matched with the index number according to the index number of the feign; the message production end is used for sending an http request to the message agent through a flight interface of a spring-closed technology stack;

the message agent receives the message in the pre-sending state, stores the message in the pre-sending state to a second database, modifies the state of the message into a sending state when the confirmation information which is sent by the message production end and is associated with the business operation result indicates that the business operation is successful, sends the message to a message queue, and deletes the message in the pre-sending state if the confirmation information indicates that the business operation is failed; wherein the message broker is configured to isolate a message sending activity from the message producer;

the message queue caches the messages sent by the message agent;

the message consumption end receives the message and processes the local service by defining a self-defined class for realizing an IMsgrceeveCallbackHandler interface;

the message state monitoring system monitors the state of the message in the full life cycle of the message, and when the state of the message is abnormal, the message agent is triggered to process the message with the abnormal state; the method specifically comprises the following steps: the message state monitoring system monitors abnormal pre-sending state messages in the message agent, the abnormal pre-sending state messages refer to messages of which the duration in a pre-sending state exceeds a preset first time threshold, and the abnormal pre-sending state messages are deleted; the message state monitoring system monitors an abnormal sending state message in the message agent, wherein the abnormal sending state message is a message of which the duration in the sending state exceeds a preset second time threshold, and triggers the message agent to resend the abnormal sending state message;

wherein the deployment mode of the message broker comprises:

the message agent comprises a plurality of partitions, only one machine is configured in at least one partition, a plurality of machines are configured in at least one partition, in the partitions where the plurality of machines are configured, the plurality of machines have a common server identifier and run corresponding clone instances, and when the machine in any one partition is down, the message agent distributes the messages to be processed to the machines of which the clone instances in other normal partitions do not fail.

9. The method of claim 8, further comprising any one or more of:

the message agent regularly cleans messages or regularly backs up messages;

the message broker deletes the dead message;

the message agent initiates the resending of the death message;

the message agent inquires the overtime message and resends the overtime message;

the message broker queries for messages that have timed out and exceeded a preset number of resends, triggering manual processing.