CN111800354A

CN111800354A - Message processing method and device, message processing equipment and storage medium

Info

Publication number: CN111800354A
Application number: CN201910277493.6A
Authority: CN
Inventors: 杨巍巍; 房耘耘; 马琪
Original assignee: China Mobile Communications Group Co Ltd; China Mobile Suzhou Software Technology Co Ltd
Current assignee: China Mobile Communications Group Co Ltd; China Mobile Suzhou Software Technology Co Ltd
Priority date: 2019-04-08
Filing date: 2019-04-08
Publication date: 2020-10-20
Anticipated expiration: 2039-04-08
Also published as: CN111800354B

Abstract

The embodiment of the invention discloses a message processing method and device, message processing equipment and a storage medium. The method comprises the following steps: receiving a target message; determining item information of the target message; according to the project information and the message strategy, transmitting the target message by using a message queue corresponding to the project information; therefore, when the middleware transmits the message between the producer and the consumer of the message by using the message queue, the message to be processed is pertinently pressed into the corresponding message queue according to the message strategy; therefore, when the message transmission fails, the message queue with the message failure can be positioned at least according to the item of the message with the current transmission failure, and the positioning and problem troubleshooting of the message error queue are facilitated to be simplified.

Description

Message processing method and device, message processing equipment and storage medium

Technical Field

The present invention relates to the field of information technologies, and in particular, to a message processing method and apparatus, a message processing device, and a storage medium.

Background

The realization of distributed services by messages has become a necessary trend in the present. Under an Openstack cloud platform, a plurality of message agent modes are available, including Rabbitmq, Qpid and the like. This tool is described in detail as Rabbitmq. Currently, in Rabbitmq implementation, as shown in fig. 1, a producer (publisher) is responsible for generating a message, the message is routed to a specified queue according to a binding rule, and a Subscriber (Subscriber) at a consumer end (consumer) is responsible for pulling the message from the queue and processing the message. Middleware (browser) performs interaction between a producer and a consumer of messages through exchange (exchange) and by using a Queue (Queue).

The service under the Openstack cloud platform is embodied in two aspects in terms of message use:

on one hand, the method is message monitoring, and in the process of service starting, a connection to a Rabbitmq server end is established simultaneously through a specified target object and a monitored message type, so that the monitoring of the service on the message is realized;

on the other hand, the client sends the message, and the client specifies the target object of the message and the channel of the context message by initializing the flow of a Remote Procedure Call (RPC) to obtain the channel connected to the Rabbitmq server, so as to further obtain the channel according to the type of the message.

The routing rules send messages to a message queue Broker (Broker).

From the two aspects, the flow process of the message on the Openstack cloud platform is known, firstly, the client acquires an RPC client connection channel, then, the message is packaged and sent to the Broker according to a message mode which the client wants to send, then, the consumer service monitors the arrival of the message, the message is immediately pulled and processed, if the message is a synchronous message, a processing result is returned, and otherwise, the processing result is not returned.

The message types under Openstack are divided into three types, namely direct, fanout and topic, wherein when a service is started, the service generally monitors two types of messages, namely a fanout message and a topic message, which indicate a monitoring matching mode and all messages sent to a route, and the direct type of messages are generally used in a process and used for receiving a message return result of a synchronization request.

In the framework of the existing message queue, firstly, once a problem occurs in a platform, operation and maintenance personnel often take measures, the cognitive ability of a back-end service does not reach the level of research and development, and even if the problem-occurring queue is known, the problem still cannot be investigated.

Second, the specific location of the message error is difficult to determine.

Finally, the message queue is an important performance bottleneck point of the cloud platform, which is not a capacity problem of the message queue itself, but is caused by insufficient customer service capacity.

Disclosure of Invention

In view of this, embodiments of the present invention are intended to provide a message processing method and device, a message processing apparatus, and a storage medium

The technical scheme of the invention is realized as follows:

a method of message processing, comprising:

receiving a target message;

determining item information of the target message;

and transmitting the target message by using a message queue corresponding to the project information according to the project information and the message strategy.

Based on the above scheme, the method further comprises:

acquiring attribute information of the target message;

determining whether to track the target message according to the attribute information;

and if the target message is determined to be tracked, recording the message record and/or the link track information of the target message into metadata.

Based on the above scheme, the obtaining of the attribute information of the target message includes:

and acquiring the attribute information of the target message according to the tracking mode.

Based on the above scheme, the obtaining of the attribute information of the target message according to the tracking mode includes one of the following steps:

acquiring project information of the target message according to a project mode;

acquiring the routing information of the target message according to a routing mode;

and acquiring the queue information of the target message according to the queue mode.

Based on the above scheme, the determining whether to track the target message according to the attribute information includes:

matching the attribute information with the attribute information set by tracking;

and if the matching is successful, determining the message record and/or the link track information for tracking the target message.

Based on the above scheme, if it is determined to track the target message, recording the message record and/or the link trace information of the target message into metadata, including at least one of:

in the project mode, if the target message is determined to be tracked, at least one of the routing information, the queue information, the routing key, the message content, the producer of the message, the receiver of the message and the information of the receiving and sending conditions of the message, which contain the target message, is recorded in the metadata;

under a routing mode, if the target message is determined to be tracked, at least one of the sender, the receiver, the message content and the message receiving and sending condition information of the target message is recorded into the metadata;

in the queue mode, if the target message is determined to be tracked, at least one of the message content of the message, the producer of the message, the receiver of the message, the receiving and sending condition information of the message and the item information of the message is recorded in the metadata.

Based on the above scheme, the method further comprises:

determining the tracking priority of the target message according to the attribute information;

and determining a recording mode for tracking the target message according to the tracking priority.

Based on the scheme, the recording mode comprises at least one of the following modes:

information info mode;

error mode;

debug mode.

Based on the above scheme, the method further comprises:

and if the first cluster where the message queue of the target message is positioned is abnormal, switching the message queue and/or the middleware for processing the message to a second cluster which is back up to the first cluster.

Based on the above scheme, the method further comprises:

and if the first cluster is abnormal, recovering the abnormal information of the project processing of the target information through the second cluster according to the recovery mechanism of the project of the target information.

Based on the above scheme, the metadata comprises at least one of the following data objects:

a queue object;

requesting an object;

a routing object;

a resource object.

Based on the scheme, the queue object comprises at least one of the following fields:

message identification, routing key, message header, message body, message producer, and message destination;

the request object includes at least one of the following fields:

an identification, the identification comprising one of: request identification, item identification, queue identification, route identification and message identification;

a creation time;

the routing object includes at least one of the following fields:

a route identifier;

a route name;

a project name;

a queue name;

the resource object includes at least one of the following fields:

resource identification of the middleware;

the node position where the host resource is located;

resource information of the queue;

service information where the service resource is located;

port information.

Based on the above scheme, the method further comprises:

based on the query request, querying the metadata to obtain the message record and/or the link track information of the message to be tracked;

and processing the information according to the message record and/or the link track information.

Based on the above scheme, the performing information processing according to the message record and/or the link track information includes at least one of:

according to the message record and/or the link track information, counting the number of messages of one project, counting the number of messages transmitted in unit time, and evaluating the service capacity of a message server for processing the messages;

and performing at least one of error node positioning, error service positioning and error reason positioning according to the message record and/or the link track information.

A message processing apparatus comprising:

the first receiving module is used for receiving the target message;

the first determining module is used for determining the item information of the target message;

and the transmission module is used for transmitting the target message by using the message queue corresponding to the item information according to the item information and the message strategy.

Based on the above scheme, the apparatus further comprises:

the first acquisition module is used for acquiring the attribute information of the target message;

the second determining module is used for determining whether to track the target message according to the attribute information;

and the recording module is used for recording the message record and/or the link track information of the target message into metadata if the target message is determined to be tracked.

Based on the above scheme, the first obtaining module is configured to obtain the attribute information of the target message according to a tracking mode.

Based on the above scheme, the first obtaining module is specifically configured to execute one of the following:

Based on the above scheme, the second determining module is configured to match the attribute information with attribute information set for tracking; and if the matching is successful, determining the message record and/or the link track information for tracking the target message.

Based on the above scheme, the recording module is specifically configured to execute one of the following:

Based on the above scheme, the apparatus further comprises:

a third determining module, configured to determine a tracking priority of the target message according to the attribute information;

and the fourth determining module is used for determining a recording mode for tracking the target message according to the tracking priority.

Based on the above scheme, the apparatus further comprises:

and the switching module is used for switching the message queue and/or the middleware for message processing to a second cluster which is back up to the first cluster if the first cluster where the message queue of the target message is located is abnormal.

Based on the above scheme, the apparatus further comprises:

and if the first cluster is abnormal, recovering the abnormal information of the item in which the target information is positioned through the second cluster according to the recovery mechanism of the item in which the target information is positioned.

Based on the above scheme, the apparatus further comprises:

the query module is used for querying the metadata to obtain the message record and/or the link track information of the message to be tracked based on the query request;

and the processing module is used for processing information according to the message record and/or the link track information.

Based on the above scheme, the processing module is configured to execute at least one of:

A message processing apparatus comprising:

a memory;

and the processor is connected with the processor and used for realizing the message processing method provided by one or more of the technical schemes by executing the computer executable instructions stored on the memory.

A computer storage medium having stored thereon computer-executable instructions; after being executed by a processor, the computer-executable instructions can implement the message processing method provided by one or more of the technical solutions.

The message processing method and device, the message processing equipment and the storage medium provided by the embodiment of the invention can set different message queues in different projects, so that when the middleware transmits messages between a producer and a consumer of the messages by using the message queues, the messages to be processed can be pertinently pressed into the corresponding message queues according to the message strategy; therefore, when the message transmission fails, the message queue with the message failure can be positioned at least according to the project where the message which fails to be transmitted currently is located, and the method is favorable for simplifying the error positioning and problem troubleshooting of the message.

Drawings

FIG. 1 is a schematic diagram of a message transmission path;

fig. 2 is a schematic flowchart of a message processing method according to an embodiment of the present invention;

fig. 3 is a schematic flowchart of another message processing method according to an embodiment of the present invention;

fig. 4 is a schematic structural diagram of a message processing apparatus according to an embodiment of the present invention;

fig. 5 is a flowchart illustrating a further message processing method according to an embodiment of the present invention;

fig. 6 is a schematic diagram of message delivery according to an embodiment of the present invention;

FIG. 7 is a diagram illustrating a data structure of metadata according to an embodiment of the present invention;

fig. 8 is a schematic diagram of a correspondence relationship between a cloud platform, a message server, and a cluster according to an embodiment of the present invention.

Detailed Description

The technical solution of the present invention is further described in detail with reference to the drawings and the specific embodiments of the specification.

As shown in fig. 2, the present embodiment provides a message processing method, including:

step S110: receiving a target message;

step S120: determining item information of the target message;

step S130: and transmitting the target message by using a message queue corresponding to the project information according to the project information and the message strategy.

In this embodiment, the message processing method may be applied to middleware of a cloud platform and a message server.

The middleware can be embedded into the cloud platform and/or the message server in a plug-in mode; therefore, the physical architecture of a cloud platform and/or a message server is not required to be changed, the compatibility with the prior art is strong, meanwhile, the message queues can be distinguished according to the items, and the processing of message forwarding, transmission and the like of item granularity can be realized.

The middleware is located between the producer (Provider) and the Consumer (Consumer) of the message. Middleware can be used at least for the relay of messages that need to be pushed into the queue or for messages that need to be pulled from the queue. For example, the middleware pushes a message provided from a producer into a corresponding message queue according to an item in which the message is located, so that a consumer of the message can conveniently pull the message from the corresponding message queue.

The items described in this embodiment may be one or more service components. If an item consists of a plurality of services, the services are associated with each other. For example, a management project for software may include: authentication service of the software, upgrade service of the software, and the like; the relevance of these several services is reflected in corresponding to the same software. As another example, in the case of a mobile communication charging item, the communication charging item may include: communication billing services, charging management services, and the like. The above are of course merely examples.

In a specific implementation process, the setting of the items may be processed as needed. In some embodiments, one said item may be a big service next to one or more sub-service compositions.

In this embodiment, the target message may be any message that needs to be transmitted between the producer and the consumer of the message, for example, a trigger message generated by a certain thread.

In this embodiment, a message policy is preset, and the message policy may be executed by the middleware. If receiving a piece of information and needing to be forwarded between the cloud platform and the message server, determining item information of the message, wherein the item information indicates an item of the target message. Different items are provided with different message queues, referred to as message queues in this embodiment.

For example, item a and item B set up various message queues for messaging, respectively, and the queue for item a may be an a queue and the queue for item B may be a B queue. When the middleware performs message relay in step S130, the message of item a may be processed by using the a queue, and the message of item B may be processed by using the B queue.

Therefore, the messages transmitted between the cloud platform and the message server realize message processing such as message transfer of different projects by using different message queues, and the message queues at the project level are distinguished, so that when a fault is positioned or messages are accumulated, targeted adjustment can be performed according to the projects, targeted performance optimization of the cloud platform and/or the message service is realized, and the message transmission quality and efficiency are improved. If errors occur in the message transmission process, the message queues with the errors can be determined according to the item where the target message is located, but not all the queues can receive all the messages, so that the problem of high difficulty in positioning the message queues which cause the messages to be transmitted normally due to the errors cannot be positioned, and the error positioning of the target message in the transmission process is facilitated.

In some embodiments, the method further comprises:

recording queue state information of the message queue in terms, wherein the queue state information can be used for indicating the set maximum length, the average occupied queue length and the like of the corresponding message queue; thus, the queue parameters corresponding to the corresponding items can be subsequently adjusted according to the queue status information, where the queue parameters include but are not limited to: the number of queues, the maximum length of a single message queue, etc., to satisfy the message queues required for different items of message transmission.

In some embodiments, the method further comprises:

when the message transmission of an item is abnormal, whether the abnormality occurs in the message queue or in the consumer receiving the message can be determined according to the queue state message of the message queue corresponding to the item, and thus, the queue state message can also be used for positioning the abnormality. If the queue status information indicates that the message queue corresponding to the item is normal and there is no overflow, a message transmission failure occurs, and the message transmission failure may occur abnormally to the consumer instead of the message queue.

In some embodiments, as shown in fig. 3, the method further comprises:

step S210: acquiring attribute information of the target message;

step S220: determining whether to track the target message according to the attribute information;

step S230: and if the target message is determined to be tracked, recording the message record and/or the link track information of the target message into metadata.

The attribute information may be determined according to a message header of the target message, or may be determined according to an attribute of a previous processing node of the target message. For example, the middleware may determine according to the attributes of the producers, different producers belong to different services or different items, and thus, the attribute information of the target message may be determined according to the attributes of the producers.

In some embodiments, there are some messages that need to be traced and some information that may not be traced. For example, important information is tracked, while unimportant information is not tracked, so that tracking load can be reduced and processing can be simplified relative to tracking all target messages.

The message records include, but are not limited to:

updating records of the messages; the update record of the message here includes: updating the package of the message and updating the record of the content of the message. The updating of the capsule of the message includes but is not limited to: and (4) encapsulating and de-encapsulating related records of the message. The update record of the message content comprises: the addition, deletion and/or replacement of message content, etc.

In some embodiments, the message record further comprises:

the sending and receiving state of the message, for example, the transmission time of the message from one node to the next node, the success or failure of the transmission, and the like.

The link track information is used for indicating a message transmission path; the producer, sender and receiver of the message can be known according to the link trace information.

Through the message recording and/or the recording of the link trace information, error analysis can be conveniently performed in the process of message transmission failure, and the error analysis includes but is not limited to: positioning of an error node, positioning of an error reason and the like.

In this embodiment, the message record and/or the link track information are stored in the metadata, and the persistent memory of the message can be realized by using the recording function of the metadata.

In some embodiments, the step S210 may include:

The attribute information required for different tracking modules is different. The tracking mode may be set according to a user instruction or may be set according to an attribute of the item.

Due to different tracking modes, attribute information of different target messages needs to be acquired to judge whether the target message needs to be tracked. Therefore, in step S210, the attribute information is acquired from the target message itself or the transceiver node according to the currently set tracking mode.

In some embodiments, the step S210 may specifically include one of the following:

In this embodiment, the tracking mode may be based on different granularities of partitioning. For example, if the item mode is used, all target messages that need to be tracked are tracked, and at this time, when it is determined whether a target message needs to be tracked, item information of a current target message needs to be acquired first.

And matching the acquired item information with the item information of the item to be tracked, and if the item information is consistent with the item information of the item to be tracked, determining that the current target message is the message to be tracked.

In the routing mode may be used to define a trace that traces information transmitted over a particular routing path. And if the current tracking mode is the routing mode, acquiring the routing information of the target message. After the routing information of the current target message is obtained, the routing information is matched with the routing message needing to be traced, and if the matching is consistent, the current target message can be determined to be the target message needing to be traced.

If the current tracking mode is a queue mode taking a queue as a granularity, the queue information of the target message needing to be pressed into the queue can be obtained, the queue information is matched with the queue information of the queue needing to be tracked in advance, and if the queue information is consistent with the queue information, the current target message can be considered as the target message needing to be tracked.

In some embodiments, the tracking mode is not limited to the project mode, the route mode, the queue mode, but may also include a service mode with service granularity. In the service mode, the service information of the service where the target message is located can be acquired, and through matching with the service information of the service needing to be tracked, if the matching is consistent, the corresponding target message is determined to be the target message needing to be tracked.

The tracking modes are various, and in some specific cases, multiple tracking modes can coexist, but if one message hits multiple tracking modes at the same time, the tracking is performed only once, and repeated redundant tracking is not involved. For example, the current trace mode includes a queue mode and an item mode, and if a target message needs to be traced in both the queue and the item, the target message may be considered to be traced, but tracing is performed only once, and the target message is not traced in both modes.

In summary, the step S220 may include: matching the attribute information with the attribute information set by tracking; and if the matching is successful, determining the message record and/or the link track information for tracking the target message.

The description of the message record and the link trace information is specifically referred to the foregoing section, and will not be repeated here.

In this embodiment, various tracking information, such as the message record and/or the link trace information, obtained when the target message is tracked, may be recorded in the metadata, so as to implement long-term recording and facilitate query.

In some embodiments, the step S220 may specifically include at least one of:

In the project mode, one or more of route information describing a route path of the target message, queue information of the target message pushed into a queue, a route key used in the route, message contents, a producer of the message, a receiver of the message, and information on the transmission/reception situation at each transit routing node are recorded in the metadata.

The transceiving situation information may include at least one of:

transmitting and receiving status information indicating whether the target message is successfully received;

first time information indicating reception of the target message;

indicating second time information for transmitting the target message.

In the routing mode, and the corresponding target message needs to be tracked, one or more of the producer, the receiver, and the sending and receiving information of the target message are recorded in the metadata.

In the queue mode, at least one of the producer of the message, the recipient of the message, and the item information of the message may be written into the metadata.

In still other embodiments, routing information and/or service messages for the target message may also be written into the metadata in the queue mode.

In some embodiments, in any one of the project mode, the route mode, and the queue mode, one or more of identification information of middleware that processes the target message, resource information that processes the target message, node information that processes the information to be processed, port information of a transmission port, and the like may be further written to the metadata.

The resource information may include: calculating resource identification of resources, storage identification of storage resources and bandwidth identification of bandwidth resources. The node information is used for indicating the node identification of the physical node and/or the virtual node which processes the information to be processed. Including but not limited to a host or a cluster. The virtual node includes, but is not limited to, a virtual machine or a cluster of virtual machines.

In some embodiments, the metadata stores message records and/or link trace information for the target message, at least for determining path information for error-prone routing segments, or may be used to determine routing information for a complete routing path for the target message.

Thus, various information required for analysis according to error elimination in error analysis is facilitated.

In some embodiments, the step S220 further comprises:

In this embodiment, the tracking priority of the target message is also determined according to the attribute information of the target message.

The target messages with different attribute information have different tracking priorities, and the higher the tracking priority is, the more detailed recording mode is adopted to record the target messages; thus, detailed tracking of the target message with high priority is realized.

Therefore, the tracking information can be recorded in a corresponding recording mode according to the user setting or the importance of the message, and unnecessary acquisition and/or recording of the tracking information is reduced.

In some embodiments, there are multiple ways of recording, and several alternatives are provided below:

information info mode;

error mode;

debug mode.

In this embodiment, the recording details of the error mode are higher than those of the information mode; the recording details of the debug mode are higher than those of the error mode.

For example, in some cases, the information mode only records some general information in the target message tracking process; the error mode may record error information, such as the routing node in error, the type of error, and the like, in addition to general information. Besides general information and error information, the debugging mode also records some description information, which is further detailed description of one or more information points in the general information and/or the error information, or detailed information of other information besides the general information and/or the error information.

The general information includes but is not limited to: queue information, routing information, project information, message type, service in which the message is located, etc.

In other embodiments, the method further comprises:

In this embodiment, in order to reduce the problem that information cannot be normally transmitted due to an abnormality of a single middleware or a single message queue, in this embodiment, backups are formed in both the middleware and the message queue. For example, the main middleware and the main message queue are configured in a first cluster, the backup middleware and the backup message queue are configured in a second cluster, and if the first cluster is abnormal, the main middleware and the backup message queue are automatically switched to the middleware and the backup message queue on the second cluster to process messages.

In some embodiments, the backup middleware and the backup message queue are configured on the second cluster in advance, but in some cases, the backup middleware and the backup message queue may also be configured on the preset second cluster temporarily when the abnormality of the main middleware and the main message queue is detected.

In some embodiments, the middleware and message queue backups may be item-granular. For example, when different items use different middleware and message queues, if the item a has a transmission abnormality, the message transmission for the whole item a can be migrated to the second cluster of the backup.

In some embodiments, a middleware may process target messages of multiple items, and the message queue is a sub-item, in which case, only the message queue and the middleware may be separated from the master backup switch.

There are many ways to determine the abnormality of the first cluster, and several alternatives are provided below:

a message sent by a first node is not received when a second node is overtime, for example, a message sent by a client is not received by a receiving end within set time such as 60 seconds;

the message retransmission rate reaches a retransmission threshold.

In some embodiments, the method further comprises:

In this embodiment, in order to provide services normally, each project may be provided with a recovery mechanism, for example, if a message of a certain service in the project is not successfully transmitted, after waiting for a certain time or once it is determined that the transmission is abnormal, the recovery mechanism is started to automatically retransmit, and the like. In this manner, after migrating the middleware and/or message queue onto the second cluster, supplemental provisioning of the non-provisioned services for the exception period may be achieved based on the recovery mechanism.

In this embodiment, the metadata sets different data objects. Different recording modes can be used for recording the message of the target message and/or the tracking information such as the link track information from one or more data objects. The metadata includes at least one of the following data objects:

a queue object;

requesting an object;

a routing object;

a resource object.

The data object includes at least one field.

Different data objects may include a different field.

For example, the queue object includes at least one of the following fields:

message identification, routing key, message header, message body, message producer, and message destination.

The message identifier is used for recording information of the unique identification message, such as the serial number of the message.

The routing key may be a key for routing queries for messages.

The message header may include: a header of the target message.

The message body may include: the body of the target message.

The producer of the message may be the source of the message.

The purpose of the message may be the recipient or consumer of the message.

The request object includes at least one of the following fields:

the creation time.

The creation time field herein may be used to store the time at which various data objects in the metadata were created.

The routing object includes at least one of the following fields:

a route identifier;

a route name;

a project name;

a queue name;

the resource object includes at least one of the following fields:

resource identification of the middleware;

the node position where the host resource is located;

resource information of the queue;

service information where the service resource is located;

port information.

In some embodiments, the method further comprises: based on the query request, querying the metadata to obtain the message record and/or the link track information of the message to be tracked; and processing the information according to the message record and/or the link track information.

After tracing, various tracing information is recorded by using the metadata, and the metadata can be inquired based on a query request subsequently, so that required information is obtained for information processing.

Specifically, the performing information processing according to the message record and/or the link track information includes: according to the message record and/or the link track information, counting the number of messages of one project, counting the number of messages transmitted in unit time, and evaluating the service capacity of a message server for processing the messages; and performing at least one of error node positioning, error service positioning and error reason positioning according to the message record and/or the link track information.

As shown in fig. 4, the present embodiment provides a message processing apparatus, including:

a first receiving module 110, configured to receive a target message;

a first determining module 120, configured to determine item information of the target message;

a transmission module 130, configured to transmit the target message by using a message queue corresponding to the item information according to the item information and the message policy.

In some embodiments, the first receiving module 110, the first determining module 120, and the transmitting module 130 may be program modules; the program module can realize the operations of receiving the target message, determining the project information, transmitting the target message and the like after being executed by the processor.

In other embodiments, the first receiving module 110, the first determining module 120 and the transmitting module 130 may be a hardware-software module; the soft and hard combining module can comprise various programmable arrays, which can be complex programmable arrays or field programmable arrays, etc.

In still other embodiments, the first receiving module 110, the first determining module 120, and the transmitting module 130 may be pure hardware modules; including but not limited to application specific integrated circuits.

In some embodiments, the apparatus further comprises:

In some embodiments, the first obtaining module is configured to obtain the attribute information of the target message according to a tracking mode.

In some embodiments, the first obtaining module is specifically configured to perform one of:

In some embodiments, the second determining module is configured to match the attribute information with attribute information of a tracking setting; and if the matching is successful, determining the message record and/or the link track information for tracking the target message.

In some embodiments, the recording module is specifically configured to perform one of:

In some embodiments, the apparatus further comprises:

In some embodiments, the recording mode includes at least one of:

information info mode;

error mode;

debug mode.

In some embodiments, the apparatus further comprises:

In some embodiments, the metadata includes at least one of the following data objects:

a queue object;

requesting an object;

a routing object;

a resource object.

In some embodiments, the queue object includes at least one of the following fields:

the request object includes at least one of the following fields:

a creation time;

the routing object includes at least one of the following fields:

a route identifier;

a route name;

a project name;

a queue name;

the resource object includes at least one of the following fields:

resource identification of the middleware;

the node position where the host resource is located;

resource information of the queue;

service information where the service resource is located;

port information.

In some embodiments, the apparatus further comprises:

In some embodiments, the processing module is configured to perform at least one of:

Several specific examples are provided below in connection with any of the embodiments described above:

example 1:

one strategy for message queuing proposed by this example is based on the combined use of middleware technology and the capabilities of the middleware for notification messages.

The system architecture of the cloud platform message policy proposed in this example is shown in fig. 5, and mainly includes four parts, namely a cloud platform, a message policy, middleware, and a message server.

The cloud platform is a platform for providing services, and includes two execution subjects, a Consumer (Consumer) and a Producer (Producer).

And the middleware encapsulates the message at a cloud platform and message server transmission interface layer.

The message server refers to a message server for providing message store-and-forward.

In the initialization process, firstly, the service of the cloud platform is started to serve as the role of a Consumer, a channel to a message server is established, and the message server is informed of establishing information such as a route, a queue and the like; then monitors the corresponding queue, at this time, the service start is completed, and the service can be provided.

When a user initiates a request through the cloud platform, the client sends a message to a corresponding service queue by means of a producer (Publisher) object in the cloud platform. Therefore, when the service is started, the message flows in the framework, as shown in fig. 5; the specific steps can be as follows:

step 1: the producer (Publisher) establishes a connection to a message server, the message passing through a middleware interface;

step 2: the middleware carries out message processing screening based on the message strategy;

and step 3: then sending to the message server;

and 4, step 4: the message server pushes the message to a message queue after passing through the route;

and 5: the cloud platform service monitors that a message comes in the queue, and the middleware pulls the message out of the message queue and screens the message strategy;

and 6, the message passes through an interface of the middleware and is finally transmitted to the corresponding cloud platform service, and the service finishes the processing of the message.

The detailed design framework diagram of the message strategy proposed by this example is shown in fig. 6. The message strategy part is added into cloud platforms such as Openstack framework and the like in a plug-in (Plugin) mode, so that the message strategy part can be compatible with a message using mode of a previous version, and is flexible and convenient.

The configurable plug-in primarily designs two parts of content, Metadata (Metadata) and Policy (Policy).

Metadata is used to collect Metadata information in producer, consumer connections and needs to be stored in a persisteable storage.

Policy is used to set policies such as specifying items or specifying queues or specifying services.

The two parts of content need to be supported by notification technology in the Openstack platform and the Trace (Trace) plug-in provided by the example. According to the two technologies, the valuable information is fully explored, the exploration of the message is realized, and therefore the service capability of the consumer can be analyzed.

The notification technology is a notification capability under an Openstack platform, and can record user operation behaviors and perform the user operation behaviors in a classified manner. The Trace plug-in records all message links assigned to the mode behavior completely according to the mode set by the user, including the routing of the message, routing keys, sending queues, sending and receiving messages, and the like.

In this example, the proposed design of Metadata includes 4 objects, namely queue (queue), request (request), route (router), and resource (resource), as shown in FIG. 7. The queue object includes: id message Id, routing _ key message routing key, headers message header, body message body, producer of from _ Id message, message resource received by to _ Id, type receiving or sending message type.

The request object includes: the id requests a unique mark of an object, the value corresponds to a request id in an openstack project, if the value does not correspond to the request id, the value is randomly generated, a queue id corresponding to a queue _ id, an id corresponding to a queue object, a message unique id contained in msg _ id message and creation time of a create _ time request are generated, and the value can be used for analyzing the message according to a time period.

The router object includes: the unique id of the id route, the exchange _ name route name, the project _ name project name and the queue _ name queue name, and with the design, the information of the queue resource can be counted according to the content of the project.

resource objects include: the id mark of the internal resource of the id, the node position where the host resource appears, the service where the service resource is located, the port monitored by the port service, and the design of the object, the complete link track of the message can be counted at any time.

The policy concept proposed by this example is the core content of the policy. Policy can be set according to the user's requirement, and is designed to count message queues, track the priority of Notification, track the route of a specified item, track all message transceiving at a certain time, query message links, and switch between master and slave message clusters.

The concrete implementation steps of the example are that according to the policy set by the user, metadata information of the resource is counted, and then the metadata information is used for making a system response. Policy, referred to in this example, is a policy that processes the collected valuable information, which is stored in a format designed by metadata.

In this example, as shown in fig. 5, the statistical message queue in Policy does not include a message Policy part in the prior art, that is, after the service in the cloud platform is started, the queue in the message server is monitored, and when a message arrives, the message is processed, where the cloud platform has many items and services of the items are also many. Therefore, the number of queues in the message server will reach a large number, the display is not intuitive, and the user cannot directly confirm the belonged situation of each queue.

The message policy proposed in this example is mainly implemented by recording through a router object in metadata, establishing connection to a message queue through a consumer object by a service in a cloud platform, performing initialization work, and initiating operation for establishing information such as a queue and a route by the service in the cloud platform. Project information, routing information and queue information are extracted from the context by the middleware and recorded into data of the router object, but repeated data cannot be recorded, and at the moment, the collected queue information is stored in a metadata layer, so that the system can inquire corresponding information according to the persistent data storage of metadata and the project name or the queue name, namely the system can have the capacity of dividing the queue, and the positioning of the message is visual and simple.

In this example, the priority of tracking Notification in Policy is a record level set according to an Openstack project, and includes several parts such as info, error, debug, and the like, and a user can set a Policy by himself, such as an error level, so that all error message records of the system and complete link track information can be collected, and these tracks are completed by a trace module designed in this example. The specific implementation process is as follows:

the Notification is an existing capability of the cloud platform for notifying messages, and can record according to the type of the message, for example, a message of an error operation is sent to an error record queue, but in the related art, only the content of the message is stored in the queue, and the message is lost due to other reasons, and is not recorded or notified. The executor of the trace module designed in this example is middleware, three modes are set, including 3 modes of item, message routing, and queue name, if the item mode is set, the trace of the following middleware content is continued according to the item information in the message sent by the publisher object of the current cloud platform, and if the trace is the trace of the following middleware content, the complete paths of the message routing, the message queue, the routing key, the message content, the producer of the message, the receiver of the message, the transmitting and receiving conditions, and the like from the publisher to the message server are recorded in the metadata layer, otherwise, the complete paths are not recorded.

If the queue mode is set, the message sent by the publisher object is distributed to the queue of the specified message, whether the currently set trace queue is consistent with the message queue sent by the pulisher is checked, and if so, the related content of the message, including the message content, the producer of the message, the receiver of the message, the receiving and sending conditions of the message, the item to which the message belongs and the like, is extracted in the metadata layer through the middleware.

If the queue routing mode is set, messages sent by the pulisher object all have corresponding message routing information in the message server, whether the route is the same with the currently set trace queue routing or not is judged through the routing information, if so, all messages passing the route are recorded, and the sending and receiving conditions of the sender, the receiver, the message content and the messages are recorded, otherwise, the messages are not recorded.

With the record for the message in the metadata layer, the user can query the location of the message error, the service of the error, and find the root cause.

For example, in metadata, a specific message content is recorded in several objects of metadata according to the schema of trace, and a user can query a link of a message and an erroneous service location according to a request id or an id of a message. If a user inquires the id of a specified message, a corresponding queue object is found through the msg _ id content in the request object in the metadata, information such as a routing key, message content, message source id and the like of the message is recorded in the queue object, and the id in the resource object is corresponded according to the message source id, so that the service of the message can be found, the information such as a node, a port and the like where the service is located can be known, and the problem can be conveniently located.

In this example, Policy tracks the sending and receiving of all messages at a certain time, and counts the sending and receiving of all messages and the links of the messages according to the trace module, and is mainly used for counting the upper limit of the pressure of the system, so that an architect can reasonably arrange the deployment nodes and services conveniently. For example, in an Openstack project, before the project is online, a system pressure test needs to be performed, this function is useful, statistics of a time dimension of 8 hours can be designed, all messages passing through a message server are recorded and stored in several objects of a metadata layer, an interval screening can be performed according to a field of create _ time in a request object of metadata, the number of messages in the interval can be found, a source id is obtained according to a message id corresponding to a queue object in the metadata object, a corresponding service is found according to a resource object, and information such as pressure time, pressure position and the like of the project is known.

In this example, the query message link in Policy refers to finding the queue object according to the record of the message collected by the metadata layer in a message ID or request ID manner, further finding the service information in the resource object according to the producer ID of the message, and presenting the service information to the user.

In this example, Policy tracks the route of a specified item, finds relevant routing information according to a router object in metadata, monitors these routing messages through a trace module, records the content, route, queue, and relevant service information in the messages, records the content of the service in a resource object in the metadata layer, and can detect and count the capability of the item with these data, provide valuable analysis data for users or developers, and analyze big data according to a time sequence with the information detected by trace. The following illustrates a specific implementation process, and sets and tracks all the routes of the nova project in the Openstack platform, where the trace module records all message links passing through all the routes of the project, has the content of the messages, can count the number of messages of the core service of the nova project, can quantify the number of messages per hour, and can roughly estimate the capacity of the service, which is embodied as the number of requests processed by each service in unit time, with the data.

Finally, this example proposes that Policy provides switching between master and slave clusters, in an Openstack project, multiple resource pools are divided for a message queue cluster, the resource pools are master and slave, and the control right of the resource pools is charged by Policy, a message queue parameter in the Openstack project is configured as a VIP address of the cluster, whenever a problem occurs in a cluster in use, the cluster in use can be automatically switched to the slave cluster, and then the cluster in question can complete self-repair, and the repair capabilities depend on some monitoring scripts, and when the cluster returns to normal, the cluster is added again to the cluster to wait for the next continued use. The specific implementation method is as follows:

in an Openstack cloud platform project, an address of a message queue server may be set for a service of each project, and in the scheme proposed in this example, as shown in fig. 8, policy control is added to an address of a message server, a plurality of active and standby clusters are designed by using Virtual IP (VIP) addresses, and when a certain class of service in a cloud platform is not connected or a message is blocked in a certain cluster, another cluster can be switched in real time, where a determination method is that the service is in a starting process or a client is in a process of connecting a message queue, the cluster is not connected or the client times out (the default is to wait for 60s) after sending the message, and the cluster connected with the service is considered to have a problem. Policy can detect problems in real time and reassign a cluster to a service from the cluster, but it should be noted that the cloud platform was previously connected to the message server and reconnected due to cluster switching. But the contents of the queues in the previous cluster do not need to be synchronized for several reasons: a retry mechanism exists in the items on the cloud platform, namely lost messages can be repeatedly sent; the queues do not need to be synchronized, and the process of connection reestablishment, namely the process equivalent to service reestablishment, can automatically create related information such as queues, routes and the like. In addition, the policy needs to clear the problem queue in the last cluster. To ensure that the last cluster can continue to be recycled. The specific cleaning method needs to use an API (application programming interface) of a back-end message server. In fig. 8, clusters 1 to n, a master backup middleware and a message queue of one item are shown, which may be arranged on any two clusters of clusters 1 to n.

The present embodiment provides a message processing apparatus, including:

a memory;

and the processor is connected with the processor and used for realizing the message processing method provided by any of the above technical schemes by executing the computer executable instructions stored on the memory, for example, the message processing method shown in fig. 2, fig. 3 or fig. 5.

The embodiment of the invention provides a computer storage medium, which stores computer executable instructions; after being executed by a processor, the computer-executable instructions can implement the message processing method provided by any of the foregoing technical solutions, for example, one or more of the message processing methods shown in fig. 2, fig. 3, and fig. 5 can be executed.

The computer storage media provided by the present embodiments may be non-transitory storage media.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described device embodiments are merely illustrative, for example, the division of the unit is only a logical functional division, and there may be other division ways in actual implementation, such as: multiple units or components may be combined, or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the coupling, direct coupling or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between the devices or units may be electrical, mechanical or other forms.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, all the functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may be separately used as one unit, or two or more units may be integrated into one unit; the integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.

Those of ordinary skill in the art will understand that: all or part of the steps for implementing the method embodiments may be implemented by hardware related to program instructions, and the program may be stored in a computer readable storage medium, and when executed, the program performs the steps including the method embodiments; and the aforementioned storage medium includes: a mobile storage device, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.

Claims

1. A message processing method, comprising:

receiving a target message;

determining item information of the target message;

2. The method of claim 1, further comprising:

acquiring attribute information of the target message;

3. The method of claim 2, wherein the obtaining of the attribute information of the target message comprises:

4. The method of claim 3, wherein the obtaining of the attribute information of the target message according to the tracking mode comprises one of:

5. The method of claim 2, wherein the determining whether to track the target message according to the attribute information comprises:

6. The method according to any one of claims 2 to 5, wherein if it is determined to track the target message, recording the message record and/or the link trace information of the target message into metadata comprises at least one of:

7. The method according to any one of claims 2 to 5, further comprising:

8. The method of claim 7,

the recording mode comprises at least one of the following modes:

information info mode;

error mode;

debug mode.

9. The method of claim 1, further comprising:

10. The method of claim 9, further comprising:

11. The method according to any one of claims 2 to 5,

the metadata includes at least one of the following data objects:

a queue object;

requesting an object;

a routing object;

a resource object.

12. The method of claim 11, wherein the queue object comprises at least one of the following fields:

the request object includes at least one of the following fields:

a creation time;

the routing object includes at least one of the following fields:

a route identifier;

a route name;

a project name;

a queue name;

the resource object includes at least one of the following fields:

resource identification of the middleware;

the node position where the host resource is located;

resource information of the queue;

service information where the service resource is located;

port information.

13. The method according to any one of claims 2 to 5, further comprising:

14. The method of claim 13,

and processing information according to the message record and/or the link track information, wherein the processing information comprises at least one of the following steps:

15. A message processing apparatus, comprising:

the first receiving module is used for receiving the target message;

16. The apparatus of claim 15, further comprising:

17. The apparatus of claim 16, wherein the first obtaining module is configured to obtain the attribute information of the target message according to a tracking mode.

18. The apparatus of claim 17, wherein the first obtaining module is specifically configured to perform one of:

19. The apparatus of claim 16, wherein the second determining module is configured to match the attribute information with attribute information of a tracking setting; and if the matching is successful, determining the message record and/or the link track information for tracking the target message.

20. The apparatus according to any one of claims 16 to 19, wherein the recording module is specifically configured to perform one of:

21. The apparatus of any one of claims 16 to 19, further comprising:

22. The apparatus of claim 15, further comprising:

23. The apparatus of claim 22, further comprising:

24. The apparatus of any one of claims 16 to 19, further comprising:

25. The apparatus of claim 24,

the processing module is configured to perform at least one of:

26. A message processing device, comprising:

a memory;

a processor coupled to the processor for enabling implementation of the method provided in any one of claims 1 to 14 by executing computer executable instructions stored on the memory.

27. A computer storage medium having stored thereon computer-executable instructions; the computer-executable instructions, when executed by a processor, are capable of performing the method provided by any one of claims 1 to 14.