CN115242787B

CN115242787B - Message processing system and method

Info

Publication number: CN115242787B
Application number: CN202210872959.9A
Authority: CN
Inventors: 杨宇; 马进朝; 董攀峰; 徐进; 张君华; 童世红; 周翔
Original assignee: Hundsun Technologies Inc
Current assignee: Hundsun Technologies Inc
Priority date: 2022-07-22
Filing date: 2022-07-22
Publication date: 2023-09-05
Anticipated expiration: 2042-07-22
Also published as: CN115242787A

Abstract

The present specification provides a message processing system and method, wherein the message processing system comprises: the producer client is used for collecting the message to be processed corresponding to the target business theme, processing the message to be processed according to the effective configuration information to obtain a target message, and sending the target message to the Kafka cluster for storage; the configuration information is used for recording the mapping relation between at least two business topics and the message storage path; and the consumer client is used for subscribing a target partition in the Kafka cluster according to the configuration information under the condition of receiving a message processing request associated with the target business theme, and pulling the target message in the target partition in response to the message processing request.

Description

Message processing system and method

Technical Field

The present disclosure relates to the field of computer technology, and in particular, to a message processing system and method.

Background

Kafka, as a high-throughput distributed publish-subscribe messaging system, can be implemented to handle all action flow data of consumers in websites. Such action flow data is a key factor for many social functions on a network, and this part of the data is usually needed to be solved by handling logs and log aggregations due to throughput requirements. And the throughput capacity of the Kafka is maintained by the modes of sequential read-write, zero copy, direct Memory Access (DMA) and the like of the disk, so that the Kafka has better performance in the scene. However, in the Kafka cluster, topic is used as a basis for classifying message types, if it increases, the partition will increase, and accordingly, sequential read-write becomes random read-write, and if the topic is too much, throughput will decrease, so an effective solution is needed to solve the above problem.

Disclosure of Invention

In view of this, the present description embodiments provide a message processing system. The present specification also relates to a message processing method, another message processing system, another message processing method, a computing device, and a computer readable storage medium, which solve the technical drawbacks of the prior art.

According to a first aspect of embodiments of the present specification, there is provided a message processing system comprising a producer client and a consumer client;

the producer client is used for collecting the message to be processed corresponding to the target business theme, processing the message to be processed according to the effective configuration information to obtain a target message, and sending the target message to the Kafka cluster for storage; the configuration information is used for recording the mapping relation between at least two business topics and the message storage path;

and the consumer client is used for subscribing a target partition in the Kafka cluster according to the configuration information under the condition of receiving a message processing request associated with the target business theme, and pulling the target message in the target partition in response to the message processing request.

Optionally, the producer client determines a message template corresponding to the target service theme according to the configuration information, adds the message to be processed in a message header of the message template, and generates the target message according to an addition result.

Optionally, the producer client analyzes the target message to obtain a target message storage path added in a message header of the message to be processed, and sends the target message to the Kafka cluster according to the target message storage path; and the Kafka cluster stores the target message to the target partition according to the target message storage path.

Optionally, the consumer client receives the message processing request, parses the message processing request, determines the target service topic according to a parsing result, determines partition information corresponding to the target service topic in the configuration information, and subscribes to the target partition in the Kafka cluster according to the partition information.

Optionally, the consumer client sends a message consumption request to the Kafka cluster in response to the message processing request, receives a partition message fed back by the Kafka cluster for the message consumption request, and determines the target message in the partition message; the partition message is a global message stored in the target partition.

Optionally, the system also comprises a message management end; the message management end determines category information associated with the Kafka cluster, establishes mapping relation between at least two business topics and a message storage path according to the category information, creates configuration information according to the mapping relation, and sends the configuration information to the producer client and the consumer client.

Optionally, the message management end receives a message recording request, determines a message consumption link and a recording time interval according to the message recording request, records the message consumption link in the recording time interval, and generates and stores a message record to be traced according to a recording result; and under the condition that a backtracking request associated with the recording time interval is received, responding to the backtracking request and executing the record of the message to be backtracked.

Optionally, the producer client receives the configuration information, determines a mapping relationship between at least two service topics and a message storage path according to the configuration information under the condition that the configuration information is effective, creates a message link corresponding to the mapping relationship, and uses the message link as a message template corresponding to the target service topic.

According to a second aspect of embodiments of the present specification, there is provided a message processing method, comprising:

the method comprises the steps that a producer client collects information to be processed corresponding to a target service theme, processes the information to be processed according to effective configuration information to obtain a target message, and sends the target message to a Kafka cluster for storage; the configuration information is used for recording the mapping relation between at least two business topics and the message storage path;

and under the condition that the consumer client receives a message processing request associated with the target business theme, subscribing a target partition in the Kafka cluster according to the configuration information, and pulling the target message in the target partition in response to the message processing request.

According to a third aspect of embodiments of the present specification, there is provided another message processing system comprising:

a message management end, a producer client and a consumer client;

the message management end is used for establishing a mapping relation between at least two business topics and a message storage path, creating configuration information according to the mapping relation and sending the configuration information to the producer client and the consumer client;

the producer client is used for collecting a message to be processed corresponding to a target service theme under the condition that the configuration information is effective, processing the message to be processed according to the effective configuration information to obtain a target message, and sending the target message to a Kafka cluster for storage;

According to a fourth aspect of embodiments of the present specification, there is provided another message processing method, comprising:

the method comprises the steps that a message management end establishes a mapping relation between at least two business topics and a message storage path, creates configuration information according to the mapping relation and sends the configuration information to a producer client and a consumer client;

under the condition that the configuration information is effective, the producer client acquires a message to be processed corresponding to a target service theme, processes the message to be processed according to the effective configuration information to obtain a target message, and sends the target message to a Kafka cluster for storage;

According to a fifth aspect of embodiments of the present specification, there is provided a computing device comprising:

a memory and a processor;

the memory is configured to store computer-executable instructions that, when executed, implement the steps of the message processing method.

According to a sixth aspect of embodiments of the present description, there is provided a computer readable storage medium storing computer executable instructions which, when executed by a processor, implement the steps of the message processing method.

In the message processing system provided by the specification, after the producer client generates the message to be processed associated with the target service subject, in order to ensure that the problem of throughput degradation does not occur when the Kafka cluster processes the message, the limited resource of the Kafka cluster can be fully utilized, the message to be processed can be processed according to the effective configuration information to obtain the target message, and then the target message is sent to the Kafka cluster, wherein the configuration information records the mapping relation between at least two service subjects and the message storage path. On the basis, when the consumer client terminal performs message consumption, the target partition in the Kafka cluster can be determined according to the configuration information, and the effect that the target message can be pulled from the target partition for consumption in response to the message processing request is realized. The method and the device have the advantages that the mapping relation between at least two business topics and the message storage path is established through configuration information, and the message storage and consumption are carried out on the basis of the mapping relation, so that the topic number and the partition number of the Kafka cluster can be effectively controlled, and the throughput capacity of the Kafka cluster is effectively ensured.

Drawings

FIG. 1 is a schematic diagram of a message processing system according to an embodiment of the present disclosure;

FIG. 2 is a flow diagram of a message consumption process according to an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of a message processing system according to one embodiment of the present disclosure;

FIG. 4 is a flow chart of a message processing method according to an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of another message processing system according to an embodiment of the present disclosure;

FIG. 6 is a flow chart of another message processing method provided by an embodiment of the present disclosure;

FIG. 7 is a process flow diagram of a message processing system according to one embodiment of the present disclosure;

FIG. 8 is a block diagram of a computing device according to one embodiment of the present disclosure.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be embodied in many other forms than described herein and similarly generalized by those skilled in the art to whom this disclosure pertains without departing from the spirit of the disclosure and, therefore, this disclosure is not limited by the specific implementations disclosed below.

The terminology used in the one or more embodiments of the specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of the specification. As used in this specification, one or more embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any or all possible combinations of one or more of the associated listed items.

It should be understood that, although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first may also be referred to as a second, and similarly, a second may also be referred to as a first, without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.

First, terms related to one or more embodiments of the present specification will be explained.

Kafka: is a high throughput distributed publish-subscribe messaging system that can handle all action flow data for consumers in a web site. Such actions (web browsing, searching and other user actions) are a key factor in many social functions on modern networks. These data typically need to be addressed by processing logs and log aggregations due to throughput requirements.

topic: each message issued to the Kafka cluster has a category called topic.

part is: the partitions are represented in Kafka, each topic in Kafka can be divided into a plurality of partitions (at least one partition is arranged in each topic), and messages contained in different partitions under the same topic are different.

DMA (Direct Memory Access ): the DMA is an independent chip on the main board, and when the data transmission of the memory and the IO equipment is carried out, the data transmission is not controlled by the CPU, but is directly realized by the DMA controller.

In the present specification, a message processing system is provided, and the present specification relates to a message processing method, another message processing system, another message processing method, a computing device, and a computer-readable storage medium, one by one, which are described in detail in the following embodiments.

Referring to fig. 2, in the prior art, after a producer a or B issues a message, if the message belongs to the same category, the message is sent to the same topic, and then the message is written into partition 1, partition 2 or partition 3 in the topic, respectively, and after the writing of the message is completed; when the consumer needs to consume the information, the consumer subscribes to the partition corresponding to the needed information, and pulls the information from the partition for consumption. Although Kafka has higher throughput capability, once topic increases, the number of partitions increases, and accordingly sequential read-write becomes random read-write, when the pagecache used by Kafka is too large, a large amount of disk I/O is triggered, and the load is exceeded, so that the CPU time of each application is occupied, the main flow is influenced, and the main flow is slowed down, so that Kafka in practical application can be largely used in log centers, big data calculation and the like, and under the scene of small demand on topic and relatively simple logic. If the problem of reduced Kafka throughput can be solved, it can be applied to complex scenarios, so that the problem of excessive partition count needs to be solved, then topic needs to be managed.

In view of this, in the message processing system provided in the present disclosure, after the producer client generates the message to be processed associated with the target service topic, in order to ensure that the Kafka cluster does not have a problem of throughput degradation during processing of the message, and at the same time, the limited resource of the Kafka cluster can be fully utilized, the message to be processed may be processed according to the validated configuration information, so as to obtain the target message, and then the target message is sent to the Kafka cluster, where the configuration information records the mapping relationship between at least two service topics and the message storage path. On the basis, when the consumer client terminal performs message consumption, the target partition in the Kafka cluster can be determined according to the configuration information, and the effect that the target message can be pulled from the target partition for consumption in response to the message processing request is realized. The method and the device have the advantages that the mapping relation between at least two business topics and the message storage path is established through configuration information, and the message storage and consumption are carried out on the basis of the mapping relation, so that the topic number and the partition number of the Kafka cluster can be effectively controlled, and the throughput capacity of the Kafka cluster is effectively ensured.

FIG. 1 shows a schematic diagram of a message processing system provided in accordance with an embodiment of the present description, the message processing system 100 including a producer client 110 and a consumer client 120, and a Kafka cluster 130;

The producer client 110 is configured to collect a message to be processed corresponding to a target service theme, process the message to be processed according to the validated configuration information, obtain a target message, and send the target message to the Kafka cluster 130 for storage; the configuration information is used for recording the mapping relation between at least two business topics and the message storage path;

consumer client 120 is configured to subscribe to a target partition in Kafka cluster 130 according to the configuration information, and pull the target message in the target partition in response to the message processing request, when receiving the message processing request associated with the target service topic.

Since Kafka has the advantages of better throughput, built-in partitioning, redundancy and fault tolerance, it can be applied to different message processing scenarios; if the behavior tracking scene is adopted, kafka can be used for tracking the page browsing, searching and other behaviors of a user, the behaviors are recorded in the corresponding topic in real time through a publish-subscribe mode, and real-time monitoring can be realized through access processing analysis of a rear-end big data platform. Or, as in the log collection scenario, the log aggregation can be realized through Kafka, so that the log files are collected from the server and then processed through a centralized platform.

In this embodiment, the message processing system is described by taking an example that Kafka is applied to a behavior tracking scene, and the message processing processes in other scenes can be referred to the same or corresponding description contents in this embodiment, which is not repeated here.

Specifically, the producer client specifically refers to a client corresponding to the message producer, namely a client for sending a message to the Kafka cluster; accordingly, the consumer client specifically refers to a client corresponding to the message consumer, that is, a client that fetches a message to the Kafka cluster.

Further, the target business topic specifically refers to a topic of a category to which the message to be processed belongs, including but not limited to a market information topic, a rights and interests log topic, a combined message change topic, a user information change topic, etc., that is, a topic carrying a business definition. The target business theme can make clear which topic in the Kafka cluster needs to be written into the message to be processed after the message is processed. Correspondingly, the message to be processed specifically refers to the message issued by the producer client; correspondingly, the target message specifically refers to a message processed according to the configuration information, and the message finally written into the Kafka cluster is the target message. The configuration information is information specifying the service theme and the message storage path, and since the message issued by the producer client side must belong to one service theme, the storage path of the message can be specified through the configuration information, so as to define the partition where the message needs to be written, after the message is issued, the message can be processed according to the configuration information, so as to obtain the target message recording the routing information, and the target message is sent to the Kafka cluster for storage.

That is, the configuration message can process the message to be processed into the target message, and define the message storage path of the target service topic associated message in the target message, so as to realize that the message for different service topics can be explicitly pointed to topic through the configuration information, and the message associated with each service topic is stored and consumed by using the effective topic.

Correspondingly, the message processing request specifically refers to a request that the consumer client needs to perform message consumption processing, and the request is sent by the Kafka cluster; accordingly, the target partition specifically refers to a partition in topic associated with the target business topic in the Kafka cluster, and is used for storing target messages written into the Kafka cluster by the producer client.

Based on the above, when the configuration information is validated by both the producer client and the consumer client, and the message to be processed associated with the target business theme is collected, the message to be processed can be processed according to the validated configuration information, so that the relevant information with storage directivity is written in the message to be processed, and the target message is obtained. And then the target message is sent to the Kafka cluster for storage. Because the mapping relation between at least two business topics and the message storage path is recorded in the configuration information, after the Kafka cluster receives the target message, the writing of the target message into the target partition in the designated topic can be completed according to the related information with storage directivity carried in the target message.

Further, under the condition that the consumer client receives the message processing request associated with the target service theme, the target partition under topic associated with the target service theme can be determined according to the effective configuration information, and then the consumer client can subscribe to the target partition in the Kafka cluster, so that the target message can be pulled in the target partition for consumption in response to the message processing request.

It should be noted that, because the producer client and the consumer client both take effect configuration information, the message storage path corresponding to any service theme can be defined through the configuration information, meanwhile, in order to improve the utilization of effective topic to complete the storage of messages associated with various service themes, a mapping relationship between at least two service themes and one topic can be established, so as to achieve the purposes that various service themes can be written into the same topic, and meanwhile, the to-be-processed message is processed into the target message before writing, so that the difference between the different service theme messages can be distinguished through the related information carried in the target message, and the consumer client can more conveniently and accurately pull the required target message in the same topic for consumption.

For example, the configuration information records that the service A corresponds to topica_1:sub1, and the service B corresponds to topica_2:sub1, wherein the service A is different from the service B in category; after the producer client issues the message to be processed of the association service A, the message to be processed can be processed according to the configuration information, so that the message header topica_1:sub1 of the message to be processed is filled according to the configuration information, and after filling is completed, the obtained target message is sent to the Kafka cluster, and the information of the association configuration information is recorded in the message header of the target message, so that the first partition in the topicA theme corresponding to the target message can be determined according to the topica_1, and the target message can be sent to the first partition of the topicA theme and stored. Further, when the consumer client needs to consume the message, it is determined that the message is stored in the first partition of the topicA theme according to the configuration information, the first partition of the topicA theme in the Kafka cluster may be subscribed, the message is pulled from the first partition, and the target message is filtered and used.

In summary, after the producer client generates the message to be processed associated with the target service topic, in order to ensure that the problem of throughput degradation does not occur when the Kafka cluster processes the message, the limited resource of the Kafka cluster can be fully utilized, the message to be processed can be processed according to the validated configuration information to obtain the target message, and then the target message is sent to the Kafka cluster, wherein the configuration information records the mapping relationship between at least two service topics and the message storage path. On the basis, when the consumer client terminal performs message consumption, the target partition in the Kafka cluster can be determined according to the configuration information, and the effect that the target message can be pulled from the target partition for consumption in response to the message processing request is realized. The method and the device have the advantages that the mapping relation between at least two business topics and the message storage path is established through configuration information, and the message storage and consumption are carried out on the basis of the mapping relation, so that the topic number and the partition number of the Kafka cluster can be effectively controlled, and the throughput capacity of the Kafka cluster is effectively ensured.

Further, since the configuration information defines the mapping relationship between different service topics and topics, and after the messages corresponding to the different service topics are issued, storage in the Kafka cluster is completed according to the mapping relationship, so that the Kafka cluster can define which topic the messages of the different service topics should be written in, a message template of the mapping relationship between the service topics and the topics can be created according to the configuration information, and after the messages associated with the target service topics are obtained, creation of the target message can be completed through the corresponding message template.

Specifically, the message template specifically refers to a template for specifying mapping relations between different service topics and different topics according to effective configuration information, after obtaining a message to be processed associated with a target service topic, the message template corresponding to the target service topic can be determined first, and then the message to be processed is added into the message template, so that the target message recorded with message routing information can be obtained, and the storage of the message by a subsequent Kafka cluster and the pulling of the message by a consumer client are facilitated.

Based on the above, when the producer client needs to issue the message to be processed associated with the target business theme, the message template corresponding to the target business theme can be determined according to the configuration information, then the message to be processed is added in the message header of the message template, and the target message can be obtained according to the addition result, so that the message can be stored and pulled according to the information recorded in the message header.

In a specific implementation, the message template may exist in a message link manner, so that corresponding information may be written in a link in a message storage stage, in this embodiment, the producer client is further configured to receive the configuration information, determine, according to the configuration information, a mapping relationship between at least two service topics and a message storage path when the configuration information is effective, create a message link corresponding to the mapping relationship, and use the message link as a message template corresponding to the target service topic.

The message link specifically refers to a link created according to a mapping relationship, and is used as a message template to facilitate message insertion and recording of routing information.

In summary, by determining a message template associated with the target service theme in combination with the configuration information and determining the target message based on the message template, it is ensured that the routing information of the associated message is recorded in the message header of the target message, so that the Kafka cluster can clearly store the location, and when the consumer client consumes the message, the required message can be filtered out according to the information in the message header.

Further, after the producer client finishes determining the target message, it also needs to send the message to the Kafka cluster, so as to assist the Kafka cluster to determine the storage location of the target message, analyze the target message to obtain a target message storage path, and send and store the message according to the path. And the Kafka cluster stores the target message to the target partition according to the target message storage path.

Specifically, the target message storage path specifically refers to a logical topic associated with the target message, and is used to specify a storage partition of the target message. Based on the above, after the information is filled in the message header to obtain the target message, the target message can be parsed to obtain a target message storage path in the message header to which the message to be processed is added, and then the target message is sent to the Kafka cluster according to the target message storage path, at this time, the Kafka cluster can determine the target partition according to the target message storage path, and sequentially write the target message in the target partition.

Along the above example, after the producer client issues the message to be processed of the association service a, the producer client obtains the target message by filling the message header topica_1:sub1, wherein the message header is filled with the routing information of the message to be processed. When the message is required to be stored, the first stage of the message header can be resolved as a sending logic theme, namely the resolving result is topica_1, the fact that the message of the associated service A needs to be written into the first partition of the topicA theme in the Kafka cluster is determined, and at the moment, the producer client can send the target message to the Kafka cluster according to the topica_1 so as to write the target message into the first partition of the topicA theme.

In summary, by defining the topic and the partition according to the target message storage path, the Kafka cluster can quickly and accurately write the target message into the target partition, and meanwhile, the message is specified to the partition, so that the sequential storage of the messages can be ensured.

On the basis, when the consumer client performs message consumption processing, because the mapping relationship between the theme and the topic is recorded in the configuration information, in order to accurately pull the target message related to the target service theme, the partition to which the message belongs needs to be determined first.

Specifically, the partition information is information recorded in the configuration information, and a message related to the related target business theme is written into the target partition, so that the partition to which the message belongs needs to be explicitly pulled. Based on this, after the consumer client receives the message processing request, in order to accurately pull and cancel the message processing request for use, the message processing request may be parsed first to determine a target service topic according to a parsing result, that is, a category to which a message explicitly needs to be consumed belongs, then determine partition information corresponding to the target service topic in the configuration information, and finally subscribe to a target partition in the Kafka cluster, that is, a target partition in a topic associated with the target service topic according to the partition information.

In summary, in order to ensure that the consumer client can accurately pull the message, the target service theme can be determined through the message processing request, and then the partition information is determined by matching with the configuration information, so that the partition associated with the target service theme can be subscribed according to the partition information, and the subsequent pulling of the target message is facilitated.

Further, in the process of pulling the target message, considering that not only the related message of the target service theme but also the related message of other service themes exist in the same partition, when the target message is pulled, the target message needs to be determined from the partition message, in this embodiment, the consumer client is further configured to send a message consumption request to the Kafka cluster in response to the message processing request, receive the partition message fed back by the Kafka cluster for the message consumption request, and determine the target message in the partition message; the partition message is a global message stored in the target partition.

Specifically, the partition message specifically refers to all messages stored in the target partition; based on this, in the process of pulling the message by the consumer client, a message consumption request may be sent to the Kafka cluster in response to the message processing request, where the Kafka cluster may determine that the consumer client needs to pull the message of the target service topic, so that topic associated with the target service topic may be determined first, then a target partition storing the target service topic is determined in the topic, and then the partitioned message is fed back to the consumer client. And finally, the consumer client can filter out the target message for use according to the partition message fed back by the Kafka cluster.

Along the above example, when it is determined that the consumer client needs to use the message of service a, a message consumption request may be sent to the Kafka cluster, and the related message of service a is determined to be stored in the first partition of topicA according to topica_1. And because a large number of messages are stored in the first partition, the target message of the associated service A can be filtered out according to sub1 in the message header. In the process, the consumer client determines the message pulled from the first partition of topicA, then judges whether the message header information is matched with the message header sub1 mapped by the service A, if not, the message is discarded until the target message matched with the sub1 is pulled for use.

In conclusion, by determining the target message according to the message header matching mode, the consumer client can be ensured to accurately pull the target message for use.

In addition, since the configuration information is the basis of the mapping relationship between the standard service theme and the topic, in order to effectively control the topic and the partition number in the Kafka cluster, the message management end can set the configuration information for controlling the increase of the topic and the partition number, so that the effective topic and the partition number are utilized to map a plurality of service themes, and in this embodiment, the message processing system further comprises a message management end; the message management end is used for determining category information associated with the Kafka cluster, establishing mapping relation between at least two business topics and a message storage path according to the category information, establishing configuration information according to the mapping relation, and sending the configuration information to the producer client and the consumer client.

The message management end is a mapping management end responsible for service topics and Kafka message topics, can use annotation identification service topic identification during service coding, and can convert SDK (Software Development Kit ) into a message header for generating configuration information by combining the service topics and the message header.

Based on the above, in the configuration stage, the message management end may determine the category information associated with the Kafka cluster, then establish a mapping relationship between at least two service topics and the message storage path according to the category information, then create configuration information according to the mapping relationship, and send the configuration information to the producer client and the consumer client.

That is, the message manager may identify the service topic using annotations while converting the SDK into a message header at the time of service encoding, and thereafter may tear down the first level from the target message as the message topic of Kafka at the time of message storage, and may customize the partition identifier, such as topica_1, to be sent to a designated partition for sequential delivery and sequential consumption. The service theme and the message header can form complete message configuration, and can be issued to the producer client and the consumer client through a configuration center or multiplexing Kafka.

In the implementation, in the service topic encoding stage, a service message identifier of a message can be implanted, then a message management end can edit the mapping between a service topic and a message header to realize service A-topica_1:sub1, the message management end sends configuration information to a producer client and a consumer client through a configuration center or multiplexing Kafka, after the producer client and the consumer client acquire the configuration information, validation can be carried out, and the client can convert the service topic list into a real topic list topicA according to a service message list for subsequent message sending and consumption.

Referring to fig. 3, after publishing the message, the producer client will write topicA in the message header: sub1 and topicA: sub2, topicB: sub3 and topicB: sub4, kafka cluster may write topicA according to the routing information written in the message header: sub1 and topicA: writing a message corresponding to sub2 into topicA, and writing topicB: sub3 and topicB: the message corresponding to sub4 is written into topicB, and when the consumer client consumes data, topicA can be pulled from topicA: sub1 and topicA: sub2, pulling topicB in topicB: sub3 and topicB: sub4 for subsequent use.

In summary, the configuration information is created and managed by the message management end, so that the management and control of the limited resources in the Kafka cluster can be realized, the sending and consumption of the message by using the limited resources can be realized, and the throughput of the Kafka cluster can be ensured.

In addition, in order to monitor the life cycle of the important message, the message management end can also provide recording and backtracking functions, in this embodiment, the message management end is further configured to receive a message recording request, determine a message consumption link and a recording time interval according to the message recording request, record the message consumption link in the recording time interval, and generate and store a message record to be backtracked according to a recording result; and under the condition that a backtracking request associated with the recording time interval is received, responding to the backtracking request and executing the record of the message to be backtracked.

Specifically, the message recording request specifically refers to a request for recording the storage and consumption processes of the message; correspondingly, the message consumption link specifically refers to a link corresponding to the message consumption process from message generation to message storage, and then to the message storage, namely, a life cycle corresponding to the message to be recorded; correspondingly, the recording time interval specifically refers to the designated time of the whole life cycle of the recording message. Correspondingly, the backtracking request specifically refers to a request for playing back the life cycle of the recording time interval message.

Based on the above, when the message management end receives the message recording request, it indicates that the recording needs to be performed for the storage and consumption process of a certain message, so as to facilitate the backtracking at other times, therefore, the message consumption link and the recording time interval can be determined according to the message recording request, the message consumption link can be recorded in the recording time interval, and the message record to be backtracked is generated and stored according to the recording result; and under the condition that the backtracking request of the associated recording time interval is received, the record of the message to be backtracked can be executed in response to the backtracking request, so that the storage and consumption processes of the message are replayed. That is, the message recording of the designated message link can be started at the designated time, and the recorded content is retransmitted to the message server when needed, so that the service scene reproduction of the recorded time interval is realized.

For example, in the time interval t1-t2, the message corresponding to the service B is written into the second partition of topicB of the Kafka cluster, and when the consumer client consumes, the information use of sub3 of the associated service B is pulled in the second partition by subscribing to the second partition of topicB of the Kafka cluster. When the backtracking is needed for the message of the service B in the time interval of t1-t2, the above-mentioned flow can be re-executed for assisting the processing of problem positioning and the like in the backtracking stage.

In summary, the message management end provides the functions of recording and backtracking the messages, so that the message storage and consumption processes in the key time interval can be monitored, the problem can be accurately positioned, and the standardization of the application scene is improved.

Corresponding to the above system embodiment, the present disclosure further provides a message processing method embodiment, and fig. 4 shows a flowchart of a message processing method provided in an embodiment of the present disclosure. As shown in fig. 4, the method includes:

step S402, a producer client collects a message to be processed corresponding to a target service theme, processes the message to be processed according to effective configuration information to obtain a target message, and sends the target message to a Kafka cluster for storage; the configuration information is used for recording the mapping relation between at least two business topics and the message storage path;

In step S404, the consumer client subscribes to the target partition in the Kafka cluster according to the configuration information under the condition of receiving the message processing request associated with the target service theme, and pulls the target message in the target partition in response to the message processing request.

In an optional embodiment, the producer client determines a message template corresponding to the target service theme according to the configuration information, adds the message to be processed in a message header of the message template, and generates the target message according to an addition result.

In an optional embodiment, the producer client parses the target message to obtain a target message storage path in a message header to which the message to be processed is added, and sends the target message to the Kafka cluster according to the target message storage path; and the Kafka cluster stores the target message to the target partition according to the target message storage path.

In an optional embodiment, the consumer client receives the message processing request, parses the message processing request, determines the target service topic according to a parsing result, determines partition information corresponding to the target service topic in the configuration information, and subscribes to the target partition in the Kafka cluster according to the partition information.

In an optional embodiment, the consumer client sends a message consumption request to the Kafka cluster in response to the message processing request, receives a partition message fed back by the Kafka cluster for the message consumption request, and determines the target message in the partition message; the partition message is a global message stored in the target partition.

In an alternative embodiment, the method further comprises a message management end; the message management end determines category information associated with the Kafka cluster, establishes mapping relation between at least two business topics and a message storage path according to the category information, creates configuration information according to the mapping relation, and sends the configuration information to the producer client and the consumer client.

In an optional embodiment, the message management end receives a message recording request, determines a message consumption link and a recording time interval according to the message recording request, records the message consumption link in the recording time interval, and generates and stores a message record to be traced according to a recording result; and under the condition that a backtracking request associated with the recording time interval is received, responding to the backtracking request and executing the record of the message to be backtracked.

In an optional embodiment, the producer client receives the configuration information, determines a mapping relationship between at least two service topics and a message storage path according to the configuration information under the condition that the configuration information is effective, creates a message link corresponding to the mapping relationship, and uses the message link as a message template corresponding to the target service topic.

According to the message processing method provided by the specification, after the producer client generates the message to be processed associated with the target business theme, in order to ensure that the problem of throughput reduction does not occur when the Kafka cluster processes the message, the limited resource of the Kafka cluster can be fully utilized, the message to be processed can be processed according to the effective configuration information to obtain the target message, and then the target message is sent to the Kafka cluster, wherein the configuration information records the mapping relation between at least two business themes and a message storage path. On the basis, when the consumer client terminal performs message consumption, the target partition in the Kafka cluster can be determined according to the configuration information, and the effect that the target message can be pulled from the target partition for consumption in response to the message processing request is realized. The method and the device have the advantages that the mapping relation between at least two business topics and the message storage path is established through configuration information, and the message storage and consumption are carried out on the basis of the mapping relation, so that the topic number and the partition number of the Kafka cluster can be effectively controlled, and the throughput capacity of the Kafka cluster is effectively ensured.

The above is an exemplary scheme of a message processing method of the present embodiment. It should be noted that, the technical solution of the message processing method and the technical solution of the message processing system belong to the same conception, and details of the technical solution of the message processing method, which are not described in detail, can be referred to the description of the technical solution of the message processing system.

In correspondence with the above-described embodiment, the present embodiment further provides another message processing system, and fig. 5 shows a schematic structural diagram of another message processing system provided according to an embodiment of the present disclosure, where the message processing system 500 includes a message management end 510, a producer client 520, and a consumer client 530, and a Kafka cluster 540;

the message management end 510 is configured to establish a mapping relationship between at least two service topics and a message storage path, create configuration information according to the mapping relationship, and send the configuration information to the producer client and the consumer client;

the producer client 520 is configured to collect a message to be processed corresponding to a target service topic under the condition that the configuration information is effective, process the message to be processed according to the effective configuration information, obtain a target message, and send the target message to the Kafka cluster 540 for storage;

The consumer client 530 is configured to subscribe to a target partition in the Kafka cluster 540 according to the configuration information, and pull the target message in the target partition in response to the message processing request when receiving the message processing request associated with the target service topic.

According to the message processing system provided by the specification, after the configuration information is synchronized to the producer client and the consumer client through the message management end, the mapping relation between at least two business topics and the message storage path can be established, so that after the producer client generates the message to be processed associated with the target business topic, in order to ensure that the problem of throughput reduction can not occur when the Kafka cluster processes the message, the limited resource of the Kafka cluster can be fully utilized, the message to be processed can be processed according to the validated configuration information so as to obtain the target message, and then the target message is sent to the Kafka cluster. The method and the device have the advantages that the mapping relation between at least two business topics and the message storage path is established through configuration information, and the message storage and consumption are carried out on the basis of the mapping relation, so that the topic number and the partition number of the Kafka cluster can be effectively controlled, and the throughput capacity of the Kafka cluster is effectively ensured.

Corresponding to the above system embodiment, the present disclosure further provides another embodiment of a message processing method, and fig. 6 shows a flowchart of another message processing method provided in an embodiment of the present disclosure. As shown in fig. 6, the method includes:

step S602, a message management end establishes a mapping relation between at least two business topics and a message storage path, creates configuration information according to the mapping relation and sends the configuration information to a producer client and a consumer client;

step S604, under the condition that the configuration information is effective, the producer client acquires a message to be processed corresponding to a target service theme, processes the message to be processed according to the effective configuration information to obtain a target message, and sends the target message to a Kafka cluster for storage;

in step S606, the consumer client subscribes to the target partition in the Kafka cluster according to the configuration information under the condition of receiving the message processing request associated with the target service topic, and pulls the target message in the target partition in response to the message processing request.

According to the message processing method provided by the specification, after the configuration information is synchronized to the producer client and the consumer client through the message management end, the mapping relation between at least two business topics and the message storage path can be established, so that after the producer client generates the message to be processed associated with the target business topic, in order to ensure that the problem of throughput reduction does not occur when the Kafka cluster processes the message, the limited resource of the Kafka cluster can be fully utilized, the message to be processed can be processed according to the validated configuration information so as to obtain the target message, and then the target message is sent to the Kafka cluster. The method and the device have the advantages that the mapping relation between at least two business topics and the message storage path is established through configuration information, and the message storage and consumption are carried out on the basis of the mapping relation, so that the topic number and the partition number of the Kafka cluster can be effectively controlled, and the throughput capacity of the Kafka cluster is effectively ensured.

The application of the message processing system provided in the present specification in the log processing scenario is taken as an example, and the message processing system is further described below with reference to fig. 7. Fig. 7 shows a process flow diagram of a message processing system according to an embodiment of the present disclosure, which specifically includes the following steps:

in step S702, the message management end determines category information associated with the Kafka cluster, and establishes a mapping relationship between at least two service topics and the message storage path according to the category information.

In step S704, the message management end creates configuration information according to the mapping relationship, and sends the configuration information to the producer client and the consumer client.

Step S706, the producer client collects the message to be processed corresponding to the target business theme, and determines a message template corresponding to the target business theme according to the configuration information.

In step S708, the producer client adds the message to be processed in the message header of the message template, and generates the target message according to the addition result.

In step S710, the producer client parses the target message to obtain a target message storage path in the message header to which the message to be processed is added, and sends the target message to the Kafka cluster according to the target message storage path.

And the Kafka cluster stores the target message to the target partition according to the target message storage path, and the configuration information is used for recording the mapping relation between at least two business topics and the message storage path.

Step S712, the consumer client receives the message processing request, analyzes the message processing request, and determines the target business topic according to the analysis result.

In step S714, the consumer client determines partition information corresponding to the target service topic in the configuration information, and subscribes to the target partition in the Kafka cluster according to the partition information.

In step S716, the consumer client sends a message consumption request to the Kafka cluster in response to the message processing request, and receives the partition message fed back by the Kafka cluster for the message consumption request.

In step S718, the consumer client determines the target message in the partition message.

The partition message is a global message stored in the target partition.

In summary, the mapping relation between at least two service topics and the message storage path is established through the configuration information, and the storage and consumption of the messages are carried out on the basis of the mapping relation, so that the topic number and the partition number of the Kafka cluster can be effectively controlled, and the throughput capacity of the Kafka cluster is effectively ensured.

Fig. 8 illustrates a block diagram of a computing device 800 provided in accordance with an embodiment of the present specification. The components of computing device 800 include, but are not limited to, memory 810 and processor 820. Processor 820 is coupled to memory 810 through bus 830 and database 850 is used to hold data.

Computing device 800 also includes access device 840, access device 840 enabling computing device 800 to communicate via one or more networks 860. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. Access device 840 may include one or more of any type of network interface, wired or wireless (e.g., a Network Interface Card (NIC)), such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.

In one embodiment of the present description, the above-described components of computing device 800, as well as other components not shown in FIG. 8, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device illustrated in FIG. 8 is for exemplary purposes only and is not intended to limit the scope of the present description. Those skilled in the art may add or replace other components as desired.

Computing device 800 may be any type of stationary or mobile computing device including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smart phone), wearable computing device (e.g., smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 800 may also be a mobile or stationary server.

Wherein processor 820 is configured to implement the steps of the message processing method described above when executing computer-executable instructions as follows.

The foregoing is a schematic illustration of a computing device of this embodiment. It should be noted that, the technical solution of the computing device and the technical solution of the message processing method belong to the same concept, and details of the technical solution of the computing device, which are not described in detail, can be referred to the description of the technical solution of the message processing method.

An embodiment of the present disclosure also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the message processing method described above. The above is an exemplary version of a computer-readable storage medium of the present embodiment. It should be noted that, the technical solution of the storage medium and the technical solution of the message processing method belong to the same concept, and details of the technical solution of the storage medium which are not described in detail can be referred to the description of the technical solution of the message processing method.

The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. The computer instructions include computer program code that may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

It should be noted that, for the sake of simplicity of description, the foregoing method embodiments are all expressed as a series of combinations of actions, but it should be understood by those skilled in the art that the present description is not limited by the order of actions described, as some steps may be performed in other order or simultaneously in accordance with the present description. Further, those skilled in the art will appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily all necessary in the specification. In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments.

The preferred embodiments of the present specification disclosed above are merely used to help clarify the present specification. Alternative embodiments are not intended to be exhaustive or to limit the invention to the precise form disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the disclosure and the practical application, to thereby enable others skilled in the art to best understand and utilize the disclosure. This specification is to be limited only by the claims and the full scope and equivalents thereof.

Claims

1. A message processing system comprising a producer client and a consumer client;

the producer client is used for collecting the message to be processed corresponding to the target business theme, determining a message template corresponding to the target business theme according to the validated configuration information, adding the message to be processed in the message header of the message template, generating a target message according to the addition result, and sending the target message to the Kafka cluster for storage; the configuration information is used for recording mapping relations between at least two business topics and a message storage path, creating a message link corresponding to the mapping relations, and taking the message link as a message template corresponding to the target business topic;

2. The system of claim 1, wherein the producer client is further configured to parse the target message to obtain a target message storage path for adding to a message header of the message to be processed, and send the target message to the Kafka cluster according to the target message storage path; and the Kafka cluster stores the target message to the target partition according to the target message storage path.

3. The system of claim 1, wherein the consumer client is further configured to receive the message processing request, parse the message processing request, determine the target service topic according to the parsing result, determine partition information corresponding to the target service topic in the configuration information, and subscribe to the target partition in the Kafka cluster according to the partition information.

4. A system according to claim 1 or 3, wherein the consumer client is further configured to send a message consumption request to the Kafka cluster in response to the message processing request, receive a partition message fed back by the Kafka cluster for the message consumption request, and determine the target message in the partition message; the partition message is a global message stored in the target partition.

5. The system of claim 1, further comprising a message manager;

the message management end is used for determining category information associated with the Kafka cluster, establishing mapping relation between at least two business topics and a message storage path according to the category information, establishing configuration information according to the mapping relation, and sending the configuration information to the producer client and the consumer client.

6. The system of claim 5, wherein the message management terminal is further configured to receive a message recording request, determine a message consumption link and a recording time interval according to the message recording request, record the message consumption link in the recording time interval, and generate and store a message record to be traced according to a recording result; and under the condition that a backtracking request associated with the recording time interval is received, responding to the backtracking request and executing the record of the message to be backtracked.

7. The system of claim 1, wherein the producer client is further configured to receive the configuration information, and determine a mapping relationship between at least two business topics and a message storage path according to the configuration information if the configuration information is valid.

8. A method of message processing comprising:

the method comprises the steps that a producer client collects information to be processed corresponding to a target business theme, a message template corresponding to the target business theme is determined according to effective configuration information, the information to be processed is added in a message header of the message template, a target message is generated according to an addition result, and the target message is sent to a Kafka cluster to be stored; the configuration information is used for recording mapping relations between at least two business topics and a message storage path, creating a message link corresponding to the mapping relations, and taking the message link as a message template corresponding to the target business topic;

9. A message processing system, comprising:

a message management end, a producer client and a consumer client;

the producer client is configured to collect a message to be processed corresponding to a target service topic under the condition that the configuration information is effective, determine a message template corresponding to the target service topic according to the effective configuration information, add the message to be processed in a message header of the message template, generate a target message according to an addition result, send the target message to a Kafka cluster for storage, and create a message link corresponding to the mapping relationship, and use the message link as the message template corresponding to the target service topic;

10. A method of message processing comprising:

under the condition that the configuration information is effective, a producer client acquires a message to be processed corresponding to a target service theme, determines a message template corresponding to the target service theme according to the effective configuration information, adds the message to be processed into a message header of the message template, generates a target message according to an addition result, and sends the target message to a Kafka cluster for storage, wherein a message link corresponding to the mapping relation is created, and the message link is used as the message template corresponding to the target service theme;

11. A computing device comprising a memory and a processor; the memory is for storing computer executable instructions and the processor is for executing the computer executable instructions to implement the steps of the method of claim 8 or 10.

12. A computer readable storage medium storing computer instructions which, when executed by a processor, implement the steps of the method of claim 8 or 10.