CN115242787A - Message processing system and method - Google Patents

Message processing system and method Download PDF

Info

Publication number
CN115242787A
CN115242787A CN202210872959.9A CN202210872959A CN115242787A CN 115242787 A CN115242787 A CN 115242787A CN 202210872959 A CN202210872959 A CN 202210872959A CN 115242787 A CN115242787 A CN 115242787A
Authority
CN
China
Prior art keywords
message
target
configuration information
partition
client
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210872959.9A
Other languages
Chinese (zh)
Other versions
CN115242787B (en
Inventor
杨宇
马进朝
董攀峰
徐进
张君华
童世红
周翔
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hundsun Technologies Inc
Original Assignee
Hundsun Technologies Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hundsun Technologies Inc filed Critical Hundsun Technologies Inc
Priority to CN202210872959.9A priority Critical patent/CN115242787B/en
Publication of CN115242787A publication Critical patent/CN115242787A/en
Application granted granted Critical
Publication of CN115242787B publication Critical patent/CN115242787B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/22—Parsing or analysis of headers
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Transfer Between Computers (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

This specification provides a message processing system and method, wherein the message processing system includes: the producer client is used for acquiring the message to be processed corresponding to the target service theme, processing the message to be processed according to the valid configuration information to obtain the target message, and sending the target message to the Kafka cluster for storage; the configuration information is used for recording the mapping relation between at least two service themes and a message storage path; and the consumer client is used for subscribing a target partition in the Kafka cluster according to the configuration information under the condition of receiving a message processing request associated with the target service theme, and pulling the target message in the target partition in response to the message processing request.

Description

Message processing system and method
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a message processing system and method.
Background
Kafka, a high-throughput distributed publish-subscribe messaging system, can be implemented to handle all the action flow data of a consumer in a website. Such action flow data is a key factor for many social functions on the network, and this portion of data often needs to be addressed by handling logs and log aggregations due to throughput requirements. The Kafka adds the throughput capacity of the Kafka by means of sequential reading and writing, zero copy, DMA direct memory access and the like of the disk, and achieves that the Kafka has better performance in the scene. However, in Kafka cluster, topic is used as a basis for dividing message types, and once increasing, partitions will also increase, and consequently sequential read-write becomes random read-write, and too much topic will also cause throughput to decrease, so an effective solution is needed to solve the above problems.
Disclosure of Invention
In view of this, this specification provides a message processing system. The present specification also relates to a message processing method, another message processing system, another message processing method, a computing device, and a computer-readable storage medium, which are used for solving the technical defects in the prior art.
According to a first aspect of embodiments herein, there is provided a message processing system comprising a producer client and a consumer client;
the producer client is used for acquiring the message to be processed corresponding to the target service theme, processing the message to be processed according to the valid configuration information to obtain the target message, and sending the target message to the Kafka cluster for storage; the configuration information is used for recording the mapping relation between at least two service themes and a message storage path;
and the consumer client is used for subscribing a target partition in the Kafka cluster according to the configuration information under the condition of receiving a message processing request associated with the target service theme, and pulling the target message in the target partition in response to the message processing request.
Optionally, the producer client determines a message template corresponding to the target service theme according to the configuration information, adds the to-be-processed message to a message header of the message template, and generates the target message according to an addition result.
Optionally, the producer client analyzes the target message, obtains a target message storage path in a message header to which the message to be processed is added, and sends the target message to the Kafka cluster according to the target message storage path; and the Kafka cluster stores the target message to the target partition according to the target message storage path.
Optionally, the consumer client receives the message processing request, parses the message processing request, determines the target service topic according to a parsing result, determines partition information corresponding to the target service topic in the configuration information, and subscribes to the target partition in the Kafka cluster according to the partition information.
Optionally, the consumer client sends a message consumption request to the Kafka cluster in response to the message processing request, receives a partition message fed back by the Kafka cluster for the message consumption request, and determines the target message in the partition message; wherein the partition message is a global message stored in the target partition.
Optionally, the system further comprises a message management end; the message management terminal determines the category information associated with the Kafka cluster, establishes a mapping relation between at least two service themes and a message storage path according to the category information, establishes configuration information according to the mapping relation, and sends the configuration information to the producer client and the consumer client.
Optionally, the message management terminal receives a message recording request, determines a message consumption link and a recording time interval according to the message recording request, records the message consumption link in the recording time interval, and generates and stores a message record to be backtracked according to a recording result; and under the condition that a backtracking request associated with the recording time interval is received, responding to the backtracking request to execute the message record to be backtracked.
Optionally, the producer client receives the configuration information, determines a mapping relationship between at least two service topics and a message storage path according to the configuration information under the condition that the configuration information is valid, creates a message link corresponding to the mapping relationship, and takes the message link as a message template corresponding to the target service topic.
According to a second aspect of embodiments of the present specification, there is provided a message processing method including:
a producer client acquires a message to be processed corresponding to a target service theme, processes the message to be processed according to the valid configuration information to obtain a target message, and sends the target message to a Kafka cluster for storage; the configuration information is used for recording the mapping relation between at least two service themes and a message storage path;
and under the condition that a consumer client receives a message processing request associated with the target service theme, subscribing a target partition in the Kafka cluster according to the configuration information, and pulling the target message in the target partition in response to the message processing request.
According to a third aspect of embodiments herein, there is provided another message processing system including:
the system comprises a message management terminal, a producer client and a consumer client;
the message management terminal is used for establishing a mapping relation between at least two service themes and a message storage path, establishing configuration information according to the mapping relation, and sending the configuration information to the producer client and the consumer client;
the producer client is used for acquiring a message to be processed corresponding to a target service theme under the condition that the configuration information is effective, processing the message to be processed according to the effective configuration information to obtain a target message, and sending the target message to the Kafka cluster for storage;
and the consumer client is used for subscribing a target partition in the Kafka cluster according to the configuration information under the condition of receiving a message processing request associated with the target service theme, and pulling the target message in the target partition in response to the message processing request.
According to a fourth aspect of the embodiments of the present specification, there is provided another message processing method including:
the message management end establishes a mapping relation between at least two service themes and a message storage path, creates configuration information according to the mapping relation, and sends the configuration information to the producer client and the consumer client;
under the condition that the configuration information is effective, the client of the producer collects the message to be processed corresponding to the target service theme, processes the message to be processed according to the effective configuration information to obtain the target message, and sends the target message to the Kafka cluster for storage;
and under the condition that the consumer client receives a message processing request associated with the target service theme, subscribing the target partition in the Kafka cluster according to the configuration information, and pulling the target message in the target partition in response to the message processing request.
According to a fifth aspect of embodiments herein, there is provided a computing device comprising:
a memory and a processor;
the memory is configured to store computer-executable instructions, and the processor is configured to implement the steps of the message processing method when executing the computer-executable instructions.
According to a sixth aspect of embodiments herein, there is provided a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the message processing method.
In the message processing system provided in this specification, after a producer client generates a to-be-processed message associated with a target service topic, in order to ensure that a Kafka cluster does not suffer from throughput degradation when processing the message and make full use of limited resources of the Kafka cluster, the to-be-processed message may be processed according to valid configuration information to obtain the target message, and then the target message is sent to the Kafka cluster, where the configuration information records a mapping relationship between at least two service topics and a message storage path. On the basis, when the consumer client side consumes the message, the target partition in the Kafka cluster can be determined according to the configuration information, and the purpose that the target message can be pulled from the target partition for consumption in response to the message processing request is achieved. The mapping relation between at least two service themes and the message storage path is established through the configuration information, the messages are stored and consumed on the basis, the topic number and the partition number of the Kafka cluster can be effectively controlled, and the throughput capacity of the Kafka cluster is effectively guaranteed.
Drawings
Fig. 1 is a schematic structural diagram of a message processing system according to an embodiment of the present specification;
FIG. 2 is a flow diagram of a message consumption process provided by an embodiment of the present description;
FIG. 3 is a schematic diagram of a message processing system provided in one embodiment of the present specification;
fig. 4 is a flowchart of a message processing method provided in an embodiment of the present specification;
FIG. 5 is a block diagram of another message processing system provided in an embodiment of the present disclosure;
fig. 6 is a flowchart of another message processing method provided in an embodiment of the present specification;
FIG. 7 is a process flow diagram of a message processing system provided by an embodiment of the present description;
fig. 8 is a block diagram of a computing device according to an embodiment of the present disclosure.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, as those skilled in the art will be able to make and use the present disclosure without departing from the spirit and scope of the present disclosure.
The terminology used in the description of the one or more embodiments is for the purpose of describing the particular embodiments only and is not intended to be limiting of the description of the one or more embodiments. As used in one or more embodiments of the present specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, etc. may be used herein in one or more embodiments to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first can be termed a second and, similarly, a second can be termed a first without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at" \8230; "or" when 8230; \8230; "or" in response to a determination ", depending on the context.
First, the noun terms to which one or more embodiments of the present specification relate are explained.
Kafka: is a high-throughput distributed publish-subscribe messaging system that can handle all the action flow data of a consumer in a web site. This action (web browsing, searching and other user actions) is a key factor in many social functions on modern networks. These data typically need to be addressed by handling logs and log aggregations due to throughput requirements.
topic: each message issued to the Kafka cluster has a category, which is called topic.
partition: partitions are indicated in Kafka, each topic in Kafka can be divided into a plurality of partitions (each topic has at least one partition), and messages contained in different partitions under the same topic are different.
DMA (Direct Memory Access): the DMA is an independent chip on the mainboard, and when data transmission of the memory and the IO equipment is carried out, the data transmission is not controlled by the CPU any more, but is directly realized by the DMA controller.
In this specification, a message processing system is provided, and the specification simultaneously relates to a message processing method, another message processing system, another message processing method, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
Referring to fig. 2, in the prior art, after a producer a or B issues a message, if the message belongs to the same category, the message is sent to the same topic, and then the message is written into a partition 1, a partition 2, or a partition 3 in the topic, respectively, and after the message writing is completed; when the consumer needs to consume the message, the consumer subscribes to the partition corresponding to the needed message, and then pulls the message from the partition for consumption. Although Kafka has high throughput capacity, once topic increases, the number of partitions also increases, and therefore sequential read-write becomes random read-write, when pagecache used by Kafka is too large, a large number of disk I/O triggers may exceed load, occupy CPU time of each application, and affect main flow, so that the main flow slows down, and therefore Kafka in actual application may be largely used in a log center, big data calculation, and the like, and is in a scene with a small demand for topic and a relatively simple logic. If the Kafka throughput degradation problem can be solved, the Kafka throughput degradation problem can be applied to a complex scene, and therefore the problem of excessive partition number needs to be solved, topic needs to be managed and controlled.
In view of this, in the message processing system provided in this specification, after the producer client generates the to-be-processed message related to the target service topic, in order to ensure that the Kafka cluster does not suffer from throughput degradation when processing the message and make full use of the limited resource of the Kafka cluster, the to-be-processed message may be processed according to the valid configuration information to obtain the target message, and then the target message is sent to the Kafka cluster, where the configuration information records a mapping relationship between at least two service topics and a message storage path. On the basis, when the consumer client side consumes the message, the target partition in the Kafka cluster can be determined according to the configuration information, and the purpose that the target message can be pulled from the target partition for consumption in response to the message processing request is achieved. The mapping relation between at least two service themes and the message storage path is established through the configuration information, the messages are stored and consumed on the basis, the topic number and the partition number of the Kafka cluster can be effectively controlled, and the throughput capacity of the Kafka cluster is effectively guaranteed.
Fig. 1 is a schematic structural diagram of a message processing system provided according to an embodiment of the present specification, where the message processing system 100 includes a producer client 110 and a consumer client 120, and a Kafka cluster 130;
the producer client 110 is configured to collect a to-be-processed message corresponding to a target service theme, process the to-be-processed message according to the validated configuration information, obtain a target message, and send the target message to the Kafka cluster 130 for storage; the configuration information is used for recording the mapping relation between at least two service themes and a message storage path;
and the consumer client 120 is configured to, in a case that a message processing request associated with the target service topic is received, subscribe a target partition in the Kafka cluster 130 according to the configuration information, and pull the target message in the target partition in response to the message processing request.
Because Kafka has the advantages of better throughput, built-in partitioning, redundancy and fault tolerance, it can be applied to different message processing scenarios; for example, in a behavior tracking scenario, kafka may be used to track user browsing pages, searches, and other behaviors, and may be recorded in real time in a corresponding topic through a publish-subscribe mode, and may be accessed, processed, and analyzed through a backend big data platform, so as to implement real-time monitoring. Or as a log collection scene, the log aggregation can be realized through Kafka, so that the log files are collected from the server and then processed through a centralized platform.
In this embodiment, the message processing system is described by taking an example of application of Kafka to a behavior tracking scenario, and for a message processing process in other scenarios, the same or corresponding description in this embodiment may be referred to, and this embodiment is not described in detail herein.
Specifically, the producer client specifically refers to a client corresponding to the message producer, that is, a client that sends a message to the Kafka cluster; correspondingly, the consumer client specifically refers to a client corresponding to the message consumer, that is, a client that fetches a message from the Kafka cluster.
Further, the target service theme specifically refers to a theme of a category to which the message to be processed belongs, and includes but is not limited to a market information theme, a rights and interests log theme, a combined message change theme, a user information change theme, and the like, that is, a theme carrying a service definition. And the target business topic can be used for determining which topic in the Kafka cluster the message to be processed needs to be written into after the message to be processed is processed. Correspondingly, the message to be processed specifically refers to a message issued by a producer client; correspondingly, the target message specifically refers to a message processed according to the configuration information, and the message written into the Kafka cluster is the target message. The configuration information is specifically information normalized for a service theme and a message storage path, and since a message issued by a producer client always belongs to one service theme, the storage path of the message can be normalized through the configuration information to clarify a partition into which the message needs to be written, so that after the message is issued, the message can be processed according to the configuration information to obtain a target message for recording routing information, and the target message is sent to a Kafka cluster for storage.
That is to say, the configuration message can process the message to be processed into the target message, and the message storage path of the target service topic association message is specified in the target message, so that the message of different service topics can be specifically pointed to topic through the configuration information, and the configuration message is used for storing and consuming the message associated with each service topic by using the effective topic.
Correspondingly, the message processing request specifically refers to a request that the consumer client needs to perform message consumption processing, and the request is sent by the Kafka cluster; correspondingly, the target partition specifically refers to a partition in topic associated with a target service topic in the Kafka cluster, and is used for storing a target message written into the Kafka cluster by a producer client.
Based on this, after the configuration information is validated by both the producer client and the consumer client, when the to-be-processed message associated with the target service theme is acquired, the to-be-processed message can be processed according to the validated configuration information, so that the relevant information with storage directivity is written in the to-be-processed message to obtain the target message. And then sending the target message to the Kafka cluster for storage. Since the mapping relationship between at least two service topics and the message storage path is recorded in the configuration information, after the Kafka cluster receives the target message, the writing of the target message into the target partition in the designated topic can be completed according to the related information with storage directivity carried in the target message.
Further, when the consumer client receives a message processing request associated with a target service theme, the target partition under topic associated with the target service theme can be determined according to the configuration information which has been validated, and then the consumer client can subscribe the target partition in the Kafka cluster, so that the purpose that the target message can be pulled in the target partition for consumption in response to the message processing request subsequently is achieved.
It should be noted that, because both the producer client and the consumer client take the configuration information in effect, a message storage path corresponding to any business topic can be determined through the configuration information, and meanwhile, in order to improve the effect of using the effective topic to complete the storage of messages associated with multiple different types of business topics, a mapping relationship between at least two business topics and one topic can be established, so that multiple types of business topics can be written into the same topic, and meanwhile, because the message to be processed is processed into the target message before writing, the difference between the different business topic messages can be distinguished through the related information carried in the target message, and the consumer client can more conveniently accurately pull the required target message in the same topic for consumption.
For example, recording that a service a corresponds to topicA _1 and a service B corresponds to topicA _2 in the configuration information; after a producer client issues a to-be-processed message of the associated service A, the to-be-processed message can be processed according to the configuration information, so that a message header topicA _1 of the to-be-processed message is filled according to the configuration information, and after the filling is completed, the obtained target message is sent to the Kafka cluster, and because the information of the associated configuration information is recorded in the message header of the target message, the target message can be determined to correspond to a first partition in a topicA theme according to the topicA _1, and therefore the target message can be sent to the first partition of the topicA theme and stored. Further, when the consumer client needs to consume the message, the message is determined to be stored in the first partition of the topicA topic according to the configuration information, the first partition of the topicA topic in the Kafka cluster can be subscribed, the message is pulled from the first partition, and the target message is filtered and used.
In summary, after the producer client generates the to-be-processed message associated with the target service topic, in order to ensure that the Kafka cluster does not suffer from throughput degradation when processing the message and fully utilize the limited resources of the Kafka cluster, the to-be-processed message may be processed according to the valid configuration information to obtain the target message, and then the target message is sent to the Kafka cluster, where the configuration information records the mapping relationship between at least two service topics and the message storage path. On the basis, when the consumer client side consumes the message, the target partition in the Kafka cluster can be determined according to the configuration information, and the purpose that the target message can be pulled from the target partition for consumption in response to the message processing request is achieved. The mapping relation between at least two service topics and the message storage path is established through the configuration information, the messages are stored and consumed on the basis, the topic number and the partition number of the Kafka cluster can be effectively controlled, and the throughput capacity of the Kafka cluster is effectively guaranteed.
Further, since the configuration information defines the mapping relationship between different service topics and topics, and after messages corresponding to different service topics are published, the storage of a Kafka cluster is completed according to the mapping relationship, so that the Kafka cluster can define which topic messages of different service topics should be written into, a message template of the mapping relationship between the service topics and the topic can be created according to the configuration information, and after a message associated with a target service topic is obtained, the creation of the target message can be completed through a corresponding message template.
Specifically, the message template specifically refers to a template that specifies mapping relationships between different service topics and different topics according to the valid configuration information, after the to-be-processed message associated with the target service topic is obtained, the message template corresponding to the target service topic may be determined, and then the to-be-processed message is added to the message template, so that the target message in which the message routing information is recorded may be obtained, which is convenient for the subsequent Kafka cluster to store the message and for the consumer client to pull the message.
Based on this, when the producer client needs to issue the message to be processed associated with the target service theme, the message template corresponding to the target service theme may be determined according to the configuration information, and then the message to be processed is added to the message header of the message template, that is, the target message may be obtained according to the addition result, so that the message may be stored and pulled according to the information recorded in the message header subsequently.
In this embodiment, the producer client is further configured to receive the configuration information, determine, according to the configuration information, a mapping relationship between at least two service topics and a message storage path, create a message link corresponding to the mapping relationship, and use the message link as the message template corresponding to the target service topic.
The message link is specifically a link created according to the mapping relationship, and is used as a message template to facilitate message insertion and record of routing information.
In summary, by determining the message template associated with the target service topic by combining the configuration information and determining the target message based on the message template, it can be ensured that the routing information of the associated message is recorded in the message header of the target message, so that the Kafka cluster can clearly store the location, and when the consumer client consumes the message, the required message can be filtered out according to the information in the message header.
Furthermore, after the producer client completes the determination of the target message, the producer client needs to send a message to the Kafka cluster, and in order to assist the Kafka cluster in defining the storage location of the target message, the target message may be parsed to obtain a target message storage path, and the message is sent and stored according to the path; and the Kafka cluster stores the target message to the target partition according to the target message storage path.
Specifically, the target message storage path specifically refers to a logical topic associated with the target message, and is used for specifying a storage partition of the target message. Based on this, after the information is filled in the message header to obtain the target message, the target message can be analyzed to obtain the target message storage path in the message header to which the message to be processed is added, and then the target message is sent to the Kafka cluster according to the target message storage path, at this time, the Kafka cluster can determine the target partition according to the target message storage path and sequentially write the target message in the target partition.
Along with the above example, after the producer client issues the to-be-processed message of the associated service a, the producer client obtains the target message by filling the message header topicA _ 1. When the message storage is needed, the first level of the message header can be analyzed to be used as a sending logic theme, namely, the analysis result is topicA _1, the message of the associated service A is determined to be written into the first partition of the topicA theme in the Kafka cluster, and at this time, the producer client can send the target message to the Kafka cluster according to the topicA _1 so as to write the target message into the first partition of the topicA theme.
In summary, by defining topic and the partition according to the target message storage path, the Kafka cluster can write the target message into the target partition quickly and accurately, and meanwhile, since the message specifies the partition, the sequential storage of the message can be ensured.
On this basis, when the consumer client performs message consumption processing, since the mapping relationship between the topic and the topic is recorded in the configuration information, in order to accurately pull the target message associated with the target service topic, it is necessary to determine the partition to which the message belongs.
Specifically, the partition information specifically refers to information recorded in the configuration information, in which a message related to a target service theme is written into the target partition, and is used to specify a partition to which the message needs to be pulled. Based on this, after the consumer client receives the message processing request, in order to accurately pull the message for use, the message processing request may be analyzed first, so as to determine the target service topic according to the analysis result, that is, to specify the category to which the message to be consumed belongs, then determine the partition information corresponding to the target service topic in the configuration information, and finally subscribe the target partition in the Kafka cluster, that is, the target partition in topic associated with the target service topic according to the partition information.
In summary, in order to ensure that the consumer client can accurately pull the message, the target service theme can be determined first through the message processing request, and then the partition information can be determined in cooperation with the configuration information, at this time, the partition associated with the target service theme can be subscribed according to the partition information, so that the target message can be pulled subsequently.
Further, in the process of pulling the target message, considering that not only the related message of the target service topic but also related messages of other service topics exist in the same partition, when pulling the target message, the target message needs to be determined from the partition messages, in this embodiment, the consumer client is further configured to send a message consumption request to the Kafka cluster in response to the message processing request, receive a partition message fed back by the Kafka cluster for the message consumption request, and determine the target message in the partition message; wherein the partition message is a global message stored in the target partition.
Specifically, the partition message specifically refers to all messages stored in the target partition; based on this, in the process of pulling the message by the consumer client, the message consumption request may be sent to the Kafka cluster in response to the message processing request, at this time, the Kafka cluster may specify that the consumer client needs to pull the message of the target service theme, so that the topic associated with the target service theme may be determined first, then the target partition storing the target service theme is determined in the topic, and then the partition message is fed back to the consumer client. And finally, the consumer client can filter out the target message for use according to the partition message fed back by the Kafka cluster.
Following the above example, when it is determined that the consumer client needs to use the message for service a, a message consumption request may be sent to Kafka cluster, determining from topicA _1 that the relevant message for service a is stored in the first partition of topicA. And because a large number of messages are stored in the first partition, the target message of the associated service a can be filtered according to sub1 in the message header. In the process, the consumer client determines the message pulled from the first partition of topicA, then judges whether the message header information is matched with the message header sub1 mapped by the service A, and discards the message header information if the message header information is not matched with the message header sub1 mapped by the service A until the target message matched with the sub1 is pulled for use.
In conclusion, the target message is determined in a message header matching mode, so that the consumer client can be ensured to accurately pull the target message for use.
In addition, since the configuration information is a basis for specifying the mapping relationship between the service topics and the topic, in order to effectively control the number of topic and the number of partitions in the Kafka cluster, the configuration information may be set by the message management end, so as to manage and control the increase of the number of topic and the number of partitions, and implement mapping of a plurality of service topics by using the effective number of topic and the number of partitions, in this embodiment, the message processing system further includes a message management end; the message management terminal is configured to determine category information associated with the Kafka cluster, establish a mapping relationship between at least two service topics and a message storage path according to the category information, create configuration information according to the mapping relationship, and send the configuration information to the producer client and the consumer client.
Specifically, the message management end is a mapping management end responsible for the business topic and the Kafka message topic, and when the business is coded, the annotation can be used to identify the business topic identifier, and simultaneously, the SDK (Software Development Kit) can be converted into a message header for generating configuration information by combining the business topic and the message header.
Based on this, in the configuration stage, the message management end may first determine the category information associated with the Kafka cluster, then establish a mapping relationship between at least two service topics and the message storage path according to the category information, then create configuration information according to the mapping relationship, and send the configuration information to the producer client and the consumer client.
That is, the message management side may use the annotation to identify the service subject during service encoding, and convert the SDK into a message header, and thereafter, during message storage, may remove the first level from the target message as the message subject of Kafka, and may customize the partition identifier, such as topicA _1, to be sent to the specified partition for implementing sequential sending and sequential consumption. The service theme and the message header can form complete message configuration and can be issued to the producer client and the consumer client through a configuration center or by multiplexing Kafka.
In specific implementation, in a service theme coding stage, a service message identifier of a message may be implanted, then a message management end may edit a service theme and message header mapping to implement a service a-topicA _1, the message management end issues configuration information to a producer client and a consumer client through a configuration center or by multiplexing Kafka, the producer client and the consumer client may perform validation after acquiring the configuration information, and the client may convert the configuration information into a real theme list topicA according to a service message list for subsequent message sending and consumption, and may complete the message sending and consumption according to the configuration.
Referring to fig. 3, after publishing a message, the producer client will write topicA in the message header: sub1 and topicA: sub2, and topicB: sub3 and topicB: sub4, the Kafka cluster can transmit the address of topicA: sub1 and topicA: sub2 corresponding message is written to topicA, and topicB: sub3 and topicB: and writing the message corresponding to sub4 into topicB, and when the consumer client consumes data, pulling the topicA from the topicA: sub1 and topicA: sub2, and pulling topicB in topicB: sub3 and topicB: sub4 for subsequent use.
In summary, the limited resources in the Kafka cluster can be managed and controlled by the message management terminal for creating and managing the configuration information, the limited resources can be used for sending and consuming messages, and the throughput of the Kafka cluster can be ensured.
In addition, in order to monitor the life cycle of the important message, the message management terminal may further provide recording and backtracking functions, in this embodiment, the message management terminal is further configured to receive a message recording request, determine a message consumption link and a recording time interval according to the message recording request, record the message consumption link in the recording time interval, generate a message record to be backtracked according to a recording result, and store the message record; and under the condition that a backtracking request associated with the recording time interval is received, responding to the backtracking request to execute the message record to be backtracked.
Specifically, the message recording request specifically refers to a request for recording the storage and consumption processes of the message; correspondingly, the message consumption link specifically refers to a link corresponding to a message consumption process from message generation to message storage, namely, a life cycle corresponding to the message to be recorded; correspondingly, the recording time interval specifically refers to the designated time of the whole life cycle of the recorded message. Correspondingly, the backtracking request specifically refers to a request for playing back the life cycle of the recorded time interval message.
Based on this, when the message management end receives the message recording request, it indicates that the storage and consumption process of a certain message needs to be recorded so as to facilitate backtracking at other times, so that a message consumption link and a recording time interval can be determined according to the message recording request, the message consumption link is recorded at the recording time interval, and a message record to be backtracked is generated and stored according to the recording result; under the condition of receiving a backtracking request of the associated recording time interval, the message to be backtracked can be recorded in response to the backtracking request, and the storage and consumption processes of the message can be played back. That is, the message recording of the specified message link can be started at the specified time, and the recorded content is retransmitted to the message server when needed, so that the service scene reproduction of the recording time interval is realized.
For example, in the time interval from t1 to t2, the message corresponding to the service B is written into the second partition of the topicB of the Kafka cluster, and when the consumer client consumes, the sub3 information of the associated service B is pulled from the second partition by subscribing to the second partition of the topicB of the Kafka cluster. When the message of the service B in the time interval from t1 to t2 needs to be backtracked, the above process can be executed again for assisting the processing of problem positioning and the like in the backtracking stage.
In summary, the message management terminal provides the recording and backtracking functions of the messages, and the message storage and consumption processes of the key time interval can be monitored, so that problems can be accurately positioned, and the normativity of an application scene is improved.
Corresponding to the above system embodiment, this specification further provides an embodiment of a message processing method, and fig. 4 shows a flowchart of a message processing method provided in an embodiment of this specification. As shown in fig. 4, the method includes:
step S402, a producer client collects a message to be processed corresponding to a target service theme, processes the message to be processed according to the valid configuration information to obtain a target message, and sends the target message to a Kafka cluster for storage; the configuration information is used for recording the mapping relation between at least two service themes and a message storage path;
step S404, under the condition that the consumer client receives the message processing request associated with the target service theme, the consumer client subscribes to the target partition in the Kafka cluster according to the configuration information, and the consumer client pulls the target message in the target partition in response to the message processing request.
In an optional embodiment, the producer client determines, according to the configuration information, a message template corresponding to the target service theme, adds the to-be-processed message to a message header of the message template, and generates the target message according to an addition result.
In an optional embodiment, the producer client parses the target message, obtains a target message storage path in a message header to which the message to be processed is added, and sends the target message to the Kafka cluster according to the target message storage path; and the Kafka cluster stores the target message to the target partition according to the target message storage path.
In an optional embodiment, the consumer client receives the message processing request, parses the message processing request, determines the target service topic according to a parsing result, determines partition information corresponding to the target service topic in the configuration information, and subscribes to the target partition in the Kafka cluster according to the partition information.
In an optional embodiment, the consumer client sends a message consumption request to the Kafka cluster in response to the message processing request, receives a partition message fed back by the Kafka cluster for the message consumption request, and determines the target message in the partition message; wherein the partition message is a global message stored in the target partition.
In an optional embodiment, the system further comprises a message management end; the message management terminal determines the category information associated with the Kafka cluster, establishes a mapping relation between at least two service themes and a message storage path according to the category information, establishes configuration information according to the mapping relation, and sends the configuration information to the producer client and the consumer client.
In an optional embodiment, the message management end receives a message recording request, determines a message consumption link and a recording time interval according to the message recording request, records the message consumption link in the recording time interval, generates a message record to be backtracked according to a recording result, and stores the message record; and under the condition that a backtracking request associated with the recording time interval is received, responding to the backtracking request to execute the message record to be backtracked.
In an optional embodiment, the producer client receives the configuration information, determines, according to the configuration information, a mapping relationship between at least two service topics and a message storage path under the condition that the configuration information is valid, creates a message link corresponding to the mapping relationship, and takes the message link as a message template corresponding to the target service topic.
In the message processing method provided by the present specification, after a producer client generates a to-be-processed message associated with a target service topic, in order to ensure that a Kafka cluster does not have a problem of throughput reduction when processing the message and simultaneously make full use of limited resources of the Kafka cluster, the to-be-processed message may be processed according to valid configuration information to obtain the target message, and then the target message is sent to the Kafka cluster, where the configuration information records a mapping relationship between at least two service topics and a message storage path. On the basis, when the consumer client side consumes the message, the target partition in the Kafka cluster can be determined according to the configuration information, and the purpose that the target message can be pulled from the target partition for consumption in response to the message processing request is achieved. The mapping relation between at least two service themes and the message storage path is established through the configuration information, the messages are stored and consumed on the basis, the topic number and the partition number of the Kafka cluster can be effectively controlled, and the throughput capacity of the Kafka cluster is effectively guaranteed.
The above is a schematic scheme of a message processing method according to this embodiment. It should be noted that the technical solution of the message processing method and the technical solution of the message processing system belong to the same concept, and details that are not described in detail in the technical solution of the message processing method can be referred to the description of the technical solution of the message processing system.
Corresponding to the foregoing embodiment, this embodiment further provides another message processing system, and fig. 5 shows a schematic structural diagram of another message processing system provided according to an embodiment of this specification, where the message processing system 500 includes a message management end 510, a producer client 520, and a consumer client 530, and a Kafka cluster 540;
the message management terminal 510 is configured to establish a mapping relationship between at least two service topics and a message storage path, create configuration information according to the mapping relationship, and send the configuration information to the producer client and the consumer client;
the producer client 520 is configured to, in a case that the configuration information is in effect, acquire a to-be-processed message corresponding to a target service topic, process the to-be-processed message according to the configuration information that has been in effect, obtain a target message, and send the target message to the Kafka cluster 540 for storage;
the consumer client 530 is configured to subscribe to a target partition in the Kafka cluster 540 according to the configuration information when receiving a message processing request associated with the target service topic, and pull the target message in the target partition in response to the message processing request.
In an optional embodiment, the producer client determines, according to the configuration information, a message template corresponding to the target service theme, adds the to-be-processed message to a message header of the message template, and generates the target message according to an addition result.
In an optional embodiment, the producer client parses the target message, obtains a target message storage path in a message header to which the message to be processed is added, and sends the target message to the Kafka cluster according to the target message storage path; and the Kafka cluster stores the target message to the target partition according to the target message storage path.
In an optional embodiment, the consumer client receives the message processing request, parses the message processing request, determines the target service topic according to a parsing result, determines partition information corresponding to the target service topic in the configuration information, and subscribes the target partition in the Kafka cluster according to the partition information.
In an optional embodiment, the consumer client sends a message consumption request to the Kafka cluster in response to the message processing request, receives a partition message fed back by the Kafka cluster for the message consumption request, and determines the target message in the partition message; wherein the partition message is a global message stored in the target partition.
In an optional embodiment, the system further comprises a message management end; the message management terminal determines the category information associated with the Kafka cluster, establishes a mapping relation between at least two service themes and a message storage path according to the category information, establishes configuration information according to the mapping relation, and sends the configuration information to the producer client and the consumer client.
In an optional embodiment, the message management end receives a message recording request, determines a message consumption link and a recording time interval according to the message recording request, records the message consumption link in the recording time interval, generates a message record to be backtracked according to a recording result, and stores the message record; and under the condition that a backtracking request associated with the recording time interval is received, responding to the backtracking request to execute the message record to be backtracked.
In an optional embodiment, the producer client receives the configuration information, determines a mapping relationship between at least two service topics and a message storage path according to the configuration information under the condition that the configuration information is valid, creates a message link corresponding to the mapping relationship, and uses the message link as a message template corresponding to the target service topic.
In the message processing system provided in this specification, after the message management end synchronously configures information to the producer client and the consumer client, a mapping relationship between at least two service topics and a message storage path may be established, so that, after the producer client generates a to-be-processed message associated with a target service topic, in order to ensure that a Kafka cluster does not suffer from throughput degradation when processing the message, at the same time, limited resources of the Kafka cluster may be fully utilized, the to-be-processed message may be processed according to the validated configuration information to obtain a target message, and then the target message is sent to the Kafka cluster. The mapping relation between at least two service topics and the message storage path is established through the configuration information, the messages are stored and consumed on the basis, the topic number and the partition number of the Kafka cluster can be effectively controlled, and the throughput capacity of the Kafka cluster is effectively guaranteed.
Corresponding to the above system embodiment, the present specification further provides another message processing method embodiment, and fig. 6 shows a flowchart of another message processing method provided in an embodiment of the present specification. As shown in fig. 6, the method includes:
step S602, the message management end establishes a mapping relation between at least two service themes and a message storage path, creates configuration information according to the mapping relation, and sends the configuration information to the producer client and the consumer client;
step S604, the client of the producer collects the message to be processed corresponding to the target service theme under the condition that the configuration information is effective, processes the message to be processed according to the effective configuration information to obtain the target message, and sends the target message to the Kafka cluster for storage;
step S606, the consumer client subscribes to the target partition in the Kafka cluster according to the configuration information under the condition that the consumer client receives the message processing request associated with the target service topic, and pulls the target message in the target partition in response to the message processing request.
In an optional embodiment, the producer client determines a message template corresponding to the target service theme according to the configuration information, adds the to-be-processed message to a message header of the message template, and generates the target message according to an addition result.
In an optional embodiment, the producer client parses the target message, obtains a target message storage path in a message header to which the message to be processed is added, and sends the target message to the Kafka cluster according to the target message storage path; and the Kafka cluster stores the target message to the target partition according to the target message storage path.
In an optional embodiment, the consumer client receives the message processing request, parses the message processing request, determines the target service topic according to a parsing result, determines partition information corresponding to the target service topic in the configuration information, and subscribes to the target partition in the Kafka cluster according to the partition information.
In an optional embodiment, the consumer client sends a message consumption request to the Kafka cluster in response to the message processing request, receives a partition message fed back by the Kafka cluster for the message consumption request, and determines the target message in the partition message; wherein the partition message is a global message stored in the target partition.
In an optional embodiment, the system further comprises a message management end; the message management terminal determines the category information associated with the Kafka cluster, establishes a mapping relation between at least two service themes and a message storage path according to the category information, establishes configuration information according to the mapping relation, and sends the configuration information to the producer client and the consumer client.
In an optional embodiment, the message management end receives a message recording request, determines a message consumption link and a recording time interval according to the message recording request, records the message consumption link in the recording time interval, generates a message record to be backtracked according to a recording result, and stores the message record; and under the condition that a backtracking request associated with the recording time interval is received, responding to the backtracking request to execute the message record to be backtracked.
In an optional embodiment, the producer client receives the configuration information, determines a mapping relationship between at least two service topics and a message storage path according to the configuration information under the condition that the configuration information is valid, creates a message link corresponding to the mapping relationship, and uses the message link as a message template corresponding to the target service topic.
In the message processing method provided by the present specification, after the message management end synchronously configures information to the producer client and the consumer client, a mapping relationship between at least two service topics and a message storage path may be established, so that, after the producer client generates a to-be-processed message associated with a target service topic, in order to ensure that a Kafka cluster does not suffer from throughput degradation when processing the message, at the same time, limited resources of the Kafka cluster may be fully utilized, the to-be-processed message may be processed according to the validated configuration information to obtain a target message, and then the target message is sent to the Kafka cluster. The mapping relation between at least two service topics and the message storage path is established through the configuration information, the messages are stored and consumed on the basis, the topic number and the partition number of the Kafka cluster can be effectively controlled, and the throughput capacity of the Kafka cluster is effectively guaranteed.
The following describes the message processing system further by taking an application of the message processing system provided in this specification in a log processing scenario as an example, with reference to fig. 7. Fig. 7 shows a processing flow chart of a message processing system provided in an embodiment of the present specification, which specifically includes the following steps:
step S702, the message management terminal determines the category information associated with the Kafka cluster, and establishes the mapping relationship between at least two service topics and the message storage path according to the category information.
Step S704, the message management end creates configuration information according to the mapping relation and sends the configuration information to the producer client and the consumer client.
Step S706, the producer client collects the message to be processed corresponding to the target business theme, and determines the message template corresponding to the target business theme according to the configuration information.
Step S708, the producer client adds the message to be processed in the message header of the message template, and generates the target message according to the adding result.
And step S710, the producer client analyzes the target message, obtains a target message storage path in the message header added with the message to be processed, and sends the target message to the Kafka cluster according to the target message storage path.
The Kafka cluster stores the target message to the target partition according to the target message storage path, and the configuration information is used for recording the mapping relation between at least two service topics and the message storage path.
And step S712, the consumer client receives the message processing request, analyzes the message processing request, and determines the target service theme according to the analysis result.
Step S714, the consumer client determines the partition information corresponding to the target service topic in the configuration information, and subscribes the target partition in the Kafka cluster according to the partition information.
Step S716, the consumer client sends a message consumption request to the Kafka cluster in response to the message processing request, and receives a partition message fed back by the Kafka cluster for the message consumption request.
In step S718, the consumer client determines a target message in the partition message.
Wherein the partition message is a global message stored in the target partition.
In summary, the mapping relationship between at least two service topics and the message storage path is established through the configuration information, and the messages are stored and consumed on the basis of the mapping relationship, so that the topic number and the partition number of the Kafka cluster can be effectively controlled, and the throughput capacity of the Kafka cluster is effectively ensured.
Fig. 8 illustrates a block diagram of a computing device 800 provided in accordance with an embodiment of the present description. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is coupled to the memory 810 via a bus 830, and the database 850 is used to store data.
Computing device 800 also includes access device 840, access device 840 enabling computing device 800 to communicate via one or more networks 860. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. Access device 840 may include one or more of any type of network interface (e.g., a Network Interface Card (NIC)) whether wired or wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.
In one embodiment of the present description, the above-described components of computing device 800, as well as other components not shown in FIG. 8, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device architecture shown in FIG. 8 is for purposes of example only and is not limiting as to the scope of the description. Those skilled in the art may add or replace other components as desired.
Computing device 800 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), a mobile phone (e.g., smartphone), a wearable computing device (e.g., smartwatch, smartglasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 800 may also be a mobile or stationary server.
The processor 820 is configured to implement the steps of the message processing method when executing the following computer-executable instructions.
The foregoing is a schematic diagram of a computing device of the present embodiment. It should be noted that the technical solution of the computing device and the technical solution of the message processing method belong to the same concept, and for details that are not described in detail in the technical solution of the computing device, reference may be made to the description of the technical solution of the message processing method.
An embodiment of the present specification further provides a computer readable storage medium, which stores computer instructions, and when the instructions are executed by a processor, the instructions implement the steps of the message processing method. The above is an illustrative scheme of a computer-readable storage medium of the present embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solution of the above-mentioned message processing method, and details that are not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the above-mentioned message processing method.
The foregoing description of specific embodiments has been presented for purposes of illustration and description. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous. The computer instructions comprise computer program code which may be in the form of source code, object code, an executable file or some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, read-Only Memory (ROM), random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
It should be noted that for simplicity and convenience of description, the above-described method embodiments are shown as a series of combinations of acts, but those skilled in the art will appreciate that the present description is not limited by the order of acts described, as some steps may occur in other orders or concurrently with other steps from the present description. Further, those skilled in the art should also appreciate that the embodiments described in this specification are preferred embodiments and that acts and modules referred to are not necessarily required for this description. In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
The preferred embodiments of the present specification disclosed above are intended only to aid in the description of the specification. Alternative embodiments are not exhaustive and do not limit the invention to the precise embodiments described. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the specification and its practical application, to thereby enable others skilled in the art to best understand the specification and utilize the specification. The specification is limited only by the claims and their full scope and equivalents.

Claims (13)

1. A message processing system comprising a producer client and a consumer client;
the producer client is used for acquiring the message to be processed corresponding to the target service theme, processing the message to be processed according to the valid configuration information to obtain the target message, and sending the target message to the Kafka cluster for storage; the configuration information is used for recording the mapping relation between at least two service themes and a message storage path;
and the consumer client is used for subscribing a target partition in the Kafka cluster according to the configuration information under the condition of receiving a message processing request associated with the target service theme, and pulling the target message in the target partition in response to the message processing request.
2. The system according to claim 1, wherein the producer client is further configured to determine a message template corresponding to the target service topic according to the configuration information, add the to-be-processed message to a message header of the message template, and generate the target message according to an addition result.
3. The system according to claim 2, wherein the producer client is further configured to parse the target message, obtain a target message storage path in a message header to which the message to be processed is added, and send the target message to the Kafka cluster according to the target message storage path; and the Kafka cluster stores the target message to the target partition according to the target message storage path.
4. The system of claim 1, wherein the consumer client is further configured to receive the message processing request, parse the message processing request, determine the target service topic according to a parsing result, determine partition information corresponding to the target service topic in the configuration information, and subscribe to the target partition in the Kafka cluster according to the partition information.
5. The system of claim 1 or 4, wherein the consumer client is further configured to send a message consumption request to the Kafka cluster in response to the message processing request, receive a partition message fed back by the Kafka cluster for the message consumption request, and determine the target message in the partition message; wherein the partition message is a global message stored in the target partition.
6. The system of claim 1, further comprising a message management side;
the message management terminal is configured to determine category information associated with the Kafka cluster, establish a mapping relationship between at least two service topics and a message storage path according to the category information, create configuration information according to the mapping relationship, and send the configuration information to the producer client and the consumer client.
7. The system of claim 6, wherein the message management terminal is further configured to receive a message recording request, determine a message consumption link and a recording time interval according to the message recording request, record the message consumption link in the recording time interval, generate a message record to be traced back according to a recording result, and store the message record; and under the condition that a backtracking request associated with the recording time interval is received, responding to the backtracking request to execute the message record to be backtracked.
8. The system according to claim 2, wherein the producer client is further configured to receive the configuration information, determine, according to the configuration information, a mapping relationship between at least two business topics and a message storage path if the configuration information is valid, create a message link corresponding to the mapping relationship, and use the message link as a message template corresponding to the target business topic.
9. A message processing method, comprising:
a producer client acquires a message to be processed corresponding to a target service theme, processes the message to be processed according to the valid configuration information to obtain a target message, and sends the target message to a Kafka cluster for storage; the configuration information is used for recording the mapping relation between at least two service themes and a message storage path;
and under the condition that the consumer client receives a message processing request associated with the target service theme, subscribing the target partition in the Kafka cluster according to the configuration information, and pulling the target message in the target partition in response to the message processing request.
10. A message processing system, comprising:
the system comprises a message management end, a producer client and a consumer client;
the message management terminal is used for establishing a mapping relation between at least two service themes and a message storage path, establishing configuration information according to the mapping relation, and sending the configuration information to the producer client and the consumer client;
the producer client is used for acquiring a message to be processed corresponding to a target service theme under the condition that the configuration information is effective, processing the message to be processed according to the effective configuration information to obtain a target message, and sending the target message to a Kafka cluster for storage;
and the consumer client is used for subscribing a target partition in the Kafka cluster according to the configuration information under the condition of receiving a message processing request associated with the target service theme, and pulling the target message in the target partition in response to the message processing request.
11. A message processing method, comprising:
the message management end establishes a mapping relation between at least two service themes and a message storage path, creates configuration information according to the mapping relation, and sends the configuration information to the producer client and the consumer client;
under the condition that the configuration information is effective, the client of the producer collects the message to be processed corresponding to the target service theme, processes the message to be processed according to the effective configuration information to obtain the target message, and sends the target message to the Kafka cluster for storage;
and under the condition that the consumer client receives a message processing request associated with the target service theme, subscribing the target partition in the Kafka cluster according to the configuration information, and pulling the target message in the target partition in response to the message processing request.
12. A computing device comprising a memory and a processor; the memory is for storing computer-executable instructions, and the processor is for executing the computer-executable instructions to implement the steps of the method of claim 9 or 10.
13. A computer-readable storage medium storing computer instructions, which when executed by a processor, perform the steps of the method of claim 9 or 10.
CN202210872959.9A 2022-07-22 2022-07-22 Message processing system and method Active CN115242787B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210872959.9A CN115242787B (en) 2022-07-22 2022-07-22 Message processing system and method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210872959.9A CN115242787B (en) 2022-07-22 2022-07-22 Message processing system and method

Publications (2)

Publication Number Publication Date
CN115242787A true CN115242787A (en) 2022-10-25
CN115242787B CN115242787B (en) 2023-09-05

Family

ID=83675637

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210872959.9A Active CN115242787B (en) 2022-07-22 2022-07-22 Message processing system and method

Country Status (1)

Country Link
CN (1) CN115242787B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116132395A (en) * 2022-11-15 2023-05-16 马上消费金融股份有限公司 Message processing method, electronic device and computer readable storage medium

Citations (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050021622A1 (en) * 2002-11-26 2005-01-27 William Cullen Dynamic subscription and message routing on a topic between publishing nodes and subscribing nodes
US20090287805A1 (en) * 2008-05-13 2009-11-19 International Business Machines Corporation System & method for non-http session based publish/subscribe support using pre-emptive subscriptions
CN104754036A (en) * 2015-03-06 2015-07-01 合一信息技术（北京）有限公司 Message processing system and processing method based on kafka
CN106649766A (en) * 2016-12-27 2017-05-10 北京锐安科技有限公司 Message handling method based on kafka
US20170310628A1 (en) * 2016-04-21 2017-10-26 Confluent, Inc. Distributed message queue stream verification
US20180063055A1 (en) * 2016-08-26 2018-03-01 Linkedin Corporation Repartitioning a topic in a publish-subscribe message system
CN109451072A (en) * 2018-12-29 2019-03-08 广东电网有限责任公司 A kind of message caching system and method based on Kafka
CN110377486A (en) * 2019-07-09 2019-10-25 福建南威软件有限公司 The asynchronous task processing method of stable high-throughput is realized based on kafka
CN111163118A (en) * 2018-11-07 2020-05-15 株式会社日立制作所 Message transmission method and device in Kafka cluster
CN111555957A (en) * 2020-03-26 2020-08-18 孩子王儿童用品股份有限公司 Kafka-based synchronous message service system and implementation method
CN111930538A (en) * 2020-07-31 2020-11-13 银盛支付服务股份有限公司 Production and consumption method based on kafka cluster
CN112422544A (en) * 2020-11-09 2021-02-26 深圳市华昊软件有限公司 Machine room equipment information monitoring system based on MQTT communication protocol
CN112751891A (en) * 2019-10-30 2021-05-04 中移(苏州)软件技术有限公司 Kafka-based message processing method, electronic device and storage medium
WO2021082073A1 (en) * 2019-10-30 2021-05-06 平安科技（深圳）有限公司 Method and system for dynamic selection of storage partitions, computer device, and storage medium
CN112764679A (en) * 2021-01-19 2021-05-07 福建天泉教育科技有限公司 Dynamic capacity expansion method and terminal
US20210218646A1 (en) * 2020-01-10 2021-07-15 Oracle International Corporation Methods, systems, and computer readable media for request response processing
CN113656503A (en) * 2021-08-20 2021-11-16 北京健康之家科技有限公司 Data synchronization method, device and system and computer readable storage medium
US20210385176A1 (en) * 2020-06-09 2021-12-09 Jpmorgan Chase Bank, N.A. Method and system for resolving producer and consumer affinities in interaction servicing
CN113946449A (en) * 2021-10-29 2022-01-18 北京工业大学 Kafka-based consumer side load balancing method
CN114363407A (en) * 2021-12-24 2022-04-15 上海软素科技有限公司 Message service method and device, readable storage medium and electronic equipment
CN114598662A (en) * 2022-03-15 2022-06-07 云粒智慧科技有限公司 Message queue cluster federal management system and method

Patent Citations (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050021622A1 (en) * 2002-11-26 2005-01-27 William Cullen Dynamic subscription and message routing on a topic between publishing nodes and subscribing nodes
US20090287805A1 (en) * 2008-05-13 2009-11-19 International Business Machines Corporation System & method for non-http session based publish/subscribe support using pre-emptive subscriptions
CN104754036A (en) * 2015-03-06 2015-07-01 合一信息技术（北京）有限公司 Message processing system and processing method based on kafka
US20170310628A1 (en) * 2016-04-21 2017-10-26 Confluent, Inc. Distributed message queue stream verification
US20180063055A1 (en) * 2016-08-26 2018-03-01 Linkedin Corporation Repartitioning a topic in a publish-subscribe message system
CN106649766A (en) * 2016-12-27 2017-05-10 北京锐安科技有限公司 Message handling method based on kafka
CN111163118A (en) * 2018-11-07 2020-05-15 株式会社日立制作所 Message transmission method and device in Kafka cluster
CN109451072A (en) * 2018-12-29 2019-03-08 广东电网有限责任公司 A kind of message caching system and method based on Kafka
CN110377486A (en) * 2019-07-09 2019-10-25 福建南威软件有限公司 The asynchronous task processing method of stable high-throughput is realized based on kafka
CN112751891A (en) * 2019-10-30 2021-05-04 中移(苏州)软件技术有限公司 Kafka-based message processing method, electronic device and storage medium
WO2021082073A1 (en) * 2019-10-30 2021-05-06 平安科技（深圳）有限公司 Method and system for dynamic selection of storage partitions, computer device, and storage medium
US20210218646A1 (en) * 2020-01-10 2021-07-15 Oracle International Corporation Methods, systems, and computer readable media for request response processing
CN111555957A (en) * 2020-03-26 2020-08-18 孩子王儿童用品股份有限公司 Kafka-based synchronous message service system and implementation method
US20210385176A1 (en) * 2020-06-09 2021-12-09 Jpmorgan Chase Bank, N.A. Method and system for resolving producer and consumer affinities in interaction servicing
CN111930538A (en) * 2020-07-31 2020-11-13 银盛支付服务股份有限公司 Production and consumption method based on kafka cluster
CN112422544A (en) * 2020-11-09 2021-02-26 深圳市华昊软件有限公司 Machine room equipment information monitoring system based on MQTT communication protocol
CN112764679A (en) * 2021-01-19 2021-05-07 福建天泉教育科技有限公司 Dynamic capacity expansion method and terminal
CN113656503A (en) * 2021-08-20 2021-11-16 北京健康之家科技有限公司 Data synchronization method, device and system and computer readable storage medium
CN113946449A (en) * 2021-10-29 2022-01-18 北京工业大学 Kafka-based consumer side load balancing method
CN114363407A (en) * 2021-12-24 2022-04-15 上海软素科技有限公司 Message service method and device, readable storage medium and electronic equipment
CN114598662A (en) * 2022-03-15 2022-06-07 云粒智慧科技有限公司 Message queue cluster federal management system and method

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
王仲生;: "基于kafka消息队列的文本处理技术研究", 软件导刊(教育技术), no. 12 *
马浩然;: "基于NS3的分布式消息系统Kafka的仿真实现", 软件, no. 01 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116132395A (en) * 2022-11-15 2023-05-16 马上消费金融股份有限公司 Message processing method, electronic device and computer readable storage medium

Also Published As

Publication number Publication date
CN115242787B (en) 2023-09-05

Similar Documents

Publication Publication Date Title
CN111131379B (en) Distributed flow acquisition system and edge calculation method
WO2020207371A1 (en) Data processing system and method, apparatus, and electronic device
CN113296718B (en) Data processing method and device
KR20210005043A (en) Input and output schema mapping
CN113010565B (en) Server real-time data processing method and system based on server cluster
CN111935227A (en) Method for uploading file through browser, browser and electronic equipment
CN111221793B (en) Data mining method, platform, computer equipment and storage medium
CN108427619B (en) Log management method and device, computing equipment and storage medium
CN107347062A (en) A kind of method, electronic equipment and the readable storage medium storing program for executing of daily record data processing
CN110134737A (en) Data variation monitor method and device, electronic equipment and computer readable storage medium
CN105160018A (en) Method, device and system for image copy/paste
CN111475584A (en) Data processing method, system and device
CN115242787B (en) Message processing system and method
CN112925559A (en) Resource packaging method, device, medium and electronic equipment
CN113568813A (en) Mass network performance data acquisition method, device and system
CN114579532A (en) Method, device and system for processing pre-written log
CN112860412B (en) Service data processing method and device, electronic equipment and storage medium
CN117271584A (en) Data processing method and device, computer readable storage medium and electronic equipment
CN115840765A (en) Data processing method and device based on rule engine
CN113392081B (en) Data processing system and method
CN115454666A (en) Data synchronization method and device among message queue clusters
CN111459819B (en) Software testing method and device, electronic equipment and computer readable medium
CN114461425A (en) Data processing method, device, equipment and computer readable storage medium
CN112953993B (en) Resource scheduling method, equipment, network system and storage medium
CN112597119A (en) Method and device for generating processing log and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant