WO2024104289A1 - 一种消息处理方法、装置、设备及存储介质 - Google Patents

一种消息处理方法、装置、设备及存储介质 Download PDF

Info

Publication number
WO2024104289A1
WO2024104289A1 PCT/CN2023/131278 CN2023131278W WO2024104289A1 WO 2024104289 A1 WO2024104289 A1 WO 2024104289A1 CN 2023131278 W CN2023131278 W CN 2023131278W WO 2024104289 A1 WO2024104289 A1 WO 2024104289A1
Authority
WO
WIPO (PCT)
Prior art keywords
message
processed
data pipeline
processing
original
Prior art date
Application number
PCT/CN2023/131278
Other languages
English (en)
French (fr)
Inventor
王嘉凌
Original Assignee
中移(苏州)软件技术有限公司
中国移动通信集团有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中移(苏州)软件技术有限公司, 中国移动通信集团有限公司 filed Critical 中移(苏州)软件技术有限公司
Publication of WO2024104289A1 publication Critical patent/WO2024104289A1/zh

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/04—Protocols for data compression, e.g. ROHC
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/18—Multiprotocol handlers, e.g. single devices capable of handling multiple protocols

Definitions

  • the present application relates to the field of big data, and in particular to a message processing method, apparatus, device and storage medium.
  • the consumer receives the processed message and consumes it.
  • specific schemes can usually be used to process messages in systems using message queues. For example, after the consumer reads the original message from the service node, the original message can be processed by the client where the consumer is located; however, with this scheme, the original message received by the client may contain content that the consumer does not need, which will cause a waste of network bandwidth, and may contain sensitive information that is not desired to be sent to the client, which poses a security risk.
  • a new topic (which can be expressed as Topic in English) can be created to process the content and store the processed message; however, this scheme requires additional storage space, and since the processing node corresponding to the new topic needs to consume the original message from the service node and then feed it back to the service node after processing, it will also cause bandwidth waste and reduce the throughput of the message system.
  • the embodiments of the present application provide a message processing method, apparatus, device and storage medium.
  • an embodiment of the present application provides a message processing method, the method comprising:
  • the processed message is written into the processed data pipeline, and the processed message is stored.
  • the method also includes: when it is determined that the message read by the first consumer is the original message of the target topic, instructing the first consumer to subscribe to the main data channel; when it is determined that the message read by the second consumer is the processed message of the target topic, instructing the second consumer to subscribe to the processed data pipeline.
  • the method also includes: storing the correspondence between the identification information of the original message and the identification information of the processed message; when it is determined that the first processed message is lost, determining the identification information of the first original message corresponding to the first processed message based on the correspondence and the identification information of the first processed message; based on the identification information of the first original message, reading the first original message from the main data pipeline; processing the read first original message to obtain the first processed message; writing the first processed message into the processing data pipeline, and storing the first processed message.
  • the storing of the original message includes: storing the original message in the form of multiple copies.
  • the storing of the processed message includes: storing the processed message in a single copy.
  • the method further includes: after the original message is stored, sending a reception success message to the producer.
  • an embodiment of the present application further provides a message processing device, the device comprising:
  • a configuration module configured to configure a main data pipeline and a processing data pipeline for a target topic at a service node; wherein the target topic is a topic that needs to be processed;
  • a processing module configured to write the original message sent by the producer corresponding to the target topic into the main data pipeline, and store the original message;
  • the processing module is further configured to asynchronously trigger processing of the original message to obtain a processed message; write the processed message into the processing data pipeline, and store the processed message.
  • an embodiment of the present application further provides an electronic device, comprising: a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the aforementioned method when running the computer program.
  • an embodiment of the present application further provides a computer storage medium on which a computer program is stored, wherein the computer program implements the steps of any of the above methods when executed by a processor.
  • an embodiment of the present application further provides a message system, the message system comprising: a producer, a service node and a consumer; the service node comprises: a topic manager, a data processing manager and a storage;
  • the producer is configured to send an original message to the service node
  • the topic manager is configured to configure a main data pipeline and a processing data pipeline for the target topic when determining that the target topic is a topic that needs content processing; and write the original message sent by the producer into the main data pipeline;
  • the data processing manager is configured to asynchronously trigger the data processor to process the original message to obtain a processed message; and write the processed message into the processed data pipeline;
  • the memory is configured to store messages in the main data pipeline and the processed data pipeline;
  • the consumer is configured to read the original message from the main data pipeline, or read the processed message from the processed data pipeline.
  • the message processing method, device, equipment and storage medium disclosed in the embodiments of the present application process and store the original message in an asynchronous manner by configuring a main data pipeline and a processing data pipeline for the subject, so that the consumers of the subject can directly read the original message and the processed message from the main data pipeline and the processing data pipeline of the service node.
  • the technical solution provided in the embodiments of the present application at least solves the problem of additional network resource consumption and data security caused by the need to transmit the original message in full to the client; compared with the solution of creating a new subject to process and store the processed message, it at least solves the problem of network loss caused by the need to transmit the original message and the processed message between different service nodes and processing nodes, thereby improving the throughput of the message system.
  • FIG1 is a schematic diagram of a flow chart of a message processing method according to an embodiment of the present application.
  • FIG2 is a schematic diagram of the structure of a message processing device according to an embodiment of the present application.
  • FIG3 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application.
  • FIG4 is a schematic diagram of the structure of a message system according to an embodiment of the present application.
  • FIG5 is a schematic diagram of the process of producing, processing and consuming messages according to an embodiment of the present application.
  • FIG6 is a schematic diagram of a process for restoring and processing messages according to an embodiment of the present application.
  • the present application embodiment proposes a message processing method, as shown in FIG1 , the message processing method includes:
  • Step 101 configuring a main data pipeline and a processing data pipeline for a target topic at a service node; wherein the target topic is a topic that needs to be processed;
  • Step 102 writing the original message sent by the producer corresponding to the target topic into the main data pipeline, and storing the original message;
  • Step 103 asynchronously triggering processing of the original message to obtain a processed message
  • Step 104 Write the processed message into the processed data pipeline and store the processed message.
  • the service node is a server node corresponding to the target topic, which can receive message data transmitted by the producer and provide message data to the consumer.
  • the service node can be a broker node in a Kafka cluster.
  • each topic can be configured with multiple data pipelines in the service node.
  • the data pipeline can be understood as an ordered queue used to store message content.
  • the main data pipeline is used to store the original message sent by the producer (also called original data), and the processed data pipeline is used to store the processed message (also called processed data) after the original message is processed.
  • different processing methods can be used to process the original message to obtain different processed messages.
  • a main data pipeline can correspond to one or more processed data pipelines, which are used to store processed messages obtained by different processing methods.
  • the topic when a topic contains only one main data pipeline, the topic is writable (storing the original data sent by the producer) and readable (the consumer can specify the data pipeline and consume data from it); when a subject contains one or more processed data pipelines, the topic is only used to store the processed message content, that is, the topic is readable only.
  • the producer is a module for producing messages.
  • the producer may be an application that publishes messages; the consumer is a module for consuming messages.
  • the consumer may be an application that subscribes to messages.
  • the original message is the message directly sent by the producer corresponding to the target topic.
  • the consumer of the topic can directly read the original message from the main data pipeline.
  • storing the original message may include: storing the original message in the form of multiple copies.
  • the probability of loss of the original message can be reduced.
  • the method may further include: after the original message is stored, sending a reception success message to the producer.
  • the service node corresponding to the target topic (such as the Broker node) only needs to store the original data when receiving the message. It can give the producer a notification of successful reception without processing, allowing the producer to send the next batch of messages, thereby improving the message throughput when producing messages.
  • the processing of the original message may include some specific conversion of the encoding format of the message content so that the message can be received and consumed by consumer clients of different protocols.
  • a message body encoded in Kafka protocol is sent through the open source stream processing platform Kafka client, and after the message encoding is converted to the cloud native distributed message stream platform Pulsar protocol, it is received and consumed by the Pulsar client.
  • the processing of the original message may include some specific conversion of the message content. The content is processed, such as desensitizing some specific fields in the original message and deleting some sensitive information (such as user name, password, etc.).
  • the processing of the original message may include processing the original message by a data processor.
  • the processing of the original message may be triggered asynchronously, which may specifically include: controlling the data processor to read the original message from the main data pipeline in an asynchronous manner; processing the read original message to obtain a processed message.
  • the processed messages obtained by different data processing methods are stored in different processed data pipelines.
  • the original message is processed in a first processing method to obtain a first processed message; the first processed message is written into a first processed data pipeline for storage; the original message is processed in a second processing method to obtain a second processed message; the second processed message is written into a second processed data pipeline for storage.
  • Each data processor can include a processing data pipeline name for storing processed data and a data processing function.
  • the data processing function is used to process messages.
  • one data processing function can correspond to one processing data pipeline.
  • the processing may include processing of the encoding format.
  • the message system can support native client programs of different protocols.
  • step 104 by writing the processed message into the processed data pipeline, consumers of the topic can directly read the processed message from the processed data pipeline.
  • consumers of the topic can directly read the processed message from the processed data pipeline.
  • the storage for storing processed data is increased, consumers can choose to consume the original data or the processed data, and each piece of data will only be processed once. When multiple consumers consume the same processed data at the same time, the original data will not be processed multiple times.
  • storing the processed message may include: storing the processed message in a single copy.
  • the storage space required for storing the processed message can be reduced.
  • the method may further include: when it is determined that the message read by the first consumer is the original message of the target topic, instructing the first consumer to subscribe to the main data channel; when it is determined that the message read by the second consumer is the processed message of the target topic, instructing the second consumer to subscribe to the processed data pipeline.
  • instructing the first consumer to subscribe to the main data channel can enable the first consumer to read the original message from the main data channel; instructing the second consumer to subscribe to the processed data pipeline can enable the second consumer to read the processed message from the processed data pipeline.
  • the method may further include: storing the correspondence between the identification information of the original message and the identification information of the processed message; when it is determined that the first processed message is lost, determining the identification information of the first original message corresponding to the first processed message based on the correspondence and the identification information of the first processed message; based on the identification information of the first original message, reading the first original message from the main data pipeline; processing the read first original message to obtain the first processed message; writing the first processed message into the processing data pipeline, and storing the first processed message.
  • the identification information of the original message and the processed message may include a message ID, and a persistently stored index may be set to store the correspondence between the processed message ID and the original message ID.
  • a persistently stored index may be set to store the correspondence between the processed message ID and the original message ID.
  • the execution entity of steps 101 to 104 may be a processor of an electronic device that implements the message processing method in the embodiment of the present application.
  • the technical solution provided by the embodiment of the present application processes and stores the original message in an asynchronous manner by configuring the main data pipeline and the processing data pipeline for the target topic in the service node, so that the consumers of the topic can directly read the original message and the processed message from the main data pipeline and the processing data pipeline of the service node.
  • the technical solution provided by the embodiment of the present application at least solves the problem of additional network resource consumption and data security caused by the need to transmit the original message in full to the client; by processing the encoding format of the message on the service node, the message system can support native client programs of different protocols; compared with the solution of creating a new topic to process and store processed messages by stream processing, at least the network loss problem caused by the need to transmit the original message and the processed message between different service nodes and processing nodes is solved, and the throughput of the message system is improved.
  • the embodiment of the present application also provides an implementation device of a message processing method. As shown in FIG2 , the message processing device 200 specifically includes:
  • Configuration module 201 configured to configure a main data pipeline and a processing data pipeline for a target topic at a service node; wherein the target topic is a topic that needs to be processed;
  • the processing module 202 is configured to write the original message sent by the producer corresponding to the target topic into the main data pipeline and store the original message;
  • the processing module 202 is further configured to asynchronously trigger processing of the original message to obtain a processed message; write the processed message into the processing data pipeline, and store the processed message.
  • the technical solution provided by the embodiment of the present application processes and stores the original message in an asynchronous manner by configuring the main data pipeline and the processing data pipeline for the target topic in the service node, so that the consumer of the topic can directly read the original message and the processed message from the main data pipeline and the processing data pipeline of the service node.
  • the technical solution provided by the embodiment of the present application at least solves the problem of needing to transmit the original message in full to the client.
  • the message system can support native client programs of different protocols; compared with the solution of creating a new topic to process and store processed messages through stream processing, it at least solves the network loss problem caused by the need to transmit the original message and the processed message between different service nodes and processing nodes, and improves the throughput of the message system.
  • the processing module 202 is also configured to instruct the first consumer to subscribe to the main data channel when it is determined that the message read by the first consumer is the original message of the target topic; and to instruct the second consumer to subscribe to the processed data pipeline when it is determined that the message read by the second consumer is the processed message of the target topic.
  • the processing module 202 is further configured to store the correspondence between the identification information of the original message and the identification information of the processed message; when it is determined that the first processed message is lost, the identification information of the first original message corresponding to the first processed message is determined based on the correspondence and the identification information of the first processed message; based on the identification information of the first original message, the first original message is read from the main data pipeline; the read first original message is processed to obtain the first processed message; the first processed message is written into the processed data pipeline, and the first processed message is stored.
  • the processing module 202 is further configured to store the original message in multiple copies.
  • the processing module 202 is further configured to store the processed message in a single copy.
  • the processing module 202 is further configured to send a reception success message to the producer after the original message is stored.
  • the configuration module 201 and the processing module 202 may be implemented by a processor in the message processing device 200 .
  • an embodiment of the present application further provides an electronic device, as shown in FIG3 , the electronic device 300 includes: a processor 301 and a memory 302 configured to store a computer program that can be run on the processor;
  • the processor 301 is configured to execute the steps of the method in the aforementioned embodiment when running a computer program.
  • bus system 303 is used to realize the connection and communication between these components.
  • bus system 303 also includes a power bus, a control bus, and a status signal bus.
  • various buses are labeled as bus system 303 in FIG3 .
  • the processor may be an application specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), At least one of a controller, a microcontroller, and a microprocessor.
  • ASIC application specific integrated circuit
  • DSPD digital signal processing device
  • PLD programmable logic device
  • FPGA field programmable gate array
  • the above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD, Hard Disk Drive) or a solid-state drive (SSD, Solid-State Drive); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.
  • volatile memory such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD, Hard Disk Drive) or a solid-state drive (SSD, Solid-State Drive); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.
  • RAM random access memory
  • non-volatile memory such as a read-only memory (ROM), a flash memory, a
  • the embodiment of the present application further provides a computer-readable storage medium, such as a memory including a computer program, and the computer program can be executed by a processor of an electronic device to complete the steps of the aforementioned message processing method.
  • a computer-readable storage medium such as a memory including a computer program
  • the embodiment of the present application also provides a message system, which can be a distributed message system.
  • the distributed message system is an extensible multi-node message queue system.
  • the message system 40 may include: a producer, a service node and a consumer; the service node may include: a topic manager 401, a data processing manager 402 and a memory 403;
  • the service node may be a Broker node.
  • the producer is configured to send an original message to the service node
  • the topic manager 401 is configured to configure a main data pipeline and a processing data pipeline for the target topic when it is determined that the target topic is a topic that needs content processing; write the original message sent by the producer into the main data pipeline, so that the consumer corresponding to the target topic reads the original message from the main data pipeline.
  • multiple data pipelines can be added to each topic (which can be expressed as Topic in English).
  • only one main data pipeline is writable (storing the original data sent by the producer) and readable (the consumer can specify this data pipeline and consume data from it), and other processing data pipelines are only used to store processed message content, that is, other processing data pipelines are only readable.
  • the processed messages obtained by different data processors are stored in different processed data pipelines.
  • the first data processor processes the original message in a first processing manner to obtain a first processed message; writes the first processed message into the first processed data pipeline for storage;
  • the second data processor processes the original message in a second processing manner to obtain a second processed message; writes the second processed message into the second processed data pipeline for storage.
  • the data processing manager 402 is configured to asynchronously trigger the data processor to process the original message to obtain a processed message; write the processed message into the processing data pipeline so that the consumer corresponding to the target topic reads the processed message from the processing data pipeline.
  • the data processing manager 402 provides a message processing management interface, through which the manager can dynamically add and modify the message content processing method.
  • the processor 402 is responsible for receiving and processing management commands of the data processor.
  • a command to add a data processor may include: the name of the subject corresponding to the data processor, the name of the processed data pipeline for storing the processed results, a script file (such as a Python script) or a bytecode file (such as a Java class) containing a data processing function, or other files that can realize dynamic loading of functions.
  • the memory 403 is configured to store messages in the main data pipeline and the processed data pipeline.
  • the data of the main data pipeline is stored as multiple copies, that is, the original message is stored in multiple copies.
  • the probability of loss of the original message can be reduced.
  • the data of the processed data pipeline is stored as a single copy. That is, the processed message is stored in a single copy.
  • the storage space required for storing the processed message can be reduced.
  • the service node is further configured to send a reception success message to the producer after the storage of the original message is completed.
  • the service node corresponding to the target topic only needs to store the original data when receiving the message. It can give the producer a notification of successful reception without processing, allowing the producer to send the next batch of messages, which can improve the message throughput when producing messages.
  • the message system may include: producers, service nodes, consumers, and a storage layer; the service nodes include: a topic manager, a data processing manager, and a storage manager, and the storage manager is configured to store the data in the main data pipeline and the processed data pipeline to the storage layer or read data from the storage layer.
  • the consumer is configured to read the original message from the main data pipeline, or read the processed message from the processed data pipeline.
  • the message system supports processing and storing the message content in an asynchronous manner after the service node receives the message sent by the producer, so that the consumer can receive both the original message content and the processed message content. Since the processing of the message content is performed on the service node Broker that receives the message, compared with the solution of message processing by the client where the consumer is located, at least the problem of extra network resource consumption and data security caused by the need to transmit the original message in full to the client is solved; by processing the encoding format of the message on the service node Broker, the message system can support native client programs of different protocols; compared with the solution of processing and storing processed messages by creating a new topic through stream processing, it is not necessary to send the message to other processing nodes for processing through network transmission, and the throughput of the entire message system is not affected; since the processing is performed asynchronously, the service node Broker corresponding to the target topic only needs to store the original data when receiving the message, and can give the producer feedback notification of successful reception without processing, so that the
  • the process of using the message system 40 to process messages may include: pre-configuring Placement process; production, processing and consumption of message processes.
  • the pre-configuration process may include: configuring a main data pipeline, a processing data pipeline, and a data processor A for Topic A. Creating a consumer A to subscribe to the main data pipeline of Topic A and consume the original message. Creating a consumer B to subscribe to the processing data pipeline of Topic A and consume the processed message.
  • Figure 5 is a schematic diagram of a specific process of producing, processing and consuming messages in an embodiment of the present application. As shown in Figure 5, the process may specifically include:
  • the service node Broker After the service node Broker receives the original message sent by the producer to the corresponding TopicA, it writes the original message into the main data pipeline and triggers the storage to store the original message in multiple copies to the disk. After the storage is completed, a message of successful sending is returned to the producer.
  • the data processor A After the data is written into the main data pipeline, the data processor A is asynchronously triggered to read the original message from the main data pipeline, process it, and write the result of the data processing (i.e., the processed message) into the processing data pipeline, and triggers the storage to store the data in a single copy to the disk, and stores the corresponding relationship between the processed message ID and the original message ID in the data index.
  • Consumer A reads the original message from the main data pipeline of TopicA; Consumer B reads the processed message from the processing data pipeline of TopicA.
  • consumer A reads the original message from the main data pipeline of TopicA, which may specifically include: the main data pipeline receives the request for consuming the original message sent by consumer A; triggers the storage to read the original message from the disk; after reading the original message, returns the original message to consumer A.
  • consumer B consumes the processed message from the processing data pipeline of TopicA, which may specifically include: the processing data pipeline receives the request for consuming the processed message sent by consumer B; triggers the storage to read the processed message from the disk; after reading the processed message, returns the processed message to consumer B.
  • the processing data pipeline receives the request for consuming the processed message sent by consumer B; triggers the storage to read the processed message from the disk; after reading the processed message, returns the processed message to consumer B.
  • the data in the processed data pipeline is processed by the data processor after reading the original message from the main data pipeline in an asynchronous manner, and the processed message and the mapping relationship with the original message number are stored. In order to reduce storage usage, only a single copy is stored. Therefore, if the data stored in the processed data pipeline is lost, the processed message needs to be restored.
  • the process of using the message system to process messages may also include: a process of recovering processed messages. For example, as shown in FIG6 , when reading a processed message fails (or a processed message is lost), it may be determined to recover the processed message.
  • the process of recovering the processed message may specifically include:
  • Step 601 Obtain the corresponding original message ID from the message ID index according to the ID of the processed message to be restored;
  • the message ID index is an index that can be persistently stored and is used to store the correspondence between the processed message ID and the original message ID. This step can be implemented by the storage manager.
  • Step 602 The data processor reads the original message corresponding to the original message ID from the main data pipeline;
  • Step 603 The data processor processes the read original message to obtain a processed message
  • Step 604 rewrite the processed message into the processed data pipeline for storage.
  • the technical solution provided in the embodiment of the present application can effectively deal with the situation where the processed message is lost by triggering the message recovery mechanism when the data in the processed data pipeline is lost, reading the original data of the corresponding message from the main data pipeline and re-processing and storing the data.
  • first, second, third, etc. may be used in this application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and are not necessarily used to describe a specific order or sequence.
  • first information may also be referred to as the second information
  • second information may also be referred to as the first information.
  • the technical solutions recorded in the embodiments of this application can be arbitrarily combined without conflict.
  • the disclosed methods, devices and equipment can be implemented in other ways.
  • the division of units is only a logical function division, and there may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed.
  • the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms of coupling.
  • the units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
  • all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

本申请公开了一种消息处理方法、装置、设备及存储介质。其中，方法包括：为目标主题在服务节点配置主数据管道和加工数据管道；将目标主题对应的生产者发送的原始消息写入主数据管道，并对原始消息进行存储；异步触发针对原始消息的加工处理，得到加工消息；将加工消息写入加工数据管道，并对加工消息进行存储。

Description

一种消息处理方法、装置、设备及存储介质
相关申请的交叉引用
本申请基于申请号为202211419131.4、申请日为2022年11月14日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本申请作为参考。
技术领域
本申请涉及大数据领域，尤其涉及一种消息处理方法、装置、设备及存储介质。
背景技术
在消息系统的一些业务中，存在一些场景，需要对生产者发布的消息进行一些内容加工处理，然后再由消费者来接收加工后的消息并进行消费。针对这些场景，目前在使用消息队列的系统中通常可以采用特定方案进行消息处理。比如，消费者从服务节点读取原始消息后，可以由消费者所在的客户端对原始消息进行内容加工；然而，采用该方案，客户端接收的原始消息一方面可能包含消费者不需要的内容，会造成网络带宽的浪费，另一方面可能包含不希望发给客户端的敏感信息，存在安全隐患。再比如，在接受到生产者发送的原始消息后，可以通过新建主题(英文可以表达为Topic)来进行内容加工并存储加工后的消息；然而，采用该方案需要消耗额外的存储空间，且由于新建主题对应的加工节点需要从服务节点消费原始消息，加工完成后再反馈给服务节点，还会造成带宽浪费，会降低消息系统的吞吐量。
从上面的描述可以看出，相关技术中的消息处理方法均存在一些缺陷。
发明内容
为解决相关技术问题，本申请实施例提供一种消息处理方法、装置、设备及存储介质。
本申请实施例的技术方案是这样实现的：
第一方面，本申请实施例提供一种消息处理方法，所述方法包括：
为目标主题在服务节点配置主数据管道和加工数据管道；其中，所述目标主题为需要进行内容加工的主题；
将所述目标主题对应的生产者发送的原始消息写入所述主数据管道，并对所述原始消息进行存储；
异步触发针对所述原始消息的加工处理，得到加工消息；
将所述加工消息写入所述加工数据管道，并对所述加工消息进行存储。
上述方案中，所述方法还包括：确定第一消费者读取的消息为所述目标主题的原始消息时，指示所述第一消费者订阅所述主数据通道；确定第二消费者读取的消息为所述目标主题的加工消息时，指示所述第二消费者订阅所述加工数据管道。
上述方案中，所述方法还包括：存储所述原始消息的标识信息与所述加工消息的标识信息之间的对应关系；确定第一加工消息丢失时，基于所述对应关系和所述第一加工消息的标识信息，确定所述第一加工消息对应的第一原始消息的标识信息；基于所述第一原始消息的标识信息，从所述主数据管道中读取所述第一原始消息；对读取到的所述第一原始消息进行加工处理，得到所述第一加工消息；将所述第一加工消息写入所述加工数据管道，并对所述第一加工消息进行存储。
上述方案中，所述对所述原始消息进行存储，包括：对所述原始消息以多副本形式进行存储。
上述方案中，所述对所述加工消息进行存储，包括：对所述加工消息以单副本形式进行存储。
上述方案中，所述方法还包括：所述原始消息存储完成后，向所述生产者发送接收成功消息。
第二方面，本申请实施例还提供一种消息处理装置，所述装置包括：
配置模块，配置为为目标主题在服务节点配置主数据管道和加工数据管道；其中，所述目标主题为需要进行内容加工的主题；
处理模块，配置为将所述目标主题对应的生产者发送的原始消息写入所述主数据管道，并对所述原始消息进行存储；
所述处理模块，还配置为异步触发针对所述原始消息的加工处理，得到加工消息；将所述加工消息写入所述加工数据管道，并对所述加工消息进行存储。
第三方面，本申请实施例还提供一种电子设备，所述电子设备包括：处理器和配置为存储能够在处理器上运行的计算机程序的存储器，其中，所述处理器配置为运行所述计算机程序时，执行前述方法的步骤。
第四方面，本申请实施例还提供一种计算机存储介质，其上存储有计算机程序，其中，该计算机程序被处理器执行时实现上述任一方法的步骤。
第五方面，本申请实施例还提供一种消息系统，所述消息系统包括：生产者、服务节点和消费者；所述服务节点包括：主题管理器、数据加工管理器和存储器；
所述生产者，配置为发送原始消息至所述服务节点；
所述主题管理器，配置为确定目标主题为需要进行内容加工的主题时，为所述目标主题配置主数据管道和加工数据管道；将所述生产者发送的所述原始消息写入所述主数据管道；
所述数据加工管理器，配置为异步触发数据加工器对所述原始消息进行加工处理，得到加工消息；将所述加工消息写入所述加工数据管道；
所述存储器，配置为对所述主数据管道和所述加工数据管道中的消息进行存储；
所述消费者，配置为从所述主数据管道读取所述原始消息，或者从所述加工数据管道中读取所述加工消息。
本申请实施例公开的消息处理方法、装置、设备及存储介质，通过为主题配置主数据管道和加工数据管道，以异步的方式对原始消息进行加工及存储，使得该主题的消费者可以直接从服务节点的主数据管道、加工数据管道中读取到原始消息和加工消息。本申请实施例提供的技术方案，相比于由消费者所在客户端进行消息加工的方案，至少解决了需要将原始消息全量传给客户端带来的额外网络资源消耗和数据安全性问题；相比于新建主题来加工并存储加工消息的方案，至少解决了原始消息和加工消息需要在不同服务节点和加工节点之间传输所带来的网络损耗问题，提高了消息系统的吞吐量。
附图说明
图1为本申请实施例消息处理方法的流程示意图；
图2为本申请实施例消息处理装置结构示意图；
图3为本申请实施例电子设备结构示意图；
图4为本申请实施例消息系统结构示意图；
图5为本申请实施例生产、加工及消费消息的流程示意图；
图6为本申请实施例恢复加工消息的流程示意图。
具体实施方式
为了能够更加详尽地了解本申请实施例的特点与技术内容，下面结合附图对本申请实施例的实现进行详细阐述，所附附图仅供参考说明之用，并非用来限定本申请实施例。
本申请实施例提出一种消息处理方法，如图1所示，该消息处理方法包括：
步骤101：为目标主题在服务节点配置主数据管道和加工数据管道；其中，所述目标主题为需要进行内容加工的主题；
步骤102：将所述目标主题对应的生产者发送的原始消息写入所述主数据管道，并对所述原始消息进行存储；
步骤103：异步触发针对所述原始消息的加工处理，得到加工消息；
步骤104：将所述加工消息写入所述加工数据管道，并对所述加工消息进行存储。
其中，步骤101中，服务节点为目标主题对应的服务器节点，可以接收生产者传输的消息数据，并向消费者提供消息数据。示例性地，实际应用中，服务节点可以为卡夫卡(Kafka)集群中的经济人(Broker)节点。
实际应用中，每个主题在服务节点可以配置多个数据管道。数据管道可以理解为用来存储消息内容的有序队列。主数据管道用于存储生产者发送的原始消息(也可称原始数据)，加工数据管道用于存储对原始消息进行加工后的加工消息(也可称加工数据)。示例性地，可以采用不同的加工方式对原始消息进行加工，得到不同的加工消息。一个主数据管道可以对应一个或多个加工数据管道，用于存储通过不同加工方式加工得到的加工消息。
示例性地，当一个主题中仅包含一个主数据管道时，该主题可写(存储生产者发送的原始数据)并且可读(消费者可指定该数据管道并从中消费数据)；当一个主体中包含一个或多个加工数据管道时，该主题只用于存储加工后的消息内容，即该主题只可读。
步骤102中，生产者为用于生产消息的模块，实际应用中，生产者可以为发布消息的应用；消费者为用于消费消息的模块，实际应用中，消费者可以为订阅消息的应用。
这里，原始消息为目标主题对应的生产者直接发送的消息。通过将原始消息写入主数据管道，可以使得该主题的消费者可以直接从主数据管道中读取到原始消息。
示例性地，在一些实施例中，所述对所述原始消息进行存储，可以包括：对所述原始消息以多副本形式进行存储。
这里，通过以多副本形式对原始消息进行存储，可以减小原始消息的丢失概率。
示例性地，在另一些实施例中，所述方法还可以包括：所述原始消息存储完成后，向所述生产者发送接收成功消息。
这里，由于加工处理是异步进行的，目标主题对应的服务节点(如Broker节点)接收消息时只需要存储原始数据，不需要进行加工处理就可以给生产者反馈接收成功的通知，让生产者发送下一批消息，从而提高了生产消息时的消息吞吐量。
步骤103中，对所述原始消息的加工处理，可以包括对消息内容的编码格式做一些特定转换，让消息可以由不同协议的消费客户端来接收并消费。比如通过开源流处理平台Kafka客户端发送Kafka协议编码的消息体，在进行了消息编码转换到云原生分布式消息流平台Pulsar协议后由Pulsar客户端接收并消费。或者，对所述原始消息的加工处理可以包括对消息内 容进行加工处理，比如对原始消息中的一些特定字段进行脱敏处理，删除一些敏感信息(比如用户名、密码等)。
实际应用中，对所述原始消息的加工处理，可以包括通过数据加工器对原始消息进行加工处理。示例性地，可以异步触发针对原始消息的加工处理，具体可以包括：控制数据加工器以异步的方式从主数据管道中读取原始消息；对读取到的原始消息进行加工处理，得到加工消息。
需要说明的是，通过不同数据加工方式加工得到的加工消息，存储于不同的加工数据管道中。示例性地，以第一加工方式对原始消息进行加工，得到第一加工消息；将所述第一加工消息写入第一加工数据管道进行存储；以第二加工方式对原始消息进行加工，得到第二加工消息；将所述第二加工消息写入第二加工数据管道进行存储。
示例性地，每个主题可以添加多个数据加工器。每个数据加工器可以包含一个用于存储加工后数据的加工数据管道名称，以及一个数据加工函数。数据加工函数用于加工消息。示例性地，一个数据加工函数可以对应一个加工数据管道。
示例性地，在实际应用中，加工处理可以包括对编码格式的加工处理，通过在服务节点上对原始消息的编码格式进行加工，使得消息系统可以支持不同协议的原生客户端程序。
步骤104中，通过将加工消息写入加工数据管道，可以使得该主题的消费者可以直接从加工数据管道中读取到加工消息。虽然增加了用于存储加工后数据的存储，但是消费者可以选择消费原始数据还是加工后的加工数据，并且每份数据只会加工一次，当有多个消费者同时消费同一份加工后数据时不会对原始数据多次加工。
示例性地，在一些实施例中，所述对所述加工消息进行存储，可以包括：对所述加工消息以单副本形式进行存储。
这里，通过将加工消息以单副本形式进行存储，可以减少存储加工消息所需要的存储空间。
实际应用中，目标主题的消费者消费消息时，可以从目标主题的主数据管道消费原始消息，也可以从目标主题的加工数据管道消费加工后的加工消息。
示例性地，在一些实施例中，所述方法还可以包括：确定第一消费者读取的消息为所述目标主题的原始消息时，指示所述第一消费者订阅所述主数据通道；确定第二消费者读取的消息为所述目标主题的加工消息时，指示所述第二消费者订阅所述加工数据管道。
这里，指示所述第一消费者订阅所述主数据通道，可以使得所述第一消费者从所述主数据通道中读取所述原始消息；指示所述第二消费者订阅所述加工数据管道，可以使得所述第二消费者从所述加工数据管道中读取所述加工消息。
示例性地，在一些实施例中，所述方法还可以包括：存储所述原始消息的标识信息与所述加工消息的标识信息之间的对应关系；确定第一加工消息丢失时，基于所述对应关系和所述第一加工消息的标识信息，确定所述第一加工消息对应的第一原始消息的标识信息；基于所述第一原始消息的标识信息，从所述主数据管道中读取所述第一原始消息；对读取到的所述第一原始消息进行加工处理，得到所述第一加工消息；将所述第一加工消息写入所述加工数据管道，并对所述第一加工消息进行存储。
示例性地，原始消息和加工消息的标识信息可以包括消息ID，并可以设置一个可持久化存储的索引，用于存储加工消息ID和原始消息ID的对应关系。当加工数据管道中的数据丢失时，可以通过触发消息恢复机制，从主数据管道中读取对应消息的原始数据并重新进行加工和存储，从而有效应对加工消息丢失的情况。
这里，步骤101至步骤104的执行主体可以为实现本申请实施例中消息处理方法的电子设备的处理器。
本申请实施例提供的技术方案，通过为目标主题在服务节点配置主数据管道和加工数据管道，以异步的方式对原始消息进行加工及存储，使得该主题的消费者可以直接从服务节点的主数据管道、加工数据管道中读取到原始消息和加工消息。本申请实施例提供的技术方案，相比于由消费者所在客户端进行消息加工的方案，至少解决了需要将原始消息全量传给客户端带来的额外网络资源消耗和数据安全性问题；通过在服务节点上对消息的编码格式进行加工，让消息系统可以支持不同协议的原生客户端程序；相比于通过流处理的方式新建主题来加工并存储加工消息的方案，至少解决了原始消息和加工消息需要在不同服务节点和加工节点之间传输所带来的网络损耗问题，提高了消息系统的吞吐量。
本申请实施例还提供了一种消息处理方法的实现装置，如图2所示，该消息处理装置200具体包括：
配置模块201，配置为为目标主题在服务节点配置主数据管道和加工数据管道；其中，所述目标主题为需要进行内容加工的主题；
处理模块202，配置为将所述目标主题对应的生产者发送的原始消息写入所述主数据管道，并对所述原始消息进行存储；
所述处理模块202，还配置为异步触发针对所述原始消息的加工处理，得到加工消息；将所述加工消息写入所述加工数据管道，并对所述加工消息进行存储。
本申请实施例提供的技术方案，通过为目标主题在服务节点配置主数据管道和加工数据管道，以异步的方式对原始消息进行加工及存储，使得该主题的消费者可以直接从服务节点的主数据管道、加工数据管道中读取到原始消息和加工消息。本申请实施例提供的技术方案，相比于由消费者所在客户端进行消息加工的方案，至少解决了需要将原始消息全量传给客 户端带来的额外网络资源消耗和数据安全性问题；通过在服务节点(如Broker节点)上对消息的编码格式进行加工，让消息系统可以支持不同协议的原生客户端程序；相比于通过流处理的方式新建主题来加工并存储加工消息的方案，至少解决了原始消息和加工消息需要在不同服务节点和加工节点之间传输所带来的网络损耗问题，提高了消息系统的吞吐量。
在一些实施例中，所述处理模块202，还配置为确定第一消费者读取的消息为所述目标主题的原始消息时，指示所述第一消费者订阅所述主数据通道；确定第二消费者读取的消息为所述目标主题的加工消息时，指示所述第二消费者订阅所述加工数据管道。
在一些实施例中，所述处理模块202，还配置为存储所述原始消息的标识信息与所述加工消息的标识信息之间的对应关系；确定第一加工消息丢失时，基于所述对应关系和所述第一加工消息的标识信息，确定所述第一加工消息对应的第一原始消息的标识信息；基于所述第一原始消息的标识信息，从所述主数据管道中读取所述第一原始消息；对读取到的所述第一原始消息进行加工处理，得到所述第一加工消息；将所述第一加工消息写入所述加工数据管道，并对所述第一加工消息进行存储。
在一些实施例中，所述处理模块202，还配置为对所述原始消息以多副本形式进行存储。
在一些实施例中，所述处理模块202，还配置为对所述加工消息以单副本形式进行存储。
在一些实施例中，所述处理模块202，还配置为所述原始消息存储完成后，向所述生产者发送接收成功消息。
实际应用时，所述配置模块201和处理模块202可由消息处理装置200中的处理器实现。
基于上述消息处理装置中各单元的硬件实现，本申请实施例还提供了一种电子设备，如图3所示，该电子设备300包括：处理器301和配置为存储能够在处理器上运行的计算机程序的存储器302；
其中，处理器301配置为运行计算机程序时，执行前述实施例中的方法的步骤。
当然，实际应用时，如图3所示，该电子设备中的各个组件通过总线系统303耦合在一起。可理解，总线系统303用于实现这些组件之间的连接通信。总线系统303除包括数据总线之外，还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起见，在图3中将各种总线都标为总线系统303。
在实际应用中，上述处理器可以为特定用途集成电路(ASIC，Application Specific Integrated Circuit)、数字信号处理装置(DSPD，Digital Signal Processing Device)、可编程逻辑装置(PLD，Programmable Logic Device)、现场可编程门阵列(Field－Programmable Gate Array，FPGA)、 控制器、微控制器、微处理器中的至少一种。可以理解地，对于不同的设备，用于实现上述处理器功能的电子器件还可以为其他电子器件，本申请实施例不作具体限定。
上述存储器可以是易失性存储器(volatile memory)，例如随机存取存储器(RAM，Random-Access Memory)；或者非易失性存储器(non-volatile memory)，例如只读存储器(ROM，Read-Only Memory)，快闪存储器(flash memory)，硬盘(HDD，Hard Disk Drive)或固态硬盘(SSD，Solid-State Drive)；或者上述种类的存储器的组合，并向处理器提供指令和数据。
在示例性实施例中，本申请实施例还提供了一种计算机可读存储介质，例如包括计算机程序的存储器，计算机程序可由电子设备的处理器执行，以完成前述消息处理方法的步骤。
为了能更加体现本申请的目的，在本申请上述实施例的基础上，下面结合消息系统实施例对本申请再作进一步详细的描述。
本申请实施例还提供一种消息系统，该消息系统可以为分布式消息系统。其中，分布式消息系统是一种可扩展的多节点消息队列系统。如图4所示，该消息系统40可以包括：生产者、服务节点和消费者；所述服务节点可以包括：主题管理器401、数据加工管理器402和存储器403；
示例性地，实际应用中，服务节点可以为Broker节点。
如图4所示，所述生产者，配置为发送原始消息至所述服务节点；
如图4所示，所述主题管理器401，配置为确定目标主题为需要进行内容加工的主题时，为所述目标主题配置主数据管道和加工数据管道；将生产者发送的原始消息写入所述主数据管道，以使得所述目标主题对应的消费者从所述主数据管道中读取所述原始消息。
实际应用中，每个主题(英文可以表达为Topic)可以添加多个数据管道。其中只有一个主数据管道可写(存储生产者发送的原始数据)并且可读(消费者可指定该数据管道并从中消费数据)，其他加工数据管道只用于存储加工后的消息内容，即其他加工数据管道只可读。
需要说明的是，通过不同数据加工器加工得到的加工消息，存储于不同的加工数据管道中。示例性地，第一数据加工器以第一加工方式对原始消息进行加工，得到第一加工消息；将所述第一加工消息写入第一加工数据管道进行存储；第二数据加工器以第二加工方式对原始消息进行加工，得到第二加工消息；将所述第二加工消息写入第二加工数据管道进行存储。
如图4所示，所述数据加工管理器402，配置为异步触发数据加工器对原始消息进行加工处理，得到加工消息；将所述加工消息写入所述加工数据管道，以使得所述目标主题对应的消费者从所述加工数据管道中读取所述加工消息。
如图4所示，数据加工管理器402提供一个消息加工的管理接口，管理者可通过改接口动态添加和修改对消息内容的加工方式。数据加工管理 器402，负责接收和处理数据加工器的管理命令。示例性地，一条添加数据加工器的命令可以包含：数据加工器对应主题的名称、用于存储加工结果的加工数据管道名称、包含了数据加工函数的脚本文件(比如Python脚本)或字节码文件(比如Java class)或其他可实现动态加载函数的文件。
如图4所示，所述存储器403，配置为对所述主数据管道和所述加工数据管道中的消息进行存储。
示例性地，在一些实施例中，主数据管道的数据存储为多副本，即对原始消息进行多副本存储。通过以多副本形式对原始消息进行存储，可以减小原始消息的丢失概率。
示例性地，在一些实施例中，加工数据管道的数据存储为单副本。即对加工消息进行单副本存储。通过将加工消息以单副本形式进行存储，可以减少存储加工消息所需要的存储空间。
示例性地，在一些实施例中，服务节点还配置为在所述原始消息存储完成后，向所述生产者发送接收成功消息。
由于加工处理是异步进行的，目标主题对应的服务节点接收消息时只需要存储原始数据，不需要进行加工处理就可以给生产者反馈接收成功的通知，让生产者发送下一批消息，可以提高生产消息时的消息吞吐量。
示例性地，在实际应用中，一些服务节点不具备存储功能。因此，在一些实施例中，消息系统可以包括：生产者、服务节点、消费者和存储层；服务节点包括：主题管理器、数据加工管理器、存储管理器，存储管理器配置为将所述主数据管道和所述加工数据管道中的数据存储至所述存储层或从存储层中读取数据。
如图4所示，所述消费者，配置为从所述主数据管道读取所述原始消息，或者从所述加工数据管道中读取所述加工消息。
本申请实施例提供的消息系统，支持在服务节点接收到生产者发送的消息后，对消息内容以异步的方式进行加工并存储，使得消费者即可以接收原始消息内容，也可以接收加工后的消息内容。由于对消息内容的加工是在接收到消息的服务节点Broker上进行的，相比于由消费者所在客户端进行消息加工的方案，至少解决了需要将原始消息全量传给客户端带来的额外网络资源消耗和数据安全性问题；通过在服务节点Broker上对消息的编码格式进行加工，让消息系统可以支持不同协议的原生客户端程序；相比于通过流处理的方式新建主题来加工并存储加工消息的方案，不需要通过网络传输将消息发送到其他加工节点上进行处理，不影响整个消息系统的吞吐量；由于加工处理是异步进行的，目标主题对应的服务节点Broker接收消息时只需要存储原始数据，不需要进行加工处理就可以给生产者反馈接收成功的通知，让生产者发送下一批消息，可以提高生产消息时的消息吞吐量。
示例性地，使用该消息系统40进行消息处理的过程可以包括：预先配 置过程；生产、加工及消费消息过程。
具体地，预先配置过程可以包括：为主题A(TopicA)配置主数据管道、加工数据管道和数据加工器A。创建消费者A订阅TopicA的主数据管道，消费原始消息。创建消费者B订阅TopicA的加工数据管道，消费加工后的加工消息。
示例性地，图5为本申请实施例中生产、加工及消费消息的具体流程示意图。如图5所示，该流程具体可以包括：
服务节点Broker接收生产者向对应TopicA发送的原始消息后，将原始消息写入主数据管道，并触发存储器将原始消息以多副本的存储至磁盘，存储完成后给生产者返回发送成功消息；数据写入主数据管道后，异步触发数据加工器A，从主数据管道中读取原始消息，进行加工处理，并把数据加工后的结果(即加工消息)写入加工数据管道，并触发存储器将数据以单副本的形式存储至磁盘，以及在数据索引中存入加工消息ID和原始消息ID的对应关系；消费者A从TopicA的主数据管道中读取原始消息；消费者B从TopicA的加工数据管道中读取加工后的加工消息。
如图5所示，消费者A从TopicA的主数据管道中读取原始消息，具体可以包括：主数据管道接收消费者A发送的消费原始消息的请求；触发存储器从磁盘中读取原始消息；读取原始消息完成后，返回原始消息至消费者A。
如图5所示，消费者B从TopicA的加工数据管道中消费加工后的加工消息，具体可以包括：加工数据管道接收消费者B发送的消费加工消息的请求；触发存储器从磁盘中读取加工消息；读取加工消息完成后，返回加工消息至消费者B。
消费者需要接收加工消息时，订阅该Topic中用于存储加工后数据的加工数据管道。不同于主数据管道内的数据，加工数据管道中的数据由数据加工器以异步的方式从主数据管道中读取原始消息后进行加工处理，并存储加工消息以及和原始消息编号的映射关系。且为了减少存储使用量，只存储单副本。因此，如果发生加工数据管道中存储数据丢失的情况，则需要对加工消息进行恢复。
在一些实施例中，使用消息系统进行消息处理的过程还可以包括：加工消息恢复过程。示例性地，如图6所示，读取加工消息失败(或加工消息丢失)时，可以确定恢复加工消息，恢复加工消息过程具体可以包括：
步骤601：根据要恢复的加工消息的ID从消息ID索引中获取对应的原始消息ID；
消息ID索引为一个可持久化存储的索引，用于存储加工消息ID和原始消息ID的对应关系，该步骤可以由存储管理器实现。
步骤602：数据加工器从主数据管道中读取该原始消息ID对应的原始消息；
步骤603：由数据加工器对读取到的原始消息进行加工处理，得到加工消息；
步骤604：将加工消息重新写入加工数据管道进行存储。
实际应用中，整个加工消息恢复过程消费者不感知。
本申请实施例提供的技术方案，通过当加工数据管道中的数据丢失时，触发消息恢复机制，从主数据管道中读取对应消息的原始数据并重新进行数据加工和存储，可有效应对加工消息丢失的情况。
应当理解，在本申请使用的术语是仅仅出于描述特定实施例的目的，而非旨在限制本申请。在本申请和所附权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其他含义。还应当理解，本文中使用的术语“和/或”是指并包含一个或多个相关联的列出项目的任何或所有可能组合。本申请中表述“具有”、“可以具有”、“包括”和“包含”、或者“可以包括”和“可以包含”在本文中可以用于指示存在对应的特征(例如，诸如数值、功能、操作或组件等元素)，但不排除附加特征的存在。
应当理解，尽管在本申请可能采用术语第一、第二、第三等来描述各种信息，但这些信息不应限于这些术语。这些术语仅用来将同一类型的信息彼此区分开，不必用于描述特定的顺序或先后次序。例如，在不脱离本申请范围的情况下，第一信息也可以被称为第二信息，类似地，第二信息也可以被称为第一信息。本申请实施例所记载的技术方案之间，在不冲突的情况下，可以任意组合。在本申请所提供的几个实施例中，应该理解到，所揭露的方法、装置和设备，可以通过其他的方式实现。以上所描述的实施例仅仅是示意性的，例如，单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，如：多个单元或组件可以结合，或可以集成到另一个系统，或一些特征可以忽略，或不执行。另外，所显示或讨论的各组成部分相互之间的耦合、或直接耦合、或通信连接可以是通过一些接口，设备或单元的间接耦合或通信连接，可以是电性的、机械的或其他形式的耦合。
上述作为分离部件说明的单元可以是、或也可以不是物理上分开的，作为单元显示的部件可以是、或也可以不是物理单元，即可以位于一个地方，也可以分布到多个网络单元上；可以根据实际的需要选择其中的部分或全部单元来实现本实施例方案的目的。
另外，在本申请各实施例中的各功能单元可以全部集成在一个处理单元中，也可以是各单元分别单独作为一个单元，也可以两个或两个以上单元集成在一个单元中；上述集成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可 轻易想到变化或替换，都应涵盖在本申请的保护范围之内。

Claims (10)

  1. 一种消息处理方法，所述方法包括：
    为目标主题在服务节点配置主数据管道和加工数据管道；其中，所述目标主题为需要进行内容加工的主题；
    将所述目标主题对应的生产者发送的原始消息写入所述主数据管道，并对所述原始消息进行存储；
    异步触发针对所述原始消息的加工处理，得到加工消息；
    将所述加工消息写入所述加工数据管道，并对所述加工消息进行存储。
  2. 根据权利要求1所述的方法，其中，所述方法还包括：
    确定第一消费者读取的消息为所述目标主题的原始消息时，指示所述第一消费者订阅所述主数据通道；
    确定第二消费者读取的消息为所述目标主题的加工消息时，指示所述第二消费者订阅所述加工数据管道。
  3. 根据权利要求1所述的方法，其中，所述方法还包括：
    存储所述原始消息的标识信息与所述加工消息的标识信息之间的对应关系；
    确定第一加工消息丢失时，基于所述对应关系和所述第一加工消息的标识信息，确定所述第一加工消息对应的第一原始消息的标识信息；
    基于所述第一原始消息的标识信息，从所述主数据管道中读取所述第一原始消息；
    对读取到的所述第一原始消息进行加工处理，得到所述第一加工消息；
    将所述第一加工消息写入所述加工数据管道，并对所述第一加工消息进行存储。
  4. 根据权利要求1至3任一项所述的方法，其中，所述对所述原始消息进行存储，包括：对所述原始消息以多副本形式进行存储。
  5. 根据权利要求1至3任一项所述的方法，其中，所述对所述加工消息进行存储，包括：对所述加工消息以单副本形式进行存储。
  6. 根据权利要求1至3任一项所述的方法，其中，所述方法还包括：所述原始消息存储完成后，向所述生产者发送接收成功消息。
  7. 一种消息处理装置，所述装置包括：
    配置模块，配置为为目标主题在服务节点配置主数据管道和加工数据管道；其中，所述目标主题为需要进行内容加工的主题；
    处理模块，配置为将所述目标主题对应的生产者发送的原始消息写入所述主数据管道，并对所述原始消息进行存储；
    所述处理模块，还配置为异步触发针对所述原始消息的加工处理，得到加工消息；将所述加工消息写入所述加工数据管道，并对所述加工消息进行存储。
  8. 一种电子设备，所述电子设备包括：处理器和配置为存储能够在处理器上运行的计算机程序的存储器，
    其中，所述处理器配置为运行所述计算机程序时，执行权利要求1至6任一项所述方法的步骤。
  9. 一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现权利要求1至6任一项所述的方法的步骤。
  10. 一种消息系统，所述消息系统包括：生产者、服务节点和消费者；所述服务节点包括：主题管理器、数据加工管理器和存储器；
    所述生产者，配置为发送原始消息至所述服务节点；
    所述主题管理器，配置为确定目标主题为需要进行内容加工的主题时，为所述目标主题配置主数据管道和加工数据管道；将所述生产者发送的所述原始消息写入所述主数据管道；
    所述数据加工管理器，配置为异步触发数据加工器对所述原始消息进行加工处理，得到加工消息；将所述加工消息写入所述加工数据管道；
    所述存储器，配置为对所述主数据管道和所述加工数据管道中的消息进行存储；
    所述消费者，配置为从所述主数据管道读取所述原始消息，或者从所述加工数据管道中读取所述加工消息。
PCT/CN2023/131278 2022-11-14 2023-11-13 一种消息处理方法、装置、设备及存储介质 WO2024104289A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211419131.4A CN116800741A (zh) 2022-11-14 2022-11-14 一种消息处理方法、装置、设备及存储介质
CN202211419131.4 2022-11-14

Publications (1)

Publication Number Publication Date
WO2024104289A1 true WO2024104289A1 (zh) 2024-05-23

Family

ID=88048677

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/131278 WO2024104289A1 (zh) 2022-11-14 2023-11-13 一种消息处理方法、装置、设备及存储介质

Country Status (2)

Country Link
CN (1) CN116800741A (zh)
WO (1) WO2024104289A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180159731A1 (en) * 2015-01-23 2018-06-07 Ebay Inc. Processing high volume network data
CN109582637A (zh) * 2017-09-28 2019-04-05 韩国电子通信研究院 网络基础结构系统和使用其的数据处理和数据共享的方法
KR20210110113A (ko) * 2020-02-28 2021-09-07 심윤찬 명칭변환서비스 시스템 및 제공 방법
CN114500546A (zh) * 2022-01-25 2022-05-13 中国农业银行股份有限公司 基于负载均衡集群的通信消息发送方法、装置及设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180159731A1 (en) * 2015-01-23 2018-06-07 Ebay Inc. Processing high volume network data
CN109582637A (zh) * 2017-09-28 2019-04-05 韩国电子通信研究院 网络基础结构系统和使用其的数据处理和数据共享的方法
KR20210110113A (ko) * 2020-02-28 2021-09-07 심윤찬 명칭변환서비스 시스템 및 제공 방법
CN114500546A (zh) * 2022-01-25 2022-05-13 中国农业银行股份有限公司 基于负载均衡集群的通信消息发送方法、装置及设备

Also Published As

Publication number Publication date
CN116800741A (zh) 2023-09-22

Similar Documents

Publication Publication Date Title
US20220188003A1 (en) Distributed Storage Method and Device
JP4696089B2 (ja) 分散ストレージシステム
US7209972B1 (en) High speed data transfer mechanism
US20230205603A1 (en) Communications Framework for Multi-Cluster Management Systems
US20160328168A1 (en) Optimized write performance at block-based storage during volume snapshot operations
US7676503B2 (en) Hybrid computer restore using network service
US20210160319A1 (en) Data Sending Method and Apparatus, and Data Receiving Method and Apparatus
WO2018054079A1 (zh) 一种存储文件的方法、第一虚拟机及名称节点
US11416176B2 (en) Function processing using storage controllers for load sharing
CN113157658B (zh) 客户端日志收集分发方法、装置及计算机设备
WO2024104289A1 (zh) 一种消息处理方法、装置、设备及存储介质
Hegde et al. Low latency message brokers
US11210089B2 (en) Vector send operation for message-based communication
CN110288309B (zh) 数据交互方法、装置、系统、计算机设备以及存储介质
CN114461595A (zh) 发送消息的方法、装置、介质和电子设备
CN116466973B (zh) 一种多并发固件升级方法、系统、计算机设备及存储介质
CN116820430B (zh) 异步读写方法、装置、计算机设备及存储介质
WO2024082773A1 (zh) 一种迁移文件系统的方法和相关设备
CN114637475B (zh) 一种分布式存储系统控制方法、装置及可读存储介质
CN116302007B (zh) 磁盘阵列固件升级方法、装置、设备、介质及磁盘阵列卡
JP7326234B2 (ja) 情報処理装置、情報処理方法及びコンピュータプログラム
CN114911421B (zh) 基于csi插件的数据存储方法、系统、装置及存储介质
CN114051047B (zh) 一种会话消息的备份方法、装置、网络设备和存储介质
WO2023116438A1 (zh) 一种数据访问方法、装置以及设备
EP1256050B1 (en) High speed data transfer mechanism