CN116893910A - Message processing method, device, platform, computer equipment and storage medium - Google Patents

Message processing method, device, platform, computer equipment and storage medium Download PDF

Info

Publication number
CN116893910A
CN116893910A CN202310891508.4A CN202310891508A CN116893910A CN 116893910 A CN116893910 A CN 116893910A CN 202310891508 A CN202310891508 A CN 202310891508A CN 116893910 A CN116893910 A CN 116893910A
Authority
CN
China
Prior art keywords
message
event
queue
hot
processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310891508.4A
Other languages
Chinese (zh)
Inventor
王旭
王伟锋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Bilibili Technology Co Ltd
Original Assignee
Shanghai Bilibili Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Bilibili Technology Co Ltd filed Critical Shanghai Bilibili Technology Co Ltd
Priority to CN202310891508.4A priority Critical patent/CN116893910A/en
Publication of CN116893910A publication Critical patent/CN116893910A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/54Interprogram communication
    • G06F9/546Message passing systems or structures, e.g. queues
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/252Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/54Interprogram communication
    • G06F9/542Event management; Broadcasting; Multicasting; Notifications
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00Indexing scheme relating to G06F9/00
    • G06F2209/54Indexing scheme relating to G06F9/54
    • G06F2209/547Messaging middleware

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application discloses a message processing platform, which comprises: the producer node is used for generating an event message, and sending the event message to a target message storage partition corresponding to the type in the message middleware for storage according to the type of the event message; the event acquisition module is used for acquiring event messages from the target message storage partition; the analysis module is used for analyzing the acquired event message to obtain an analysis result, and distributing the acquired event message to a corresponding concurrent message storage queue according to the analysis result; each concurrent message storage queue is used for storing event messages sent by the analysis module; each message processing thread is used for acquiring the event message from the corresponding concurrent message storage queue and writing the acquired event message into the database.

Description

Message processing method, device, platform, computer equipment and storage medium
Technical Field
The present application relates to the field of internet technologies, and in particular, to a message processing method, device, platform, computer device, and storage medium.
Background
In a highly concurrent service scenario, in order to be able to process service messages in time, the throughput of the messages is often improved by the continuous expansion of the service system. However, the expansion of the service system requires a large cost, and even if the service system is expanded, a problem of data skew occurs due to a hot spot phenomenon found in the system.
In view of the foregoing, it is desirable to provide a method for improving throughput of messages and avoiding data skew without expanding the service system.
Disclosure of Invention
In view of this, a message processing method, apparatus, platform, computer device and storage medium are provided to solve the problem that in the prior art, when the throughput of a message is improved, a service system needs to be continuously expanded, and the problem that the expanded system can generate data tilting is solved.
The application provides a message processing platform, which comprises a producer node and a consumer node, wherein the consumer node comprises a message middleware, an event acquisition module, an analysis module and a message concurrency processing module, the message middleware comprises a plurality of message storage partitions, the message concurrency processing module comprises a plurality of concurrent message storage queues, a plurality of message processing threads, a hot spot analysis unit, a hot spot transfer unit, a hot spot queue and a hot spot thread, and each message processing thread corresponds to one concurrent message storage queue one by one, wherein:
The producer node is used for generating an event message, and sending the event message to a target message storage partition corresponding to the type in the message middleware for storage according to the type of the event message;
the event acquisition module is used for acquiring event messages from the target message storage partition;
the analysis module is used for analyzing the acquired event message to obtain an analysis result, and distributing the acquired event message to a corresponding concurrent message storage queue according to the analysis result;
each concurrent message storage queue is used for storing event messages sent by the analysis module;
the hot spot analysis unit is used for analyzing the event messages stored in each concurrent message storage queue and determining hot spot event messages according to analysis results;
the hotspot transferring unit is used for transferring the determined hotspot event message to a hotspot message storage partition in the message middleware;
the event acquisition module is further configured to acquire a hotspot event message from the hotspot message storage partition, and store the hotspot event message acquired from the hotspot message storage partition into the hotspot queue;
The hot spot thread is used for acquiring hot spot event messages from the hot spot queue and writing the hot spot event messages acquired from the hot spot queue into the database;
each message processing thread is used for acquiring event messages from the corresponding concurrent message storage queues and writing the acquired non-hot event messages into the database.
Optionally, the hotspot analysis unit is further configured to analyze occurrence frequencies and processing time consumption in each event message stored in each concurrent message storage queue, and determine a hotspot event message according to an analysis result.
Optionally, the message concurrency processing module further includes a repayment unit, a repayment queue, and a repayment thread, wherein:
the repacking unit is used for repacking the event message with the writing failure into the repacking queue when the event message is detected to be written into the database with the writing failure;
the repacking thread is used for acquiring the event message from the repacking queue after the time of repacking the failed event message to the repacking queue reaches the preset time, and writing the acquired event message into the database.
Optionally, the consumer node further includes a message processing mode determining module and a message aggregation processing module, where the message aggregation processing module includes a plurality of aggregate message storage queues and a plurality of message aggregation threads, each message aggregation thread corresponds to one aggregate message storage queue one by one, where:
the message processing mode determining module is used for determining a message processing mode according to preset configuration information, wherein the message processing mode comprises a concurrent processing mode and an aggregation processing mode;
each aggregation message storage queue is used for storing event messages sent by the analysis module;
each message aggregation thread is used for acquiring event messages from the corresponding aggregation message storage queue, carrying out aggregation processing on the acquired event messages, and writing the aggregated event messages into the database in batches when the number of the aggregated event messages reaches a preset number or the aggregation time reaches a preset time.
Optionally, the producer node is further configured to detect whether a transmission channel between the producer node and the message middleware is faulty, and if so, send the generated event message to the event acquisition module.
Optionally, the message processing platform further comprises a validation character module, wherein:
the event acquisition module is further used for orderly connecting acquired event messages in series in a linked list mode and adding a processing state for each event message;
the event acquisition module is further used for modifying the processing state of the processed event message into processed event message after the event message is processed;
the confirmation character module is used for detecting the processing state of each event message in the linked list at fixed time, and marking the position of the target event message as a confirmation character position when the processing state of all the previous event messages of the target event message in the linked list is processed.
The application also provides a message processing method which is applied to the message processing platform, and the method comprises the following steps:
analyzing event messages stored in a plurality of concurrent message storage queues, and determining hot event messages according to analysis results;
transferring the determined hot event message to a hot message storage partition in the message middleware;
acquiring a hot spot event message from the hot spot message storage partition, and storing the hot spot event message acquired from the hot spot message storage partition into the hot spot queue;
And acquiring the hot event message from the hot queue, and writing the hot event message acquired from the hot queue into a database.
Optionally, the analyzing the event messages stored in the multiple concurrent message storage queues, and determining the hot event message according to the analysis result includes:
analyzing the occurrence frequency and the processing time consumption of each event message stored in each concurrent message storage queue to obtain a hot spot analysis result;
and determining a hot spot event message according to the hot spot analysis result.
The application also provides a message processing device, which comprises:
the analysis module is used for analyzing the event messages stored in the concurrent message storage queues and determining hot event messages according to analysis results;
the transfer module is used for transferring the determined hot event message to a hot message storage partition in the message middleware;
the storage module is used for acquiring the hot spot event message from the hot spot message storage partition and storing the hot spot event message acquired from the hot spot message storage partition into the hot spot queue;
and the writing module is used for acquiring the hot event message from the hot queue and writing the hot event message acquired from the hot queue into a database.
The application also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the above method when executing the computer program.
The application also provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the above method.
In this embodiment, the message processing platform adds some queues under the condition that the partitions are kept unchanged, so that after receiving the event message, the event message stored in the same message storage partition can be distributed to different queues, and then the event message in the queue is processed by binding independent processing threads through each queue, thereby realizing the concurrent processing of the event message and improving the throughput of the message. In addition, in this embodiment, by analyzing the event messages stored in each queue in real time, the hot event message can be found, and after the hot event message is found, the hot event message is transferred to the hot queue, and the hot thread is synchronously started to write the hot event message in the hot queue into the database, so that the hot event message is isolated from other event messages, and the hot event message can be normally processed, thereby avoiding the phenomenon of data tilting.
Drawings
FIG. 1 is a schematic view of an application environment of an embodiment of a message processing platform according to an embodiment of the present application;
FIG. 2 is a block diagram of one embodiment of a message processing platform according to the present application;
FIG. 3 is a flow chart of one embodiment of a message processing method according to the present application;
FIG. 4 is a flow chart of the message concurrency processing module for processing event messages according to the present application;
FIG. 5 is a flow chart of the message aggregation processing module of the present application for processing an event message;
FIG. 6 is a schematic diagram of hot event message analysis according to the present application;
FIG. 7 is a flow chart of another embodiment of a message processing method according to the present application; the method comprises the steps of carrying out a first treatment on the surface of the
FIG. 8 is a block diagram illustrating a message processing apparatus according to an embodiment of the present application;
fig. 9 is a schematic hardware structure of a computer device for executing a message processing method according to an embodiment of the present application.
Detailed Description
Advantages of the application are further illustrated in the following description, taken in conjunction with the accompanying drawings and detailed description.
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.
The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present disclosure. Depending on the context, the word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination".
In the description of the present application, it should be understood that the numerical references before the steps do not identify the order in which the steps are performed, but are merely used to facilitate description of the present application and to distinguish between each step, and thus should not be construed as limiting the present application.
An exemplary application environment for the present application is provided below. Fig. 1 schematically shows an environmental application schematic according to an embodiment of the application.
The message processing platform 2 comprises a producer node and a consumer node. Wherein the producer node is configured to produce an event message. The consumer node is configured to consume the event message generated by the producer node, and store the consumed event message in the database 4.
The database 4 may be various types of databases, such as a relational database oracle, mysql, etc., a non-relational database mongod, cassandra, redis, hbase, etc.
The message processing platform 2 and the database 4 may be connected via a network. The network may include various network devices such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, and/or proxy devices, etc. The network may include physical links such as coaxial cable links, twisted pair cable links, fiber optic links, combinations thereof, and/or the like. The network may include wireless links, such as cellular links, satellite links, wi-Fi links, and/or the like.
In the following, several embodiments will be provided in the above-described exemplary application environment to illustrate comment data processing schemes in the present application.
Referring to fig. 2, an architecture diagram of a message processing platform according to an embodiment of the application is shown. The message processing platform comprises a producer node 20 and a consumer node 21. The consumer node 21 includes a message middleware 210, an event obtaining module 211, an parsing module 212, and a message concurrency processing module 212, where the message middleware 210 includes a plurality of message storage partitions, and the message concurrency processing module 212 includes a plurality of concurrent message storage queues 2120, a plurality of message processing threads 2121, a hotspot analysis unit 2122, a hotspot transfer unit 2123, and a hotspot thread 2124, and each message processing thread 2120 corresponds to one concurrent message storage queue 2121 one by one.
Producer node 20, which is the producer of the event message, may be a variety of types of devices. For example, the server may be a tablet computer, a notebook computer, a desktop computer, a rack-mounted server, a blade server, a tower server, or a rack server (including a stand-alone server or a server cluster formed by a plurality of servers), etc.
And the producer node 20 is used for generating an event message, and sending the event message to a target message storage partition corresponding to the type in the message middleware according to the type of the event message for storage.
Specifically, the event message is a message generated based on a behavior event of the user, for example, the event message may be a praise message generated based on a praise behavior of the user, or a comment message generated based on a comment behavior of the user, or the like.
In this embodiment, the event message may be abstracted into the following 4 types according to the format of the data:
sendbytes, send byte type messages;
sendstring, send a string type message;
sendjson, sending a message of JSON type strings;
sendbatch, send byte type messages in batches.
It will be appreciated that, after generating the event message, the producer node 20 may perform data format conversion on the event message after generating the event message to convert the data format of the event message into an event message of a preset data format, so that the event messages of different data types may be conveniently written into the message middleware.
The message middleware is a supportive software system which provides synchronous or asynchronous and reliable message transmission for an application system in a network environment based on a queue and message transmission technology. Wherein the message middleware may be RabbitMQ, rocketMQ, kafka, pulsar, databus or the like. In this embodiment, the message middleware is preferably Kafka, and accordingly, the plurality of message storage partitions in the message middleware are a plurality of topics (Topic) in the Kafka.
In one embodiment, different types of event messages generated for producer node 20 are stored in a target message storage partition in message middleware corresponding to the type.
It should be noted that, the above-mentioned different types of event messages refer to different types of event messages obtained by dividing the function types of the data objects, for example, a praise message for the video a and a bullet screen message for the video a, which belong to two different types of event messages.
As an example, when the message middleware 210 is Kafka and the message storage partition is Topic (Topic) in Kafka, then for praise messages, they will be stored to Topic a in Kafka and for comment messages, they will be stored to Topic B in Kafka.
Wherein Kafka is an open source stream processing platform developed by the Apache software foundation, written by Scala and Java. Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action flow data for consumers in a web site. Such actions (web browsing, searching and other user actions) are a key factor in many social functions on modern networks. Topic (Topic) is the name of a category, and a generic message is sent under the same Topic. For each Topic, there may be multiple Partition (Partition) log files below.
As an example, when the message middleware 210 is a rabitmq and the message storage partition is a queue in the rabitmq, the message is stored to a queue a in the rabitmq for the like-to-like message, and to a queue B in the rabitmq for the comment message.
Wherein RabbitMQ is a set of open source (MPL) message queue service software, implemented by an open source of Advanced Message Queuing Protocol (AMQP) provided by LShift, written by Erlang, which is named high performance, robust and scalable. RabbitMQ is used to process asynchronous messages from clients. The server puts the message to be sent into the queue pool. The receiving end can receive the message sent by the server according to the forwarding mechanism configured by RabbitMQ. The RabbitMQ performs the operations of forwarding, buffering and persistence of the message according to the designated forwarding rules.
An event acquisition module 211 is configured to acquire an event message from the target message storage partition.
Specifically, the target storage partition refers to a storage interval storing the event message.
As an example, when the target message storage Partition is a target Topic (Topic) in kafka, the event acquisition module 211 acquires an event message from a plurality of partitions (Partition) in the target Topic if the target Topic contains the event message. For example, if the target Topic includes 3 partitions, the event acquisition module 211 acquires an event message from the 3 partitions.
As an example, when the target message storage partition is a queue in a rabitmq, then the event retrieval module 211 retrieves the event message stored therein from the target queue.
The parsing module 212 is configured to parse the obtained event message to obtain a parsing result, and distribute the obtained event message to a corresponding concurrent message storage queue according to the parsing result.
Specifically, after the event obtaining module 211 obtains the event message, it invokes the parsing module 212 to parse the event message, so as to obtain a parsing result, where the parsing result is used to determine to which concurrent message storage queue the event message needs to be distributed.
In an embodiment, when the parsing module 212 parses the event message, a parsing method for the event message may be determined according to a preset configuration file, and then the event message is parsed according to the parsing method. For example, if the event message is a praise message of the video, the parsing method of the praise message of the video may be configured in the configuration file in advance to determine the storage queue according to the hash value of the video ID. Thus, when the parsing module 212 parses the praise message, the video ID information is extracted from the praise message, then hash calculation is performed according to the video ID to obtain a hash value, and finally the storage queue is determined according to the hash value. For example, when the hash value is 1, determining the storage queue as the concurrent message storage queue 1; when the hash value is 2, the storage queue is determined to be the concurrent message storage queue 2, and the like.
Each concurrent message store queue 2120 is used to store event messages sent by parsing module 212.
Specifically, after parsing the event message, the parsing module 212 distributes each event message to the corresponding concurrent message storage queues 2120 according to different parsing results, so that each concurrent message storage queue 2120 can store the event message matched with the concurrent message storage queue.
It should be noted that, the number of the concurrent message storage queues 2120 may be set in the configuration information by the user according to the actual situation. When the user needs to increase the throughput of messages, only the number of concurrent message store queues 2120 need be increased.
To facilitate understanding of the message concurrency processing module 212 in the present application, the following details are described in connection with fig. 4.
The message concurrency processing module, after obtaining the event message from the message middleware (Kafka, for example), distributes the event message to a different concurrency message storage queue (the memory queue in fig. 4), and then each message processing Thread (the Thread processing Thread in fig. 4) bound to the concurrency message storage queue obtains the event message from the message processing Thread.
The hotspot analysis unit 2122 is configured to analyze the event messages stored in each concurrent message storage queue 2120, and determine a hotspot event message according to the analysis result.
In particular, when hot events occur in the system, such as a large number of users gathering under the same video, the system may generate serious data tilt for such a scenario. In order to solve the above-mentioned problem, in this embodiment, a hotspot analysis unit 2122 is provided in the message processing platform to analyze the event messages stored in each concurrent message storage queue 2120, so that the hotspot event messages can be found in time, and thus, the hotspot event messages can be isolated from the common event messages later, and the hotspot event messages are prevented from affecting the processing of other event messages.
In an exemplary embodiment, the hotspot analysis unit 2122 is configured to analyze the occurrence frequency and the processing time consumption of each event message stored in each concurrent message storage queue, and determine a hotspot event message according to the analysis result.
Specifically, the hotspot analysis unit 2122 may analyze the occurrence frequency of each event message according to the key identification information in each event message. For example, if the event message is a praise message of a video, the number of occurrences of praise information having the same video ID is counted when the frequency of occurrence in the event message is analyzed.
In one embodiment, the hotspot analysis unit 2122 may also determine the occurrence frequency or the processing time of each event message separately when performing the analysis of the hotspot event message.
The hotspot transferring unit 2123 is configured to transfer the determined hotspot event message to a hotspot message storage partition in the message middleware.
Specifically, the hot spot message storage partition is a message storage partition for storing hot spot event messages, where the hot spot message storage partition may be preset, or may be selected according to actual situations, and is not limited in this embodiment.
In this embodiment, after determining which event messages belong to the hotspot event messages, the hotspot analyzing unit 2122 may immediately notify the hotspot transferring unit 2123 to transfer the hotspot event messages to the hotspot message storage partition. In an embodiment, when the concurrent message storage queue is found to be blocked, the hotspot event message may be transferred to the hotspot message storage partition through the hotspot analysis unit 2122.
In an exemplary embodiment, when transferring the determined hot event message, the hot spot transferring unit 2123 may transfer all the hot event messages to the hot spot message storage partition, or may determine whether to transfer only part of the hot event messages to the hot spot message storage partition according to the processing pressure (how much resources are consumed for transferring).
The event obtaining module 211 is further configured to obtain a hotspot event message from the hotspot message storage partition, and store the hotspot event message obtained from the hotspot message storage partition to the hotspot queue.
Specifically, after transferring the hot event message, the event obtaining module 211 further transfers the hot event message to an independent hot queue in time, so as to isolate the hot event message from the normal event message, and avoid the situation of data congestion.
The hotspot thread 2124 is configured to obtain a hotspot event message from the hotspot queue, and write the obtained hotspot event message into the database.
Specifically, after storing the hotspot event message in the hotspot queue, in order to enable the hotspot event message to be written into the database in time, the hotspot thread 2124 is started to consume the hotspot event message from the hotspot queue simultaneously, and the consumed hotspot event message is written into the database.
Each message processing thread 2121 is configured to obtain an event message from the corresponding concurrent message storage queue, and write the obtained non-hotspot event message into the database.
Specifically, each message processing thread 2121 binds to one concurrent message store queue 2120, such that each message processing thread 2121 can retrieve event messages from its corresponding concurrent message store queue 2120 and write the retrieved non-hotspot event messages to the database.
Wherein, the non-hot event message refers to a common event message.
In this embodiment, the message processing platform adds some queues under the condition that the partitions are kept unchanged, so that after receiving the event message, the event message stored in the same message storage partition can be distributed to different queues, and then the event message in the queue is processed by binding independent processing threads through each queue, thereby realizing the concurrent processing of the event message and improving the throughput of the message. In addition, in this embodiment, by analyzing the event messages stored in each queue in real time, the hot event message can be found, and after the hot event message is found, the hot event message is transferred to the hot queue, and the hot thread is synchronously started to write the hot event message in the hot queue into the database, so that the hot event message is isolated from other event messages, and the hot event message can be normally processed, thereby avoiding the phenomenon of data tilting.
In an exemplary embodiment, the message concurrency processing module 212 may further include a repayment unit, a repayment queue, and a repayment thread, wherein:
the repacking unit is used for repacking the event message with the writing failure into the repacking queue when the event message is detected to be written into the database with the writing failure.
Specifically, the reparse queue is a queue for storing event messages of write failures.
The repacking thread is used for acquiring the event message from the repacking queue after the time of repacking the failed event message to the repacking queue reaches the preset time, and writing the acquired event message into the database.
Specifically, the repayment thread is a thread for processing event eliminations stored in the repayment queue. The preset time may be preset, or may be set and adjusted according to actual situations, which is not limited in this embodiment. For example, the preset time is 1 minute.
In this embodiment, when writing of the event message into the database fails, the event message is redelivered, and the redelivered thread delays for a period of time and then rewrites the event message, so that all the event messages can be written into the database as much as possible.
In an exemplary embodiment, the consumer node 21 may further include a message processing mode determining module and a message aggregation processing module, where the message aggregation processing module includes a plurality of aggregate message storage queues, and a plurality of message aggregation threads, each message aggregation thread corresponds to one aggregate message storage queue one to one, where:
the message processing mode determining module is used for determining a message processing mode according to preset configuration information, wherein the message processing mode comprises a concurrent processing mode and an aggregation processing mode.
Specifically, the concurrent processing mode is a mode of processing the event message by adopting a message concurrent processing module, and the aggregation processing mode is a mode of processing the event message by adopting the message aggregation processing module.
Each aggregate message storage queue is used for storing event messages sent by the analysis module.
Specifically, the aggregate message storage queue is a queue for storing event messages sent by the parsing module in an aggregate processing mode.
Each message aggregation thread is used for acquiring event messages from the corresponding aggregation message storage queue, carrying out aggregation processing on the acquired event messages, and writing the aggregated event messages into the database in batches when the number of the aggregated event messages reaches a preset number or the aggregation time reaches a preset time.
Specifically, the message aggregation thread is a thread responsible for aggregating event messages stored in an aggregate message storage queue.
As an example, the preset number is 100, and the preset time is 5 minutes, the message aggregation thread may batch write the aggregated event messages into the database at a time when the aggregation time reaches 100 event messages, or when the aggregation time reaches 5 minutes.
In one embodiment, when the program is closed, the aggregation event can be processed and then exited, and the panic protection is performed, so that the possibility of process crash is avoided.
In this embodiment, by adding the message aggregation processing module, when processing the event message, the number of requests for writing into the database can be greatly reduced, and the performance of the message processing platform can be improved.
In order to facilitate understanding of the aggregation processing flow of the event message acquired from the message middleware by the message aggregation processing module in the present application, the following description will be given in detail with reference to fig. 5.
After the message aggregation processing module acquires the event message from the message middleware, the event message is analyzed, then the event message is distributed to different aggregation message storage queues according to the analysis result, then each message aggregation thread bound with the aggregation message storage queues acquires the event message from the event message aggregation thread, the acquired event message is subjected to aggregation processing, a plurality of event messages obtained by the aggregation processing can be placed in one or more data aggregation channels (channels), and finally, when the number or aggregation time of the event messages in each data aggregation channel reaches a preset time, the event messages are written into a database.
In an exemplary embodiment, the producer node 20 is further configured to detect whether a transmission channel between the producer node 20 and the message middleware is faulty, and if so, send the generated event message to the event acquisition module 211.
Specifically, in order to enable the event message generated by the producer node 20 to be still processed when the transmission channel between the producer node 20 and the message middleware fails, the producer node 20 directly sends the generated event message to the event acquisition module 211 through the API interface provided by the event acquisition module 211 when detecting that the transmission channel between the producer node 20 and the message middleware fails, and uses a plurality of concurrent message storage queues as buffers, so that the availability of a message link can be ensured when the transmission channel fails.
In this embodiment, if the transmission channel between the producer node 20 and the message middleware does not fail, the event message is sent to the target message storage partition corresponding to the type in the message middleware according to the type of the event message for storage.
In this embodiment, when a failure occurs in a transmission channel with a message middleware is detected, the message is automatically switched to an HTTP channel, directly pushed to a consumer, enters a queue of the consumer, and uses a ready-made multi-stage queue as a buffer, so that the availability of a message link in the event of a failure of a main channel is ensured.
In an exemplary embodiment, the message processing platform further comprises an acknowledgement character (ACK, acknowledge character) module, wherein:
the event obtaining module 211 is further configured to orderly concatenate the obtained event messages in a linked list manner, and add a processing status to each event message.
Specifically, in order to maintain the sequence of each event message, the event acquisition module 211 orderly concatenates each acquired event message according to the generation time thereof through a linked list, that is, forms the following linked list:
event message 1- > event message 2- > event message 3- > … - > event message N-1- > event message N.
It should be noted that the generation time of the event message N is later than the generation time of the event N-1.
The processing state is used to indicate whether the event message was processed, i.e. whether the event message was successfully written into the database.
In an embodiment, the processing states may include four states of unprocessed, processed, and processing failure.
The event obtaining module 211 is further configured to modify a processing state of the processed event message to processed after the event message is processed.
Specifically, the event acquisition module 211 marks the event message as processed in time after detecting that the event message has been successfully processed, so that it can be determined whether each event message is processed later according to the processing state.
The confirmation character module is used for detecting the processing state of each event message in the linked list at fixed time, and marking the position of the target event message as the position of a confirmation character (ACK) when the processing state of all the previous event messages of the target event message in the linked list is processed.
Specifically, a character position (ACK) position for realizing recovery of data upon occurrence of a failure is determined for recording an offset (offset address) of an event message successfully acquired from a message memory partition.
In this embodiment, the event messages are orderly concatenated in a linked list manner, and a processing flag is added to each event message, and after the message is processed, the message is marked as processed, the linked list state is checked periodically, and after all events in front of a certain message are processed, the message is set to confirm the character state, so that the problem of event loss is solved.
In order to facilitate understanding of the processing flow of event messages by the message processing platform in the present application, the following description is provided in detail in connection with fig. 7.
After the event message is obtained from the message middleware, when the processing mode of the event message is determined to be the concurrent processing mode, the event message is distributed to different queues (the memory queue 1-the memory queue N in fig. 7), and then the event message is obtained from the corresponding queue through threads (Thread 1-Thread N in fig. 7) bound with the queues. In addition, when the existence of the hot event message in the queue is detected, the hot event message in the queue is subjected to hot spot transfer so as to transfer the hot event message into the hot spot queue. When the writing of the event message into the database fails, the event message is written into a retry queue so as to realize the repayment of the event message. When the processing mode of the event message is determined to be the aggregation processing mode, the event message stored in the queue is subjected to aggregation processing, and after the aggregation processing is carried out to a certain amount or a certain time, the aggregated event message is subjected to batch processing through threads.
In an exemplary embodiment, referring to fig. 3, the method includes:
Step S30, analyzing event messages stored in a plurality of concurrent message storage queues, and determining hot event messages according to analysis results;
in an exemplary embodiment, analyzing event messages stored in a plurality of concurrent message storage queues and determining hot event messages based on the analysis results includes: analyzing the occurrence frequency and the processing time consumption of each event message stored in each concurrent message storage queue, and determining the hot event message according to the analysis result.
In a specific scenario, referring to fig. 5, when determining a hotspot event message, statistics is performed on the occurrence number and the processing time consumption of the event message stored in each concurrent message storage queue according to key identification information (such as key in fig. 5) in each event message, so as to obtain the total occurrence number and the total processing time consumption of each message event, and then, the hotspot event message can be determined according to the statistics information.
Step S31, transferring the determined hot spot event information to a hot spot information storage partition in the information middleware;
step S32, obtaining a hot spot event message from the hot spot message storage partition, and storing the hot spot event message obtained from the hot spot message storage partition into the hot spot queue;
Step S33, obtaining a hot event message from the hot queue, and writing the hot event message obtained from the hot queue into the database.
In an exemplary embodiment, the method further comprises:
when the failure of writing the event message into the database is detected, re-delivering the event message with the failure of writing into the re-delivery queue;
and after the time for re-delivering the failed event message to the re-delivery queue reaches the preset time, acquiring the event message from the re-delivery queue, and writing the acquired event message into the database.
In an exemplary embodiment, the method further comprises:
acquiring event messages from a concurrent message storage queue;
and carrying out aggregation processing on the acquired event messages, and writing the aggregated event messages into the database in batches when the number of the aggregated event messages reaches a preset number or the aggregation time reaches a preset time.
Specifically, in order to reduce the number of requests for writing into the database and improve the performance of the message processing platform, when the event messages stored in the concurrent message storage queue are written into the database, an independent thread is used for event aggregation, and after the event aggregation reaches a certain number or time, the events are packaged in batches, and the database is updated once.
Referring to FIG. 8, a block diagram of a message processing apparatus 50 according to an embodiment of the present application is shown.
In this embodiment, the message processing device 50 includes a series of computer program instructions stored on a memory, which when executed by a processor, can implement the photographing functions of the embodiments of the present application. In some embodiments, based on the particular operations implemented by the portions of the computer program instructions, the message processing apparatus 50 may be divided into one or more modules, which may be specifically divided as follows:
the analysis module 51 is configured to analyze event messages stored in the multiple concurrent message storage queues, and determine a hot event message according to an analysis result;
a transferring module 52, configured to transfer the determined hot event message to a hot message storage partition in the message middleware;
a storage module 53, configured to obtain a hot event message from the hot message storage partition, and store the hot event message obtained from the hot message storage partition into the hot queue;
and the writing module 54 is configured to acquire a hotspot event message from the hotspot queue, and write the hotspot event message acquired from the hotspot queue into the database.
In an exemplary embodiment, the analysis module 51 is further configured to analyze the occurrence frequency and the processing time consumption of each event message stored in each concurrent message storage queue, and determine a hot event message according to the analysis result.
Fig. 9 schematically shows a hardware architecture diagram of a computer device 6 adapted to implement a message processing method according to an embodiment of the application. In the present embodiment, the computer device 6 is a device capable of automatically performing numerical calculation and/or information processing in accordance with instructions set or stored in advance. For example, the server may be a tablet computer, a notebook computer, a desktop computer, a rack-mounted server, a blade server, a tower server, or a rack server (including a stand-alone server or a server cluster formed by a plurality of servers), etc. As shown in fig. 9, the computer device 6 includes at least, but is not limited to: memory 120, processor 121, and network interface 122 may be communicatively linked to each other by a system bus. Wherein:
memory 120 includes at least one type of computer-readable storage medium that may be volatile or nonvolatile, and specifically, readable storage media include flash memory, hard disk, multimedia card, card memory (e.g., SD or DX memory, etc.), random Access Memory (RAM), static Random Access Memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 120 may be an internal storage module of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 120 may also be an external storage device of the computer device 6, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like, which are provided on the computer device 6. Of course, the memory 120 may also include both internal memory modules of the computer device 6 and external memory devices. In this embodiment, the memory 120 is typically used to store an operating system installed on the computer device 6 and various types of application software, such as program codes of a message processing method, and the like. In addition, the memory 120 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 121 may be a central processing unit (Central Processing Unit, simply CPU), controller, microcontroller, microprocessor, or other camera chip in some embodiments. The processor 121 is typically used to control the overall operation of the computer device 6, such as performing control and processing related to data interaction or communication with the computer device 6, and the like. In this embodiment, the processor 121 is configured to execute program codes or process data stored in the memory 120.
The network interface 122 may comprise a wireless network interface or a wired network interface, the network interface 122 typically being used to establish a communication link between the computer device 6 and other computer devices. For example, the network interface 122 is used to connect the computer device 6 with an external terminal through a network, establish a data transmission channel and a communication link between the computer device 6 and the external terminal, and the like. The network may be a wireless or wired network such as an Intranet (Intranet), the Internet (Internet), a global system for mobile communications (Global System of Mobile communication, abbreviated as GSM), wideband code division multiple access (Wideband Code Division Multiple Access, abbreviated as WCDMA), a 4G network, a 5G network, bluetooth (Bluetooth), wi-Fi, etc.
It should be noted that fig. 9 only shows a computer device having components 120-122, but it should be understood that not all of the illustrated components are required to be implemented, and that more or fewer components may be implemented instead.
In the present embodiment, the message processing method stored in the memory 120 may be divided into one or more program modules and executed by one or more processors (the processor 121 in the present embodiment) to complete the present application.
The present application provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the message processing method of the embodiments.
In this embodiment, the computer-readable storage medium includes a flash memory, a hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the computer readable storage medium may be an internal storage unit of a computer device, such as a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium may also be an external storage device of a computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card), etc. that are provided on the computer device. Of course, the computer-readable storage medium may also include both internal storage units of a computer device and external storage devices. In this embodiment, the computer-readable storage medium is typically used to store an operating system installed on a computer device and various types of application software, such as program codes of the message processing method in the embodiment, and the like. Furthermore, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.
The apparatus embodiments described above are merely illustrative, wherein elements illustrated as separate elements may or may not be physically separate, and elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over at least two network elements. Some or all modules in the system can be screened out according to actual needs to realize the purpose of the scheme of the embodiment of the application. Those of ordinary skill in the art will understand and implement the present application without undue burden.
From the above description of embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus a general purpose hardware platform, or may be implemented by hardware. Those skilled in the art will appreciate that all or part of the processes implementing the methods of the above embodiments may be implemented by a computer program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and where the program may include processes implementing the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-only memory (ROM), a random access memory (RandomAccessMemory, RAM), or the like.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the application.

Claims (11)

1. The message processing platform comprises a producer node and a consumer node, and is characterized in that the consumer node comprises a message middleware, an event acquisition module, an analysis module and a message concurrency processing module, the message middleware comprises a plurality of message storage partitions, the message concurrency processing module comprises a plurality of concurrent message storage queues, a plurality of message processing threads, a hot spot analysis unit, a hot spot transfer unit, a hot spot queue and hot spot threads, and each message processing thread corresponds to one concurrent message storage queue one by one, wherein:
the producer node is used for generating an event message, and sending the event message to a target message storage partition corresponding to the type in the message middleware for storage according to the type of the event message;
The event acquisition module is used for acquiring event messages from the target message storage partition;
the analysis module is used for analyzing the acquired event message to obtain an analysis result, and distributing the acquired event message to a corresponding concurrent message storage queue according to the analysis result;
each concurrent message storage queue is used for storing event messages sent by the analysis module;
the hot spot analysis unit is used for analyzing the event messages stored in each concurrent message storage queue and determining hot spot event messages according to analysis results;
the hotspot transferring unit is used for transferring the determined hotspot event message to a hotspot message storage partition in the message middleware;
the event acquisition module is further configured to acquire a hotspot event message from the hotspot message storage partition, and store the hotspot event message acquired from the hotspot message storage partition into the hotspot queue;
the hot spot thread is used for acquiring hot spot event messages from the hot spot queue and writing the hot spot event messages acquired from the hot spot queue into the database;
each message processing thread is used for acquiring event messages from the corresponding concurrent message storage queues and writing the acquired non-hot event messages into the database.
2. The message processing platform according to claim 1, wherein the hotspot analysis unit is further configured to analyze occurrence frequencies and processing time consumption of each event message stored in each concurrent message storage queue, and determine a hotspot event message according to an analysis result.
3. The message processing platform of claim 1, wherein the message concurrency processing module further comprises a repayment unit, a repayment queue, and a repayment thread, wherein:
the repacking unit is used for repacking the event message with the writing failure into the repacking queue when the event message is detected to be written into the database with the writing failure;
the repacking thread is used for acquiring the event message from the repacking queue after the time of repacking the failed event message to the repacking queue reaches the preset time, and writing the acquired event message into the database.
4. A message processing platform according to any one of claims 1 to 3, wherein the consumer node further comprises a message processing mode determining module and a message aggregation processing module, the message aggregation processing module comprising a plurality of aggregated message storage queues, a plurality of message aggregation threads, each message aggregation thread being in a one-to-one correspondence with one aggregated message storage queue, wherein:
The message processing mode determining module is used for determining a message processing mode according to preset configuration information, wherein the message processing mode comprises a concurrent processing mode and an aggregation processing mode;
each aggregation message storage queue is used for storing event messages sent by the analysis module;
each message aggregation thread is used for acquiring event messages from the corresponding aggregation message storage queue, carrying out aggregation processing on the acquired event messages, and writing the aggregated event messages into the database in batches when the number of the aggregated event messages reaches a preset number or the aggregation time reaches a preset time.
5. The message processing platform of claim 1, wherein the producer node is further configured to detect whether a transmission channel between the producer node and the message middleware is faulty, and if so, send the generated event message to the event acquisition module.
6. The message processing platform of claim 5, further comprising a validation character module, wherein:
the event acquisition module is further used for orderly connecting acquired event messages in series in a linked list mode and adding a processing state for each event message;
The event acquisition module is further used for modifying the processing state of the processed event message into processed event message after the event message is processed;
the confirmation character module is used for detecting the processing state of each event message in the linked list at fixed time, and marking the position of the target event message as a confirmation character position when the processing state of all the previous event messages of the target event message in the linked list is processed.
7. A message processing method applied to the message processing platform according to any one of claims 1 to 6, characterized in that the method comprises:
analyzing event messages stored in a plurality of concurrent message storage queues, and determining hot event messages according to analysis results;
transferring the determined hot event message to a hot message storage partition in the message middleware;
acquiring a hot spot event message from the hot spot message storage partition, and storing the hot spot event message acquired from the hot spot message storage partition into the hot spot queue;
and acquiring the hot event message from the hot queue, and writing the hot event message acquired from the hot queue into a database.
8. The message processing method as claimed in claim 7, wherein analyzing the event messages stored in the plurality of concurrent message storage queues and determining the hot event message according to the analysis result comprises:
analyzing the occurrence frequency and the processing time consumption of each event message stored in each concurrent message storage queue to obtain a hot spot analysis result;
and determining a hot spot event message according to the hot spot analysis result.
9. A message processing apparatus, comprising:
the analysis module is used for analyzing the event messages stored in the concurrent message storage queues and determining hot event messages according to analysis results;
the transfer module is used for transferring the determined hot event message to a hot message storage partition in the message middleware;
the storage module is used for acquiring the hot spot event message from the hot spot message storage partition and storing the hot spot event message acquired from the hot spot message storage partition into the hot spot queue;
and the writing module is used for acquiring the hot event message from the hot queue and writing the hot event message acquired from the hot queue into a database.
10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method of claim 7 or 8 when the computer program is executed.
11. A computer-readable storage medium having stored thereon a computer program, characterized by: which computer program, when being executed by a processor, carries out the steps of the method according to claim 7 or 8.
CN202310891508.4A 2023-07-19 2023-07-19 Message processing method, device, platform, computer equipment and storage medium Pending CN116893910A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310891508.4A CN116893910A (en) 2023-07-19 2023-07-19 Message processing method, device, platform, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310891508.4A CN116893910A (en) 2023-07-19 2023-07-19 Message processing method, device, platform, computer equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116893910A true CN116893910A (en) 2023-10-17

Family

ID=88311867

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310891508.4A Pending CN116893910A (en) 2023-07-19 2023-07-19 Message processing method, device, platform, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116893910A (en)

Similar Documents

Publication Publication Date Title
US9888048B1 (en) Supporting millions of parallel light weight data streams in a distributed system
EP2998863B1 (en) Converting a serial transaction schedule to a parallel transaction schedule
KR101963917B1 (en) Automatic synchronization of most recently used document lists
CN113067883B (en) Data transmission method, device, computer equipment and storage medium
US20110066676A1 (en) Method and system for reducing web page download time
CN111555963A (en) Message pushing method and device, electronic equipment and storage medium
CN112822260B (en) File transmission method and device, electronic equipment and storage medium
CN111010364B (en) System for offline object-based storage and simulation of REST responses
CN103324713B (en) Data processing method and device in multistage server and data processing system
US10123190B2 (en) System and method for testing a CoAP server
CN107357526B (en) Method and apparatus for processing network data, server, and storage medium
CN116743619B (en) Network service testing method, device, equipment and storage medium
US20210365520A1 (en) Webpage loading method, intermediate server and webpage loading system
JP6364727B2 (en) Information processing system, distributed processing method, and program
US11190301B2 (en) Method and device for setting out-of-order value of network
CN111078975B (en) Multi-node incremental data acquisition system and acquisition method
CN112527519A (en) High-performance local cache method, system, equipment and medium
US10628242B1 (en) Message stream processor microbatching
CN109284257B (en) Log writing method and device, electronic equipment and storage medium
CN116893910A (en) Message processing method, device, platform, computer equipment and storage medium
US20150331917A1 (en) Recording medium having stored therein transmission order determination program, transmission order determination device, and transmission order determination method
CN111835801A (en) File downloading method, device, server, edge device, terminal and medium
CN113282347B (en) Plug-in operation method, device, equipment and storage medium
EP3408989B1 (en) Detecting malware on spdy connections
US10819622B2 (en) Batch checkpointing for inter-stream messaging system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination