CN112615773B - Message processing method and system - Google Patents
Message processing method and system Download PDFInfo
- Publication number
- CN112615773B CN112615773B CN202011399106.5A CN202011399106A CN112615773B CN 112615773 B CN112615773 B CN 112615773B CN 202011399106 A CN202011399106 A CN 202011399106A CN 112615773 B CN112615773 B CN 112615773B
- Authority
- CN
- China
- Prior art keywords
- message
- processing
- information
- messages
- processed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/07—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail characterised by the inclusion of specific contents
- H04L51/18—Commands or executable codes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/06—Message adaptation to terminal or network requirements
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a message processing method, which comprises the following steps: acquiring message data streams from a publish-subscribe system, grouping the acquired message data streams according to time and then aggregating the grouped message data streams into a plurality of elastic distributed data sets, wherein the message data streams are packaged and stored in the publish-subscribe system in a serialized manner; acquiring a plurality of messages included in an elastic distributed data set, and respectively acquiring predefined rules to process the messages according to identification information of message headers of the messages; when the message type is a calculation type, pulling a plurality of processed messages in each processed elastic distributed data set each time, and further processing the pulled messages according to a predefined rule; and respectively forwarding the processed messages to corresponding service systems, creating message retransmission tasks according to the information returned by the plurality of service systems, and regularly sending the messages in the message retransmission tasks to the corresponding service systems. The invention also discloses corresponding computing equipment and a storage medium.
Description
Technical Field
The present invention relates to the field of information processing technologies, and in particular, to a method and a system for processing a message.
Background
With the wide application of big data technology in various industries, the real-time processing of mass data is increasingly required. In the method for realizing real-time processing of the message in the prior art, one mode is that the local service directly and synchronously calls interfaces of other services, and the subsequent service processing is carried out after the interface returns a result. The other mode is that the local service sends a message to a message bus, the message bus performs uniform processing and forwarding, and the format, forwarding rule and error or overtime processing mechanism of the message.
Disclosure of Invention
To this end, the present invention provides a network request processing method, computing device and storage medium in an attempt to solve, or at least alleviate, at least one of the problems identified above.
According to an aspect of the present invention, there is provided a message processing method, including the steps of: acquiring message data streams from a publish-subscribe system, grouping the acquired message data streams according to time and then aggregating the grouped message data streams into a plurality of elastic distributed data sets, wherein the message data streams are packaged and stored in the publish-subscribe system in a serialized manner; for each elastic distributed data set, acquiring a plurality of messages included in the elastic distributed data set, and respectively acquiring predefined rules to process the messages according to identification information of message headers of the messages; when the message type is a calculation type, pulling a plurality of processed messages in each elastic distributed data set for the processed elastic distributed data sets each time, and further processing the pulled messages according to a predefined rule; and respectively forwarding the processed messages to corresponding service systems, creating message retransmission tasks according to the information returned by the plurality of service systems, and regularly sending the messages in the message retransmission tasks to the corresponding service systems.
Optionally, in the message processing method according to the present invention, aggregating the obtained message data packets into a plurality of elastic distributed data sets according to time includes: dividing the serialized message data stream into a plurality of groups by taking a preset time slice as a unit, performing deserialization processing on the head of the grouped message data stream, and aggregating the processed data into an elastic distributed data set.
Optionally, in the message processing method according to the present invention, the step of respectively obtaining predefined rules to process the multiple messages according to the identification information of the message headers of the multiple messages includes: if the identification information of the message head comprises rule information, deserializing the message body of the message, acquiring a corresponding predefined rule according to the identification information, and processing the deserialized message.
Optionally, in the message processing method according to the present invention, the types of the information returned by the service system include: processing success information, processing failure information, and timeout information.
Optionally, in the message processing method according to the present invention, before forwarding the finally processed message to the corresponding service system, the method further includes the steps of: and storing the finally processed message in a local cache.
Optionally, in the message processing method according to the present invention, a retransmission task is created according to information returned by a service system, and a message in the message retransmission task is sent to the corresponding service system at regular time, including the steps of: and if the processing success information is received, deleting the corresponding information in the local cache.
Optionally, in the message processing method according to the present invention, a retransmission task is created according to information returned by a service system, and a message in the message retransmission task is sent to the corresponding service system at regular time, including the steps of: if receiving the processing failure and overtime message, establishing a failure and overtime message retransmission task, and acquiring the corresponding message in the local cache for retransmission.
Optionally, in the message processing method according to the present invention, the predefined rule includes: the data mapping rule is suitable for mapping the key value pair in the deserialized message body according to a mapping table in the configuration information; a message behavior rule adapted to send messages to a specified address according to information in the deserialized message header and message body; a message merge rule adapted to merge messages.
Optionally, in the message processing method according to the present invention, after creating the retransmission task according to the information returned by the service system, the method further includes the steps of: and if the number of the processing failure information returned by the service system reaches a threshold value, acquiring the corresponding information according to the processing failure information, and forwarding the processing failure information to the responsible person corresponding to the information.
According to another aspect of the present invention, there is provided a message processing and distribution system, comprising: the publishing and subscribing system is suitable for storing the message data stream which is packaged and serialized and sending the message data stream to the message processing system in real time; the message processing system is suitable for executing the message processing method to process the received message data stream and respectively forwarding the processed messages to the corresponding service systems; and a plurality of service systems are provided, which are suitable for respectively receiving the messages from the message processing system, recording the processing success information, the processing failure information and the overtime information of the messages in the service systems, and returning the information to the message processing system.
Optionally, in the message processing and distributing system according to the present invention, the service system uses the SDK to record the processing success information, the processing failure information, and the timeout information of the message in the service system.
According to yet another aspect of the invention, there is provided a computing device comprising at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions comprising instructions for performing the message processing method according to the invention.
According to still another aspect of the present invention, there is provided a readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to execute the network request processing method of the present invention.
According to the technical scheme of the invention, message data are packaged and stored in a publishing and subscribing system in a serialized mode to form a message data stream, the message data stream is divided into a plurality of batches according to time, each batch of messages are processed according to a predefined rule, messages of a non-calculation type are directly forwarded to a corresponding service party, when the message types are calculation types such as needing to be combined, messages of a fixed size in the processed messages of each batch are pulled together for further processing each time, the processed messages are respectively forwarded to corresponding service systems, message retransmission tasks are established according to information returned by the service systems, and messages in the message retransmission tasks are sent to the corresponding service systems at regular time. The issuing and subscribing system is utilized to avoid the generation of a flow peak, so that a subsequent system can stably process message data, each batch of acquired messages are respectively processed according to a defined rule, whether further combination processing with other batches of messages is needed or not is judged according to the message types, the messages are summarized and then distributed while the processing capacity of the system is considered, the messages are directly forwarded to a corresponding service system instead of the message types, and the parallelism of message processing is improved to utilize computing resources to the minimum.
Furthermore, after the message is sent, the current thread is immediately finished without waiting for the completion of the execution of the service system, the service system records message processing success information, processing failure information and overtime information, and then establishes a retransmission task according to the returned data of the service system to avoid message errors, so that the performance consumption is dispersed to each service system for processing, and the method has the characteristics of large data processing amount, high concurrency, easiness in elastic expansion and high fault tolerance.
Drawings
To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings, which are indicative of various ways in which the principles disclosed herein may be practiced, and all aspects and equivalents thereof are intended to be within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description read in conjunction with the accompanying drawings. Throughout this disclosure, like reference numerals generally refer to like parts or elements.
FIG. 1 shows a schematic diagram of a message processing and distribution system 100 according to one embodiment of the invention;
FIG. 2 shows a schematic diagram of a computing device 200 according to an embodiment of the invention;
fig. 3 shows a flow diagram of a message processing method 300 according to one embodiment of the invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Fig. 1 shows a schematic diagram of a message processing and distribution system 100 according to one embodiment of the invention. As shown in fig. 1, the message processing system 100 includes: a publish-subscribe system 110, a message processing system 120, and a business system 130. With the continuous development of network technologies, a great deal of message data is generated when online behaviors such as online shopping, online games, financial transactions and the like occur. The client side, as a producer side of the message data, needs to store the produced message data into the publish-subscribe system 110, and the business system 130, as a consumer side, needs to use the message processing system 120 to read the business data from the publish-subscribe system and analyze and monitor the data.
Specifically, the publish-subscribe system 110 is adapted to store the message data stream through encapsulation and serialization, and send the message data stream to the message processing system 120 in real time, so as to avoid the generation of traffic peak by using the publish-subscribe system, and enable the subsequent system to smoothly process the message data; the message processing system 120 is adapted to execute a message processing method as described below to process a received message data stream, forward the processed messages to the corresponding service systems 130, create message retransmission tasks according to information returned by the plurality of service systems 130, and periodically send messages in the message retransmission tasks to the corresponding service systems 130; the plurality of service systems 130 are adapted to receive the messages from the message processing system 120, record the processing success information, the processing failure information and the timeout information of the messages in the service system 130, and return the messages to the message processing system 120. According to one embodiment of the present invention, the publish-subscribe system is implemented as Kafka, a high-throughput distributed publish-subscribe messaging system that handles all the action flow data of consumers in a web site, and is used in this scenario to store producer-generated messages. The message processing system is implemented as Spark, a fast, general-purpose computing engine designed specifically for large-scale data processing. Spark Streaming is a framework built on Spark to process Streaming data, and the basic principle is to divide the Streaming data into small time slices (several seconds) and process the small portions of data in a batch process. The processing response time of the spark streaming is generally in minutes, and the delay time when processing real-time data is in seconds, so that the current streaming processing system that can be implemented based on the spark streaming framework can be used to implement real-time processing of message data. And the subsequent processing of the streaming data realizes the functions of tracking the client interests in real time, recommending reasonable contents, refreshing a client list and the like.
In the technical solution according to the present invention, for each batch of acquired messages, the message processing system 120 processes the messages according to the defined rules, and then determines whether further merging with other batches of messages is needed or not according to the message type, and the like, and collects and then distributes the messages while considering the system processing capability, instead of directly forwarding the message types to the corresponding service systems, so as to improve the parallelism of message processing to minimize the utilization of computing resources. Further, after sending the message, the current thread is immediately ended without waiting for the completion of the execution of the service system 130, the service system 130 records the message processing success information, the processing failure information and the timeout information, and then the message processing system 120 establishes a retransmission task according to the return data of the service system 130 to avoid the message error, disperses the performance consumption to each service system 130 for processing, and has the characteristics of large data processing amount, high concurrency, easy elastic expansion and high fault tolerance.
The plurality of business systems 130, which are the consumers of the messages, are adapted to be respectively integrated into a plurality of computing devices 200, and fig. 2 shows a block diagram of the computing device 200 according to an embodiment of the present invention. In a basic configuration 202, the computing device 200 typically includes a system memory 206 and one or more processors 204. A memory bus 208 may be used for communication between the processor 204 and the system memory 206.
Depending on the desired configuration, the processor 204 may be any type of processing, including but not limited to: a microprocessor (μ P), a microcontroller (μ C), a digital information processor (DSP), or any combination thereof. The processor 204 may include one or more levels of cache, such as a level one cache 210 and a level two cache 212, processor cores ALU/FPU/DSP214, and registers 216. Example processor cores 214 may include Arithmetic Logic Units (ALUs), floating Point Units (FPUs), digital signal processing cores (DSP cores), or any combination thereof. The example memory controller 218 may be used with the processor 204, or in some implementations the memory controller 218 may be an internal part of the processor 204.
Depending on the desired configuration, system memory 206 may be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 206 may include an operating system 220, one or more programs 222, data 224, and program data 226. In some embodiments, program 222 may be arranged to operate with program data 226 on an operating system. According to one embodiment of the invention, program data 226 includes an SDK (Software Development Kit), which is generally a collection of Development tools used by some Software engineers to build application Software for a particular Software package, software framework, hardware platform, operating system, and the like. The computing device 200 integrated with a service system implements subsequent processing of processing the message from the message processing system 120 by using the SDK, records processing success information, processing failure information, and timeout information of the message in the service system, and notifies the result to the message processing system 120 as described above after the processing. After distributing tasks to the service systems 130 configured with the SDK, the message processing system 120 establishes a retransmission task according to the returned data of the service systems 130 to avoid message errors, and disperses the performance consumption to each service system 130 for processing, so that the scheme has the characteristics of large data processing amount, high concurrency, easy elastic expansion, and high fault tolerance.
Computing device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to the basic configuration 102 via the bus/interface controller 230. The example output device 242 includes a graphics processing unit 248 and an audio processing unit 250. They may be configured to facilitate communication with various external devices, such as a display or speakers, via one or more a/V ports 252. Example peripheral interfaces 244 may include a serial interface controller 254 and a parallel interface controller 256, which may be configured to facilitate communication with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device) or other peripherals (e.g., printer, scanner, etc.) via one or more I/O ports 258. An example communication device 246 may include a network controller 260, which may be arranged to facilitate communications with one or more other computing devices 262 over a network communication link via one or more communication ports 264.
The computing device 200 may also include a storage interface bus 234 and a storage device 232 coupled to the storage interface bus 234. The storage device 232 includes removable storage 236 and non-removable storage 238.
A network communication link may be one example of a communication medium. Communication media may typically be embodied by computer readable instructions, data structures, program modules, and may include any information delivery media, such as carrier waves or other transport mechanisms, in a modulated data signal. A "modulated data signal" may be a signal that has one or more of its data set or its changes in such a manner as to encode information in the signal. By way of non-limiting example, communication media may include wired media such as a wired network or private-wired network, and various wireless media such as acoustic, radio Frequency (RF), microwave, infrared (IR), or other wireless media. The term computer readable media as used herein may include both storage media and communication media. In some embodiments, one or more programs are stored in a computer readable medium, the one or more programs including instructions for performing certain methods.
The computing device 200 is installed with a mobile APP or client application supporting network file transmission and storage, including a native application or a browser such as IE, chrome, and Firefox, or a wechat, QQ, and other communication software, and stores various files locally, such as photos, audio, video, and documents (e.g., documents in Word, PDF, and other formats). The application client may run on an operating system such as Windows, macOS, android, and the like. Computing device 200 may be implemented as a portion of a small-form factor portable (or mobile) electronic device such as a cellular telephone, a digital camera, a Personal Digital Assistant (PDA), a personal media player device, a wireless web-watch device, a personal headset device, an application specific device, or a hybrid device that include any of the above functions.
Fig. 3 illustrates a flow diagram of a message processing method 300 according to one embodiment of the invention, the method 300 being suitable for execution in a message processing system, such as the message processing system 120 described above. As shown in fig. 3, the message processing method starts in step S310.
In step S310, a message data stream is obtained from the publish-subscribe system, and the obtained message data streams are grouped and aggregated into a plurality of elastic distributed data sets according to time. Wherein, the message data flow is stored to the publish-subscribe system by packaging and serialization. In a publish-subscribe system, publishers (message producers) sort messages in some way and subscribers (message consumers) subscribe to them in order to receive messages of a particular type. The client end is used as the producer end of the message data, packages the message into a message header and a message body according to a specified format, and sends the message to the publish-subscribe system according to a specified protocol.
Specifically, in the packaging process, the producer needs to convert the state information of the message object into a form that can be stored or transmitted by using the sequencer, so as to send the state information to the publish-subscribe system through the network. After the message data stream is acquired by the publish-subscribe system, the message needs to be processed, such as disassembled, calculated, and reassembled into a new message, so as to be forwarded to the specified service system. The publish-subscribe system can adopt Kafka, a message producer utilizes a Protobuf tool to serialize encapsulated message data and store the encapsulated message data into the Kafka, a topic is a basic unit of Kafka data write operation, one topic comprises a plurality of partitions, each message only belongs to one partition, the message producer needs to specify to which topic the message is published when publishing data, and a message consumer, namely a service system, needs to specify to subscribe to the message of which topic so as to receive the message of the specific type. The method 300 is implemented as a process of message processing from publication to subscription to each topic, with the message producer and publisher specified by the publish-subscribe system.
According to an embodiment of the present invention, in order to improve efficiency, aggregating acquired message data into a plurality of elastic distributed data sets according to time includes the steps of: dividing the serialized message data stream into a plurality of groups by taking a preset time slice as a unit, performing deserialization processing on the head of the grouped message data stream, and aggregating the processed data into an elastic distributed data set. That is, these large message data streams are aggregated into RDD (elastic distributed data set) according to a custom time slice, and an RDD object containing message data is obtained. The step of grouping may be performed on Spark Streaming, which takes the message stream data from Kafka in a Direct fashion into small time slices (a few seconds) and processes this small portion of data in a batch process, with delay times on the order of seconds when processing real-time data. During serialization, the object writes its current state to temporary or persistent storage, deserializes into a process of reading data from storage, recreating the object, where only the message header needs to be deserialized since the message header and message body are serialized separately. Further, the number of partitions is generally equal to the number of processing nodes in the message processing process, and the number of processing nodes in the scheme is tens of times of the number of partitions in each topic of the publish-subscribe system, so that the resource utilization is maximized, and the message throughput is improved.
Next, in step S320, for each elastic distributed data set, a plurality of messages included in the elastic distributed data set are obtained, and according to identification information of message headers of the plurality of messages, predefined rules are respectively obtained to process the plurality of messages. For each RDD, the messages therein need to be processed, e.g., disassembled, calculated, and reassembled into new messages, before being forwarded to the designated service system.
According to an embodiment of the present invention, a predefined rule is respectively obtained according to identification information of message headers of a plurality of messages to process the plurality of messages, including the steps of: if the identification information of the message head comprises rule information, deserializing the message body of the message, acquiring a corresponding predefined rule according to the identification information, and processing the deserialized message. The message which does not need to be processed does not have a corresponding predefined rule in the head mark, and the message head after the deserialization can be directly forwarded to a corresponding service system at the moment, and the deserialization of the message body is processed by a real message consumer service system, so that the distribution rate is improved. Further, the predefined rules include:
the data mapping rule is suitable for mapping key values in the message body after the deserialization according to a mapping table in configuration information, for example, the message body is deserialized, and the obtained structured data { "username": nickname "}, key is the place where username is replaced by nickname, so that {" username ": key" } - > { "nickname": key "}, wherein the structured data is usually json, and can also be xml or protobuf and the like;
the message behavior rule is suitable for sending the message to a specified address according to the information in the message header and the message body after deserialization, specifically, the key of each message exists in the message header, the key also exists in the structured message body after deserialization, and the message is sent to the specified address (url or Ip port) after the address information in the two keys is matched;
the message merging rules are suitable for merging messages, for example, an input message has { "username": "key", "generator": "large", "age": 37"} {" username ":" yangkai "," generator r ":" large "," age ": 37" } { "username": "yangkai", "generator": "male", "age": 37"}, and the message is merged by using a rule { distinct (data.username, data.generator, data.age) }, resulting in {" username ":" key "," generator ":" large "," generator ": 37": male "," age ": 37": preventing a service system from receiving repeated messages. Other self-defined rules are not limited in the present invention, such as message counting, summing, splicing, etc., and are determined according to specific service scenarios.
Subsequently, in step S330, when the message type is a calculation type, for the processed elastic distributed data sets, pulling a plurality of processed messages in each elastic distributed data set each time, and further processing the pulled messages according to a predefined rule.
After the messages are processed in batches, part of the message data, for example, the head identification of the message data only corresponds to the mapping rule, and the message type of the message data is a non-calculation type, and the message data can be directly forwarded to a corresponding service system after mapping conversion is performed on the message. And for the message of which the identifier in the message header also corresponds to the message merging rule, the message type is a calculation type, and the message is also required to be merged with the messages in other elastic distributed data sets again so as to remove redundant messages under the theme. Therefore, the messages of the non-calculation type are directly forwarded to the corresponding service side, when the message type is the calculation type, for example, the messages of the fixed size in the messages after each batch of processing are jointly pulled each time for further processing, and the combination processing of a large amount of message data contained in the whole message flow can be realized without independently improving the performance of the processing nodes during combination. It should be noted that the message type being the calculation type not only refers to the message header identifying the message corresponding to the message merging rule, but also includes the rule corresponding to the message needing to be aggregated with other batches of messages, such as message counting, summing, splicing, and the like. The message type is directly forwarded to a corresponding service system, so that the parallelism of message processing is improved to minimally utilize computing resources.
Further, when the Spark is used to further process the calculation type message, the thread of each processing node has a buffer, and each time data with the same size as the buffer is pulled from the processed elastic distributed data set, then the aggregation and other operations are performed through a Map in the memory. And after a batch of data is aggregated, pulling down a batch of data, and so on until all the data are pulled out finally, and obtaining a final result. On one hand, the file quantity is obviously reduced, on the other hand, the size of a memory occupied by writing in the cache is reduced, and the real-time processing speed of the message data stream is improved.
Finally, in step S340, the processed messages are respectively forwarded to the corresponding service systems, a message retransmission task is created according to the information returned by the plurality of service systems, and the messages in the message retransmission task are sent to the corresponding service systems at regular time. Further, the processed message is stored in a local cache to cooperate with subsequent message retransmission when the message has an error. When the message type is a non-calculation type, the processed message refers to the message processed in step S320, and when the message type is a calculation type, the processed message is the message further processed in step S330.
After sending the message, the current thread is immediately ended without waiting for the completion of the execution of the service system, and the service system returns information on whether the message is successfully processed after processing the received message, according to an embodiment of the present invention, the types of the information returned by the service system include: processing success information, processing failure information, and timeout information. Further, the service system uses the SDK to record the processing success information, the processing failure information, and the timeout information of the message in the service system. SDKs are generally a collection of development tools used by some software engineers to build application software for a particular software package, software framework, hardware platform, operating system, etc. After distributing tasks to the service systems configured with the SDK, a retransmission task is established according to the returned data of the service systems to avoid message errors, and the performance consumption is dispersed to each service system for processing.
Specifically, at the service system end, the SDK receives the message and enters the local service system for processing. And the service system end adopts an automatic exception acquisition mechanism, and if no exception exists, the service of the service system end is correctly processed and completed. After the normal processing of the service, the SDK calls back the information of successful processing of the message. And if the information of successful processing is received, deleting the corresponding information in the local cache.
If the error occurs, the SDK returns the processing error information of the message. If receiving the processing failure and overtime information, establishing a failure and overtime information retransmission task, acquiring the corresponding information in the local cache for retransmission, and starting an information bus retransmission task after the failure of message forwarding. Therefore, when the message is unavailable due to network or other faults, the message can be received after the fault is recovered, and the integrity of the message is ensured.
According to another embodiment of the invention, if the number of the processing failure information returned by the service system reaches the threshold value, the corresponding information is obtained according to the processing failure information, and the processing failure information is forwarded to the responsible person corresponding to the information. In the whole process of message processing and forwarding of the message system, the operation of each link is recorded, and a data source is provided for inquiry, monitoring and alarm of a responsible person.
According to the technical scheme of the invention, message data are packaged and stored in a publishing and subscribing system in a serialized mode to form a message data stream, the message data stream is divided into a plurality of batches according to time, each batch of messages are processed according to a predefined rule, messages of a non-calculation type are directly forwarded to a corresponding service party, when the message types are calculation types such as needing to be combined, messages of a fixed size in the processed messages of each batch are pulled together for further processing each time, the processed messages are respectively forwarded to corresponding service systems, message retransmission tasks are established according to information returned by the service systems, and messages in the message retransmission tasks are sent to the corresponding service systems at regular time. The issuing and subscribing system is utilized to avoid the generation of a flow peak, so that a subsequent system can stably process message data, each batch of acquired messages are respectively processed according to a defined rule, whether further combination processing with other batches of messages is needed or not is judged according to the message types, the messages are summarized and then distributed while the processing capacity of the system is considered, the messages are directly forwarded to a corresponding service system instead of the message types, and the parallelism of message processing is improved to utilize computing resources to the minimum.
Furthermore, after the message is sent, the current thread is immediately ended without waiting for the completion of the execution of the service system, the service system records message processing success information, processing failure information and overtime information, and then establishes a retransmission task according to the returned data of the service system to avoid message errors, so that the performance consumption is dispersed to each service system for processing, and the method has the characteristics of large data processing amount, high concurrency, easiness in elastic expansion and high fault tolerance.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
A9, the method as described in A8, after the retransmission task is created according to the information returned by the service system, further including the steps of:
and if the number of the processing failure information returned by the service system reaches a threshold value, acquiring the corresponding information according to the processing failure information, and forwarding the processing failure information to the responsible person corresponding to the information.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: rather, the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in a device as described in this embodiment, or alternatively may be located in one or more devices different from the device in this example. The modules in the foregoing examples may be combined into one module or may be further divided into multiple sub-modules.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
Furthermore, some of the described embodiments are described herein as a method or combination of method elements that can be performed by a processor of a computer system or by other means of performing the described functions. A processor having the necessary instructions for carrying out the method or method elements thus forms a means for carrying out the method or method elements. Further, the elements of the apparatus embodiments described herein are examples of the following apparatus: the means for performing the functions performed by the elements for the purpose of carrying out the invention.
As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this description, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as disclosed herein. Furthermore, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the appended claims. The present invention has been disclosed with respect to the scope of the invention, which is to be considered as illustrative and not restrictive, and the scope of the invention is defined by the appended claims.
Claims (10)
1. A message processing method, comprising the steps of:
acquiring message data streams from a publish-subscribe system, grouping the acquired message data streams according to time and then aggregating the grouped message data streams into a plurality of elastic distributed data sets, wherein the message data streams are packaged and stored in the publish-subscribe system in a serialized manner;
for each elastic distributed data set, acquiring a plurality of messages included in the elastic distributed data set, and respectively acquiring predefined rules to process the messages according to identification information of message headers of the messages;
when the message type is a calculation type, pulling a plurality of processed messages in each elastic distributed data set for the processed elastic distributed data sets each time, and further processing the pulled messages according to a predefined rule;
respectively forwarding the processed messages to corresponding service systems, creating message retransmission tasks according to information returned by a plurality of service systems, and regularly sending messages in the message retransmission tasks to the corresponding service systems;
wherein, according to the identification information of the message headers of the plurality of messages, respectively obtaining a predefined rule to process the plurality of messages, comprising:
if the identification information of the message head comprises rule information, deserializing the message body of the message, acquiring a corresponding predefined rule according to the identification information, and processing the deserialized message;
further processing the pulled message according to predefined rules, including:
if the message type of the message is a non-calculation type, mapping and converting the message, and sending the message to a corresponding service system;
and if the message type of the message is not the calculation type, combining the message with the messages in other elastic distributed data sets.
2. The method of claim 1, said post-aggregating the retrieved message data packets into a plurality of elastically distributed data sets in time, comprising the steps of:
dividing the serialized message data stream into a plurality of groups by taking a preset time slice as a unit, performing deserialization processing on the head of the grouped message data stream, and aggregating the processed data into an elastic distributed data set.
3. The method of claim 1, wherein the type of information returned by the business system comprises: processing success information, processing failure information, and timeout information.
4. A method according to any of claims 1-3, further comprising, before forwarding the final processed message to the corresponding service system, the steps of:
and storing the finally processed message in a local cache.
5. The method according to claim 4, wherein a retransmission task is created according to the information returned by the service system, and the message in the message retransmission task is sent to the corresponding service system at regular time, comprising the steps of:
and if the information of successful processing is received, deleting the corresponding information in the local cache.
6. The method according to claim 5, wherein the creating of the retransmission task according to the information returned by the service system and the timed sending of the message in the message retransmission task to the corresponding service system comprise the steps of:
if receiving the processing failure and overtime message, establishing a failure and overtime message retransmission task, and acquiring the corresponding message in the local cache for retransmission.
7. The method of any of claims 1-3, the predefined rule comprising:
the data mapping rule is suitable for mapping the key value pair in the deserialized message body according to a mapping table in the configuration information;
a message behavior rule adapted to send a message to a specified address according to information in the deserialized message header and message body;
a message merge rule adapted to merge messages.
8. The method of claim 7, after creating the retransmission task according to the information returned by the service system, further comprising the steps of:
and if the number of the processing failure information returned by the service system reaches a threshold value, acquiring the corresponding information according to the processing failure information, and forwarding the processing failure information to the responsible person corresponding to the information.
9. A message processing and distribution system comprising:
the system comprises a publishing and subscribing system, a message processing system and a message processing system, wherein the publishing and subscribing system is suitable for storing a message data stream which is packaged and serialized and sending the message data stream to the message processing system in real time;
a message processing system, adapted to execute the message processing method according to any one of claims 1 to 8 to process the received message data stream and forward the processed messages to the corresponding service systems, respectively;
and a plurality of service systems which are suitable for respectively receiving the messages from the message processing system, recording the processing success information, the processing failure information and the overtime information of the messages in the service systems and returning the information to the message processing system.
10. The message processing and distribution system of claim 9, the service system using the SDK to implement recording of message processing success information, processing failure information, and timeout information at the service system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011399106.5A CN112615773B (en) | 2020-12-02 | 2020-12-02 | Message processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011399106.5A CN112615773B (en) | 2020-12-02 | 2020-12-02 | Message processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112615773A CN112615773A (en) | 2021-04-06 |
| CN112615773B true CN112615773B (en) | 2023-02-28 |
Family
ID=75228837
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011399106.5A Active CN112615773B (en) | 2020-12-02 | 2020-12-02 | Message processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112615773B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114257582A (en) * | 2021-12-17 | 2022-03-29 | 中国农业银行股份有限公司 | Batch job processing method, distributed system and batch job processing architecture |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106776855A (en) * | 2016-11-29 | 2017-05-31 | 上海轻维软件有限公司 | The processing method of Kafka data is read based on Spark Streaming |
| CN108921188A (en) * | 2018-05-23 | 2018-11-30 | 重庆邮电大学 | A kind of parallel C RF algorithm based on Spark big data platform |
| CN110647570A (en) * | 2019-09-20 | 2020-01-03 | 百度在线网络技术（北京）有限公司 | Data processing method and device and electronic equipment |
| CN110908788A (en) * | 2019-12-02 | 2020-03-24 | 北京锐安科技有限公司 | Spark Streaming based data processing method and device, computer equipment and storage medium |
| CN110912803A (en) * | 2019-11-14 | 2020-03-24 | 富途网络科技(深圳)有限公司 | IM message transmission method and terminal |
| CN111367953A (en) * | 2020-03-30 | 2020-07-03 | 中国建设银行股份有限公司 | Streaming processing method and device for information data |
| CN111756779A (en) * | 2019-03-26 | 2020-10-09 | 阿里巴巴集团控股有限公司 | Data serialization processing method, device and computer system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108427725B (en) * | 2018-02-11 | 2021-08-03 | 华为技术有限公司 | Data processing method, device and system |
-
2020
- 2020-12-02 CN CN202011399106.5A patent/CN112615773B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106776855A (en) * | 2016-11-29 | 2017-05-31 | 上海轻维软件有限公司 | The processing method of Kafka data is read based on Spark Streaming |
| CN108921188A (en) * | 2018-05-23 | 2018-11-30 | 重庆邮电大学 | A kind of parallel C RF algorithm based on Spark big data platform |
| CN111756779A (en) * | 2019-03-26 | 2020-10-09 | 阿里巴巴集团控股有限公司 | Data serialization processing method, device and computer system |
| CN110647570A (en) * | 2019-09-20 | 2020-01-03 | 百度在线网络技术（北京）有限公司 | Data processing method and device and electronic equipment |
| CN110912803A (en) * | 2019-11-14 | 2020-03-24 | 富途网络科技(深圳)有限公司 | IM message transmission method and terminal |
| CN110908788A (en) * | 2019-12-02 | 2020-03-24 | 北京锐安科技有限公司 | Spark Streaming based data processing method and device, computer equipment and storage medium |
| CN111367953A (en) * | 2020-03-30 | 2020-07-03 | 中国建设银行股份有限公司 | Streaming processing method and device for information data |
Non-Patent Citations (2)
| Title |
|---|
| "Performance Evaluation of Intrusion Detection Streaming Transactions Using Apache Kafka and Spark Streaming";MayThet Tun等;《IEEE》;20191205;全文 * |
| "基于弹性分布式数据集的流数据聚类分析";张媛;《CNKI》;20160916;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112615773A (en) | 2021-04-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210406068A1 (en) | Method and system for stream computation based on directed acyclic graph (dag) interaction | |
| WO2020147336A1 (en) | Micro-service full-link monitoring system and method | |
| CN112783672B (en) | Remote procedure call processing method and system | |
| CN109254854A (en) | Asynchronous invoking method, computer installation and storage medium | |
| CN111698233B (en) | Message processing method and device | |
| CN112615773B (en) | Message processing method and system | |
| CN111210340A (en) | Automatic task processing method and device, server and storage medium | |
| WO2023020242A1 (en) | Blockchain-based data processing method and apparatus, computer device, computer-readable storage medium, and computer program product | |
| CN111200606A (en) | Deep learning model task processing method, system, server and storage medium | |
| CN111338769B (en) | Data processing method, device and computer readable storage medium | |
| CN114615096A (en) | Telecommunication charging method, system and related equipment based on event-driven architecture | |
| CN115174472A (en) | Message forwarding processing method and related device | |
| CN107249019A (en) | Data handling system, method, device and server based on business | |
| CN104731564A (en) | Generation method and device for system interface call records | |
| CN103299298A (en) | Service processing method and system | |
| CN116009985A (en) | Interface calling method, device, computer equipment and storage medium | |
| CN114780353B (en) | File log monitoring method and system and computing device | |
| CN113992644B (en) | Internet of things system based on no-service technology and data processing method thereof | |
| CN115460265A (en) | Interface calling method, device, equipment and medium | |
| CN110865973A (en) | Data processing method and equipment and related device | |
| CN116701452A (en) | Data processing method, related device, storage medium and program product | |
| CN112417015A (en) | Data distribution method and device, storage medium and electronic device | |
| CN102253940B (en) | Method and device for processing data by tree view | |
| CN112433870A (en) | Data calling method and device, computer readable storage medium and electronic equipment | |
| CN112087373B (en) | Message sending method and service device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |