CN108509299B - Message processing method, device and computer readable storage medium - Google Patents
Message processing method, device and computer readable storage medium Download PDFInfo
- Publication number
- CN108509299B CN108509299B CN201810274276.7A CN201810274276A CN108509299B CN 108509299 B CN108509299 B CN 108509299B CN 201810274276 A CN201810274276 A CN 201810274276A CN 108509299 B CN108509299 B CN 108509299B
- Authority
- CN
- China
- Prior art keywords
- offset position
- consumer thread
- message
- message processing
- standby
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1415—Saving, restoring, recovering or retrying at system level
- G06F11/1417—Boot up procedures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/55—Push-based network services
Abstract
The invention discloses a message processing method, a device and a computer readable storage medium, wherein the method comprises the following steps: starting a standby consumer thread and a new consumer thread after the old consumer thread fails; the standby consumer thread acquires a first offset position from a preset cache and acquires a second offset position from a manager zookeeper; consuming the messages written by the server in non-real time according to the first offset position and the second offset position; and the new consumer thread consumes the messages written by the server in real time according to the second offset position. The invention consumes the non-real-time and real-time written messages of the server through the standby consumer thread and the new consumer thread; on the one hand, the latest messages can be guaranteed to be processed, the delay of message processing is reduced, and on the other hand, the messages during the failure of the consumer thread can be guaranteed to be processed in a supplementing mode and not lost.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a message processing method, device, and computer-readable storage medium.
Background
Kafka is a high throughput, distributed publish-subscribe messaging system that can handle all the action flow data in a consumer-sized web site. This action (web browsing, searching and other user actions) is a key factor in many social functions on modern networks. These data are typically addressed by handling logs and log aggregations due to throughput requirements. The purpose of Kafka is to unify online and offline message processing through the parallel loading mechanism of Hadoop (a distributed system infrastructure developed by the Apache foundation) and also to provide real-time consumption through a cluster machine.
The Kafka application is used for generating messages and consuming messages for large data amount data in a big data statistics process. Wherein, the production of the message is directly sent to the Kafka by the producer; the consumption of the message is performed by a consumer thread. Since in many production environments, a Kafka cluster must be used to eliminate single point failures, the consumer threads are all connected to the entire Kafka cluster to perform the consuming action of the message. Kafka's consumer thread number has a relationship with the topic's partition (there is a topic in a class of messages, and each topic is created, the partition number of the topic needs to be fixed). At a certain time, only one consumer thread can be connected to a partition.
The existing consumption mode has the following problems: when a large amount of messages are written and the consumer threads are down and stopped, restarting the consumer threads, and enabling the consumer threads to consume data from the positions recorded by the zookeeper in the kafka; if the downtime is long and more data to be consumed are accumulated, the processing of the non-real-time data takes time, which causes that the latest data cannot be consumed in real time, and the processing delay time is increased.
Disclosure of Invention
Embodiments of the present invention mainly aim to provide a message processing method, a message processing device, and a computer-readable storage medium, which are used to solve the problems of the existing consumption methods.
In order to achieve the above object, a first aspect of the embodiments of the present invention provides a message processing method, where the method includes:
starting a standby consumer thread and a new consumer thread after the old consumer thread fails;
the standby consumer thread acquires a first offset position from a preset cache and acquires a second offset position from a manager zookeeper; consuming the message written by the server in non-real time according to the first offset position and the second offset position; wherein the first offset position is an offset position after the old consumer thread consumes the server message;
and the new consumer thread consumes the messages written by the server in real time according to the second offset position.
Optionally, the consuming a message written by a server in non-real time according to the first offset position and the second offset position includes:
judging whether the first offset position and the second offset position are the same;
if the first offset position and the second offset position are not the same, the standby consumer thread consumes the message between the first offset position and the second offset position.
Optionally, the standby consumer thread consuming the message between the first offset position and the second offset position comprises the steps of:
the standby consumer thread initiates multithreading and consumes messages between the first offset position and the second offset position via the multithreading.
Optionally, if the first offset position and the second offset position are not the same, the method further includes, after the standby consumer thread consumes the message between the first offset position and the second offset position:
closing the standby consumer thread if the message between the first offset position and the second offset position has been consumed.
Optionally, the standby consumer thread acquires a first offset position from a preset cache and acquires a second offset position from a zookeeper; consuming the message written by the server in non-real time according to the first offset position and the second offset position, and comprising the following steps:
setting a second offset position in the zookeeper.
Optionally, the step of consuming, by the new consumer thread, the message written in real time by the server according to the second offset position further includes:
and if the new consumer thread is consumed completely, updating the first offset position in the preset cache.
Optionally, the starting the standby consumer thread and the new consumer thread after the old consumer thread fails comprises the steps of:
detecting whether the old consumer thread fails;
and if the old consumer thread fails, starting a standby consumer thread and a new consumer thread.
Optionally, the failure comprises a failure that the old consumer thread is down and/or a failure that the old consumer thread is down.
Further, to achieve the above object, a second aspect of embodiments of the present invention provides a message processing apparatus, including: a memory, a processor and a message processing program stored on the memory and executable on the processor, the message processing program, when executed by the processor, implementing the steps of the message processing method of the first aspect.
Furthermore, to achieve the above object, a third aspect of embodiments of the present invention provides a computer-readable storage medium, on which a message processing program is stored, and the message processing program, when executed by a processor, implements the steps of the message processing method according to the first aspect.
According to the message processing method, the message processing device and the computer readable storage medium, the standby consumer thread and the new consumer thread consume the messages written by the server in a non-real-time and real-time manner; on the one hand, the latest messages can be guaranteed to be processed, the delay of message processing is reduced, and on the other hand, the messages during the failure of the consumer thread can be guaranteed to be processed in a supplementing mode and not lost.
Drawings
Fig. 1 is a schematic diagram of a hardware structure of a mobile terminal implementing various embodiments of the present invention;
fig. 2 is a diagram of a communication network system architecture according to an embodiment of the present invention;
FIG. 3 is a flow chart of a message processing method according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating an offset structure in a message processing process according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a message processing device according to an embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In the following description, suffixes such as "module", "component", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no specific meaning in itself. Thus, "module", "component" or "unit" may be used mixedly.
The terminal may be implemented in various forms. For example, the terminal described in the present invention may include a mobile terminal such as a mobile phone, a tablet computer, a notebook computer, a palmtop computer, a Personal Digital Assistant (PDA), a Portable Media Player (PMP), a navigation device, a wearable device, a smart band, a pedometer, and the like, and a fixed terminal such as a Digital TV, a desktop computer, and the like.
The following description will be given by way of example of a mobile terminal, and it will be understood by those skilled in the art that the construction according to the embodiment of the present invention can be applied to a fixed type terminal, in addition to elements particularly used for mobile purposes.
Referring to fig. 1, which is a schematic diagram of a hardware structure of a mobile terminal for implementing various embodiments of the present invention, the mobile terminal 100 may include: RF (Radio Frequency) unit 101, WiFi module 102, audio output unit 103, a/V (audio/video) input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, processor 110, and power supply 111. Those skilled in the art will appreciate that the mobile terminal architecture shown in fig. 1 is not intended to be limiting of mobile terminals, which may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
The following describes each component of the mobile terminal in detail with reference to fig. 1:
the radio frequency unit 101 may be configured to receive and transmit signals during information transmission and reception or during a call, and specifically, receive downlink information of a base station and then process the downlink information to the processor 110; in addition, uplink data is transmitted to the base station. Typically, radio frequency unit 101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, and the like. In addition, the radio frequency unit 101 can also communicate with a network and other devices through wireless communication. The wireless communication may use any communication standard or protocol, including but not limited to GSM (Global System for Mobile communications), GPRS (General Packet Radio Service), CDMA2000(Code Division Multiple Access 2000), WCDMA (Wideband Code Division Multiple Access), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access), FDD-LTE (Frequency Division duplex Long Term Evolution), and TDD-LTE (Time Division duplex Long Term Evolution).
WiFi belongs to short-distance wireless transmission technology, and the mobile terminal can help a user to receive and send e-mails, browse webpages, access streaming media and the like through the WiFi module 102, and provides wireless broadband internet access for the user. Although fig. 1 shows the WiFi module 102, it is understood that it does not belong to the essential constitution of the mobile terminal, and may be omitted entirely as needed within the scope not changing the essence of the invention.
The audio output unit 103 may convert audio data received by the radio frequency unit 101 or the WiFi module 102 or stored in the memory 109 into an audio signal and output as sound when the mobile terminal 100 is in a call signal reception mode, a call mode, a recording mode, a voice recognition mode, a broadcast reception mode, or the like. Also, the audio output unit 103 may also provide audio output related to a specific function performed by the mobile terminal 100 (e.g., a call signal reception sound, a message reception sound, etc.). The audio output unit 103 may include a speaker, a buzzer, and the like.
The a/V input unit 104 is used to receive audio or video signals. The a/V input Unit 104 may include a Graphics Processing Unit (GPU) 1041 and a microphone 1042, the Graphics processor 1041 Processing image data of still pictures or video obtained by an image capturing device (e.g., a camera) in a video capturing mode or an image capturing mode. The processed image frames may be displayed on the display unit 106. The image frames processed by the graphic processor 1041 may be stored in the memory 109 (or other storage medium) or transmitted via the radio frequency unit 101 or the WiFi module 102. The microphone 1042 may receive sounds (audio data) via the microphone 1042 in a phone call mode, a recording mode, a voice recognition mode, or the like, and may be capable of processing such sounds into audio data. The processed audio (voice) data may be converted into a format output transmittable to a mobile communication base station via the radio frequency unit 101 in case of a phone call mode. The microphone 1042 may implement various types of noise cancellation (or suppression) algorithms to cancel (or suppress) noise or interference generated in the course of receiving and transmitting audio signals.
The mobile terminal 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor that can adjust the brightness of the display panel 1061 according to the brightness of ambient light, and a proximity sensor that can turn off the display panel 1061 and/or a backlight when the mobile terminal 100 is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally, three axes), can detect the magnitude and direction of gravity when stationary, and can be used for applications of recognizing the posture of a mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer and tapping), and the like; as for other sensors such as a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which can be configured on the mobile phone, further description is omitted here.
The display unit 106 is used to display information input by a user or information provided to the user. The Display unit 106 may include a Display panel 1061, and the Display panel 1061 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like.
The user input unit 107 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the mobile terminal. Specifically, the user input unit 107 may include a touch panel 1071 and other input devices 1072. The touch panel 1071, also referred to as a touch screen, can collect touch operations of a user (e.g., operations of a user on the touch panel 1071 or near the touch panel 1071 using a finger, a stylus, or any other suitable object or accessory) thereon or nearby and drive the corresponding connection device according to a predetermined program. The touch panel 1071 may include two parts of a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 110, and can receive and execute commands sent by the processor 110. In addition, the touch panel 1071 may be implemented in various types, such as a resistive type, a capacitive type, an infrared ray, and a surface acoustic wave. In addition to the touch panel 1071, the user input unit 107 may include other input devices 1072. In particular, other input devices 1072 may include, but are not limited to, one or more of a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like, and are not limited to these specific examples.
Further, the touch panel 1071 may cover the display panel 1061, and when the touch panel 1071 detects a touch operation thereon or nearby, the touch panel 1071 transmits the touch operation to the processor 110 to determine the type of the touch event, and then the processor 110 provides a corresponding visual output on the display panel 1061 according to the type of the touch event. Although the touch panel 1071 and the display panel 1061 are shown in fig. 1 as two separate components to implement the input and output functions of the mobile terminal, in some embodiments, the touch panel 1071 and the display panel 1061 may be integrated to implement the input and output functions of the mobile terminal, and is not limited herein.
The interface unit 108 serves as an interface through which at least one external device is connected to the mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, an audio input/output (I/O) port, a video I/O port, an earphone port, and the like. The interface unit 108 may be used to receive input (e.g., data information, power, etc.) from external devices and transmit the received input to one or more elements within the mobile terminal 100 or may be used to transmit data between the mobile terminal 100 and external devices.
The memory 109 may be used to store software programs as well as various data. The memory 109 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. Further, the memory 109 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
The processor 110 is a control center of the mobile terminal, connects various parts of the entire mobile terminal using various interfaces and lines, and performs various functions of the mobile terminal and processes data by operating or executing software programs and/or modules stored in the memory 109 and calling data stored in the memory 109, thereby performing overall monitoring of the mobile terminal. Processor 110 may include one or more processing units; preferably, the processor 110 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 110.
The mobile terminal 100 may further include a power supply 111 (e.g., a battery) for supplying power to various components, and preferably, the power supply 111 may be logically connected to the processor 110 via a power management system, so as to manage charging, discharging, and power consumption management functions via the power management system.
Although not shown in fig. 1, the mobile terminal 100 may further include a bluetooth module or the like, which is not described in detail herein.
In order to facilitate understanding of the embodiments of the present invention, a communication network system on which the mobile terminal of the present invention is based is described below.
Referring to fig. 2, fig. 2 is an architecture diagram of a communication Network system according to an embodiment of the present invention, where the communication Network system is an LTE system of a universal mobile telecommunications technology, and the LTE system includes a UE (User Equipment) 201, an E-UTRAN (Evolved UMTS Terrestrial Radio Access Network) 202, an EPC (Evolved Packet Core) 203, and an IP service 204 of an operator, which are in communication connection in sequence.
Specifically, the UE201 may be the terminal 100 described above, and is not described herein again.
The E-UTRAN202 includes eNodeB2021 and other eNodeBs 2022, among others. Among them, the eNodeB2021 may be connected with other eNodeB2022 through backhaul (e.g., X2 interface), the eNodeB2021 is connected to the EPC203, and the eNodeB2021 may provide the UE201 access to the EPC 203.
The EPC203 may include an MME (Mobility Management Entity) 2031, an HSS (Home Subscriber Server) 2032, other MMEs 2033, an SGW (Serving gateway) 2034, a PGW (PDN gateway) 2035, and a PCRF (Policy and Charging Rules Function) 2036, and the like. The MME2031 is a control node that handles signaling between the UE201 and the EPC203, and provides bearer and connection management. HSS2032 is used to provide registers to manage functions such as home location register (not shown) and holds subscriber specific information about service characteristics, data rates, etc. All user data may be sent through SGW2034, PGW2035 may provide IP address assignment for UE201 and other functions, and PCRF2036 is a policy and charging control policy decision point for traffic data flow and IP bearer resources, which selects and provides available policy and charging control decisions for a policy and charging enforcement function (not shown).
The IP services 204 may include the internet, intranets, IMS (IP Multimedia Subsystem), or other IP services, among others.
Although the LTE system is described as an example, it should be understood by those skilled in the art that the present invention is not limited to the LTE system, but may also be applied to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA, and future new network systems.
Based on the above mobile terminal hardware structure and communication network system, the present invention provides various embodiments of the method.
First embodiment
Before describing the present embodiment, the kafka terminology in the examples is described and illustrated below:
the message producer: the Producer, is the source of the message generation and is responsible for generating and sending the message to the Kafka server.
The message consumer: i.e., Consumer, is the user of the message, responsible for consuming messages on the Kafka server.
Subject matter: i.e., Topic, is defined by the user and configured at the Kafka server for establishing a subscription relationship between the producer and the messenger: the producer sends a message to a designated Topic from which the message consumer consumes the message.
Message partitioning: i.e. Partition, a Topic is divided into many partitions below, for example: the Topic of "kafka-test" can be divided into 6 partitions, which are provided by two servers, and it can be configured to provide 3 partitions for each server, and if the server ID is 0 and 1, all the partitions are 0-0, 0-1, 0-2 and 1-0, 1-1, 1-2. A topoc is physically grouped, and a topoc may be divided into multiple partitions, each partition being an ordered queue. Each message in partition is assigned an ordered id (offset).
Grouping consumers: i.e. Group, for grouping consumers of the same class, in Kafka, a plurality of consumers may collectively consume a message under Topic, each consumer consumes a part of the message therein, and the consumers form a Group, and have the same Group name, which is also commonly referred to as a consumer cluster.
Offset: the messages are stored on the Broker of Kafka and the consumer needs to know the Offset of the message in the file, which is called Offset, when pulling the message data.
And (4) Broker: i.e., Kafka's server, where the user stores messages, and one or more servers in the Kafka cluster are collectively referred to as brookers.
Zookeeper is a widely used in distributed systems as: distributed state management, distributed coordination management, distributed configuration management, and clustering of distributed lock services. The Kafka increase and decrease servers trigger corresponding events on the Zookeeper node, the Kafka system captures the events to perform a new round of load balancing, and the client captures the events to perform a new round of processing.
As shown in fig. 3, a first embodiment of the present invention provides a message processing method, where the method includes:
and S31, starting the standby consumer thread and the new consumer thread after the old consumer thread fails.
Firstly, building a related environment of a related distributed publish-subscribe message system cluster, downloading a related tool jar package, writing a related distributed publish-subscribe message system consumer client code, and setting the maximum number of messages processed by a local thread (for example, defining the maximum number of messages processed by the local thread as TASKCOUNT) and the timeout time of the distributed publish-subscribe message system (for example, defining the timeout time of the distributed publish-subscribe message system as session.
Before the old consumer thread fails, the old consumer thread acquires data from the Kafka server in batch, and offset position information of the consumption Kafka server message is recorded in zookeeper. Meanwhile, a Redis cache component is newly added, and after the consumed offset position information is recorded in the zookeeper, the consumed offset position information is recorded in the Redis cache.
After the old consumer thread fails, the consumer thread needs to be restarted to consume the messages written by the Kafka server in real time and the messages written by the Kafka server in non-real time. The present embodiment addresses the consumption of real-time write messages and non-real-time write messages by starting a standby consumer thread and a new consumer thread.
In this embodiment, the failure includes a failure that the old consumer thread is down and/or a failure that the old consumer thread is down. The standby consumer thread and the new consumer thread may be consumers in the same consumer cluster, or may not be consumers in the same consumer cluster, which is not limited herein.
In one embodiment, said starting the standby consumer thread and the new consumer thread after the failure of the old consumer thread comprises the steps of:
detecting whether the old consumer thread fails;
and if the old consumer thread fails, starting a standby consumer thread and a new consumer thread.
S32, the standby consumer thread acquires a first offset position from a preset cache and acquires a second offset position from a manager zookeeper; consuming the messages written by the server in non-real time according to the first offset position and the second offset position; wherein the first offset position is an offset position after the old consumer thread consumes the server message.
In this embodiment, the preset cache is the foregoing Redis cache, and the first offset position is consumed offset position information recorded in the Redis cache, that is, an offset position after the Kafka server message is consumed before the old consumer thread fails.
In this embodiment, the second offset position is the offset position of the latest message, i.e., the message written to the Kafka server after the consumer thread restarts.
In one embodiment, the consuming a server non-real-time written message according to the first offset position and the second offset position comprises the steps of:
judging whether the first offset position and the second offset position are the same;
if the first offset position and the second offset position are not the same, the standby consumer thread consumes the message between the first offset position and the second offset position.
In this embodiment, the standby consumer thread consuming the message between the first offset location and the second offset location comprises the steps of:
the standby consumer thread initiates multithreading and consumes messages between the first offset position and the second offset position via the multithreading.
In this embodiment, throughput and consumption rate are improved by multithreading to process messages between the first offset position and the second offset position.
In this embodiment, if the first offset position and the second offset position are not the same, the method further includes, after the standby consumer thread consumes the message between the first offset position and the second offset position:
closing the standby consumer thread if the message between the first offset position and the second offset position has been consumed.
In another embodiment, the standby consumer thread obtains a first offset position from a preset cache and a second offset position from a supervisor zookeeper; consuming the message written by the server in non-real time according to the first offset position and the second offset position, and comprising the following steps:
setting a second offset position in the zookeeper.
In this embodiment, by setting the second offset position in zookeeper, it can be ensured that the second offset position points to the offset position of the latest message.
And S33, the new consumer thread consumes the message written by the server in real time according to the second offset position.
In this embodiment, the new consumer thread directly consumes from the offset position of the latest message, while the standby consumer thread consumes between the offset position after the old consumer thread consumes the server message and the offset position of the latest message, and the standby consumer thread can be closed after the consumption is completed. This approach ensures that, on the one hand, the latest messages can be processed, reducing delays in message processing, and on the other hand, that messages during a consumer thread failure can be supplementarily processed without being lost.
In one embodiment, the new consumer thread consuming the message written by the server in real-time according to the second offset position further comprises the steps of:
and if the new consumer thread is consumed completely, updating the first offset position in the preset cache.
To better illustrate the present embodiment, the following describes the message processing process in detail with reference to fig. 4:
as shown in fig. 4, assume that the consumer thread has successfully consumed processing to the location where offset is 20 (i.e., offset1 in the figure) before the consumer thread goes down or stops. After restarting the consumer thread, the offset position of the latest message is the offset position of 40 (i.e., offset2 in the figure).
Existing solutions typically handle this case by starting to process the message from offset 1. The problems with this solution are: if there are more messages between offset2 and offset1 (the downtime is long, and the amount of written messages is too much, which results in more messages between offset2 and offset1), it takes a while for the consumer thread to process the messages between offset2 and offset1 to start processing the messages from the offset position offset2 of the latest message, which results in a delay in message processing.
For this case, a standby consumer thread and a new consumer thread may be started, with messages between offset 2-offset 1 being processed by the standby consumer thread, and the new consumer thread starting processing messages directly from the offset position offset2 of the most recent message. The standby consumer thread is turned off directly after it has processed the messages between offset2 and offset 1.
According to the message processing method provided by the embodiment of the invention, the standby consumer thread and the new consumer thread consume the messages written by the server in a non-real-time and real-time manner; on the one hand, the latest messages can be guaranteed to be processed, the delay of message processing is reduced, and on the other hand, the messages during the failure of the consumer thread can be guaranteed to be processed in a supplementing mode and not lost.
Second embodiment
Referring to fig. 5, fig. 5 provides a message processing apparatus for the second embodiment of the present invention, where the apparatus 40 includes: a memory 41, a processor 42 and a message processing program stored on the memory 41 and operable on the processor 42, the message processing program, when executed by the processor 42, is configured to implement the following steps of the message processing method:
starting a standby consumer thread and a new consumer thread after the old consumer thread fails;
the standby consumer thread acquires a first offset position from a preset cache and acquires a second offset position from a manager zookeeper; consuming the messages written by the server in non-real time according to the first offset position and the second offset position; wherein the first offset position is an offset position after the old consumer thread consumes the server message;
and the new consumer thread consumes the messages written by the server in real time according to the second offset position.
The message processing program, when executed by the processor 42, is further configured to implement the steps of the message processing method as follows:
judging whether the first offset position and the second offset position are the same;
if the first offset position and the second offset position are not the same, the standby consumer thread consumes the message between the first offset position and the second offset position.
The message processing program, when executed by the processor 42, is further configured to implement the steps of the message processing method described below:
the standby consumer thread initiates multithreading and consumes messages between the first offset position and the second offset position via the multithreading.
The message processing program, when executed by the processor 42, is further configured to implement the steps of the message processing method as follows:
closing the standby consumer thread if the message between the first offset position and the second offset position has been consumed.
The message processing program, when executed by the processor 42, is further configured to implement the steps of the message processing method as follows:
setting a second offset position in the zookeeper.
The message processing program, when executed by the processor 42, is further configured to implement the steps of the message processing method described below:
and if the new consumer thread is consumed completely, updating the first offset position in the preset cache.
The message processing program, when executed by the processor 42, is further configured to implement the steps of the message processing method as follows:
detecting whether the old consumer thread fails;
and if the old consumer thread fails, starting a standby consumer thread and a new consumer thread.
The message processing program, when executed by the processor 42, is further configured to implement the steps of the message processing method as follows:
the failure includes a failure that the old consumer thread is down and/or a failure that the old consumer thread is down.
The message processing equipment provided by the embodiment of the invention consumes the messages written in the server in a non-real-time and real-time manner through the standby consumer thread and the new consumer thread; on the one hand, the latest messages can be guaranteed to be processed, the delay of message processing is reduced, and on the other hand, the messages during the failure of the consumer thread can be guaranteed to be processed in a supplementing mode and not lost.
Third embodiment
A third embodiment of the present invention provides a computer-readable storage medium, on which a message processing program is stored, and the message processing program, when executed by a processor, implements the steps of the message processing method described in the first embodiment.
The computer-readable storage medium provided by the embodiment of the invention consumes the messages written in the server in non-real time and real time through the standby consumer thread and the new consumer thread; on the one hand, the latest messages can be guaranteed to be processed, the delay of message processing is reduced, and on the other hand, the messages during the failure of the consumer thread can be guaranteed to be processed in a supplementing mode and not lost.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
While the present invention has been described with reference to the embodiments shown in the drawings, the present invention is not limited to the embodiments, which are illustrative and not restrictive, and it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (10)
1. A method of message processing, the method comprising:
starting a standby consumer thread and a new consumer thread after the old consumer thread fails;
the standby consumer thread acquires a first offset position from a preset cache and acquires a second offset position from a manager zookeeper; consuming the messages written by the server in non-real time according to the first offset position and the second offset position; wherein the first offset position is an offset position after the old consumer thread consumes the server message;
and the new consumer thread consumes the messages written by the server in real time according to the second offset position.
2. The message processing method of claim 1, wherein consuming the server non-real-time written message according to the first offset position and the second offset position comprises:
judging whether the first offset position and the second offset position are the same;
if the first offset position and the second offset position are not the same, the standby consumer thread consumes the message between the first offset position and the second offset position.
3. The message processing method of claim 2, wherein the standby consumer thread consuming the message between the first offset location and the second offset location comprises the steps of:
the standby consumer thread initiates multithreading and consumes messages between the first offset position and the second offset position via the multithreading.
4. The message processing method of claim 2, wherein if the first offset location and the second offset location are not the same, the standby consumer thread further comprises, after consuming the message between the first offset location and the second offset location:
closing the standby consumer thread if the message between the first offset position and the second offset position has been consumed.
5. The message processing method of claim 1, wherein the standby consumer thread obtains a first offset position from a preset cache and a second offset position from a supervisor zookeeper; consuming the message written by the server in non-real time according to the first offset position and the second offset position, and comprising the following steps:
setting a second offset position in the zookeeper.
6. The message processing method of claim 1, wherein the new consumer thread consumes the message written by the server in real time according to the second offset position further comprising the steps of:
and if the new consumer thread is consumed completely, updating the first offset position in the preset cache.
7. The message processing method according to claim 1, wherein the starting of the standby consumer thread and the new consumer thread after the failure of the old consumer thread comprises the steps of:
detecting whether the old consumer thread fails;
and if the old consumer thread fails, starting a standby consumer thread and a new consumer thread.
8. The message processing method according to claim 7, wherein the failure comprises a failure that an old consumer thread is down and/or a failure that an old consumer thread is down.
9. A message processing device, characterized in that the device comprises: memory, a processor and a message handling program stored on the memory and executable on the processor, the message handling program when executed by the processor implementing the steps of the message handling method according to any of claims 1 to 8.
10. A computer-readable storage medium, having stored thereon a message processing program which, when executed by a processor, implements the steps of the message processing method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810274276.7A CN108509299B (en) | 2018-03-29 | 2018-03-29 | Message processing method, device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810274276.7A CN108509299B (en) | 2018-03-29 | 2018-03-29 | Message processing method, device and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108509299A CN108509299A (en) | 2018-09-07 |
| CN108509299B true CN108509299B (en) | 2022-08-12 |
Family
ID=63379249
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810274276.7A Active CN108509299B (en) | 2018-03-29 | 2018-03-29 | Message processing method, device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108509299B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110968370B (en) * | 2019-11-18 | 2024-02-23 | 中国银行股份有限公司 | Data processing method and device |
| CN111124710A (en) * | 2019-12-17 | 2020-05-08 | 联想(北京)有限公司 | Data processing method and electronic equipment |
| CN111124650B (en) * | 2019-12-26 | 2023-10-24 | 中国建设银行股份有限公司 | Stream data processing method and device |
| CN112000489A (en) * | 2020-07-29 | 2020-11-27 | 新华三大数据技术有限公司 | Kafka data processing method and server |
| CN112286723A (en) * | 2020-09-30 | 2021-01-29 | 北京大米科技有限公司 | Computer room disaster recovery control method, terminal and storage medium |
| CN112506992B (en) * | 2020-12-04 | 2024-04-16 | 中国人寿保险股份有限公司 | Fuzzy query method and device for Kafka data, electronic equipment and storage medium |
| CN112612606A (en) * | 2020-12-16 | 2021-04-06 | 平安消费金融有限公司 | Message theme processing method and device, computer equipment and readable storage medium |
| CN113297309B (en) * | 2021-05-31 | 2023-11-10 | 平安证券股份有限公司 | Stream data writing method, device, equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105306552A (en) * | 2015-09-30 | 2016-02-03 | 中国科学院计算技术研究所 | Consumption equilibrium method and system based on message queues |
| CN106657349A (en) * | 2016-12-29 | 2017-05-10 | 上海理想信息产业（集团）有限公司 | Message subscription processing device, system and method |
| CN106648933A (en) * | 2016-12-26 | 2017-05-10 | 北京奇虎科技有限公司 | Consuming method and device of message queue |
| CN106817295A (en) * | 2016-12-08 | 2017-06-09 | 努比亚技术有限公司 | A kind of message processing apparatus and method |
| CN107423145A (en) * | 2017-07-11 | 2017-12-01 | 北京潘达互娱科技有限公司 | A kind of method and apparatus for avoiding information drop-out |
| CN107423141A (en) * | 2017-06-20 | 2017-12-01 | 北京小度信息科技有限公司 | Information processing method and device |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050021836A1 (en) * | 2003-05-01 | 2005-01-27 | Reed Carl J. | System and method for message processing and routing |
| US9330005B2 (en) * | 2011-12-13 | 2016-05-03 | International Business Machines Corporation | Interface and method for inter-thread communication |
| CN104424186B (en) * | 2013-08-19 | 2018-04-03 | 阿里巴巴集团控股有限公司 | The method and device of persistence is realized in a kind of stream calculation application |
| US10481948B2 (en) * | 2015-08-25 | 2019-11-19 | Box, Inc. | Data transfer in a collaborative file sharing system |
| US10409650B2 (en) * | 2016-02-24 | 2019-09-10 | Salesforce.Com, Inc. | Efficient access scheduling for super scaled stream processing systems |
| US9917913B2 (en) * | 2016-05-23 | 2018-03-13 | Microsoft Technology Licensing, Llc | Large message support for a publish-subscribe messaging system |
-
2018
- 2018-03-29 CN CN201810274276.7A patent/CN108509299B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105306552A (en) * | 2015-09-30 | 2016-02-03 | 中国科学院计算技术研究所 | Consumption equilibrium method and system based on message queues |
| CN106817295A (en) * | 2016-12-08 | 2017-06-09 | 努比亚技术有限公司 | A kind of message processing apparatus and method |
| CN106648933A (en) * | 2016-12-26 | 2017-05-10 | 北京奇虎科技有限公司 | Consuming method and device of message queue |
| CN106657349A (en) * | 2016-12-29 | 2017-05-10 | 上海理想信息产业（集团）有限公司 | Message subscription processing device, system and method |
| CN107423141A (en) * | 2017-06-20 | 2017-12-01 | 北京小度信息科技有限公司 | Information processing method and device |
| CN107423145A (en) * | 2017-07-11 | 2017-12-01 | 北京潘达互娱科技有限公司 | A kind of method and apparatus for avoiding information drop-out |
Non-Patent Citations (2)
| Title |
|---|
| 一种基于Kafka的可靠的Consumer的设计方案;王岩等;《软件》;20160115(第01期);全文 * |
| 基于NS3的分布式消息系统Kafka的仿真实现;马浩然;《软件》;20150115(第01期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108509299A (en) | 2018-09-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108509299B (en) | Message processing method, device and computer readable storage medium | |
| CN108768775B (en) | Information processing method, electronic device, and computer storage medium | |
| CN112799577B (en) | Method, terminal and storage medium for projecting small window | |
| CN107135128B (en) | Call chain data acquisition method, mobile terminal and computer-readable storage medium | |
| CN110162418B (en) | Application cold start method, mobile terminal and computer readable storage medium | |
| CN107862217B (en) | Position information acquisition method, mobile terminal and computer storage medium | |
| CN109766119B (en) | Recovery partition upgrade method, terminal and computer readable storage medium | |
| CN112822538A (en) | Screen projection display method, screen projection device, terminal and storage medium | |
| CN112399215A (en) | Screen projection parameter regulation and control method and device and computer readable storage medium | |
| CN107818022B (en) | Application program interface merging method, mobile terminal and computer readable storage medium | |
| CN108011937B (en) | Message pushing method, server, intelligent terminal and computer readable storage medium | |
| CN109254972B (en) | Offline command word bank updating method, terminal and computer readable storage medium | |
| CN109889651B (en) | Notification message processing method, terminal and computer readable storage medium | |
| CN113067734B (en) | Electronic price tag upgrading method, electronic price tag system, electronic terminal and storage medium | |
| CN107688497B (en) | Memory regulation and control method, equipment and computer readable storage medium | |
| CN112102780B (en) | Display frame rate regulation and control method, device and computer readable storage medium | |
| CN107239208B (en) | Method, apparatus, and computer-readable storage medium for processing screenshot | |
| CN112494932A (en) | Game frame supplementing method and device, terminal and computer readable storage medium | |
| CN112395106A (en) | Process management method, mobile terminal, and computer-readable storage medium | |
| CN112347067A (en) | Log information management and control method and device and computer readable storage medium | |
| CN112612372A (en) | Touch event regulation and control method and device and computer readable storage medium | |
| CN109818967B (en) | Notification method, server, mobile terminal and computer readable storage medium | |
| CN110955397A (en) | Method for setting frame rate of game terminal, game terminal and storage medium | |
| CN107678622B (en) | Application icon display method, terminal and storage medium | |
| CN108304302B (en) | Interface merging method, equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220728 Address after: No. 6, Minzhu Road, Xingning District, Nanning City, Guangxi Zhuang Autonomous Region, 530012 Applicant after: GUANGXI POWER GRID Co.,Ltd. Address before: 518000 6-8 / F, 10-11 / F, 6 / F, 6-10 / F, C zone, Han's innovation building, No. 9018, Beihuan Avenue, high tech Zone, Nanshan District, Shenzhen City, Guangdong Province Applicant before: NUBIA TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |