CN108509299A - Message treatment method, equipment and computer readable storage medium - Google Patents
Message treatment method, equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN108509299A CN108509299A CN201810274276.7A CN201810274276A CN108509299A CN 108509299 A CN108509299 A CN 108509299A CN 201810274276 A CN201810274276 A CN 201810274276A CN 108509299 A CN108509299 A CN 108509299A
- Authority
- CN
- China
- Prior art keywords
- message
- offset location
- consumer thread
- offset
- consumer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1415—Saving, restoring, recovering or retrying at system level
- G06F11/1417—Boot up procedures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/55—Push-based network services
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Retry When Errors Occur (AREA)
- Mobile Radio Communication Systems (AREA)
Abstract
The invention discloses a kind of message treatment method, equipment and computer readable storage medium, the method includes the steps：After old consumer thread breaks down, start spare consumer thread and Xin consumer threads；The spare consumer thread obtains the first offset location from preset buffer memory, and the second offset location is obtained from manager zookeeper；According to the message of first offset location and the non real-time write-in of the second offset location consumption service device；The message that the new consumer thread is written in real time according to the second offset location consumption service device.The present invention passes through the message that spare consumer thread and Xin consumer thread's consumption service devices are non real-time and are written in real time；On the one hand it can guarantee that latest news can be handled, and the delay of Message Processing is reduced, on the other hand can ensure that the message during consumer thread's failure can obtain supplement process without losing.
Description
Technical field
The present invention relates to a kind of field of communication technology more particularly to message treatment method, equipment and computer-readable storages
Medium.
Background technology
Kafka is a kind of distributed post subscription message system of high-throughput, it can handle the net of consumer's scale
Everything flow data in standing.This action (web page browsing, the action of search and other users) is on modern network
One key factor of many social functions.These data be often as the requirement of handling capacity and by handling daily record and daily record
It polymerize to solve.The purpose of Kafka is by a Hadoop (distributed system foundation frame developed by Apache funds club
Structure) loaded in parallel mechanism unify Message Processing on line and offline, disappear in real time to provide also for by cluster machine
Take.
Kafka is applied in big data statistic processes, is generation, the message of the data progress message for being directed to big data quantity
It consumes to use.Wherein, the production of message is the transmission that message is directly carried out from the producer to Kafka；The consumption of message,
It is to be operated by consumer thread to carry out the consumption of message.Because in many production environments, it is necessary to use Kafka clusters
Mode eliminate Single Point of Faliure, so the consumption that consumer thread is attached to carry out in entire Kafka clusters message is dynamic
Make.Consumer thread's number of Kafka and the partition (differentiation) of topic (theme) have relationship, and (there are one main for a kind of message
Topic needs the number of partitions for fixing the theme when each theme creates).At a time, partition can only there are one
Consumer thread can connect.
Existing consumption pattern the problem is that：When a large amount of message write-in and consumer thread's delay machine stop after, open again
Consumer thread is moved, then consumption data the consumer thread position that included zookeeper is recorded since the inside kafka；
If downtime is longer, the more data for needing to consume are accumulated, then has handled these non real-time data and needs the time, this is just
Cause newest data that cannot carry out consumption processing in real time, increases processing delay time.
Invention content
The main purpose of the embodiment of the present invention is to provide a kind of message treatment method, equipment and computer-readable storage medium
Matter, it is intended to solve the problems, such as that existing consumption pattern exists.
To achieve the above object, first aspect of the embodiment of the present invention provides a kind of message treatment method, the method includes
Step：
After old consumer thread breaks down, start spare consumer thread and Xin consumer threads；
The spare consumer thread obtains the first offset location from preset buffer memory, and from manager zookeeper
The second offset location of middle acquisition；According to first offset location and the non-reality of the second offset location consumption service device
When the message that is written；Wherein, first offset bit is set to the offset after old consumer thread's consumption service device message
Measure position；
The message that the new consumer thread is written in real time according to the second offset location consumption service device.
Optionally, described non real-time according to first offset location and the second offset location consumption service device
The message of write-in includes step：
Judge whether first offset location and second offset location are identical；
If first offset location and second offset location differ, the spare consumer thread disappears
Take the message between first offset location and second offset location.
Optionally, the spare consumer thread consume first offset location and second offset location it
Between message include step：
The spare consumer thread starts multithreading, and by the multithreading consume first offset location and
Message between second offset location.
Optionally, described spare if first offset location and second offset location differ
Consumer thread further includes step after consuming the message between first offset location and second offset location：
It finishes, closes if the message between first offset location and second offset location has been consumed
The spare consumer thread.
Optionally, the spare consumer thread obtains the first offset location from preset buffer memory, and from manager
The second offset location is obtained in zookeeper；It is consumed according to first offset location and second offset location
Further include step before the message of the non real-time write-in of server：
The second offset location in the zookeeper is set.
Optionally, the message that the new consumer thread is written in real time according to the second offset location consumption service device
Further include step later：
It is finished if the new consumer thread has consumed, updates the first offset location in the preset buffer memory.
Optionally, described after old consumer thread breaks down, start spare consumer thread and Xin consumer threads
Including step：
Detect whether the old consumer thread breaks down；
If the old consumer thread breaks down, start spare consumer thread and Xin consumer threads.
Optionally, the failure includes that old consumer thread's delay machine failure and/or old consumer thread stop failure.
In addition, to achieve the above object, second aspect of the embodiment of the present invention provides a kind of message processing device, the equipment
Including：Memory, processor and it is stored in the message processing program that can be run on the memory and on the processor, institute
State the step of message treatment method described in first aspect is realized when message processing program is executed by the processor.
Furthermore to achieve the above object, the third aspect of the embodiment of the present invention provides a kind of computer readable storage medium, institute
It states and is stored with message processing program on computer readable storage medium, the message processing program realizes when being executed by processor
The step of message treatment method described in one side.
Message treatment method, equipment and computer readable storage medium provided in an embodiment of the present invention, pass through spare consumption
Person's thread and new consumer thread's consumption service device is non real-time and the message that is written in real time；On the one hand it can guarantee latest news energy
Processing is accessed, the delay of Message Processing is reduced, on the other hand can ensure that the message during consumer thread's failure can obtain
Supplement process is without losing.
Description of the drawings
The hardware architecture diagram of Fig. 1 mobile terminals of each embodiment to realize the present invention；
Fig. 2 is a kind of communications network system Organization Chart provided in an embodiment of the present invention；
Fig. 3 is the message treatment method flow diagram of the embodiment of the present invention；
Fig. 4 is offset structural schematic diagram in the message processing procedure of the embodiment of the present invention；
Fig. 5 is the message processing device structural schematic diagram of the embodiment of the present invention.
The embodiments will be further described with reference to the accompanying drawings for the realization, the function and the advantages of the object of the present invention.
Specific implementation mode
It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not intended to limit the present invention.
In subsequent description, using for indicating that the suffix of such as " module ", " component " or " unit " of element is only
The explanation for being conducive to the present invention, itself does not have a specific meaning.Therefore, " module ", " component " or " unit " can mix
Ground uses.
Terminal can be implemented in a variety of manners.For example, terminal described in the present invention may include such as mobile phone, tablet
Computer, laptop, palm PC, personal digital assistant (Personal Digital Assistant, PDA), portable
The shiftings such as media player (Portable Media Player, PMP), navigation device, wearable device, Intelligent bracelet, pedometer
The fixed terminals such as dynamic terminal, and number TV, desktop computer.
It will be illustrated by taking mobile terminal as an example in subsequent descriptions, it will be appreciated by those skilled in the art that in addition to special
Except element for moving purpose, construction according to the embodiment of the present invention can also apply to the terminal of fixed type.
Referring to Fig. 1, a kind of hardware architecture diagram of its mobile terminal of each embodiment to realize the present invention, the shifting
Moving terminal 100 may include：RF (Radio Frequency, radio frequency) unit 101, WiFi module 102, audio output unit
103, A/V (audio/video) input unit 104, sensor 105, display unit 106, user input unit 107, interface unit
108, the components such as memory 109, processor 110 and power supply 111.It will be understood by those skilled in the art that shown in Fig. 1
Mobile terminal structure does not constitute the restriction to mobile terminal, and mobile terminal may include components more more or fewer than diagram,
Either combine certain components or different components arrangement.
The all parts of mobile terminal are specifically introduced with reference to Fig. 1：
Radio frequency unit 101 can be used for receiving and sending messages or communication process in, signal sends and receivees, specifically, by base station
Downlink information receive after, to processor 110 handle；In addition, the data of uplink are sent to base station.In general, radio frequency unit 101
Including but not limited to antenna, at least one amplifier, transceiver, coupler, low-noise amplifier, duplexer etc..In addition, penetrating
Frequency unit 101 can also be communicated with network and other equipment by radio communication.Above-mentioned wireless communication can use any communication
Standard or agreement, including but not limited to GSM (Global System of Mobile communication, global system for mobile telecommunications
System), GPRS (General Packet Radio Service, general packet radio service), CDMA2000 (Code
Division Multiple Access 2000, CDMA 2000), WCDMA (Wideband Code Division
Multiple Access, wideband code division multiple access), TD-SCDMA (Time Division-Synchronous Code
Division Multiple Access, TD SDMA), FDD-LTE (Frequency Division
Duplexing-Long Term Evolution, frequency division duplex long term evolution) and TDD-LTE (Time Division
Duplexing-Long Term Evolution, time division duplex long term evolution) etc..
WiFi belongs to short range wireless transmission technology, and mobile terminal can help user to receive and dispatch electricity by WiFi module 102
Sub- mail, browsing webpage and access streaming video etc., it has provided wireless broadband internet to the user and has accessed.Although Fig. 1 shows
Go out WiFi module 102, but it is understood that, and it is not belonging to must be configured into for mobile terminal, it completely can be according to need
It to be omitted in the range for the essence for not changing invention.
Audio output unit 103 can be in call signal reception pattern, call mode, record mould in mobile terminal 100
When under the isotypes such as formula, speech recognition mode, broadcast reception mode, it is that radio frequency unit 101 or WiFi module 102 are received or
The audio data stored in memory 109 is converted into audio signal and exports to be sound.Moreover, audio output unit 103
The relevant audio output of specific function executed with mobile terminal 100 can also be provided (for example, call signal receives sound, disappears
Breath receives sound etc.).Audio output unit 103 may include loud speaker, buzzer etc..
A/V input units 104 are for receiving audio or video signal.A/V input units 104 may include graphics processor
(Graphics Processing Unit, GPU) 1041 and microphone 1042, graphics processor 1041 is in video acquisition mode
Or the image data of the static images or video obtained by image capture apparatus (such as camera) in image capture mode carries out
Reason.Treated, and picture frame may be displayed on display unit 106.Through graphics processor 1041, treated that picture frame can be deposited
Storage is sent in memory 109 (or other storage mediums) or via radio frequency unit 101 or WiFi module 102.Mike
Wind 1042 can connect in telephone calling model, logging mode, speech recognition mode etc. operational mode via microphone 1042
Quiet down sound (audio data), and can be audio data by such acoustic processing.Audio that treated (voice) data can
To be converted to the format output that can be sent to mobile communication base station via radio frequency unit 101 in the case of telephone calling model.
Microphone 1042 can implement various types of noises elimination (or inhibition) algorithms and send and receive sound to eliminate (or inhibition)
The noise generated during frequency signal or interference.
Mobile terminal 100 further includes at least one sensor 105, such as optical sensor, motion sensor and other biographies
Sensor.Specifically, optical sensor includes ambient light sensor and proximity sensor, wherein ambient light sensor can be according to environment
The light and shade of light adjusts the brightness of display panel 1061, and proximity sensor can close when mobile terminal 100 is moved in one's ear
Display panel 1061 and/or backlight.As a kind of motion sensor, accelerometer sensor can detect in all directions (general
For three axis) size of acceleration, size and the direction of gravity are can detect that when static, can be used to identify the application of mobile phone posture
(such as horizontal/vertical screen switching, dependent game, magnetometer pose calibrating), Vibration identification correlation function (such as pedometer, percussion) etc.；
The fingerprint sensor that can also configure as mobile phone, pressure sensor, iris sensor, molecule sensor, gyroscope, barometer,
The other sensors such as hygrometer, thermometer, infrared sensor, details are not described herein.
Display unit 106 is for showing information input by user or being supplied to the information of user.Display unit 106 can wrap
Display panel 1061 is included, liquid crystal display (Liquid Crystal Display, LCD), Organic Light Emitting Diode may be used
Forms such as (Organic Light-Emitting Diode, OLED) configure display panel 1061.
User input unit 107 can be used for receiving the number or character information of input, and generate the use with mobile terminal
Family is arranged and the related key signals input of function control.Specifically, user input unit 107 may include touch panel 1071 with
And other input equipments 1072.Touch panel 1071, also referred to as touch screen collect user on it or neighbouring touch operation
(for example user uses any suitable objects or attachment such as finger, stylus on touch panel 1071 or in touch panel 1071
Neighbouring operation), and corresponding attachment device is driven according to preset formula.Touch panel 1071 may include touch detection
Two parts of device and touch controller.Wherein, the touch orientation of touch detecting apparatus detection user, and detect touch operation band
The signal come, transmits a signal to touch controller；Touch controller receives touch information from touch detecting apparatus, and by it
It is converted into contact coordinate, then gives processor 110, and order that processor 110 is sent can be received and executed.In addition, can
To realize touch panel 1071 using multiple types such as resistance-type, condenser type, infrared ray and surface acoustic waves.In addition to touch panel
1071, user input unit 107 can also include other input equipments 1072.Specifically, other input equipments 1072 can wrap
It includes but is not limited in physical keyboard, function key (such as volume control button, switch key etc.), trace ball, mouse, operating lever etc.
It is one or more, do not limit herein specifically.
Further, touch panel 1071 can cover display panel 1061, when touch panel 1071 detect on it or
After neighbouring touch operation, processor 110 is sent to determine the type of touch event, is followed by subsequent processing device 110 according to touch thing
The type of part provides corresponding visual output on display panel 1061.Although in Fig. 1, touch panel 1071 and display panel
1061 be to realize the function that outputs and inputs of mobile terminal as two independent components, but in certain embodiments, can
The function that outputs and inputs of mobile terminal is realized so that touch panel 1071 and display panel 1061 is integrated, is not done herein specifically
It limits.
Interface unit 108 be used as at least one external device (ED) connect with mobile terminal 100 can by interface.For example,
External device (ED) may include wired or wireless headphone port, external power supply (or battery charger) port, wired or nothing
Line data port, memory card port, the port for connecting the device with identification module, audio input/output (I/O) end
Mouth, video i/o port, ear port etc..Interface unit 108 can be used for receiving the input from external device (ED) (for example, number
It is believed that breath, electric power etc.) and the input received is transferred to one or more elements in mobile terminal 100 or can be with
For the transmission data between mobile terminal 100 and external device (ED).
Memory 109 can be used for storing software program and various data.Memory 109 can include mainly storing program area
And storage data field, wherein storing program area can storage program area, application program (such as the sound needed at least one function
Sound playing function, image player function etc.) etc.；Storage data field can store according to mobile phone use created data (such as
Audio data, phone directory etc.) etc..In addition, memory 109 may include high-speed random access memory, can also include non-easy
The property lost memory, a for example, at least disk memory, flush memory device or other volatile solid-state parts.
Processor 110 is the control centre of mobile terminal, utilizes each of various interfaces and the entire mobile terminal of connection
A part by running or execute the software program and/or module that are stored in memory 109, and calls and is stored in storage
Data in device 109 execute the various functions and processing data of mobile terminal, to carry out integral monitoring to mobile terminal.Place
Reason device 110 may include one or more processing units；Preferably, processor 110 can integrate application processor and modulatedemodulate is mediated
Manage device, wherein the main processing operation system of application processor, user interface and application program etc., modem processor is main
Processing wireless communication.It is understood that above-mentioned modem processor can not also be integrated into processor 110.
Mobile terminal 100 can also include the power supply 111 (such as battery) powered to all parts, it is preferred that power supply 111
Can be logically contiguous by power-supply management system and processor 110, to realize management charging by power-supply management system, put
The functions such as electricity and power managed.
Although Fig. 1 is not shown, mobile terminal 100 can also be including bluetooth module etc., and details are not described herein.
Embodiment to facilitate the understanding of the present invention, below to the communications network system that is based on of mobile terminal of the present invention into
Row description.
Referring to Fig. 2, Fig. 2 is a kind of communications network system Organization Chart provided in an embodiment of the present invention, the communication network system
System is the LTE system of universal mobile communications technology, which includes communicating UE (User Equipment, the use of connection successively
Family equipment) (the lands Evolved UMTS Terrestrial Radio Access Network, evolved UMTS 201, E-UTRAN
Ground wireless access network) 202, EPC (Evolved Packet Core, evolved packet-based core networks) 203 and operator IP operation
204。
Specifically, UE201 can be above-mentioned terminal 100, and details are not described herein again.
E-UTRAN202 includes eNodeB2021 and other eNodeB2022 etc..Wherein, eNodeB2021 can be by returning
Journey (backhaul) (such as X2 interface) is connect with other eNodeB2022, and eNodeB2021 is connected to EPC203,
ENodeB2021 can provide the access of UE201 to EPC203.
EPC203 may include MME (Mobility Management Entity, mobility management entity) 2031, HSS
(Home Subscriber Server, home subscriber server) 2032, other MME2033, SGW (Serving Gate Way,
Gateway) 2034, PGW (PDN Gate Way, grouped data network gateway) 2035 and PCRF (Policy and
Charging Rules Function, policy and rate functional entity) 2036 etc..Wherein, MME2031 be processing UE201 and
The control node of signaling, provides carrying and connection management between EPC203.HSS2032 is all to manage for providing some registers
Such as the function of home location register (not shown) etc, and some are preserved in relation to use such as service features, data rates
The dedicated information in family.All customer data can be sent by SGW2034, and PGW2035 can provide the IP of UE 201
Address is distributed and other functions, and PCRF2036 is strategy and the charging control strategic decision-making of business data flow and IP bearing resources
Point, it selects and provides available strategy and charging control decision with charge execution function unit (not shown) for strategy.
IP operation 204 may include internet, Intranet, IMS (IP Multimedia Subsystem, IP multimedia
System) or other IP operations etc..
Although above-mentioned be described by taking LTE system as an example, those skilled in the art it is to be understood that the present invention not only
Suitable for LTE system, be readily applicable to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA with
And the following new network system etc., it does not limit herein.
Based on above-mentioned mobile terminal hardware configuration and communications network system, each embodiment of the method for the present invention is proposed.
First embodiment
Before illustrating the present embodiment, the kafka terms in embodiment are introduced and are illustrated below：
The message producer：That is Producer is the source of the generation of message, is responsible for generating information and sending to Kafka clothes
It is engaged on device.
Message consumer：That is Consumer is the user of message, the message being responsible on consumption Kafka servers.
Theme：That is Topic is defined by the user and configures in Kafka servers, for establishing between the producer and message person
Subscribing relationship：The producer sends a message under specified Topic, and message person consumes message under this Topic.
Message partition：That is it can be divided into many subregions below Partition, a Topic, such as：" kafka-test " this
6 subregions can be divided under a Topic, provided respectively by two-server, then being usually configurable to allow every server
3 subregions are provided, if server ID is respectively 0,1, then all subregions are 0-0,0-1,0-2 and 1-0,1-1,1-2.
The groupings of Topic physically, a topic can be divided into multiple partition, and each partition is an orderly team
Row.Every message in partition can all be assigned an orderly id (offset).
Consumer is grouped：That is Group, for being grouped into similar consumer, in Kafka, multiple consumers can disappear jointly
Take the message under a Topic, each consumer spending part messages therein, these consumers just constitute a grouping,
Possess the same group names, is also generally referred to as consumer's cluster.
Offset：Message is stored on the Broker of Kafka, and consumer needs to know during pulling message data to disappear
The offset of breath hereof, this offset is exactly so-called Offset.
Broker：That is the server of Kafka, user store message, one or more server in Kafka clusters is referred to as
For broker.
Zookeeper is that one kind being widely used for conduct in a distributed system：Distributed state management, distributed coordination
The cluster of management, decentralized configuration management and distributed lock service.Kafka increases and decreases server all can be in Zookeeper
Corresponding event is triggered on node, Kafka systems can capture these events, carry out the load balancing of a new round, and client also can
These events are captured to carry out the processing of a new round.
As shown in figure 3, first embodiment of the invention provides a kind of message treatment method, the method includes the steps：
S31, after old consumer thread breaks down, start spare consumer thread and Xin consumer threads.
Correlation distribution formula distribution subscription message system cluster relevant environment is built first, and downloads related tool jar packets,
Correlation distribution formula distribution subscription message system consumer client's code is write, and the maximum number of local thread processing message is set
Amount (for example, the maximum quantity for defining local thread processing message is TASKCOUNT) and distributed post subscribe to message system
Time-out time (for example, define distributed post subscribe to message system time-out time be session.timeout.ms),
In, distributed post subscribe to message system the max-timeout time be distributed post subscribe to message system maximum heartbeat between
Every the time, the numerical value of system default, or sets itself according to actual needs could be provided as.
Before old consumer thread does not break down, old consuming thread obtains data in batches from Kafka servers,
The offset location informations of consumption Kafka server messages can be recorded in zookeeper.Newly-increased Redis caching components simultaneously,
After the offset location informations for recording consumption in zookeeper, while the positions offset of consumption are recorded in Redis cachings
Information.
After old consumer thread breaks down, it need to restart what Kafka servers were written in consumer thread in real time
Message and the message of non real-time write-in are consumed.The present embodiment by start spare consumer thread and Xin consumer threads come
Solve the consumption of write-in message in real time and non real-time write-in message.
In the present embodiment, the failure includes that old consumer thread's delay machine failure and/or old consumer thread stop event
Barrier.Spare consumer thread and Xin consumer threads can be the consumer in same consumer's cluster, can not also be same
Consumer in consumer's cluster, this is not restricted.
In one embodiment, described after old consumer thread breaks down, start spare consumer thread and Xin
Consumer thread includes step：
Detect whether the old consumer thread breaks down；
If the old consumer thread breaks down, start spare consumer thread and Xin consumer threads.
S32, the spare consumer thread obtain the first offset location from preset buffer memory, and from manager
The second offset location is obtained in zookeeper；It is consumed according to first offset location and second offset location
The message of the non real-time write-in of server；Wherein, first offset bit is set to old consumer thread's consumption service device and disappears
Offset location after breath.
In the present embodiment, preset buffer memory caches for Redis above-mentioned, and the first offset bit, which is set in Redis cachings, to be remembered
The offset location informations of the consumption of record before that is, old consumer thread does not break down, are consumed inclined after Kafka server messages
Shifting amount position.
In the present embodiment, the second offset bit is set to the offset location of latest news, i.e. consumer thread opens again
After dynamic, the message that is written in Kafka servers.
In one embodiment, described according to first offset location and second offset location consumption clothes
The message of the business non real-time write-in of device includes step：
Judge whether first offset location and second offset location are identical；
If first offset location and second offset location differ, the spare consumer thread disappears
Take the message between first offset location and second offset location.
In this embodiment, the spare consumer thread consumes first offset location and second offset
The message measured between position includes step：
The spare consumer thread starts multithreading, and by the multithreading consume first offset location and
Message between second offset location.
In this embodiment, disappearing between the first offset location and the second offset location is handled by multithreading
Breath, improves handling capacity and Consumption rate.
In this embodiment, if first offset location and second offset location differ,
The spare consumer thread goes back after consuming the message between first offset location and second offset location
Including step：
It finishes, closes if the message between first offset location and second offset location has been consumed
The spare consumer thread.
In another embodiment, the spare consumer thread obtains the first offset location from preset buffer memory,
And the second offset location is obtained from manager zookeeper；According to first offset location and second offset
The message for measuring the non real-time write-in of position consumption service device further includes before step：
The second offset location in the zookeeper is set.
In this embodiment, by being configured to the second offset location in zookeeper, it can be ensured that second partially
It is directed toward the offset location of latest news in shifting amount position.
The message that S33, the new consumer thread are written in real time according to the second offset location consumption service device.
In the present embodiment, new consumer thread directly carries out consumption processing from the offset location of latest news, and standby
With the offset location of offset location and latest news of the consumer thread after old consumer thread's consumption service device message
Between carry out consumption processing, spare consumer thread can close after consumption finishes.On the one hand this mode can guarantee
Latest news can be handled, and the delay of Message Processing is reduced, and on the other hand can ensure that disappearing during consumer thread's failure
Breath can obtain supplement process without losing.
In one embodiment, the new consumer thread is real-time according to the second offset location consumption service device
Further include step after the message of write-in：
It is finished if the new consumer thread has consumed, updates the first offset location in the preset buffer memory.
In order to preferably illustrate the present embodiment, message processing procedure is described in detail below in conjunction with Fig. 4：
It is assumed that in consumer thread's delay machine or before stopping, successfully consumption is handled consumer thread
To the position (offset1 i.e. in figure) of offset=20.After restarting consumer thread, the offset bit of latest news
It is set to the position (offset2 i.e. in figure) of offset=40.
Existing scheme is typically to the processing mode of this situation, from start to process message from the positions offset1.This scheme
The problem is that：If offset2 to the message between offset1 it is more if (downtime is longer, message writing mistake
It is more, can all cause offset2 more to the message between offset1), then consumer thread needs that a period of time is spent to handle
Offset2, could be from the offset location offset2 start to process message of latest news, in turn to the message between offset1
Lead to the delay of Message Processing.
For such situation, spare consumer thread and Xin consumer threads can be started, handled by spare consumer thread
Offset2 is to the message between offset1, and new consumer thread is directly since the offset location offset2 of latest news
Handle message.After spare consumer thread has handled offset2 to the message between offset1, spare consumption is directly closed
Person's thread.
Message treatment method provided in an embodiment of the present invention consumes clothes by spare consumer thread and Xin consumer threads
Business device is non real-time and the message that is written in real time；On the one hand it can guarantee that latest news can be handled, and Message Processing is reduced
On the other hand delay can ensure that the message during consumer thread's failure can obtain supplement process without losing.
Second embodiment
A kind of message processing device is provided for second embodiment of the invention with reference to Fig. 5, Fig. 5, the equipment 40 includes：Storage
Device 41, processor 42 and it is stored in the message processing program that can be run on the memory 41 and on the processor 42, institute
When stating message processing program and being executed by the processor 42, the step of for realizing message treatment method as described below：
After old consumer thread breaks down, start spare consumer thread and Xin consumer threads；
The spare consumer thread obtains the first offset location from preset buffer memory, and from manager zookeeper
The second offset location of middle acquisition；According to first offset location and the non-reality of the second offset location consumption service device
When the message that is written；Wherein, first offset bit is set to the offset after old consumer thread's consumption service device message
Measure position；
The message that the new consumer thread is written in real time according to the second offset location consumption service device.
When the message processing program is executed by the processor 42, it is additionally operable to realize message treatment method as described below
The step of：
Judge whether first offset location and second offset location are identical；
If first offset location and second offset location differ, the spare consumer thread disappears
Take the message between first offset location and second offset location.
When the message processing program is executed by the processor 42, it is additionally operable to realize message treatment method as described below
The step of：
The spare consumer thread starts multithreading, and by the multithreading consume first offset location and
Message between second offset location.
When the message processing program is executed by the processor 42, it is additionally operable to realize message treatment method as described below
The step of：
It finishes, closes if the message between first offset location and second offset location has been consumed
The spare consumer thread.
When the message processing program is executed by the processor 42, it is additionally operable to realize message treatment method as described below
The step of：
The second offset location in the zookeeper is set.
When the message processing program is executed by the processor 42, it is additionally operable to realize message treatment method as described below
The step of：
It is finished if the new consumer thread has consumed, updates the first offset location in the preset buffer memory.
When the message processing program is executed by the processor 42, it is additionally operable to realize message treatment method as described below
The step of：
Detect whether the old consumer thread breaks down；
If the old consumer thread breaks down, start spare consumer thread and Xin consumer threads.
When the message processing program is executed by the processor 42, it is additionally operable to realize message treatment method as described below
The step of：
The failure includes that old consumer thread's delay machine failure and/or old consumer thread stop failure.
Message processing device provided in an embodiment of the present invention consumes clothes by spare consumer thread and Xin consumer threads
Business device is non real-time and the message that is written in real time；On the one hand it can guarantee that latest news can be handled, and Message Processing is reduced
On the other hand delay can ensure that the message during consumer thread's failure can obtain supplement process without losing.
3rd embodiment
Third embodiment of the invention provides a kind of computer readable storage medium, is deposited on the computer readable storage medium
Message processing program is contained, the message processing described in first embodiment is realized when the message processing program is executed by processor
The step of method.
Computer readable storage medium provided in an embodiment of the present invention passes through spare consumer thread and Xin consumer threads
Consumption service device is non real-time and the message that is written in real time；On the one hand it can guarantee that latest news can be handled, and message is reduced
On the other hand the delay of processing can ensure that the message during consumer thread's failure can obtain supplement process without losing.
It should be noted that herein, the terms "include", "comprise" or its any other variant are intended to non-row
His property includes, so that process, method, article or device including a series of elements include not only those elements, and
And further include other elements that are not explicitly listed, or further include for this process, method, article or device institute it is intrinsic
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including this
There is also other identical elements in the process of element, method, article or device.
The embodiments of the present invention are for illustration only, can not represent the quality of embodiment.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can add the mode of required general hardware platform to realize by software, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on this understanding, technical scheme of the present invention substantially in other words does the prior art
Going out the part of contribution can be expressed in the form of software products, which is stored in a storage medium
In (such as ROM/RAM, magnetic disc, CD), including some instructions are used so that a station terminal (can be mobile phone, computer, service
Device, air conditioner or network equipment etc.) execute method described in each embodiment of the present invention.
The embodiment of the present invention is described with above attached drawing, but the invention is not limited in above-mentioned specific
Embodiment, the above mentioned embodiment is only schematical, rather than restrictive, those skilled in the art
Under the inspiration of the present invention, without breaking away from the scope protected by the purposes and claims of the present invention, it can also make very much
Form, all of these belong to the protection of the present invention.
Claims (10)
1. a kind of message treatment method, which is characterized in that the method includes：
After old consumer thread breaks down, start spare consumer thread and Xin consumer threads；
The spare consumer thread obtains the first offset location from preset buffer memory, and is obtained from manager zookeeper
Take the second offset location；According to first offset location and the second offset location consumption service device is non real-time writes
The message entered；Wherein, first offset bit is set to the offset bit after old consumer thread's consumption service device message
It sets；
The message that the new consumer thread is written in real time according to the second offset location consumption service device.
2. message treatment method according to claim 1, which is characterized in that it is described according to first offset location and
The message of the non real-time write-in of the second offset location consumption service device includes step：
Judge whether first offset location and second offset location are identical；
If first offset location and second offset location differ, the spare consumer thread consumes institute
State the message between the first offset location and second offset location.
3. message treatment method according to claim 2, which is characterized in that the spare consumer thread consumption described the
Message between one offset location and second offset location includes step：
The spare consumer thread starts multithreading, and consumes first offset location and described by the multithreading
Message between second offset location.
4. message treatment method according to claim 2, which is characterized in that if first offset location and institute
It states the second offset location to differ, then the spare consumer thread consumes first offset location and described second partially
Further include step after message between shifting amount position：
It is finished if the message between first offset location and second offset location has been consumed, described in closing
Spare consumer thread.
5. message treatment method according to claim 1, which is characterized in that the spare consumer thread is from preset buffer memory
The first offset location of middle acquisition, and the second offset location is obtained from manager zookeeper；It is deviated according to described first
The message for measuring position and the non real-time write-in of the second offset location consumption service device further includes before step：
The second offset location in the zookeeper is set.
6. message treatment method according to claim 1, which is characterized in that the new consumer thread is according to described second
Further include step after the message that offset location consumption service device is written in real time：
It is finished if the new consumer thread has consumed, updates the first offset location in the preset buffer memory.
7. message treatment method according to claim 1, which is characterized in that described to break down in old consumer thread
Afterwards, it includes step to start spare consumer thread and Xin consumer threads：
Detect whether the old consumer thread breaks down；
If the old consumer thread breaks down, start spare consumer thread and Xin consumer threads.
8. message treatment method according to claim 7, which is characterized in that the failure includes old consumer thread's delay machine
Failure and/or old consumer thread stop failure.
9. a kind of message processing device, which is characterized in that the equipment includes：Memory, processor and it is stored in the storage
It is real when the message processing program is executed by the processor on device and the message processing program that can run on the processor
Now such as the step of message treatment method described in any item of the claim 1 to 8.
10. a kind of computer readable storage medium, which is characterized in that be stored at message on the computer readable storage medium
Program is managed, such as Message Processing described in any item of the claim 1 to 8 is realized when the message processing program is executed by processor
The step of method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810274276.7A CN108509299B (en) | 2018-03-29 | 2018-03-29 | Message processing method, device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810274276.7A CN108509299B (en) | 2018-03-29 | 2018-03-29 | Message processing method, device and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108509299A true CN108509299A (en) | 2018-09-07 |
| CN108509299B CN108509299B (en) | 2022-08-12 |
Family
ID=63379249
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810274276.7A Active CN108509299B (en) | 2018-03-29 | 2018-03-29 | Message processing method, device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108509299B (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110474917A (en) * | 2019-08-16 | 2019-11-19 | 深圳前海微众银行股份有限公司 | The upper and lower line method of message-oriented middleware, device, equipment and readable storage medium storing program for executing |
| CN110968370A (en) * | 2019-11-18 | 2020-04-07 | 中国银行股份有限公司 | Data processing method and device |
| CN111124650A (en) * | 2019-12-26 | 2020-05-08 | 中国建设银行股份有限公司 | Streaming data processing method and device |
| CN111124710A (en) * | 2019-12-17 | 2020-05-08 | 联想(北京)有限公司 | Data processing method and electronic equipment |
| CN112000489A (en) * | 2020-07-29 | 2020-11-27 | 新华三大数据技术有限公司 | Kafka data processing method and server |
| CN112286723A (en) * | 2020-09-30 | 2021-01-29 | 北京大米科技有限公司 | Computer room disaster recovery control method, terminal and storage medium |
| CN112380030A (en) * | 2020-11-09 | 2021-02-19 | 中国建设银行股份有限公司 | Main task processing method, device, server and storage medium |
| CN112506992A (en) * | 2020-12-04 | 2021-03-16 | 中国人寿保险股份有限公司 | Fuzzy query method and device for Kafka data, electronic equipment and storage medium |
| CN112612606A (en) * | 2020-12-16 | 2021-04-06 | 平安消费金融有限公司 | Message theme processing method and device, computer equipment and readable storage medium |
| CN113297309A (en) * | 2021-05-31 | 2021-08-24 | 平安证券股份有限公司 | Stream data writing method, device, equipment and storage medium |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050021836A1 (en) * | 2003-05-01 | 2005-01-27 | Reed Carl J. | System and method for message processing and routing |
| US20130151783A1 (en) * | 2011-12-13 | 2013-06-13 | International Business Machines Corporation | Interface and method for inter-thread communication |
| CN104424186A (en) * | 2013-08-19 | 2015-03-18 | 阿里巴巴集团控股有限公司 | Method and device for realizing persistence in flow calculation application |
| CN105306552A (en) * | 2015-09-30 | 2016-02-03 | 中国科学院计算技术研究所 | Consumption equilibrium method and system based on message queues |
| US20170063965A1 (en) * | 2015-08-25 | 2017-03-02 | Denis Grenader | Data transfer in a collaborative file sharing system |
| CN106657349A (en) * | 2016-12-29 | 2017-05-10 | 上海理想信息产业（集团）有限公司 | Message subscription processing device, system and method |
| CN106648933A (en) * | 2016-12-26 | 2017-05-10 | 北京奇虎科技有限公司 | Consuming method and device of message queue |
| CN106817295A (en) * | 2016-12-08 | 2017-06-09 | 努比亚技术有限公司 | A kind of message processing apparatus and method |
| US20170242887A1 (en) * | 2016-02-24 | 2017-08-24 | Salesforce.Com, Inc. | Efficient access scheduling for super scaled stream processing systems |
| US20170339238A1 (en) * | 2016-05-23 | 2017-11-23 | Linkedin Corporation | Large message support for a publish-subscribe messaging system |
| CN107423141A (en) * | 2017-06-20 | 2017-12-01 | 北京小度信息科技有限公司 | Information processing method and device |
| CN107423145A (en) * | 2017-07-11 | 2017-12-01 | 北京潘达互娱科技有限公司 | A kind of method and apparatus for avoiding information drop-out |
-
2018
- 2018-03-29 CN CN201810274276.7A patent/CN108509299B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050021836A1 (en) * | 2003-05-01 | 2005-01-27 | Reed Carl J. | System and method for message processing and routing |
| US20130151783A1 (en) * | 2011-12-13 | 2013-06-13 | International Business Machines Corporation | Interface and method for inter-thread communication |
| CN104424186A (en) * | 2013-08-19 | 2015-03-18 | 阿里巴巴集团控股有限公司 | Method and device for realizing persistence in flow calculation application |
| US20170063965A1 (en) * | 2015-08-25 | 2017-03-02 | Denis Grenader | Data transfer in a collaborative file sharing system |
| CN105306552A (en) * | 2015-09-30 | 2016-02-03 | 中国科学院计算技术研究所 | Consumption equilibrium method and system based on message queues |
| US20170242887A1 (en) * | 2016-02-24 | 2017-08-24 | Salesforce.Com, Inc. | Efficient access scheduling for super scaled stream processing systems |
| US20170339238A1 (en) * | 2016-05-23 | 2017-11-23 | Linkedin Corporation | Large message support for a publish-subscribe messaging system |
| CN106817295A (en) * | 2016-12-08 | 2017-06-09 | 努比亚技术有限公司 | A kind of message processing apparatus and method |
| CN106648933A (en) * | 2016-12-26 | 2017-05-10 | 北京奇虎科技有限公司 | Consuming method and device of message queue |
| CN106657349A (en) * | 2016-12-29 | 2017-05-10 | 上海理想信息产业（集团）有限公司 | Message subscription processing device, system and method |
| CN107423141A (en) * | 2017-06-20 | 2017-12-01 | 北京小度信息科技有限公司 | Information processing method and device |
| CN107423145A (en) * | 2017-07-11 | 2017-12-01 | 北京潘达互娱科技有限公司 | A kind of method and apparatus for avoiding information drop-out |
Non-Patent Citations (4)
| Title |
|---|
| 王岩等: "一种基于Kafka的可靠的Consumer的设计方案", 《软件》 * |
| 王岩等: "一种基于Kafka的可靠的Consumer的设计方案", 《软件》, no. 01, 15 January 2016 (2016-01-15) * |
| 马浩然: "基于NS3的分布式消息系统Kafka的仿真实现", 《软件》 * |
| 马浩然: "基于NS3的分布式消息系统Kafka的仿真实现", 《软件》, no. 01, 15 January 2015 (2015-01-15) * |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110474917A (en) * | 2019-08-16 | 2019-11-19 | 深圳前海微众银行股份有限公司 | The upper and lower line method of message-oriented middleware, device, equipment and readable storage medium storing program for executing |
| CN110968370A (en) * | 2019-11-18 | 2020-04-07 | 中国银行股份有限公司 | Data processing method and device |
| CN110968370B (en) * | 2019-11-18 | 2024-02-23 | 中国银行股份有限公司 | Data processing method and device |
| CN111124710A (en) * | 2019-12-17 | 2020-05-08 | 联想(北京)有限公司 | Data processing method and electronic equipment |
| CN111124650B (en) * | 2019-12-26 | 2023-10-24 | 中国建设银行股份有限公司 | Stream data processing method and device |
| CN111124650A (en) * | 2019-12-26 | 2020-05-08 | 中国建设银行股份有限公司 | Streaming data processing method and device |
| CN112000489A (en) * | 2020-07-29 | 2020-11-27 | 新华三大数据技术有限公司 | Kafka data processing method and server |
| CN112286723A (en) * | 2020-09-30 | 2021-01-29 | 北京大米科技有限公司 | Computer room disaster recovery control method, terminal and storage medium |
| CN112380030A (en) * | 2020-11-09 | 2021-02-19 | 中国建设银行股份有限公司 | Main task processing method, device, server and storage medium |
| CN112506992A (en) * | 2020-12-04 | 2021-03-16 | 中国人寿保险股份有限公司 | Fuzzy query method and device for Kafka data, electronic equipment and storage medium |
| CN112506992B (en) * | 2020-12-04 | 2024-04-16 | 中国人寿保险股份有限公司 | Fuzzy query method and device for Kafka data, electronic equipment and storage medium |
| CN112612606A (en) * | 2020-12-16 | 2021-04-06 | 平安消费金融有限公司 | Message theme processing method and device, computer equipment and readable storage medium |
| CN113297309A (en) * | 2021-05-31 | 2021-08-24 | 平安证券股份有限公司 | Stream data writing method, device, equipment and storage medium |
| CN113297309B (en) * | 2021-05-31 | 2023-11-10 | 平安证券股份有限公司 | Stream data writing method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108509299B (en) | 2022-08-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108509299A (en) | Message treatment method, equipment and computer readable storage medium | |
| CN108536481A (en) | A kind of application program launching method, mobile terminal and computer storage media | |
| CN108521658A (en) | Reduce interference method, mobile terminal and computer readable storage medium | |
| CN109040441A (en) | Using display methods of attending to anything else, mobile terminal and computer readable storage medium | |
| CN107807767A (en) | A kind of processing method of communication traffic, terminal and computer-readable recording medium | |
| CN108600513A (en) | A kind of record screen control method, equipment and computer readable storage medium | |
| CN108958936A (en) | Application programe switch-over method, mobile terminal and computer readable storage medium | |
| CN109582565A (en) | Prevent the method, terminal and computer storage medium of application crashes | |
| CN108762926A (en) | A kind of system optimization method, terminal and computer readable storage medium | |
| CN109545120A (en) | Screen photosensitive sensors adjusting method, mobile terminal and computer readable storage medium | |
| CN109309762A (en) | Message treatment method, device, mobile terminal and storage medium | |
| CN108536529A (en) | A kind of multitask regulation and control method, equipment and computer readable storage medium | |
| CN108696909A (en) | Based reminding method, terminal and the computer readable storage medium of network connection | |
| CN108449736A (en) | A kind of data transmission method, terminal and computer readable storage medium | |
| CN108449806A (en) | A kind of information processing method, terminal and readable storage medium storing program for executing | |
| CN108306856A (en) | A kind of interface merging method, client, server and computer readable storage medium | |
| CN108366303A (en) | A kind of video broadcasting method, mobile terminal and computer readable storage medium | |
| CN109947519A (en) | Hardware-accelerated control method, terminal and computer readable storage medium | |
| CN110046151A (en) | A kind of data cleaning method, server and computer readable storage medium | |
| CN110096194A (en) | Terminal unlocking control method, device, mobile terminal and readable storage medium storing program for executing | |
| CN109388947A (en) | A kind of back-stage management method, equipment and computer readable storage medium | |
| CN108364595A (en) | A kind of Electro-static Driven Comb detection method, mobile terminal and computer readable storage medium | |
| CN108566476A (en) | A kind of information processing method, terminal and computer readable storage medium | |
| CN110278124A (en) | The monitoring method of interface, device and computer readable storage medium on line | |
| CN109451181A (en) | Information display method, dual-screen mobile terminal and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220728 Address after: No. 6, Minzhu Road, Xingning District, Nanning City, Guangxi Zhuang Autonomous Region, 530012 Applicant after: GUANGXI POWER GRID Co.,Ltd. Address before: 518000 6-8 / F, 10-11 / F, 6 / F, 6-10 / F, C zone, Han's innovation building, No. 9018, Beihuan Avenue, high tech Zone, Nanshan District, Shenzhen City, Guangdong Province Applicant before: NUBIA TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |