CN106302385A - A kind of message distribution device and method - Google Patents

A kind of message distribution device and method Download PDF

Info

Publication number
CN106302385A
CN106302385A CN201610595099.3A CN201610595099A CN106302385A CN 106302385 A CN106302385 A CN 106302385A CN 201610595099 A CN201610595099 A CN 201610595099A CN 106302385 A CN106302385 A CN 106302385A
Authority
CN
China
Prior art keywords
message
sub
read
line journey
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201610595099.3A
Other languages
Chinese (zh)
Other versions
CN106302385B (en
Inventor
林伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nubia Technology Co Ltd
Original Assignee
Nubia Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nubia Technology Co Ltd filed Critical Nubia Technology Co Ltd
Priority to CN201610595099.3A priority Critical patent/CN106302385B/en
Publication of CN106302385A publication Critical patent/CN106302385A/en
Application granted granted Critical
Publication of CN106302385B publication Critical patent/CN106302385B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60Network streaming of media packets
    • H04L65/61Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
    • H04L65/611Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio for multicast or broadcast
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/02Marketing; Price estimation or determination; Fundraising
    • G06Q30/0241Advertisements
    • G06Q30/0251Targeted advertisements
    • G06Q30/0257User requested

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Business, Economics & Management (AREA)
  • Strategic Management (AREA)
  • Accounting & Taxation (AREA)
  • Development Economics (AREA)
  • Finance (AREA)
  • Economics (AREA)
  • Game Theory and Decision Science (AREA)
  • Marketing (AREA)
  • Physics & Mathematics (AREA)
  • General Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Mobile Radio Communication Systems (AREA)

Abstract

Present invention is primarily targeted at a kind of message distribution device and method of proposition, it is intended to solving the problem of information consumption rate-constrained based on Kafka system in prior art, the method includes: from Kafka system, read message;The message read is divided into and organizes message more;Create multiple sub-line journey;Many groups message after packet is respectively sent to each sub-line journey, and the program improves the treatment effeciency of message.

Description

A kind of message distribution device and method
Technical field
Present invention distribution is distribution subscription message system, particularly relates to a kind of message distribution device and method.
Background technology
Kafka is that the distributed post of a kind of high-throughput subscribes to message system, and it can process the net of consumer's scale All of stream of action data in standing (such as, web page browsing, search and the action of other users) it is the many on modern network One key factor of social function.These data be often as the requirement of handling capacity and by processing daily record and log aggregation Solve.For the daily record data as Hadoop and off-line analysis system, the restriction processed time the most realistic, this is one Individual feasible solution.The purpose of Kafka is that the loaded in parallel mechanism by Hadoop is unified on line and the message of off-line Process, provide real-time consumption also for by cluster machine.
Consuming thread number and the partition (subregion) of topic (theme) in Kafka framework have a relation (class message Have a theme, when each theme creates, need to fix the number of partitions of this theme), at a time, a subregion can only have One consumer thread is connected, and kafka design is i.e. such.So, if project needs size of message to be processed to reach one Determine bottleneck, the Consumption rate of message will be limited.
At present, system is typically by newly increasing the quantity of the subregion of theme, thus the implementation increasing consumer is come Solve the problem that Consumption rate is limited, but this mode makes to need newly-increased customer count, is required for deleting theme every time Re-establish the quantity of the relative subregion of theme, or when newly setting up theme for the first time, bigger the dividing of quantity is just set District's quantity, but this mode underaction, and the expansion that the quantity of subregion can not be unlimited.
Summary of the invention
Present invention is primarily targeted at a kind of message distribution device and method of proposition, it is intended to solve in prior art based on The problem of the information consumption rate-constrained of Kafka system.
The invention provides a kind of message distribution device, this device includes: read module, for reading from Kafka system Cancel breath;Divide module, organize message for the message read being divided into more;Creation module, is used for creating multiple sub-line journey; Sending module, for being respectively sent to each sub-line journey by organizing message more.
Optionally, above-mentioned division module specifically for: the message read is divided into impartial many groups of message number and disappears Breath.
Optionally, above-mentioned creation module is specifically for the son of the group number same number of: the many groups message after creating and dividing Thread.
Optionally, above-mentioned read module specifically for: call the interface of Kafka consumer client to read Kafka system Message in system.
Optionally, above-mentioned creation module specifically for: create predetermined number sub-line journey;Or refer to according to the outside received Order creates the sub-line journey of number corresponding with external command.
According to another aspect of the present invention, it is provided that a kind of message distributing method, the method comprising the steps of: from Kafka System reads message;The message read is divided into and organizes message more;Create multiple sub-line journey;
It is respectively sent to each sub-line journey by organizing message more.
Optionally, the message read is divided into and organizes message more, including: the message read is divided into message number Impartial many groups message.
Optionally, create multiple sub-line journey, including: the sub-line of the group number same number of the many groups message after creating and dividing Journey.
Optionally, from Kafka system, read message, including: the interface calling Kafka consumer client reads Message in Kafka system.
Optionally, create multiple sub-line journey, including: create the sub-line journey of predetermined number;Or refer to according to the outside received Order creates the sub-line journey of number corresponding with external command.
The scheme that the embodiment of the present invention provides, is grouped the message read from Kafka system, and based on packet count Mesh sets up sub-line journey, processes the message after packet based on the sub-line journey set up, improves the treatment effeciency of message.
Accompanying drawing explanation
Fig. 1 is the optional terminal realizing each embodiment of the present invention for carrying Kafka consumer client Hardware architecture diagram;
Fig. 2 is the wireless communication system schematic diagram of terminal as shown in Figure 1;
Fig. 3 is the structural relation schematic diagram of Kafka cluster and the consumer related in first embodiment of the invention;
Fig. 4 is the structured flowchart of the message distribution device provided in first embodiment of the invention;
Fig. 5 is the flow chart of the message distributing method provided in third embodiment of the invention.
The realization of the object of the invention, functional characteristics and advantage will in conjunction with the embodiments, are described further referring to the drawings.
Detailed description of the invention
Should be appreciated that specific embodiment described herein, only in order to explain the present invention, is not intended to limit the present invention.
The mobile terminal realizing each embodiment of the present invention is described referring now to accompanying drawing.In follow-up description, use For representing the suffix explanation only for the beneficially present invention of such as " module ", " parts " or " unit " of element, itself Not specific meaning.Therefore, " module " can mixedly use with " parts ".
Mobile terminal can be implemented in a variety of manners.Such as, the terminal described in the present invention can include such as moving Phone, smart phone, notebook computer, digit broadcasting receiver, PDA (personal digital assistant), PAD (panel computer), PMP The mobile terminal of (portable media player), guider etc. and consolidating of such as numeral TV, desk computer etc. Determine terminal.Hereinafter it is assumed that terminal is mobile terminal.However, it will be understood by those skilled in the art that, mobile except being used in particular for Outside the element of purpose, structure according to the embodiment of the present invention can also apply to the terminal of fixed type.
Fig. 1 is the hardware configuration signal of the mobile terminal realizing each embodiment of the present invention.
Mobile terminal 100 can include wireless communication unit 110, A/V (audio/video) input block 120, user's input Unit 130, output unit 150, memorizer 160, interface unit 170, controller 180 and power subsystem 190 etc..Fig. 1 illustrates There is the mobile terminal of various assembly, it should be understood that be not required for implementing all assemblies illustrated.Can be alternatively Implement more or less of assembly.Will be discussed in more detail below the element of mobile terminal.
Wireless communication unit 110 generally includes one or more assembly, and it allows mobile terminal 100 and wireless communication system Or the radio communication between network.Such as, wireless communication unit can include broadcast reception module 111, mobile communication module 112, at least one in wireless Internet module 113, short range communication module 114 and positional information module 115.
Broadcast reception module 111 receives broadcast singal and/or broadcast via broadcast channel from external broadcasting management server Relevant information.Broadcast channel can include satellite channel and/or terrestrial channel.Broadcast management server can be to generate and send Broadcast singal and/or the server of broadcast related information or the broadcast singal generated before receiving and/or broadcast related information And send it to the server of terminal.Broadcast singal can include that TV broadcast singal, radio signals, data are broadcasted Signal etc..And, broadcast singal may further include the broadcast singal combined with TV or radio signals.Broadcast phase Pass information can also provide via mobile communications network, and in this case, broadcast related information can be by mobile communication mould Block 112 receives.Broadcast singal can exist in a variety of manners, and such as, it can be with the electronics of DMB (DMB) Program guide (EPG), the form of electronic service guidebooks (ESG) etc. of digital video broadcast-handheld (DVB-H) and exist.Broadcast Receiver module 111 can be broadcasted by using various types of broadcast systems to receive signal.Especially, broadcast reception module 111 Can be by using such as multimedia broadcasting-ground (DMB-T), DMB-satellite (DMB-S), digital video wide Broadcast-hand-held (DVB-H), forward link media (MediaFLO@) Radio Data System, received terrestrial digital broadcasting integrated service Etc. (ISDB-T) digit broadcasting system receives digital broadcasting.Broadcast reception module 111 may be constructed such that and is adapted to provide for extensively Broadcast the various broadcast systems of signal and above-mentioned digit broadcasting system.Via broadcast reception module 111 receive broadcast singal and/ Or broadcast related information can be stored in memorizer 160 (or other type of storage medium).
Mobile communication module 112 sends radio signals to base station (such as, access point, node B etc.), exterior terminal And in server at least one and/or receive from it radio signal.Such radio signal can include that voice leads to Words signal, video calling signal or the various types of data sending according to text and/or Multimedia Message and/or receiving.
Wireless Internet module 113 supports the Wi-Fi (Wireless Internet Access) of mobile terminal.This module can be internally or externally It is couple to terminal.Wi-Fi (Wireless Internet Access) technology involved by this module can include WLAN (WLAN) (Wi-Fi), Wibro (WiMAX), Wimax (worldwide interoperability for microwave access), HSDPA (high-speed downlink packet access) etc..
Short range communication module 114 is the module for supporting junction service.Some examples of short-range communication technology include indigo plant ToothTM, RF identification (RFID), Infrared Data Association (IrDA), ultra broadband (UWB), purple honeybeeTMEtc..
Positional information module 115 is the module of positional information for checking or obtain mobile terminal.Positional information module Typical case be GPS (global positioning system).According to current technology, GPS module 115 calculates from three or more satellites Range information and correct time information and for the Information application triangulation calculated, thus according to longitude, latitude Highly accurately calculate three-dimensional current location information.Currently, the method use three being used for calculating position and temporal information is defended Star and by using the position and the error of temporal information that other satellite correction calculates.Additionally, GPS module 115 Velocity information can be calculated by Continuous plus current location information in real time.
A/V input block 120 is used for receiving audio or video signal.A/V input block 120 can include camera 121 He Mike 122, the camera 121 static images to being obtained by image capture apparatus in Video Capture pattern or image capture mode Or the view data of video processes.Picture frame after process may be displayed on display unit 151.Process through camera 121 After picture frame can be stored in memorizer 160 (or other storage medium) or via wireless communication unit 110 carry out send out Send, two or more cameras 121 can be provided according to the structure of mobile terminal.Mike 122 can be in telephone calling model, note Record pattern, speech recognition mode etc. operational mode receives sound (voice data) via mike, and can be by so Acoustic processing be voice data.Audio frequency (voice) data after process can be converted in the case of telephone calling model can The form output of mobile communication base station it is sent to via mobile communication module 112.Mike 122 can implement various types of making an uproar Sound eliminates (or suppression) algorithm and with the noise of elimination (or suppression) generation during receiving and send audio signal or does Disturb.
User input unit 130 can generate key input data to control each of mobile terminal according to the order of user's input Plant operation.User input unit 130 allows user to input various types of information, and can include keyboard, metal dome, touch Plate (such as, detection due to touched and cause resistance, pressure, the sensitive component of change of electric capacity etc.), roller, rocking bar etc. Deng.Especially, when touch pad is superimposed upon on display unit 151 as a layer, touch screen can be formed.
Interface unit 170 is used as at least one external device (ED) and is connected, with mobile terminal 100, the interface that can pass through.Such as, External device (ED) can include wired or wireless head-band earphone port, external power source (or battery charger) port, wired or nothing Line FPDP, memory card port, for connect have the port of device of identification module, audio frequency input/output (I/O) end Mouth, video i/o port, ear port etc..Identification module can be that storage is for verifying that user uses each of mobile terminal 100 Kind of information and subscriber identification module (UIM), client identification module (SIM), Universal Subscriber identification module (USIM) can be included Etc..It addition, the device (hereinafter referred to as " identifying device ") with identification module can be to take the form of smart card, therefore, know Other device can be connected with mobile terminal 100 via port or other attachment means.Interface unit 170 may be used for receive from The input (such as, data message, electric power etc.) of external device (ED) and the input received is transferred in mobile terminal 100 One or more elements or may be used between mobile terminal and external device (ED) transmit data.
It addition, when mobile terminal 100 is connected with external base, interface unit 170 can serve as allowing electricity by it Power provides the path of mobile terminal 100 from base or can serve as allowing from the various command signals of base input by it It is transferred to the path of mobile terminal.May serve as identifying that mobile terminal is from various command signals or the electric power of base input The no signal being accurately fitted within base.Output unit 150 is configured to provide defeated with vision, audio frequency and/or tactile manner Go out signal (such as, audio signal, video signal, alarm signal, vibration signal etc.).Output unit 150 can include display Unit 151, dio Output Modules 152, alarm unit 153 etc..
Display unit 151 may be displayed on the information processed in mobile terminal 100.Such as, it is in electricity when mobile terminal 100 During words call mode, display unit 151 can show and call or other (such as, text messaging, multimedia file that communicate Download etc.) relevant user interface (UI) or graphic user interface (GUI).When mobile terminal 100 is in video calling pattern Or during image capture mode, display unit 151 can show image and/or the image of reception of capture, illustrate video or figure UI or GUI of picture and correlation function etc..
Meanwhile, when display unit 151 and touch pad the most superposed on one another with formed touch screen time, display unit 151 can serve as input equipment and output device.Display unit 151 can include liquid crystal display (LCD), thin film transistor (TFT) In LCD (TFT-LCD), Organic Light Emitting Diode (OLED) display, flexible display, three-dimensional (3D) display etc. at least A kind of.Some in these display may be constructed such that transparence is watched from outside with permission user, and this is properly termed as transparent Display, typical transparent display can for example, TOLED (transparent organic light emitting diode) display etc..According to specific The embodiment wanted, mobile terminal 100 can include two or more display units (or other display device), such as, move Dynamic terminal can include outernal display unit (not shown) and inner display unit (not shown).Touch screen can be used for detection and touches Input pressure and touch input position and touch input area.
Dio Output Modules 152 can mobile terminal be in call signal receive pattern, call mode, logging mode, Time under the isotype such as speech recognition mode, broadcast reception mode, that wireless communication unit 110 is received or at memorizer 160 The voice data transducing audio signal of middle storage and be output as sound.And, dio Output Modules 152 can provide with mobile The audio frequency output (such as, call signal receives sound, message sink sound etc.) that the specific function that terminal 100 performs is relevant. Dio Output Modules 152 can include speaker, buzzer etc..
Alarm unit 153 can provide output to notify event to mobile terminal 100.Typical event is permissible Including calling reception, message sink, key signals input, touch input etc..In addition to audio or video exports, alarm unit 153 can provide output with the generation of notification event in a different manner.Such as, alarm unit 153 can be with the form of vibration Output is provided, when receiving calling, message or some other entrance communication (incomingcommunication), alarm list Unit 153 can provide sense of touch output (that is, vibration) to notify to user.By providing such sense of touch to export, even if When the mobile phone of user is in the pocket of user, user also is able to identify the generation of various event.Alarm unit 153 is also The output of the generation of notification event can be provided via display unit 151 or dio Output Modules 152.
Memorizer 160 can store the process performed by controller 180 and the software program controlling operation etc., or can Temporarily to store the data (such as, telephone directory, message, still image, video etc.) that oneself maybe will export through output.And And, memorizer 160 can with storage about when touch be applied to touch screen time the vibration of various modes of output and audio signal Data.
Memorizer 160 can include that the storage medium of at least one type, described storage medium include flash memory, hard disk, many Media card, card-type memorizer (such as, SD or DX memorizer etc.), random access storage device (RAM), static random-access store Device (SRAM), read only memory (ROM), Electrically Erasable Read Only Memory (EEPROM), programmable read only memory (PROM), magnetic storage, disk, CD etc..And, mobile terminal 100 can be connected execution memorizer with by network The network storage device cooperation of the storage function of 160.
Controller 180 generally controls the overall operation of mobile terminal.Such as, controller 180 performs and voice call, data Control that communication, video calling etc. are relevant and process.It addition, controller 180 can include for reproducing (or playback) many matchmakers The multi-media module 181 of volume data, multi-media module 181 can construct in controller 180, or it is so structured that with control Device 180 separates.Controller 180 can perform pattern recognition process, with the handwriting input that will perform on the touchscreen or picture Draw input and be identified as character or image.
Power subsystem 190 receives external power or internal power under the control of controller 180 and provides operation each unit Suitable electric power needed for part and assembly.
Various embodiment described herein can be to use such as computer software, hardware or its any combination of calculating Machine computer-readable recording medium is implemented.Implementing for hardware, embodiment described herein can be by using application-specific IC (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), scene can Program gate array (FPGA), processor, controller, microcontroller, microprocessor, be designed to perform function described herein At least one in electronic unit is implemented, and in some cases, such embodiment can be implemented in controller 180. Software is implemented, the embodiment of such as process or function can with allow to perform the single of at least one function or operation Software module is implemented.Software code can be come by the software application (or program) write with any suitable programming language Implementing, software code can be stored in memorizer 160 and be performed by controller 180.
So far, oneself is through describing mobile terminal according to its function.Below, for the sake of brevity, will describe such as folded form, Slide type mobile terminal in various types of mobile terminals of board-type, oscillating-type, slide type mobile terminal etc. is as showing Example.Therefore, the present invention can be applied to any kind of mobile terminal, and is not limited to slide type mobile terminal.
As shown in Figure 1 mobile terminal 100 may be constructed such that utilize via frame or packet transmission data all if any Line and wireless communication system and satellite-based communication system operate.
The communication system being wherein operable to according to the mobile terminal of the present invention is described referring now to Fig. 2.
Such communication system can use different air interfaces and/or physical layer.Such as, communication system use Air interface includes such as frequency division multiple access (FDMA), time division multiple acess (TDMA), CDMA (CDMA) and universal mobile communications system System (UMTS) (especially, Long Term Evolution (LTE)), global system for mobile communications (GSM) etc..As non-limiting example, under The description in face relates to cdma communication system, but such teaching is equally applicable to other type of system.
With reference to Fig. 2, cdma wireless communication system can include multiple mobile terminal 100, multiple base station (BS) 270, base station Controller (BSC) 275 and mobile switching centre (MSC) 280.MSC280 is configured to and Public Switched Telephony Network (PSTN) 290 form interface.MSC280 is also structured to and the BSC275 formation interface that can be couple to base station 270 via back haul link. If back haul link can construct according to any one in the interface that Ganji knows, described interface includes such as E1/T1, ATM, IP, PPP, frame relay, HDSL, ADSL or xDSL.It will be appreciated that system as shown in Figure 2 can include multiple BSC275.
Each BS270 can service one or more subregion (or region), by multidirectional antenna or the sky of sensing specific direction Each subregion that line covers is radially away from BS270.Or, each subregion can be by for two or more of diversity reception Antenna covers.Each BS270 may be constructed such that support multiple frequencies distribution, and the distribution of each frequency has specific frequency spectrum (such as, 1.25MHz, 5MHz etc.).
Intersecting that subregion and frequency are distributed can be referred to as CDMA Channel.BS270 can also be referred to as base station transceiver System (BTS) or other equivalent terms.In this case, term " base station " may be used for broadly representing single BSC275 and at least one BS270.Base station can also be referred to as " cellular station ".Or, each subregion of specific BS270 can be claimed For multiple cellular stations.
As shown in Figure 2, broadcast singal is sent in system the mobile terminal operated by broadcsting transmitter (BT) 295 100.Broadcast reception module 111 is arranged on mobile terminal 100 and sentences the broadcast that reception is sent by BT295 as shown in Figure 1 Signal.In fig. 2 it is shown that several global positioning systems (GPS) satellite 300.Satellite 300 helps to position multiple mobile terminals At least one in 100.
In fig. 2, depict multiple satellite 300, it is understood that be, it is possible to use any number of satellite obtain useful Location information.GPS module 115 is generally configured to coordinate with satellite 300 to obtain the location letter wanted as shown in Figure 1 Breath.Substitute GPS tracking technique or outside GPS tracking technique, it is possible to use other of position of mobile terminal can be followed the tracks of Technology.It addition, at least one gps satellite 300 can optionally or additionally process satellite dmb transmission.
As a typical operation of wireless communication system, BS270 receives the reverse link from various mobile terminals 100 Signal.Mobile terminal 100 generally participates in call, information receiving and transmitting communicates with other type of.Certain base station 270 receive each instead Processed in specific BS270 to link signal.The data obtained are forwarded to the BSC275 being correlated with.BSC provides call Resource distribution and the mobile management function of the coordination of soft switching process included between BS270.The number that BSC275 also will receive According to being routed to MSC280, it provides the extra route service for forming interface with PSTN290.Similarly, PSTN290 with MSC280 forms interface, MSC Yu BSC275 forms interface, and BSC275 correspondingly controls BS270 with by forward link signals It is sent to mobile terminal 100.
Based on above-mentioned mobile terminal hardware configuration and communication system, each embodiment of the inventive method is proposed.
First embodiment
In Kafka cluster, Producer (Producer of message) is responsible for the Kafka broker that gives out information, and i.e. caches Agency, Kafka cluster comprises one or more server, and this server is referred to as broker, the Consumer (consumption of message Person) it is the client reading message to Kafka broker, subscribe to topics and process its process of message issued and be called and disappear Take.One or more topic (topic) can be subscribed to by consumer, and from Broker pulling data, thus consume these and issue Message, Fig. 3 is the structural relation schematic diagram of Kafka cluster and consumer, and the device 40 provided in the present embodiment is i.e. base In the improvement that existing consumer client is carried out.The message distribution device that the present embodiment provides can be Kafka consumer One ingredient of client, it is also possible to be the independent device interacted with Kafka consumer client.
Fig. 4 is the structured flowchart of this device, and as shown in Figure 4, this device 40 includes following ingredient:
Read module 41, for reading message from Kafka system;
In the present embodiment, optionally, read module 41 can read from Kafka system according to the instruction that user sends Specify the batch message of number, or, this read module 41 can also come according to the number of the message to be read pre-set Read a collection of message of predetermined number, so so that the message number that this device reads can dynamically change, in order to adapt to The magnanimity message of burst needs situation to be processed.
This read module 41 can use the existing interface of Kafka consumer client by the way of method call Interact with Kafka system, thus read message in Kafka system, simultaneously as the message in Kafka is according to different Theme divides, therefore, this read module 41, when reading message, needs to read message according to the division of theme, such as, and this reading Delivery block 41 can read the multiple messages belonging to same subject.
The message got in the present embodiment i.e. gets from Kafka cluster, in Kafka system, and every issue Message to Kafka cluster has a classification, and this classification is also referred to as Topic (theme), and theme is packet physically, One theme can be divided into multiple subregion, each subregion to be orderly queues.Every message in subregion all can be allocated One orderly id (offset), i.e. theme is that the message of physically different themes is stored separately, in logic the disappearing an of theme Though breath be stored on one or more theme user only need to specify the theme of message to produce or consumption data without It is concerned about that data are stored in where, therefore in the present embodiment, when reading message, 41 need of read module know to be currently needed for which reads The message of theme can read message from Kafka cluster.
Divide module 42, organize message for the message read being divided into more;
The a collection of message that read module 41 reads can be divided by this division module 42 according to the disposal ability of terminal Become to organize message more, such as, on the basis of above-mentioned read module 41 reads 10000 message, divide module 42 follow-up permissible These 10000 message are divided into 10 groups, group often comprises 1000 message, similar with above-mentioned read module 41, this division module The 42 message groups numbers divided again may be by pre-setting or according to real-time reception to user instruction determine, it is preferred that In the present embodiment, the group number of the message owing to dividing decides the number creating sub-line journey, creates the number of sub-line journey then Directly affect quantity and the speed of the message that device processes, therefore in the present embodiment, can be controlled by arranging of threshold value System divides the division of the module 42 message to reading, concrete, can come really according to the quantity of the message being currently needed for process The group number of the message that surely need to divide, can preset and now set multiple different message number threshold value so that in pending message count In the case of amount reaches to specify threshold value, correspond to different packet count, for example, it is possible to pre-set, pending message number More than 100000, then packet count is 100, when pending message data is more than 10000, then packet numerical digit 10, the most permissible Realize the purpose of on-demand packet.
Creation module 43, is used for creating multiple sub-line journey;
This creation module 43 can be realized by the main thread performing read module 41 and division module 42, and it creates son The number of thread can be carried out according to the group number of the many groups message dividing Module Division, and the sub-line journey that this module 43 creates is used for Process the message after packet.
Sending module 44, for being respectively sent to each sub-line journey by organizing message more.
Many groups message decibel after packet is equally sent to create each by above-mentioned main thread by this sending module 44 Individual sub-thread, so that each sub-line journey processes these message.
In this example, it is assumed that what above-mentioned read module 41 got is the same a collection of message belonging to same subject, Then with when this theme creates, owing to having secured the number of partitions of this theme, and the corresponding main thread of subregion, right For conventional art, it is to be gone to process the message got by this main thread, and the ability of a thread process message is limited 's.It should be noted that in the present embodiment, subregion is a concept physically, and each theme comprises one or more points District.But the scheme after improving based on the present embodiment, after message can being grouped by main thread, establishment right with packet group number Answer the sub-line journey of number and each group message after packet is sent to each sub-line journey, as such, it is possible to processed point by each sub-line journey Respectively organizing message after group, such as, the sub-number of threads that main thread creates is 10, then, will be distinguished by these 10 sub-threads Process the information after each packet, it is seen then that the treating capacity of information, on the basis of conventional art, is carried by the scheme that the present embodiment provides High 10 times, and owing to multiple sub-line journeys can be with parallel processing, therefore the program also improves the efficiency processing message.
Second embodiment
Present embodiments providing another kind of message distribution device, this device includes consisting of part:
Read module, for reading message from Kafka system;Divide module, for the message read being divided into Organize message more;Creation module, is used for creating multiple sub-line journey;Sending module, the many groups message obtained after dividing is respectively Send to each sub-line journey, so that the message that each sub-thread process receives.
Compared with the device that this device is involved with above-described embodiment 1, its modules can also possess following functions:
Read module, for calling the interface of Kafka consumer client to read the message in Kafka system;Divide Module, is divided into many groups message that message number is impartial by the message read;Creation module is many after creating and dividing The sub-line journey of the group number same number of group message;Sending module, the many groups message after dividing is respectively sent to each sub-line Journey.
Wherein, due to kafka broker meeting perdurable data, broker does not has memory pressure, and therefore, client is very Be suitable for taking the mode consumption data of pull (pulling), thus above-mentioned read module can be specifically by Kafka client from Kafka system pulls message, meanwhile, the most specifically can be by user according to consuming capacity Autonomous Control message Pull speed.
Optionally, involved in the present embodiment creation module specifically can be also used for: creates the sub-line journey of predetermined number; Or the sub-line journey of number corresponding with external command is created according to the external command received.
The device that the present embodiment provides is mainly used in by pulling a collection of message from Kafka system, pull out Message is carried out in batches, sends to different sub-thread process, so can evade the original design of Kafka system so that During by message in main thread one subregion of consumption, business logic processing message can be carried out by multiple sub-line journeys Information, is greatly improved the Consumption rate of message, improves the consumption handling capacity of kafka.
3rd embodiment
Present embodiments providing a kind of message distributing method, the method can be provided by above-described embodiment 1 or embodiment 2 Device performs, and Fig. 5 is the flow chart of the method, as it is shown in figure 5, the method includes processing as follows:
Step 501: read message from Kafka system;
In this step 501, optionally, appointment number can be read from Kafka system according to the instruction that user sends Batch message, or, it is also possible to according to the number of the message to be read pre-set, read a collection of of predetermined number and disappear Breath, so so that the message number read can dynamically change, in order to the magnanimity message adapting to burst needs feelings to be processed Condition.
Owing to, in Kafka system, every message being published to Kafka cluster has a classification, this classification also by Being referred to as Topic (theme), theme is packet physically, and a theme can be divided into multiple subregion, each subregion to be one to be had The queue of sequence.Every message in subregion all can be allocated an orderly id (offset), i.e. theme is physically different main The message of topic is stored separately, although the message of a theme is stored on one or more theme in logic, but user only needs to refer to Determine the theme of message can produce or consumption data is without being concerned about that data are stored in where, therefore in the present embodiment, read message Time, 41 need of read module know that the message being currently needed for reading which theme can read message from Kafka cluster.At this In embodiment, the existing interface of Kafka consumer client can be used by the way of method call to come and Kafka system Interact, thus read message in Kafka system, simultaneously as the message in Kafka divides according to different themes, Therefore, when reading message, need to read message according to the division of theme, belong to same subject for example, it is possible to be successively read Multiple messages.
Step 502: the message read is divided into and organizes message more;
In this step 502, a collection of message read can be divided into many groups according to the disposal ability of terminal and disappear Breath, such as, on the basis of reading 10000 message, can be divided into 10 groups by these 10000 message, often comprise in group Article 1000, message, the user instruction that the group number that message divides again may be by pre-setting or arriving according to real-time reception comes really Fixed, it is preferred that in the present embodiment, the group number of the message owing to dividing decides the number creating sub-line journey, and creates sub-line The number of journey then directly affects quantity and the speed of the message that device processes, therefore in the present embodiment, can pass through threshold value The packet count of drawing controlling message is set, such as, determining according to the quantity of message being currently needed for processing and need to divide During the group number of message, can preset and now set multiple different message number threshold value so that reach to refer in pending message number In the case of determining threshold value, correspond to different packet count, for example, it is possible to pre-set, pending message number is more than 100000, then packet count is 100, when pending message data is more than 10000, then packet numerical digit 10, the most then can realize The purpose of on-demand packet.
Step 503: create multiple sub-line journey;
This step can be realized by the main thread performing above-mentioned steps 501 and 502, and its number creating sub-line journey can Carrying out with the group number according to the many groups message divided, the concrete number creating sub-line journey can be with the group several of the message divided Cause, the sub-line journey created in this step message after processing packet.
Step 504: be respectively sent to each sub-line journey by organizing message more, so that the message that each sub-thread process receives.
Assume to get in step 501 is the same a collection of message belonging to same subject, then create with at this theme Time, owing to having secured the number of partitions of this theme, and a corresponding main thread of subregion, for conventional art, It is to be gone to process the message got by this main thread, and the ability of a thread process message is limited.It should be noted that In the present embodiment, subregion is a concept physically, and each theme comprises one or more subregion.But based on the present embodiment Scheme after improvement, after message can being grouped by main thread, establishment with the sub-line journey of the packet corresponding number of group number also Each group message after packet is sent to each sub-line journey, as such, it is possible to respectively organize message, example after being processed packet by each sub-line journey As, the sub-number of threads that main thread creates is 10, then, the letter after each packet will be processed respectively by these 10 sub-threads Breath, it is seen then that the scheme that the present embodiment provides, on the basis of conventional art, improves 10 times to the treating capacity of information, and due to Multiple sub-line journeys can with parallel processing, therefore the program also improve process message efficiency.
Due to Kafka broker meeting perdurable data, broker does not has memory pressure, and therefore, client is especially suitable for adopting Take the mode consumption data of pull (pulling), therefore above-mentioned steps 501 can be specifically from Kafka system by Kafka client System pulls message, meanwhile, the most specifically can pull speed by user according to consuming capacity Autonomous Control message Degree.
Optionally, the message read is divided into many group message in the present embodiment specifically to may include that and will read Message be divided into many groups message that message number is impartial.
The step creating multiple sub-line journey specifically may include that the group number same number of the many groups message after creating and dividing Sub-line journey.
From Kafka system, read message specifically may include that the interface calling Kafka consumer client reads Message in Kafka system.
The step creating multiple sub-line journey may include that the sub-line journey creating predetermined number;Or according to the outside received Instruction creates the sub-line journey of number corresponding with external command.
4th embodiment
The present embodiment provides another kind of message distributing method on the basis of above-mentioned 3rd embodiment, and the method is by public affairs Opening more technical characteristic to illustrate the solution of the present invention, the method mainly includes processing as follows:
Step 1: build Kafka cluster relevant environment, downloads Kafka related tool jar bag;
Building Kafka cluster relevant environment in this step, including building with Kafka cluster, to carry out information mutual The relevant of Kafka client changes part, such as, download the relevant installation toolkit of Kafka consumer client.
Step 2: install Kafka consumer client on consumer end;
The consumer end related in the present embodiment is the terminal of carrying Kafka consumer client, the most permissible Use the terminal shown in above-mentioned Fig. 1.
Step 3: by the method call to consumer client, pulling the message of batch from Kafka, pull disappears The quantity of breath size and message can be arranged voluntarily by user;
In the present embodiment, due to Kafka broker meeting perdurable data, broker does not has memory pressure, therefore, and visitor Family end is especially suitable for taking the mode consumption data of pull (pulling), therefore by Kafka client from Kafka system in step 3 In pull message, meanwhile, speed can be pulled by user according to consuming capacity Autonomous Control message, in this step in this step In rapid 3, the method to set up of the quantity pulling message can be used concrete grammar described in above-mentioned 3rd embodiment, herein Repeat no more.
Step 4: obtain this bulk messages, by certain service logic condition (such as data terminal disposal ability, and The total amount of the message pulled) message is grouped.
In this step 4, the mode being grouped message can use message described in above-mentioned 3rd embodiment The method of packet, here is omitted.
Such as: pull 10000 data information, through routine processes every time, being grouped into 10 groups of data, often group 1000 disappears Breath.
Step 5: after being pulled message by the main thread in step 3 and step 4, and performing division operation, by main line Journey creates the sub-line journey of corresponding grouping number, sends in the most corresponding sub-line journey of the message after associated packets, sub-line journey carries out The process work of business datum.
In the present embodiment, create sub-number of threads and controlled voluntarily by application program, when message number needs a large amount of consumption Time, the number of increase sub-line journey that can be artificial, thus increase consumption handling capacity, in this manner it is achieved that flexible and changeable control The Consumption rate of message, improves or reduces handling up of Message Processing.
It should be noted that in this article, term " includes ", " comprising " or its any other variant are intended to non-row Comprising of his property, so that include that the process of a series of key element, method, article or device not only include those key elements, and And also include other key elements being not expressly set out, or also include intrinsic for this process, method, article or device Key element.In the case of there is no more restriction, statement " including ... " key element limited, it is not excluded that including this The process of key element, method, article or device there is also other identical element.The invention described above embodiment sequence number is only Describe, do not represent the quality of embodiment.
Through the above description of the embodiments, those skilled in the art is it can be understood that arrive above-described embodiment side Method can add the mode of required general hardware platform by software and realize, naturally it is also possible to by hardware, but a lot of in the case of The former is more preferably embodiment.Based on such understanding, prior art is done by technical scheme the most in other words The part going out contribution can embody with the form of software product, and this computer software product is stored in a storage medium In (such as ROM/RAM, magnetic disc, CD), including some instructions with so that a station terminal equipment (can be mobile phone, computer, take Business device, air-conditioner, or the network equipment etc.) perform the method described in each embodiment of the present invention.
These are only the preferred embodiments of the present invention, not thereby limit the scope of the claims of the present invention, every utilize this Equivalent structure or equivalence flow process that bright description and accompanying drawing content are made convert, or are directly or indirectly used in other relevant skills Art field, is the most in like manner included in the scope of patent protection of the present invention.

Claims (10)

1. a message distribution device, it is characterised in that described device includes:
Read module, for reading message from message system;
Divide module, organize message for the message read being divided into more;
Creation module, is used for creating multiple sub-line journey;
Sending module, for being respectively sent to each sub-line journey by described many group message.
Device the most according to claim 1, it is characterised in that described division module specifically for:
The message read is divided into many groups message that message number is impartial.
Device the most according to claim 1, it is characterised in that described creation module specifically for:
The sub-line journey of the group number same number of the many groups message after creating and dividing.
Device the most according to claim 1, described read module specifically for:
The interface of message call system consumers's client reads the message in described message system.
5. according to the device described in Claims 1-4 any one, it is characterised in that described creation module specifically for:
Create the sub-line journey of predetermined number;
Or the sub-line journey of number corresponding with described external command is created according to the external command received.
6. a message distributing method, it is characterised in that described method includes step:
Message is read from message system;
The message read is divided into and organizes message more;
Create multiple sub-line journey;
Described many group message are respectively sent to each sub-line journey.
Method the most according to claim 6, it is characterised in that the described message read being divided into organizes message more, bag Include:
The message read is divided into many groups message that message number is impartial.
Method the most according to claim 6, it is characterised in that described establishment multiple sub-line journey, including:
The sub-line journey of the group number same number of the many groups message after creating and dividing.
Method the most according to claim 6, described reading message from message system, including:
Call the interface of Kafka consumer client to read the message in described message system.
10. according to the method described in claim 6 to 9 any one, it is characterised in that described establishment multiple sub-line journey, including:
Create the sub-line journey of predetermined number;
Or the sub-line journey of number corresponding with described external command is created according to the external command received.
CN201610595099.3A 2016-07-26 2016-07-26 A kind of message distribution device and method Active CN106302385B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610595099.3A CN106302385B (en) 2016-07-26 2016-07-26 A kind of message distribution device and method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610595099.3A CN106302385B (en) 2016-07-26 2016-07-26 A kind of message distribution device and method

Publications (2)

Publication Number Publication Date
CN106302385A true CN106302385A (en) 2017-01-04
CN106302385B CN106302385B (en) 2019-11-15

Family

ID=57652457

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610595099.3A Active CN106302385B (en) 2016-07-26 2016-07-26 A kind of message distribution device and method

Country Status (1)

Country Link
CN (1) CN106302385B (en)

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108737292A (en) * 2018-04-18 2018-11-02 千寻位置网络有限公司 A kind of sending method and system, server of bulk messages
CN108874562A (en) * 2018-06-21 2018-11-23 北京顺丰同城科技有限公司 Distributed high concurrent message queue supplying system
CN108924041A (en) * 2018-08-03 2018-11-30 北京捷易迅信息技术有限公司 A kind of message efficient reliable diffusion method in instant messaging
CN109451035A (en) * 2018-12-04 2019-03-08 宁波耘瑞智能科技有限公司 A kind of method, apparatus and system of separate type collection group agent and service
CN110493342A (en) * 2019-08-21 2019-11-22 北京明朝万达科技股份有限公司 Document transmission method, device, electronic equipment and readable storage medium storing program for executing
CN110535787A (en) * 2019-07-25 2019-12-03 北京奇艺世纪科技有限公司 Information consumption method, apparatus and readable storage medium storing program for executing
CN110648178A (en) * 2019-09-24 2020-01-03 四川长虹电器股份有限公司 Method for increasing kafka consumption capacity
CN110740145A (en) * 2018-07-18 2020-01-31 北京京东尚科信息技术有限公司 Message consumption method, device, storage medium and electronic equipment
CN110825533A (en) * 2018-08-10 2020-02-21 网宿科技股份有限公司 Data transmitting method and device
CN111111178A (en) * 2019-12-25 2020-05-08 上海米哈游天命科技有限公司 Game announcement acquisition method, device, equipment and storage medium
CN112506992A (en) * 2020-12-04 2021-03-16 中国人寿保险股份有限公司 Fuzzy query method and device for Kafka data, electronic equipment and storage medium
CN112905109A (en) * 2021-01-28 2021-06-04 平安普惠企业管理有限公司 Message processing method, device, equipment and storage medium
CN113923212A (en) * 2020-06-22 2022-01-11 大唐移动通信设备有限公司 Network data packet processing method and device

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104092767A (en) * 2014-07-21 2014-10-08 北京邮电大学 Posting/subscribing system for adding message queue models and working method thereof
CN104754036A (en) * 2015-03-06 2015-07-01 合一信息技术(北京)有限公司 Message processing system and processing method based on kafka
US20150188866A1 (en) * 2010-12-15 2015-07-02 Apple Inc. Message focusing
CN104834558A (en) * 2015-05-19 2015-08-12 北京京东尚科信息技术有限公司 Method and system for processing data

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150188866A1 (en) * 2010-12-15 2015-07-02 Apple Inc. Message focusing
CN104092767A (en) * 2014-07-21 2014-10-08 北京邮电大学 Posting/subscribing system for adding message queue models and working method thereof
CN104754036A (en) * 2015-03-06 2015-07-01 合一信息技术(北京)有限公司 Message processing system and processing method based on kafka
CN104834558A (en) * 2015-05-19 2015-08-12 北京京东尚科信息技术有限公司 Method and system for processing data

Cited By (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108737292A (en) * 2018-04-18 2018-11-02 千寻位置网络有限公司 A kind of sending method and system, server of bulk messages
CN108874562A (en) * 2018-06-21 2018-11-23 北京顺丰同城科技有限公司 Distributed high concurrent message queue supplying system
CN108874562B (en) * 2018-06-21 2022-08-02 北京顺丰同城科技有限公司 Distributed high-concurrency message queue pushing system
CN110740145B (en) * 2018-07-18 2023-08-08 北京京东尚科信息技术有限公司 Message consumption method and device, storage medium and electronic equipment
CN110740145A (en) * 2018-07-18 2020-01-31 北京京东尚科信息技术有限公司 Message consumption method, device, storage medium and electronic equipment
CN108924041A (en) * 2018-08-03 2018-11-30 北京捷易迅信息技术有限公司 A kind of message efficient reliable diffusion method in instant messaging
CN110825533A (en) * 2018-08-10 2020-02-21 网宿科技股份有限公司 Data transmitting method and device
CN110825533B (en) * 2018-08-10 2022-12-20 网宿科技股份有限公司 Data transmitting method and device
CN109451035A (en) * 2018-12-04 2019-03-08 宁波耘瑞智能科技有限公司 A kind of method, apparatus and system of separate type collection group agent and service
CN110535787A (en) * 2019-07-25 2019-12-03 北京奇艺世纪科技有限公司 Information consumption method, apparatus and readable storage medium storing program for executing
CN110493342A (en) * 2019-08-21 2019-11-22 北京明朝万达科技股份有限公司 Document transmission method, device, electronic equipment and readable storage medium storing program for executing
CN110493342B (en) * 2019-08-21 2021-05-14 北京明朝万达科技股份有限公司 File transmission method and device, electronic equipment and readable storage medium
CN110648178A (en) * 2019-09-24 2020-01-03 四川长虹电器股份有限公司 Method for increasing kafka consumption capacity
CN111111178B (en) * 2019-12-25 2023-08-29 上海米哈游天命科技有限公司 Game announcement acquisition method, device, equipment and storage medium
CN111111178A (en) * 2019-12-25 2020-05-08 上海米哈游天命科技有限公司 Game announcement acquisition method, device, equipment and storage medium
CN113923212B (en) * 2020-06-22 2022-12-13 大唐移动通信设备有限公司 Network data packet processing method and device
CN113923212A (en) * 2020-06-22 2022-01-11 大唐移动通信设备有限公司 Network data packet processing method and device
CN112506992A (en) * 2020-12-04 2021-03-16 中国人寿保险股份有限公司 Fuzzy query method and device for Kafka data, electronic equipment and storage medium
CN112506992B (en) * 2020-12-04 2024-04-16 中国人寿保险股份有限公司 Fuzzy query method and device for Kafka data, electronic equipment and storage medium
CN112905109B (en) * 2021-01-28 2023-02-03 平安普惠企业管理有限公司 Message processing method, device, equipment and storage medium
CN112905109A (en) * 2021-01-28 2021-06-04 平安普惠企业管理有限公司 Message processing method, device, equipment and storage medium

Also Published As

Publication number Publication date
CN106302385B (en) 2019-11-15

Similar Documents

Publication Publication Date Title
CN106302385B (en) A kind of message distribution device and method
CN104883430B (en) A kind of mobile terminal and do not read the treating method and apparatus of footmark
CN106155694A (en) A kind of application and the display packing and device applied of attending to anything else
CN106941443A (en) One population historical record checks terminal and method
CN104967802A (en) Mobile terminal, recording method of screen multiple areas and recording device of screen multiple areas
CN106407088A (en) Method and device for detecting multi-core CPU
CN106909681A (en) A kind of information processing method and its device
CN105682150A (en) Multilink intelligent shunting method and mobile terminal
CN106249989A (en) Social networking application program icon aligning method during a kind of sharing contents and mobile terminal
CN105634921A (en) Message filtering reminding method and terminal device
CN106250081A (en) A kind of display packing based on double screen terminal and device
CN106874097A (en) The multi-screen display method and device of a kind of terminal screen
CN105975052A (en) Screen switching method and device
CN104717543B (en) Method and device for achieving interactive operation between different split screens
CN107426282A (en) A kind of picture loading method, terminal and server
CN106161790A (en) A kind of mobile terminal and control method thereof
CN107071161A (en) The aggregation display method and mobile terminal of icon in a kind of status bar
CN104898927B (en) The method and device of information search
CN106791149A (en) A kind of method of mobile terminal and control screen
CN106506364A (en) A kind of cut-in method, client and routing server
CN106327338A (en) Data processing method and device
CN104915103B (en) The method and mobile terminal of arranging desktop icons
CN106941568A (en) A kind of Toast pop-ups optimization display methods, device and Toast pop-ups dispatching method and device
CN106686211A (en) Processing method and terminal for sound playing in split screen mode
CN106200924A (en) A kind of terminal and the method that virtual reality video is controlled thereof

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant