CN104754036A - Message processing system and processing method based on kafka - Google Patents

Message processing system and processing method based on kafka Download PDF

Info

Publication number
CN104754036A
CN104754036A CN201510099439.9A CN201510099439A CN104754036A CN 104754036 A CN104754036 A CN 104754036A CN 201510099439 A CN201510099439 A CN 201510099439A CN 104754036 A CN104754036 A CN 104754036A
Authority
CN
China
Prior art keywords
message
kafka
module
consumer
cluster
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201510099439.9A
Other languages
Chinese (zh)
Inventor
文德民
潘柏宇
卢述奇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Unification Infotech (beijing) Co Ltd
Original Assignee
Unification Infotech (beijing) Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Unification Infotech (beijing) Co Ltd filed Critical Unification Infotech (beijing) Co Ltd
Priority to CN201510099439.9A priority Critical patent/CN104754036A/en
Publication of CN104754036A publication Critical patent/CN104754036A/en
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
    • H04L67/5682—Policies or rules for updating, deleting or replacing the stored data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
    • H04L51/21—Monitoring or handling of messages
    • H04L51/212—Monitoring or handling of messages using filtering or selective blocking

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer And Data Communications (AREA)

Abstract

The invention discloses a message processing system and processing method based on kafka and belongs to the technical field of message processing. The message processing system and processing method based on kafka are aimed to solve the problems of the prior art that repeated messages appear in an order system frequently, the processing for each message consumes much system resource, the system resource consumption would greatly increase the database visitor volume at the consumer end, the message processing ability is lowered, and the system stability and service effectiveness are lowered. The message processing system based on kafka comprises a message producer cluster composed of one or more message producers, a kafka cluster, and a consumer cluster composed of one or more consumers, the kafka cluster is composed of one or more servers, and the consumer cluster comprises a filter module, a local message queue caching module and a business processing module. The message processing method realized through the system is used for the system especially the order system which needs to process a large number of messages.

Description

A kind of message handling system based on kafka and processing method
Technical field
The present invention is specifically related to a kind of message handling system based on kafka and processing method, belongs to message processing technique field.
Background technology
Kafka is the distributed Message Queuing system (Message Queue) developed by Linkedin.
The main target of kafka exploitation is that structure one is used for processing massive logs, the data processing shelf of user behavior and website optimize statistics etc.Combining data mining, behavioural analysis, when the demands such as monitoring operation, needing can meet various real-time online with batch processed offline application scenario to the requirement of low delay with batch throughput performance.From demand fundamentally, high-throughput is first requirement, is secondly real-time and persistence.
The cluster of kafka is made up of multiple Broker server, the message of each type is defined as topic, the message of same topic inside is partitioned (partition) according to certain key and algorithm and is stored on different Broker, and message producer producer and consumer consumer can produce or consume topic on multiple Broker.
In large concurrent message treatment system, the general Producer-consumer problem pattern of kafka that uses carrys out asynchronous process request, reaches the object eliminating request peak, improves the stability of a system and throughput.Need the system processing a large amount of message as form ordering system for some, need to consume more system resource (such as repeatedly data base querying, access file system or network call etc.) owing to often duplicating message and processing every bar message, consumer's client database visit capacity will be caused to roll up, message handling ability declines, and the stability of a system and availability of service decline.
Summary of the invention
Therefore, the present invention is directed to form ordering system in prior art often to duplicate message and process every bar message and need to consume more system resource, consumer's client database visit capacity will be caused to roll up, message handling ability declines, the problem that the stability of a system and availability of service decline, a kind of message handling system based on kafka is provided, comprise the message producer cluster be made up of one or more message producer, kafka cluster, the consumer's cluster be made up of one or more consumer, kafka cluster is made up of one or more server, each server is according to certain theme storing message, each message producer submits message to one or more server, each consumer is to one or more server subscribe message, it is characterized in that, described consumer comprises filter module, local message queue cache module, Service Processing Module, described filter module is the container not allowing repeating data, for receiving the message that consumer subscribes to from server, described local message queue cache module is used for the message of buffer memory through filter modular filtration, described Service Processing Module is used for obtaining message from message queue cache module, complete the Business Processing to this message.
A kind of message treatment method based on kafka utilizing said system to realize, it is characterized in that, described method is: the message producer in message producer cluster submits message to kafka cluster, message is stored in corresponding server according to type of theme by kafka cluster, server sends this message to the consumer subscribing to this type of theme, the filter module of the consumer of receipt message receives this message and carries out filtration treatment to this message, this message is added to local message queue cache module when this message is not in filter module, this message is supplied to Service Processing Module by local message queue cache module, Service Processing Module completes the Business Processing to this message.
In described method, local message queue cache module arranges message queue maximum.Arranging of maximum needs according to the filter effect expected and the adjustment of Message Processing speed.When queue size major effect filter effect and internal storage data are lost (as restart, machine of delaying), information drop-out number.
After the Business Processing that in described method, Service Processing Module completes this message, prompting and filtering device module deletes this message.Can ensure that filter can not be excessive like this, and new message can be added in filter to.
Beneficial effect of the present invention is: adopt the message handling system based on kafka of the present invention and processing method, while possessing the ability of a large amount of message of high-throughput process of kafka, Service Processing Module does not directly cancel breath from Kafka, but breath is cancelled from local message queue cache module, message in local message queue cache module is through filter module filters, the message repeated is removed, a piece of news only processes once, greatly reduce the access times to consumer's client database, message handling ability rises, the stability of a system and availability of service improve.
Accompanying drawing explanation
Fig. 1 is the structural representation of the message handling system that the present invention is based on kafka.
Embodiment
Below in conjunction with accompanying drawing, the specific embodiment of the present invention is described:
As shown in Figure 1, a kind of message handling system based on kafka, comprise the message producer cluster be made up of 3 message producers, kafka cluster, the consumer's cluster be made up of 2 consumers, kafka cluster is made up of 3 servers, each server is according to certain theme storing message, theme is certain key and algorithm, each message producer submits message to one or more server, each consumer is to one or more server subscribe message, consumer comprises filter module, local message queue cache module, Service Processing Module, filter module is the container not allowing repeating data, for receiving the message that consumer subscribes to from server, local message queue cache module is used for the message of buffer memory through filter modular filtration, Service Processing Module is used for obtaining message from message queue cache module, complete the Business Processing to this message.
Utilize the message treatment method that the above-mentioned message handling system based on kafka realizes, be applied particularly in form ordering system, the message producer 2 i.e. user in message producer cluster submits message to kafka cluster, this message is user ID, user ID is stored in corresponding server 1 according to type of theme by kafka cluster, server 1 sends this message to the consumer 2 subscribing to this type of theme, the filter module receiving the consumer 2 of user ID receives user ID and carries out filtration treatment to user ID, filtered out when user ID is in filter module, this message is added to local message queue cache module when user ID is not in filter module, user ID is supplied to Service Processing Module by local message queue cache module, Service Processing Module completes the Business Processing to user ID.
As preferred embodiment, local message queue cache module arranges message queue maximum.Arranging of maximum needs according to the filter effect expected and the adjustment of Message Processing speed.When queue size major effect filter effect and internal storage data are lost (as restart, machine of delaying), information drop-out number.
As preferred embodiment, after Service Processing Module completes the Business Processing to user ID, prompting and filtering device module deletes user ID.Can ensure that filter can not be excessive like this, and new message can be added in filter to.
If adopt the message treatment method of prior art, a user may have multiple order, each order also may through repeatedly revising, consumer holds needs repeatedly to set up buffer memory for user ID, increase system burden, and use this method only need set up a buffer memory to user ID, can ensure Business Processing complete before user ID can not repeat.
Main Java language algorithm structure is as follows:
Initialization consumer
Function: initialization local message queue size, generates Kafka client, and creates thread receipt message.
Fill message queue
Function: add message to local message queue by filter, completes filtering function.
Obtain the mode of message
Service Processing Module does not directly cancel breath from Kafka, but cancel breath in local message queue, obtain and successfully from filter module, delete corresponding message simultaneously, can ensure that filter module can not be excessive like this, and new message can be added in filter module to.
The size of filter module and local message queue is arranged
Initial value and the maximum of filter capacity need not be set.Local message queue needs to arrange maximum, and arranging of maximum needs according to the filter effect expected and the adjustment of Message Processing speed.When queue size major effect filter effect and internal storage data are lost (as restart, machine of delaying), information drop-out number.
The submission pattern of at-most-once
Use the submission pattern of the at-most-once of Kafka client, need the value ensureing client configuration parameter auto.commit.enable to be true.
The realization of filter module and local message queue
Local message queue adopts supports that the BlockingQueue(of concurrent processing needs more than JDK1.5 to support).Filter module adopts the container not allowing to repeat, as Set.Because the situation of multithreading operation may be there is, therefore the ConcurrentSkipListSet(of support concurrent operations is adopted to need more than JDK1.5 to support).The message got from Kafka server is first stored in ConcurrentSkipListSet, relies on the Set container characteristic that permissible value repeats to complete filtering function.
The above is the preferred embodiment of the present invention; it should be pointed out that for those skilled in the art, under the prerequisite not departing from principle of the present invention; can also make some improvements and modifications, these improvements and modifications also should be considered as protection scope of the present invention.

Claims (4)

1. the message handling system based on kafka, comprise the message producer cluster be made up of one or more message producer, kafka cluster, the consumer's cluster be made up of one or more consumer, kafka cluster is made up of one or more server, each server is according to certain theme storing message, each message producer submits message to one or more server, each consumer is to one or more server subscribe message, it is characterized in that, described consumer comprises filter module, local message queue cache module, Service Processing Module, described filter module is the container not allowing repeating data, for receiving the message that consumer subscribes to from server, described local message queue cache module is used for the message of buffer memory through filter modular filtration, described Service Processing Module is used for obtaining message from message queue cache module, complete the Business Processing to this message.
2. the message treatment method based on kafka utilizing system described in claim 1 to realize, it is characterized in that, described method is: the message producer in message producer cluster submits message to kafka cluster, message is stored in corresponding server according to type of theme by kafka cluster, server sends this message to the consumer subscribing to this type of theme, the filter module of the consumer of receipt message receives this message and carries out filtration treatment to this message, this message is added to local message queue cache module when this message is not in filter module, this message is supplied to Service Processing Module by local message queue cache module, Service Processing Module completes the Business Processing to this message.
3., as claimed in claim 2 based on the message treatment method of kafka, it is characterized in that, in described method, local message queue cache module arranges message queue maximum.
4. as claimed in claim 2 based on the message treatment method of kafka, it is characterized in that, after the Business Processing that in described method, Service Processing Module completes this message, prompting and filtering device module deletes this message.
CN201510099439.9A 2015-03-06 2015-03-06 Message processing system and processing method based on kafka Pending CN104754036A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510099439.9A CN104754036A (en) 2015-03-06 2015-03-06 Message processing system and processing method based on kafka

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510099439.9A CN104754036A (en) 2015-03-06 2015-03-06 Message processing system and processing method based on kafka

Publications (1)

Publication Number Publication Date
CN104754036A true CN104754036A (en) 2015-07-01

Family

ID=53593124

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510099439.9A Pending CN104754036A (en) 2015-03-06 2015-03-06 Message processing system and processing method based on kafka

Country Status (1)

Country Link
CN (1) CN104754036A (en)

Cited By (55)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105141687A (en) * 2015-08-19 2015-12-09 焦点科技股份有限公司 Message producing method
CN105227662A (en) * 2015-09-30 2016-01-06 努比亚技术有限公司 Message treatment method, device and system
CN105630574A (en) * 2015-12-21 2016-06-01 中国电子科技网络信息安全有限公司 Container based message queue bus management system and method
CN106170013A (en) * 2016-06-29 2016-11-30 上海浦东发展银行股份有限公司信用卡中心 A kind of Kafka message uniqueness method based on Redis
CN106302385A (en) * 2016-07-26 2017-01-04 努比亚技术有限公司 A kind of message distribution device and method
CN106293968A (en) * 2016-08-04 2017-01-04 华中科技大学 A kind of intercommunication system based on Kafka message-oriented middleware and method
CN106375200A (en) * 2016-11-08 2017-02-01 上海找钢网信息科技股份有限公司 Message system and message processing method
CN106603689A (en) * 2016-12-26 2017-04-26 努比亚技术有限公司 Data processing method and device based on distributed message releasing and subscribing system
CN106657349A (en) * 2016-12-29 2017-05-10 上海理想信息产业（集团）有限公司 Message subscription processing device, system and method
CN106817295A (en) * 2016-12-08 2017-06-09 努比亚技术有限公司 A kind of message processing apparatus and method
WO2017114176A1 (en) * 2015-12-30 2017-07-06 阿里巴巴集团控股有限公司 Method and apparatus for coordinating consumption queue in distributed environment
CN106953901A (en) * 2017-03-10 2017-07-14 重庆邮电大学 A kind of trunked communication system and its method for improving message transmission performance
CN107329832A (en) * 2017-06-30 2017-11-07 北京奇艺世纪科技有限公司 A kind of data receiver method and device
CN107369012A (en) * 2016-07-05 2017-11-21 康存乐付保数据科技（上海）有限公司 A kind of method for processing payment information and system and the prepaid card processing method with performance bond insurance mechanism
CN107590072A (en) * 2017-08-30 2018-01-16 深圳乐信软件技术有限公司 A kind of application and development and the method and apparatus of test
CN107682169A (en) * 2016-08-02 2018-02-09 北京京东尚科信息技术有限公司 A kind of method and apparatus using Kafka collection pocket transmission message
CN107911484A (en) * 2017-12-13 2018-04-13 浪潮软件股份有限公司 A kind of method and device of Message Processing
CN108009029A (en) * 2017-11-30 2018-05-08 中电福富信息科技有限公司 Method and system based on the data cached decoupling persistence of Ignite grids
CN108076111A (en) * 2016-11-15 2018-05-25 亿阳安全技术有限公司 A kind of system and method for distributing data in big data platform
CN108076103A (en) * 2016-11-18 2018-05-25 北京优朋普乐科技有限公司 A kind of message treatment method, system and message producer's cluster
CN108132845A (en) * 2016-12-01 2018-06-08 阿里巴巴集团控股有限公司 Message storage, delivering method and device and electronic equipment
CN108282514A (en) * 2017-12-12 2018-07-13 北京奇艺世纪科技有限公司 A kind of distributed service method for building up and device
CN108874562A (en) * 2018-06-21 2018-11-23 北京顺丰同城科技有限公司 Distributed high concurrent message queue supplying system
CN108920144A (en) * 2017-04-11 2018-11-30 北京京东尚科信息技术有限公司 A kind of message treatment method and device
CN108924041A (en) * 2018-08-03 2018-11-30 北京捷易迅信息技术有限公司 A kind of message efficient reliable diffusion method in instant messaging
CN108965355A (en) * 2017-05-18 2018-12-07 北京京东尚科信息技术有限公司 Method, apparatus and computer readable storage medium for data transmission
CN108984325A (en) * 2018-07-20 2018-12-11 北京北信源信息安全技术有限公司 Message queue consuming method and device
WO2018233592A1 (en) * 2017-06-20 2018-12-27 中兴通讯股份有限公司 Method for maintaining sequencing of messages, network node, and storage medium
CN109194736A (en) * 2018-08-30 2019-01-11 百度在线网络技术（北京）有限公司 Message De-weight method, device, electronic equipment, medium and unmanned vehicle
CN109451072A (en) * 2018-12-29 2019-03-08 广东电网有限责任公司 A kind of message caching system and method based on Kafka
CN109614250A (en) * 2018-12-04 2019-04-12 贵州电网有限责任公司 A kind of message system of the Kafka for electric acquisition system
CN109684099A (en) * 2018-11-23 2019-04-26 新华三大数据技术有限公司 Message treatment method and device
CN110019240A (en) * 2017-12-29 2019-07-16 中国移动通信集团北京有限公司 A kind of service data interaction method, apparatus and system
CN110138574A (en) * 2019-03-26 2019-08-16 北京达佳互联信息技术有限公司 Group managing means, device, electronic equipment and readable storage medium storing program for executing
CN110225133A (en) * 2019-06-20 2019-09-10 恒生电子股份有限公司 Message method, node, device, system and relevant device
CN110659124A (en) * 2018-06-28 2020-01-07 北京京东尚科信息技术有限公司 Message processing method and device
CN110704208A (en) * 2019-09-19 2020-01-17 深圳证券交易所 Message processing method for multiple systems, server and storage medium
CN110740145A (en) * 2018-07-18 2020-01-31 北京京东尚科信息技术有限公司 Message consumption method, device, storage medium and electronic equipment
CN111049899A (en) * 2019-12-11 2020-04-21 贝壳技术有限公司 kafka message storage system, method, apparatus, and computer-readable storage medium
CN111131512A (en) * 2019-12-31 2020-05-08 中消云（北京）物联网科技研究院有限公司 Equipment information processing method and device, storage medium and processor
CN111163172A (en) * 2019-12-31 2020-05-15 北京奇艺世纪科技有限公司 Message processing system, method, electronic device and storage medium
CN111179081A (en) * 2019-12-24 2020-05-19 中国建设银行股份有限公司 Message processing method and processing device
CN111179080A (en) * 2019-12-23 2020-05-19 中国建设银行股份有限公司 Order processing method and order processing device
CN111200637A (en) * 2019-12-20 2020-05-26 新浪网技术(中国)有限公司 Cache processing method and device
TWI697217B (en) * 2019-01-30 2020-06-21 中華電信股份有限公司 Container control system for repeatedly executing serverless programs and method thereof
CN111459689A (en) * 2020-04-07 2020-07-28 中国建设银行股份有限公司 Message processing system and method based on distributed queue
CN111555957A (en) * 2020-03-26 2020-08-18 孩子王儿童用品股份有限公司 Kafka-based synchronous message service system and implementation method
CN112330420A (en) * 2020-11-27 2021-02-05 中国农业银行股份有限公司 Order cancellation method and related device
CN112463411A (en) * 2020-12-10 2021-03-09 重庆紫光华山智安科技有限公司 Data processing method, device, server and storage medium
CN112667476A (en) * 2020-12-30 2021-04-16 平安普惠企业管理有限公司 Task-based message filtering method, device, equipment and storage medium
CN112751789A (en) * 2021-01-05 2021-05-04 浪潮云信息技术股份公司 Method and system for realizing asymmetric SDN controller cluster
TWI735519B (en) * 2017-01-24 2021-08-11 香港商阿里巴巴集團服務有限公司 Distributed environment coordinated consumption queue method and device
CN113542314A (en) * 2020-04-13 2021-10-22 烽火通信科技股份有限公司 Method and system for realizing subscription mode based on keyword filtering
CN115242787A (en) * 2022-07-22 2022-10-25 恒生电子股份有限公司 Message processing system and method
CN115914346A (en) * 2021-08-09 2023-04-04 中移物联网有限公司 Internet of things message processing method and device, electronic equipment and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102955717A (en) * 2012-11-05 2013-03-06 北京奇虎科技有限公司 Message management equipment and method in distributed message processing system
CN103049556A (en) * 2012-12-28 2013-04-17 中国科学院深圳先进技术研究院 Fast statistical query method for mass medical data
CN104092767A (en) * 2014-07-21 2014-10-08 北京邮电大学 Posting/subscribing system for adding message queue models and working method thereof
CN104092717A (en) * 2013-12-13 2014-10-08 深圳市腾讯计算机系统有限公司 Message processing method and system, message destination end device and distributed message system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102955717A (en) * 2012-11-05 2013-03-06 北京奇虎科技有限公司 Message management equipment and method in distributed message processing system
CN103049556A (en) * 2012-12-28 2013-04-17 中国科学院深圳先进技术研究院 Fast statistical query method for mass medical data
CN104092717A (en) * 2013-12-13 2014-10-08 深圳市腾讯计算机系统有限公司 Message processing method and system, message destination end device and distributed message system
CN104092767A (en) * 2014-07-21 2014-10-08 北京邮电大学 Posting/subscribing system for adding message queue models and working method thereof

Cited By (85)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105141687B (en) * 2015-08-19 2018-05-04 焦点科技股份有限公司 A kind of method for producing message
CN105141687A (en) * 2015-08-19 2015-12-09 焦点科技股份有限公司 Message producing method
CN105227662A (en) * 2015-09-30 2016-01-06 努比亚技术有限公司 Message treatment method, device and system
CN105630574A (en) * 2015-12-21 2016-06-01 中国电子科技网络信息安全有限公司 Container based message queue bus management system and method
CN106933672B (en) * 2015-12-30 2021-04-13 阿里巴巴集团控股有限公司 Distributed environment coordinated consumption queue method and device
CN106933672A (en) * 2015-12-30 2017-07-07 阿里巴巴集团控股有限公司 A kind of distributed environment coordinates consumption queuing method and device
WO2017114176A1 (en) * 2015-12-30 2017-07-06 阿里巴巴集团控股有限公司 Method and apparatus for coordinating consumption queue in distributed environment
CN106170013A (en) * 2016-06-29 2016-11-30 上海浦东发展银行股份有限公司信用卡中心 A kind of Kafka message uniqueness method based on Redis
CN107369012A (en) * 2016-07-05 2017-11-21 康存乐付保数据科技（上海）有限公司 A kind of method for processing payment information and system and the prepaid card processing method with performance bond insurance mechanism
CN106302385A (en) * 2016-07-26 2017-01-04 努比亚技术有限公司 A kind of message distribution device and method
CN106302385B (en) * 2016-07-26 2019-11-15 努比亚技术有限公司 A kind of message distribution device and method
CN107682169B (en) * 2016-08-02 2021-03-30 北京京东尚科信息技术有限公司 Method and device for sending message by Kafka cluster
CN107682169A (en) * 2016-08-02 2018-02-09 北京京东尚科信息技术有限公司 A kind of method and apparatus using Kafka collection pocket transmission message
CN106293968A (en) * 2016-08-04 2017-01-04 华中科技大学 A kind of intercommunication system based on Kafka message-oriented middleware and method
CN106293968B (en) * 2016-08-04 2019-05-10 华中科技大学 A kind of intercommunication system and method based on Kafka message-oriented middleware
CN106375200A (en) * 2016-11-08 2017-02-01 上海找钢网信息科技股份有限公司 Message system and message processing method
CN108076111B (en) * 2016-11-15 2021-07-09 亿阳安全技术有限公司 System and method for distributing data in big data platform
CN108076111A (en) * 2016-11-15 2018-05-25 亿阳安全技术有限公司 A kind of system and method for distributing data in big data platform
CN108076103A (en) * 2016-11-18 2018-05-25 北京优朋普乐科技有限公司 A kind of message treatment method, system and message producer's cluster
CN108132845A (en) * 2016-12-01 2018-06-08 阿里巴巴集团控股有限公司 Message storage, delivering method and device and electronic equipment
CN106817295A (en) * 2016-12-08 2017-06-09 努比亚技术有限公司 A kind of message processing apparatus and method
CN106603689A (en) * 2016-12-26 2017-04-26 努比亚技术有限公司 Data processing method and device based on distributed message releasing and subscribing system
CN106603689B (en) * 2016-12-26 2021-08-03 努比亚技术有限公司 Data processing method and device based on distributed publish-subscribe message system
CN106657349A (en) * 2016-12-29 2017-05-10 上海理想信息产业（集团）有限公司 Message subscription processing device, system and method
TWI735519B (en) * 2017-01-24 2021-08-11 香港商阿里巴巴集團服務有限公司 Distributed environment coordinated consumption queue method and device
CN106953901A (en) * 2017-03-10 2017-07-14 重庆邮电大学 A kind of trunked communication system and its method for improving message transmission performance
CN106953901B (en) * 2017-03-10 2020-04-07 重庆邮电大学 Cluster communication system and method for improving message transmission performance
CN108920144A (en) * 2017-04-11 2018-11-30 北京京东尚科信息技术有限公司 A kind of message treatment method and device
CN108965355B (en) * 2017-05-18 2021-05-25 北京京东尚科信息技术有限公司 Method, apparatus and computer readable storage medium for data transmission
CN108965355A (en) * 2017-05-18 2018-12-07 北京京东尚科信息技术有限公司 Method, apparatus and computer readable storage medium for data transmission
CN109104446A (en) * 2017-06-20 2018-12-28 中兴通讯股份有限公司 A kind of message order-preserving method, network node and storage medium
WO2018233592A1 (en) * 2017-06-20 2018-12-27 中兴通讯股份有限公司 Method for maintaining sequencing of messages, network node, and storage medium
CN109104446B (en) * 2017-06-20 2022-04-15 中兴通讯股份有限公司 Message order-preserving method, network node and storage medium
CN107329832A (en) * 2017-06-30 2017-11-07 北京奇艺世纪科技有限公司 A kind of data receiver method and device
CN107590072A (en) * 2017-08-30 2018-01-16 深圳乐信软件技术有限公司 A kind of application and development and the method and apparatus of test
CN107590072B (en) * 2017-08-30 2020-08-18 深圳乐信软件技术有限公司 Application development and test method and device
CN108009029B (en) * 2017-11-30 2022-01-04 中电福富信息科技有限公司 Method and system for decoupling and persisting cache data based on Ignite grid
CN108009029A (en) * 2017-11-30 2018-05-08 中电福富信息科技有限公司 Method and system based on the data cached decoupling persistence of Ignite grids
CN108282514B (en) * 2017-12-12 2020-11-27 北京奇艺世纪科技有限公司 Distributed service establishing method and device
CN108282514A (en) * 2017-12-12 2018-07-13 北京奇艺世纪科技有限公司 A kind of distributed service method for building up and device
CN107911484A (en) * 2017-12-13 2018-04-13 浪潮软件股份有限公司 A kind of method and device of Message Processing
CN110019240B (en) * 2017-12-29 2021-11-09 中国移动通信集团北京有限公司 Service data interaction method, device and system
CN110019240A (en) * 2017-12-29 2019-07-16 中国移动通信集团北京有限公司 A kind of service data interaction method, apparatus and system
CN108874562B (en) * 2018-06-21 2022-08-02 北京顺丰同城科技有限公司 Distributed high-concurrency message queue pushing system
CN108874562A (en) * 2018-06-21 2018-11-23 北京顺丰同城科技有限公司 Distributed high concurrent message queue supplying system
CN110659124A (en) * 2018-06-28 2020-01-07 北京京东尚科信息技术有限公司 Message processing method and device
CN110740145A (en) * 2018-07-18 2020-01-31 北京京东尚科信息技术有限公司 Message consumption method, device, storage medium and electronic equipment
CN110740145B (en) * 2018-07-18 2023-08-08 北京京东尚科信息技术有限公司 Message consumption method and device, storage medium and electronic equipment
CN108984325A (en) * 2018-07-20 2018-12-11 北京北信源信息安全技术有限公司 Message queue consuming method and device
CN108924041A (en) * 2018-08-03 2018-11-30 北京捷易迅信息技术有限公司 A kind of message efficient reliable diffusion method in instant messaging
CN109194736A (en) * 2018-08-30 2019-01-11 百度在线网络技术（北京）有限公司 Message De-weight method, device, electronic equipment, medium and unmanned vehicle
US11050814B2 (en) 2018-08-30 2021-06-29 Baidu Online Network Technology (Beijing) Co., Ltd. Method, device and vehicle for message deduplication
CN109194736B (en) * 2018-08-30 2021-04-27 百度在线网络技术（北京）有限公司 Message duplicate removal method and device, electronic equipment, medium and unmanned vehicle
CN109684099B (en) * 2018-11-23 2020-12-25 新华三大数据技术有限公司 Message processing method and device
CN109684099A (en) * 2018-11-23 2019-04-26 新华三大数据技术有限公司 Message treatment method and device
CN109614250A (en) * 2018-12-04 2019-04-12 贵州电网有限责任公司 A kind of message system of the Kafka for electric acquisition system
CN109451072A (en) * 2018-12-29 2019-03-08 广东电网有限责任公司 A kind of message caching system and method based on Kafka
TWI697217B (en) * 2019-01-30 2020-06-21 中華電信股份有限公司 Container control system for repeatedly executing serverless programs and method thereof
CN110138574A (en) * 2019-03-26 2019-08-16 北京达佳互联信息技术有限公司 Group managing means, device, electronic equipment and readable storage medium storing program for executing
CN110138574B (en) * 2019-03-26 2021-10-26 北京达佳互联信息技术有限公司 Group management method, device, electronic equipment and readable storage medium
CN110225133B (en) * 2019-06-20 2022-01-21 恒生电子股份有限公司 Message sending method, node, device, system and related equipment
CN110225133A (en) * 2019-06-20 2019-09-10 恒生电子股份有限公司 Message method, node, device, system and relevant device
CN110704208A (en) * 2019-09-19 2020-01-17 深圳证券交易所 Message processing method for multiple systems, server and storage medium
CN110704208B (en) * 2019-09-19 2023-03-14 深圳证券交易所 Message processing method for multiple systems, server and storage medium
CN111049899A (en) * 2019-12-11 2020-04-21 贝壳技术有限公司 kafka message storage system, method, apparatus, and computer-readable storage medium
CN111200637B (en) * 2019-12-20 2022-07-08 新浪网技术(中国)有限公司 Cache processing method and device
CN111200637A (en) * 2019-12-20 2020-05-26 新浪网技术(中国)有限公司 Cache processing method and device
CN111179080B (en) * 2019-12-23 2023-10-27 中国建设银行股份有限公司 Order processing method and order processing device
CN111179080A (en) * 2019-12-23 2020-05-19 中国建设银行股份有限公司 Order processing method and order processing device
CN111179081A (en) * 2019-12-24 2020-05-19 中国建设银行股份有限公司 Message processing method and processing device
CN111131512B (en) * 2019-12-31 2023-02-10 中消云（北京）物联网科技研究院有限公司 Equipment information processing method and device, storage medium and processor
CN111131512A (en) * 2019-12-31 2020-05-08 中消云（北京）物联网科技研究院有限公司 Equipment information processing method and device, storage medium and processor
CN111163172A (en) * 2019-12-31 2020-05-15 北京奇艺世纪科技有限公司 Message processing system, method, electronic device and storage medium
CN111555957A (en) * 2020-03-26 2020-08-18 孩子王儿童用品股份有限公司 Kafka-based synchronous message service system and implementation method
CN111555957B (en) * 2020-03-26 2022-08-19 孩子王儿童用品股份有限公司 Kafka-based synchronous message service system and implementation method
CN111459689A (en) * 2020-04-07 2020-07-28 中国建设银行股份有限公司 Message processing system and method based on distributed queue
CN113542314A (en) * 2020-04-13 2021-10-22 烽火通信科技股份有限公司 Method and system for realizing subscription mode based on keyword filtering
CN112330420A (en) * 2020-11-27 2021-02-05 中国农业银行股份有限公司 Order cancellation method and related device
CN112463411A (en) * 2020-12-10 2021-03-09 重庆紫光华山智安科技有限公司 Data processing method, device, server and storage medium
CN112667476B (en) * 2020-12-30 2023-02-14 平安普惠企业管理有限公司 Task-based message filtering method, device, equipment and storage medium
CN112667476A (en) * 2020-12-30 2021-04-16 平安普惠企业管理有限公司 Task-based message filtering method, device, equipment and storage medium
CN112751789A (en) * 2021-01-05 2021-05-04 浪潮云信息技术股份公司 Method and system for realizing asymmetric SDN controller cluster
CN115914346A (en) * 2021-08-09 2023-04-04 中移物联网有限公司 Internet of things message processing method and device, electronic equipment and storage medium
CN115242787A (en) * 2022-07-22 2022-10-25 恒生电子股份有限公司 Message processing system and method
CN115242787B (en) * 2022-07-22 2023-09-05 恒生电子股份有限公司 Message processing system and method

Similar Documents

Publication Publication Date Title
CN104754036A (en) Message processing system and processing method based on kafka
US20180365254A1 (en) Method and apparatus for processing information flow data
CN109451072A (en) A kind of message caching system and method based on Kafka
CN107515878B (en) Data index management method and device
CN106170016A (en) A kind of method and system processing high concurrent data requests
CN105224444B (en) Log generation method and device
CN108388479A (en) Postpone information push method, device, computer equipment and storage medium
US20100106915A1 (en) Poll based cache event notifications in a distributed cache
CN103516807A (en) Cloud computing platform server load balancing system and method
CN103106249A (en) Data parallel processing system based on Cassandra
CN109450987B (en) Number generation method, device and system and storage medium
CN111651510A (en) Data processing method and device, electronic equipment and computer readable storage medium
CN104994139B (en) A kind of system and method to high concurrent network request quick response
CN110727727B (en) Statistical method and device for database
CN104202423A (en) System for extending caches by aid of software architectures
CN105095299A (en) Picture capturing method and system
CN106789377A (en) The service parameter update method of network element cluster
CN110413978B (en) Data paging export method, device, computer equipment and storage medium
CN103577251A (en) Event based Internet computing processing system and method
CN108076103A (en) A kind of message treatment method, system and message producer's cluster
CN110175152A (en) A kind of log inquiring method, transfer server cluster and log query system
CN111210340A (en) Automatic task processing method and device, server and storage medium
CN107562803B (en) Data supply system and method and terminal
CN104539583A (en) Real-time database subscription system and method
CN113360577A (en) MPP database data processing method, device, equipment and storage medium

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20150701

RJ01 Rejection of invention patent application after publication