CN109783253A - Distributed message data/address bus based on kafka - Google Patents
Distributed message data/address bus based on kafka Download PDFInfo
- Publication number
- CN109783253A CN109783253A CN201811626474.1A CN201811626474A CN109783253A CN 109783253 A CN109783253 A CN 109783253A CN 201811626474 A CN201811626474 A CN 201811626474A CN 109783253 A CN109783253 A CN 109783253A
- Authority
- CN
- China
- Prior art keywords
- kafka
- data
- interface
- general purpose
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention is based on the distributed message data/address bus of kafka to be related to information data field, especially a kind of distributed message data/address bus suitable for distributed data base and distributed computing cluster.Including kafka Universal Subscriber end interface and data unified model interface；The kafka Universal Subscriber end interface is used for kafka producer and the consumer client of unified different editions, external system is shielded to limit the use of each version of kafka, it is connected to realize with producer, consumer client of different editions kafka, guarantees that the two interface structure is consistent；The data unified model interface is used for uniform service data-interface, with kafka data decoupler；According to interface structure, unified model combination/Splitting Logic of business datum is realized.
Description
Technical field
The present invention is based on the distributed message data/address bus of kafka to be related to information data field, and especially one kind is suitable for
The distributed message data/address bus of distributed data base and distributed computing cluster.
Background technique
Kafka is developed by linkedIn company, it is desirable to provide quick, expansible, distributed Publish-subscribe disappears
Breath system.The main application direction of kafka has: massive logs collection, mass data caching etc..
Kafka provides a kind of magnanimity buffering scheme, can cache in some cycles to data, and user can store
Data can also pull the data (that is, consumption data) in caching to the caching (that is, creation data).More than cycle time, then
The data that the cycle memory is put are emptied automatically.In this way, can guarantee the recycling of data in certain time, and can be in a steady stream
Continuous storage data.
The structure of kafka is mainly as follows:
1) topic-message subject stores certain types of message data；That is, each data are stored under some theme, this
Sample data can store classifiedly, and avoid cross-reference bring chaotic；
2) producer-message producer encapsulates data into the message elements of kafka, and the message elements is sent to
Kafka system is deposited into the topic of corresponding types；
3) consumer-message consumer subscribes to one or more topic, message data needed for consuming；
4) broker-kafka nodename, message data in persistence topic to local disk.
Nowadays, with the rapid development of big data technology, the value of data increasingly embodies.More and more enterprises pay attention to
Data therefrom excavate more favorable information.And these are required to premised on the efficient storage of data, shared, exchange.
There are the following problems for current kafka information distribution subscription:
1) message data structure of topic (message subject) storage is linked up with respective operation system, and the two close-coupled, this is just
So that can not be simple and quick between different business systems share, exchange respective data；Data can not intersect operating, it is difficult to,
Then it is unable to satisfy the data value supreme epoch；
2) kafka different editions are mutually incompatible namely different editions kafka need to realize respective kafka client
(producer client, consumer client)；The compatible all kafka versions of ununified client, and then can unite
One operation information data.
In conclusion existing technology is limited to:
1) message data is stored in business coupling, it is difficult to interact；
2) it can not be compatible between kafka version, its client can not be unified.
Summary of the invention
The purpose of the present invention is providing a kind of distributed message data/address bus based on kafka in view of the above shortcomings,
It realizes unified message data transmission, decouples its business；Unified kafka client, compatible each kafka version.
The present invention adopts the following technical solutions to achieve:
Distributed message data/address bus based on kafka includes kafka Universal Subscriber end interface and data unified model interface；
The kafka Universal Subscriber end interface is used for kafka producer and the consumer client of unified different editions,
External system is shielded to limit the use of each version of kafka, thus the producer of realization and different editions kafka,
Consumer client is connected, and guarantees that the two interface structure is consistent；
The data unified model interface is used for uniform service data-interface, saves and decouples with kafka data；According to interface knot
Structure realizes unified model combination/Splitting Logic of business datum.
The kafka Universal Subscriber end interface includes kafka producer general purpose client and kafka consumer logical
Use client；The kafka producer general purpose client, that is, kafka producer, to kafka cluster PUSH message data；kafka
Consumer general purpose client, that is, kafka consumer, to kafka cluster pulling data.Kafka Universal Subscriber end interface is directed to
Kafka cluster is operated, and kafka cluster can be different version, thus general purpose client is supplied to operation system and makes
Used time shields the details of kafka primary client realization.
Kafka Universal Subscriber end interface can load the dependence of corresponding version kafka, to reach simultaneous according to version number
Hold the effect of different kafka versions.
Data unified model interface can carry out the calling of structure according to the difference of business datum.
Data unified model interface is divided into two kinds, i.e. data structure serializing interface and data structure unserializing interface.
Data structure serializes interface, for business datum to be converted to unified byte sequence data.
Data structure serializes the working method of interface, includes the following steps:
1) data structure of business datum is parsed；
2) data after parsing carry out serializing operation according to the binary specification of agreement, generate binary system byte sequence；
3) byte sequence that above-mentioned steps generate is assembled according to constrained qualification, the component part of assembling includes serializing head
Portion and truthful data.
The constrained qualification are as follows: compressing mark+real data length+data type+truthful data (after compression).
The serializing head includes real data length, truthful data compressing mark and truthful data type identification.
Data structure unserializing interface, for byte sequence data reversal to be melted into business datum.
Data structure serializes the working method of interface, includes the following steps:
1) byte sequenceization is split；
Data are split according to Data Serialization specification, parse truthful data and serializing head；
2) binary specification for indicating and arranging according to the information on serializing head, carries out unserializing for truthful data；
3) data for the unserializing for obtaining step (2) carry out regular according to business datum specification, are integrated into business datum knot
Structure.
Kafka producer general purpose client sends data to kafka by unified client-side interface.
The concrete methods of realizing of kafka producer general purpose client, comprising:
1) the kafka producer native interface of each version, interface format and primary consistent, such external system use are realized
When, it is not necessary to modify original interface formats, that is, can be used directly；
2) according to the difference of kafka version, different native interfaces is called；Meanwhile the primary dependence of the kafka version is carried out
Load；It relies on load and supports the use of kafka producer general purpose client；
3) carry out data transmission.
Kafka consumer general purpose client is pulled data from kafka by unified client-side interface.
The concrete methods of realizing of kafka consumer general purpose client, comprising:
1) realize the kafka consumer native interface of each version, interface format and primary consistent, external system it is not necessary to modify
The consumer Universal Subscriber end interface can be used directly in original interface format；
2) according to the difference of kafka version, different native interfaces is called.Meanwhile loading the primary dependence of kafka；The dependence branch
Hold the use of consumer general purpose client；
3) data in kafka are pulled.
The invention has the advantages that externally providing unified kafka Universal Subscriber end interface and data unified model interface, it was not necessarily to
More learning costs can be grasped.Save the learning cost of client differentiation between different kafka versions, node different business
The operating cost of data structure.The present invention is easy to use, only need to can meet different kafka visitors by the kafka general-purpose interface
The operation at family end；Only it need to can meet the conversion of operational data structure by data unified model interface.
Detailed description of the invention
Below with reference to attached drawing, the invention will be further described:
Fig. 1 is the structure principle chart of the specific embodiment of the invention；
Fig. 2 is the function realization principle figure of kafka Universal Subscriber end interface of the present invention；
Fig. 3 is the function realization principle figure of data unified model interface of the present invention.
Specific embodiment
Referring to attached drawing 1, application program can support data unified model and kafka general purpose client two parts, that is, call two
The interface that person provides.
Kafka general purpose client can be compatible with the client of kafka different editions, i.e., can access operation by the client
The kafka of different editions, and the unified only support " byte arrays " of type for transmitting message data.
Business datum is carried out parsing combination/fractionation integration and waited and etc. by data unified model, it is therefore an objective to unified message
Data transmission decouples business.
Kafka general purpose client includes kafka producer general purpose client and kafka consumer general purpose client
Two kinds, the kafka producer general purpose client, that is, kafka producer, to kafka cluster PUSH message data；kafka
Consumer general purpose client, that is, kafka consumer, to kafka cluster pulling data.Pass through the general visitor of kafka general purpose client
Family end is operated for kafka cluster, and the kafka cluster can be different version.Kafka general purpose client provides
It is used to operation system, shields the details of kafka primary client realization.
Data unified model is distinctive model in the present invention, is converted for business datum, is converted to byte sequence；
Kafka general purpose client unifies kafka message data transport-type, i.e. byte sequence simultaneously.That is the conversion of data and data sequence
Column/unserializing are peeled away with kafka message transport-type, operation system.It can flexibly be grasped between each system in this way
Make different data, the prior value of mining data.
Fig. 2 illustrates the function realization principle of kafka Universal Subscriber end interface of the present invention.Operation system is logical using kafka
When with client, the kafka that can load corresponding version from the background relies on and reflects the realization of primary kafka client, and then logarithm
According to being transmitted or pulled.Client need to only inform the specific version number of backstage kafka.Do not changing general purpose client
In the case where, it can flexibly increase newborn kafka version or the unwanted kafka version of disassembly.Operation system no longer needs to pay close attention to
The differentiation of specific version kafka, need to can only carry out the operation of data by interface.Learning cost than it is original reduce, it is simple
It is easy-to-use, expansible, easy to maintain.
General purpose client provides kafka partitioning strategies and supports, realizes without operation system, is wanted from the background according to partitioning strategies
It asks, voluntarily realizes specific partitioning strategies logic.Operation system only needs easy configuration to can be used.
Under normal circumstances, operation of the operation system to kafka cluster is both needed to setting configuration, and then provide must for general-purpose interface
With item, configuration complexity is reduced.If operation system need to be configured additionally, can voluntarily it configure, general-purpose interface can equally be known
Not, and to kafka cluster generation it acts on.
In addition, data type is unified into byte arrays by the present invention, i.e., transmission data and pulling data are byte arrays.
External uniform data type, so that kafka cluster-based storage data and operation system decouple, i.e. kafka need to only be absorbed in data storage
Tactful (such as partitioning strategies), without paying close attention to its operation system in operation data.
Shown in Fig. 3, data unified model interface is divided into two kinds；Realize that the data structure serializing of byte sequence assembling connects
Mouth and the data structure unserializing interface for realizing data structure integration.
Data structure serialize interface, by business datum be unified into byte sequence be transmitted to kafka cluster carry out storage order
It reads.Operation system uses data unified model interface, its business datum can be unified into defined model, be finally assembling to word
Jie Zulie.
Data structure unserializing interface, pulls the business datum in kafka cluster.Byte sequence is split, instead
Serializing, is finally integrated into complete data structure.
Operation system only needs that unified model can be carried out to business datum according to mark.The model interface usage mode spirit
It is living, i.e., increase new business data structure, the unwanted data structure of disassembly as needed.Accomplish once to write, it is permanent to use.This
Data unified model interface can be used in combination with kafka general purpose client, and also detachable use, mode are flexible.
Claims (9)
1. a kind of distributed message data/address bus based on kafka, which is characterized in that including kafka Universal Subscriber end interface and
Data unified model interface；
The kafka Universal Subscriber end interface is used for kafka producer and the consumer client of unified different editions,
External system is shielded to limit the use of each version of kafka, thus the producer of realization and different editions kafka,
Consumer client is connected, and guarantees that the two interface structure is consistent；
The data unified model interface is used for uniform service data-interface, with kafka data decoupler；It is real according to interface structure
Unified model combination/Splitting Logic of existing business datum.
2. the distributed message data/address bus according to claim 1 based on kafka, which is characterized in that the kafka
Universal Subscriber end interface includes kafka producer general purpose client and kafka consumer general purpose client；
The kafka producer general purpose client, that is, kafka producer, to kafka cluster PUSH message data；kafka
Consumer general purpose client, that is, kafka consumer, to kafka cluster pulling data.
3. the distributed message data/address bus according to claim 2 based on kafka, which is characterized in that kafka
Producer general purpose client sends data to kafka by unified client-side interface；
The concrete methods of realizing of kafka producer general purpose client, comprising:
1) the kafka producer native interface of each version, interface format and primary consistent, such external system use are realized
When, it is not necessary to modify original interface formats, that is, can be used directly；
2) according to the difference of kafka version, different native interfaces is called；Meanwhile the primary dependence of the kafka version is carried out
Load；It relies on load and supports the use of kafka producer general purpose client；
3) carry out data transmission.
4. the distributed message data/address bus according to claim 2 based on kafka, which is characterized in that kafka
Consumer general purpose client is pulled data from kafka by unified client-side interface；
The concrete methods of realizing of kafka consumer general purpose client, comprising:
1) realize the kafka consumer native interface of each version, interface format and primary consistent, external system it is not necessary to modify
The consumer Universal Subscriber end interface can be used directly in original interface format；
2) according to the difference of kafka version, different native interfaces is called；
Meanwhile loading the primary dependence of kafka；The use of dependence support consumer general purpose client；
3) data in kafka are pulled.
5. the distributed message data/address bus according to claim 1 based on kafka, which is characterized in that data unify mould
Type interface is divided into two kinds, i.e. data structure serializing interface and data structure unserializing interface；
Data structure serializes interface, for business datum to be converted to unified byte sequence data；
Data structure unserializing interface, for byte sequence data reversal to be melted into business datum.
6. the distributed message data/address bus according to claim 5 based on kafka, which is characterized in that data structure sequence
The working method of columnization interface, includes the following steps:
1) data structure of business datum is parsed；
2) data after parsing carry out serializing operation according to the binary specification of agreement, generate binary system byte sequence；
3) byte sequence that above-mentioned steps generate is assembled according to constrained qualification, the component part of assembling includes serializing head
Portion and truthful data.
7. the distributed message data/address bus according to claim 6 based on kafka, which is characterized in that the constraint rule
Model are as follows: compressing mark+real data length+data type+compressed truthful data.
8. the distributed message data/address bus according to claim 5 based on kafka, which is characterized in that the serializing
Head includes real data length, truthful data compressing mark and truthful data type identification.
9. the distributed message data/address bus according to claim 8 based on kafka, which is characterized in that data structure sequence
The working method of columnization interface, includes the following steps:
1) byte sequenceization is split；
Data are split according to Data Serialization specification, parse truthful data and serializing head；
2) binary specification for indicating and arranging according to the information on serializing head, carries out unserializing for truthful data；
3) data for the unserializing for obtaining step (2) carry out regular according to business datum specification, are integrated into business datum knot
Structure.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811626474.1A CN109783253B (en) | 2018-12-28 | 2018-12-28 | Distributed message data bus based on kafka |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811626474.1A CN109783253B (en) | 2018-12-28 | 2018-12-28 | Distributed message data bus based on kafka |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109783253A true CN109783253A (en) | 2019-05-21 |
| CN109783253B CN109783253B (en) | 2023-02-14 |
Family
ID=66498788
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811626474.1A Active CN109783253B (en) | 2018-12-28 | 2018-12-28 | Distributed message data bus based on kafka |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109783253B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114553970A (en) * | 2022-02-07 | 2022-05-27 | 上海数依数据科技有限公司 | Distributed message processing method based on Kafka and data bus system |
| CN115063123A (en) * | 2022-08-17 | 2022-09-16 | 歌尔股份有限公司 | Intelligent manufacturing method and system and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104156216A (en) * | 2014-08-14 | 2014-11-19 | 浪潮(北京)电子信息产业有限公司 | Heterogeneous storage management system and method oriented to cloud computing |
| CN106293968A (en) * | 2016-08-04 | 2017-01-04 | 华中科技大学 | A kind of intercommunication system based on Kafka message-oriented middleware and method |
| CN106487891A (en) * | 2016-10-11 | 2017-03-08 | 北京集奥聚合科技有限公司 | A kind of method of the process message based on kafka |
| CN107395729A (en) * | 2017-07-27 | 2017-11-24 | 深圳乐信软件技术有限公司 | A kind of consumption system of message queue, method and device |
-
2018
- 2018-12-28 CN CN201811626474.1A patent/CN109783253B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104156216A (en) * | 2014-08-14 | 2014-11-19 | 浪潮(北京)电子信息产业有限公司 | Heterogeneous storage management system and method oriented to cloud computing |
| CN106293968A (en) * | 2016-08-04 | 2017-01-04 | 华中科技大学 | A kind of intercommunication system based on Kafka message-oriented middleware and method |
| CN106487891A (en) * | 2016-10-11 | 2017-03-08 | 北京集奥聚合科技有限公司 | A kind of method of the process message based on kafka |
| CN107395729A (en) * | 2017-07-27 | 2017-11-24 | 深圳乐信软件技术有限公司 | A kind of consumption system of message queue, method and device |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114553970A (en) * | 2022-02-07 | 2022-05-27 | 上海数依数据科技有限公司 | Distributed message processing method based on Kafka and data bus system |
| CN115063123A (en) * | 2022-08-17 | 2022-09-16 | 歌尔股份有限公司 | Intelligent manufacturing method and system and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109783253B (en) | 2023-02-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111061715B (en) | Web and Kafka-based distributed data integration system and method | |
| US7287037B2 (en) | Method and apparatus for generating service oriented state data mapping between extensible meta-data model and state data including logical abstraction | |
| US7693973B2 (en) | Pluggable state meta-data processors based on meta information modeling in a service oriented architecture | |
| CN105760534B (en) | Data structure, hadoop cluster, server and its application method of customized serializability | |
| CA2718722C (en) | Modular transcoding pipeline | |
| US20110258598A1 (en) | Server side application integration framework | |
| US20110314456A1 (en) | System and Method for Mapping Structured Document to Structured Data of Program Language and Program for Executing Its Method | |
| CN108388622A (en) | Api interface dynamic creation method, device, computer equipment and storage medium | |
| KR100880536B1 (en) | Open framework system for heterogeneous computing and service integration | |
| CN112527899A (en) | Data synchronization method, device, equipment and storage medium | |
| CN109783253A (en) | Distributed message data/address bus based on kafka | |
| CN102566984A (en) | Method and device for configuring parameters | |
| KR20080043517A (en) | Apparatus and method for parsing domain profile in software communication architecture | |
| CN115145683A (en) | Cloud service implementation method and device | |
| CN101136780A (en) | Method, system to obtain user command information, and user command registered device | |
| CN110471654A (en) | The cloud development system and computer software program product of communication protocol | |
| US20170131980A1 (en) | Model driven architecture for network management and configuration | |
| CN112989123A (en) | Dynamic data type communication method and device based on DDS | |
| Selker et al. | Elements of visual language | |
| CN112328406A (en) | Micro-service distributed transaction system based on TCC (transmission control center) and service processing method | |
| KR20040074766A (en) | Storage Medium Having Object-Oriented Program | |
| CN116155978A (en) | Multi-registry adaptation method, device, electronic equipment and storage medium | |
| CN116227625A (en) | Intelligent model development method, medium and equipment | |
| CN113312394B (en) | Method and device for constructing data acquisition system | |
| US8752075B1 (en) | Method for data transport |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |