CN112328684A - Method for synchronizing time sequence data to Kafka in real time based on OpenTsdb - Google Patents

Method for synchronizing time sequence data to Kafka in real time based on OpenTsdb Download PDF

Info

Publication number
CN112328684A
CN112328684A CN202011219627.8A CN202011219627A CN112328684A CN 112328684 A CN112328684 A CN 112328684A CN 202011219627 A CN202011219627 A CN 202011219627A CN 112328684 A CN112328684 A CN 112328684A
Authority
CN
China
Prior art keywords
kafka
plug
opentsdb
time
rtpub
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011219627.8A
Other languages
Chinese (zh)
Inventor
苑晓龙
孙思清
贾德星
刘牧
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Inspur Cloud Information Technology Co Ltd
Original Assignee
Inspur Cloud Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Inspur Cloud Information Technology Co Ltd filed Critical Inspur Cloud Information Technology Co Ltd
Priority to CN202011219627.8A priority Critical patent/CN112328684A/en
Publication of CN112328684A publication Critical patent/CN112328684A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G06F16/275Synchronous replication
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/445Program loading or initiating
    • G06F9/44521Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
    • G06F9/44526Plug-ins; Add-ons

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention discloses a method for synchronizing time sequence data to Kafka in real time based on OpenTsdb, which relates to the technical field of OpenTsdb clusters, wherein a time sequence database OpenTsdb cluster for quickly storing and reading data in an Internet of things platform supports real-time writing and sampling analysis of a large amount of time sequence data, but cannot meet the real-time analysis application of certain specific scenes, and the technical scheme is as follows: the OpenTsdb-rtpub-Kafka plug-in is customized by using an OpenTsdb custom plug-in mechanism, the Opensdb-rtpub-Kafka plug-in comprises a Kafka real-time publishing plug-in configuration unit and a Kafka real-time publisher, the Kafka real-time publishing plug-in configuration unit is responsible for setting and resolving the address, the theme and other configuration parameters of the corresponding Kafka cluster of the customized opensdb-rtpub-Kafka plug-in, and the Kafka real-time publisher is responsible for publishing the timing data and the annotation data to the corresponding theme in the distributed message queue Kafka in real time, so that different subscribers at the downstream subscribe to the timing data or the annotation data.

Description

Method for synchronizing time sequence data to Kafka in real time based on OpenTsdb
Technical Field
The invention relates to the technical field of OpenTsdb clusters, in particular to a method for synchronizing time sequence data to Kafka in real time based on OpenTsdb.
Background
The internet of things (iot of things) system is divided into a sensing layer, a transmission layer and an application layer, and with the development of the internet of things technology, more and more objects (devices or sensors and the like) are connected with each other, so that massive time series data are generated.
Time series data refers to a series of numerical data points that are a specific index quantity over a period of time, and OpenTsdb uniquely identifies an index quantity by indexes and tags and time stamps, and the index data has indelibility. The TSD is a service process of OpenTsdb, is a stateless service based on an HBase cluster, and improves the efficiency of data I/O (input/output) by means of digital mapping of index and tag data, aggregation of index data of the same tag at a specified time interval, and Snappy, ZLib and other data compression technologies.
In an internet-of-things platform, a time sequence database OpenTsdb cluster for rapidly storing and reading data supports real-time writing and sampling analysis of a large amount of time sequence data, but cannot meet the real-time analysis application (such as the alarm requirement of a monitoring platform) of certain specific scenes.
Disclosure of Invention
Aiming at the requirements and the defects of the prior art development, the invention provides a method for synchronizing time sequence data to Kafka in real time based on OpenTsdb, and the time sequence data and annotation data are published to the corresponding subjects of the distributed message queue Kafka in real time on the premise of not influencing the performance of writing the time sequence data into a time sequence database, so that different downstream subscribers subscribe the time sequence data or the annotation data, and develop applications meeting specific requirements, especially application scenes with higher real-time requirements.
The invention discloses a method for synchronizing time sequence data to Kafka in real time based on OpenTsdb, which adopts the following technical scheme for solving the technical problems:
a method for synchronizing time sequence data to Kafka in real time based on OpenTsdb is characterized in that OpenTsdb-rtpub-Kafka plug-ins are customized by utilizing an OpenTsdb custom plug-in mechanism, the customized OpenTsdb-rtpub-Kafka plug-ins run in each TSD instance, and the time sequence data and annotation data are published to corresponding subjects in a distributed message queue Kafka in real time.
Optionally, the related opentsdb-rtpub-Kafka plug-ins comprise a Kafka real-time release plug-in configuration unit and a Kafka real-time release;
the Kafka real-time release plug-in configuration unit inherits from a Config class and is responsible for setting and analyzing the address, the subject and other configuration parameters of a corresponding Kafka cluster of the custom openstdb-rtpub-Kafka plug-in;
the Kafka real-time publisher inherits from an abstrat RTPublisher abstract class and is responsible for publishing the timing data and the annotation data to corresponding subjects in a distributed message queue Kafka in real time.
Optionally, the related customized openstdb-rtpub-kafka plug-in manages the life cycle of the plug-in based on the version of the plug-in, the plug-in initialization and the plug-in stopping method.
Further optionally, the customized opentdb-rtpub-kafka plug-in supports real-time publishing of the integer and floating-point time sequence data to a topic specified by the data point parameter item according to a time interval defined by the parameter configuration item and the maximum message number of each batch based on two data point publishing methods.
Further optionally, the related customized opentdb-rtpub-kafka plug-in supports real-time publishing of annotation data to a topic specified by an annotation parameter item according to a time interval defined by the parameter configuration item and the maximum message quantity of each batch based on a publishing annotation method.
Further optionally, the customized opentdb-rtpub-Kafka plug-in collects the time-series data quantity and the annotation data quantity sent to the corresponding subject in Kafka into two monitoring indexes of data points and annotations through a collector based on a collection status method.
Optionally, when the OpenTsdb custom plug-in mechanism is used for customizing the opensdb-rtpub-kafka plug-in, relevant configuration items of the opensdb-rtpub-kafka plug-in are added to a configuration file OpenTsdb.
Further optionally, after the OpenTsdb custom plug-in mechanism is used for customizing the opensdb-rtpub-kafka plug-in, the opensdb-rtpub-kafka plug-in needs to be copied to a path specified by a path parameter item of a core plug-in of the time sequence database, and then the opensdb-rtpub-kafka plug-in can be enabled while the TSD instance is started;
and after the opentsdb-rtpub-Kafka plug-in is started, the time sequence data and the annotation data can be asynchronously and timely sent to the corresponding subjects of the distributed message queue Kafka.
Compared with the prior art, the method for synchronizing time sequence data to Kafka in real time based on OpenTsdb has the beneficial effects that:
the invention enables the opentdb-rtpub-Kafka plug-in when the TSD instance is started, and the opentdb-rtpub-Kafka plug-in can issue the time sequence data and the annotation data to the corresponding subject of the distributed message queue Kafka in real time on the premise of not influencing the performance of writing the time sequence data into the time sequence database, so that different downstream subscribers subscribe the time sequence data or the annotation data, and develop applications meeting specific requirements, especially application scenes with higher real-time requirements.
Drawings
FIG. 1 is a schematic flow diagram of the process of the present invention;
FIG. 2 is a class diagram of the opentsdb-rtpub-kafka plug-in of the present invention.
Detailed Description
In order to make the technical scheme, the technical problems to be solved and the technical effects of the present invention more clearly apparent, the following technical scheme of the present invention is clearly and completely described with reference to the specific embodiments.
The first embodiment is as follows:
with reference to fig. 1, the embodiment provides a method for synchronizing time sequence data to Kafka in real time based on OpenTsdb, an OpenTsdb custom plug-in mechanism is used for customizing OpenTsdb-rtpub-Kafka plug-in, the customized OpenTsdb-rtpub-Kafka plug-in runs in each TSD instance, and the time sequence data and annotation data are published to corresponding subjects in a distributed message queue Kafka in real time.
In this embodiment, the openstdb-rtpub-Kafka plug-in includes a Kafka real-time publishing plug-in configuration unit and a Kafka real-time publisher.
With reference to fig. 2, the Kafka real-time release plug-in configuration unit Kafka RT pubplug-in Config inherits from the Config class and is responsible for setting and parsing the address, topic and other configuration parameters of the Kafka cluster corresponding to the custom openstdb-rtpub-Kafka plug-in.
The Kafka real-time Publisher Kafka RT Publisher inherits from an abstrat RTPublisher abstract class and is responsible for publishing the timing data and annotation data to corresponding topics in the distributed message queue Kafka in real time.
In this embodiment, when an OpenTsdb custom plug-in mechanism is used to customize an opensdb-rtpub-kafka plug-in, a configuration file opensdb.config of the OpenTsdb is added with a relevant configuration item of the opensdb-rtpub-kafka plug-in, where the added configuration item includes a release plug-in path, an enabling identifier, a plug-in full path, a kafka cluster address, a data point subject, and an annotation subject.
In this embodiment, after the OpenTsdb custom plug-in mechanism is used for customizing the opensdb-rtpub-kafka plug-in, the opensdb-rtpub-kafka plug-in needs to be copied to a path specified by a path parameter item of a core plug-in of a time sequence database, and then the opensdb-rtpub-kafka plug-in can be started while the TSD instance is started;
and after the opentsdb-rtpub-Kafka plug-in is started, the time sequence data and the annotation data can be asynchronously and timely sent to the corresponding subjects of the distributed message queue Kafka.
Example two:
with reference to fig. 1, the embodiment provides a method for synchronizing time sequence data to Kafka in real time based on OpenTsdb, an OpenTsdb custom plug-in mechanism is used for customizing OpenTsdb-rtpub-Kafka plug-in, the customized OpenTsdb-rtpub-Kafka plug-in runs in each TSD instance, and the time sequence data and annotation data are published to corresponding subjects in a distributed message queue Kafka in real time.
In this embodiment, the openstdb-rtpub-Kafka plug-in includes a Kafka real-time publishing plug-in configuration unit and a Kafka real-time publisher.
Referring to fig. 2, the Kafka real-time release plug-in configuration unit Kafka RT pubplug-in Config inherits from the Config class and is responsible for setting and parsing the address, topic and other configuration parameters of the corresponding Kafka cluster of the custom openstdb-rtpub-Kafka plug-in.
The Kafka real-time Publisher Kafka RT Publisher inherits from an abstrat RTPublisher abstract class and is responsible for publishing the timing data and annotation data to corresponding topics in the distributed message queue Kafka in real time.
In this embodiment, in order to better publish the time series data and the annotation data to the corresponding topic in the distributed message queue Kafka in real time, the customized opentdb-rtpub-Kafka plug-in may further implement the following method:
(a) the custom opentsdb-rtpub-kafka plug-in manages the life cycle of the plug-in based on the version of the plug-in, the plug-in initialization and the plug-in stop method.
(b) The self-defined opentsdb-rtpub-kafka plug-in supports real-time publishing of the time sequence data with integer and floating point values to a theme specified by the data point parameter item according to the time interval defined by the parameter configuration item and the maximum message quantity of each batch based on the two data point publishing method.
(c) The self-defined opentsdb-rtpub-kafka plug-in supports real-time publishing of annotation data to a topic specified by an annotation parameter item according to a time interval defined by the parameter configuration item and the maximum message quantity of each batch based on a publishing annotation method.
(d) The customized opentsdb-rtpub-Kafka plug-in collects the time sequence data quantity and the annotation data quantity sent to the corresponding subject in Kafka into two monitoring indexes of data points and annotations through a collector based on a collection state method.
In this embodiment, when an OpenTsdb custom plug-in mechanism is used to customize an opensdb-rtpub-kafka plug-in, a configuration file opensdb.config of the OpenTsdb is added with a relevant configuration item of the opensdb-rtpub-kafka plug-in, where the added configuration item includes a release plug-in path, an enabling identifier, a plug-in full path, a kafka cluster address, a data point subject, and an annotation subject.
In this embodiment, after the OpenTsdb custom plug-in mechanism is used for customizing the opensdb-rtpub-kafka plug-in, the opensdb-rtpub-kafka plug-in needs to be copied to a path specified by a path parameter item of a core plug-in of a time sequence database, and then the opensdb-rtpub-kafka plug-in can be started while the TSD instance is started;
and after the opentsdb-rtpub-Kafka plug-in is started, the time sequence data and the annotation data can be asynchronously and timely sent to the corresponding subjects of the distributed message queue Kafka.
In summary, by using the method for synchronizing the time series data to the Kafka in real time based on OpenTsdb of the present invention, the time series data and the annotation data can be published to the corresponding topic of the distributed message queue Kafka in real time, so that different downstream subscribers subscribe to the time series data or the annotation data.
The principles and embodiments of the present invention have been described in detail using specific examples, which are provided only to aid in understanding the core technical content of the present invention. Based on the above embodiments of the present invention, those skilled in the art should make any improvements and modifications to the present invention without departing from the principle of the present invention, and therefore, the present invention should fall into the protection scope of the present invention.

Claims (8)

1. A method for synchronizing time sequence data to Kafka in real time based on OpenTsdb is characterized in that an OpenTsdb-rtpub-Kafka plug-in is customized by using an OpenTsdb custom plug-in mechanism, the customized opensdb-rtpub-Kafka plug-in runs in each TSD instance, and the time sequence data and annotation data are published to corresponding subjects in a distributed message queue Kafka in real time.
2. The OpenTsdb-based method for synchronizing time series data to Kafka in real time according to claim 1, wherein the custom opensdb-rtpub-Kafka plug-in includes a Kafka real-time publishing plug-in configuration unit and a Kafka real-time publisher;
the Kafka real-time release plug-in configuration unit inherits from a Config class and is responsible for setting and analyzing the address, the subject and other configuration parameters of a corresponding Kafka cluster of the custom openstdb-rtpub-Kafka plug-in;
the Kafka real-time publisher inherits from an abstrat RTPublisher abstract class and is responsible for publishing the timing data and the annotation data to corresponding subjects in a distributed message queue Kafka in real time.
3. The method of claim 1, wherein the custom OpenTsdb-rtpub-Kafka plug-in manages the life cycle of the plug-in based on the version of the plug-in, the method of initializing the plug-in, and the method of stopping the plug-in.
4. The method of claim 3, wherein the custom OpenTsdb-rtpub-Kafka plug-in supports real-time publishing of integer and floating point time series data to a topic specified by the data point parameter item according to a time interval defined by the parameter configuration item and a maximum number of messages per batch based on two data point publishing methods.
5. The method of claim 4, wherein the custom OpenTsdb-rtpub-Kafka plug-in supports real-time publishing of annotation data to a topic specified by an annotation parameter item according to a time interval defined by the parameter configuration item and a maximum number of messages per batch based on a publishing annotation method.
6. The method of claim 5, wherein the custom OpenTsdb-rtpub-Kafka plug-in collects the time series data amount and the annotation data amount sent to the corresponding topic in Kafka into two monitoring indicators, namely a data point and an annotation, respectively, through a collector based on a collection status method.
7. The method as claimed in claim 5, wherein when the OpenTsdb-rtpub-Kafka plug-in is customized by using the OpenTsdb custom plug-in mechanism, the configuration file opentsdb.config of the OpenTsdb is added with the relevant configuration items of the OpenTsdb-rtpub-Kafka plug-in, and the added configuration items include a publishing plug-in path, an enabling identifier, a plug-in full path, a Kafka cluster address, a data point subject, and an annotation subject.
8. The method of claim 7, wherein after the OpenTsdb-rtpub-Kafka plug-in is customized by using an OpenTsdb custom plug-in mechanism, the OpenTsdb-rtpub-Kafka plug-in needs to be copied to a path specified by a path parameter item of a core plug-in of the timing database, and then the OpenTsdb-rtpub-Kafka plug-in can be enabled at the same time as the TSD instance is started;
and after the opentsdb-rtpub-Kafka plug-in is started, the time sequence data and the annotation data can be asynchronously and timely sent to the corresponding subjects of the distributed message queue Kafka.
CN202011219627.8A 2020-11-03 2020-11-03 Method for synchronizing time sequence data to Kafka in real time based on OpenTsdb Pending CN112328684A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011219627.8A CN112328684A (en) 2020-11-03 2020-11-03 Method for synchronizing time sequence data to Kafka in real time based on OpenTsdb

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011219627.8A CN112328684A (en) 2020-11-03 2020-11-03 Method for synchronizing time sequence data to Kafka in real time based on OpenTsdb

Publications (1)

Publication Number Publication Date
CN112328684A true CN112328684A (en) 2021-02-05

Family

ID=74316170

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011219627.8A Pending CN112328684A (en) 2020-11-03 2020-11-03 Method for synchronizing time sequence data to Kafka in real time based on OpenTsdb

Country Status (1)

Country Link
CN (1) CN112328684A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114217875A (en) * 2021-12-17 2022-03-22 平安壹钱包电子商务有限公司 Method, device and equipment for processing order and storage medium

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104092767A (en) * 2014-07-21 2014-10-08 北京邮电大学 Posting/subscribing system for adding message queue models and working method thereof
CN108460789A (en) * 2018-03-19 2018-08-28 国家基础地理信息中心 A kind of artificial earth's surface timing variations on-line detecting system and method
CN108768826A (en) * 2018-05-16 2018-11-06 华南理工大学 Based on the message route method under MQTT and Kafka high concurrent scenes
CN109086894A (en) * 2018-07-06 2018-12-25 西安热工研究院有限公司 A kind of warning message centring system of facing area genco
CN109189589A (en) * 2018-08-14 2019-01-11 北京博睿宏远数据科技股份有限公司 A kind of distribution big data computing engines and framework method
CN109271412A (en) * 2018-09-28 2019-01-25 中国-东盟信息港股份有限公司 The real-time streaming data processing method and system of smart city
CN110019085A (en) * 2017-10-31 2019-07-16 上海宝信软件股份有限公司 A kind of distributed time series database based on HBase
CN110457178A (en) * 2019-07-29 2019-11-15 江苏艾佳家居用品有限公司 A kind of full link monitoring alarm method based on log collection analysis

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104092767A (en) * 2014-07-21 2014-10-08 北京邮电大学 Posting/subscribing system for adding message queue models and working method thereof
CN110019085A (en) * 2017-10-31 2019-07-16 上海宝信软件股份有限公司 A kind of distributed time series database based on HBase
CN108460789A (en) * 2018-03-19 2018-08-28 国家基础地理信息中心 A kind of artificial earth's surface timing variations on-line detecting system and method
CN108768826A (en) * 2018-05-16 2018-11-06 华南理工大学 Based on the message route method under MQTT and Kafka high concurrent scenes
CN109086894A (en) * 2018-07-06 2018-12-25 西安热工研究院有限公司 A kind of warning message centring system of facing area genco
CN109189589A (en) * 2018-08-14 2019-01-11 北京博睿宏远数据科技股份有限公司 A kind of distribution big data computing engines and framework method
CN109271412A (en) * 2018-09-28 2019-01-25 中国-东盟信息港股份有限公司 The real-time streaming data processing method and system of smart city
CN110457178A (en) * 2019-07-29 2019-11-15 江苏艾佳家居用品有限公司 A kind of full link monitoring alarm method based on log collection analysis

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
百里燊: "《OpenTSDB技术内幕》", 31 March 2019 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114217875A (en) * 2021-12-17 2022-03-22 平安壹钱包电子商务有限公司 Method, device and equipment for processing order and storage medium
CN114217875B (en) * 2021-12-17 2024-08-27 平安壹钱包电子商务有限公司 Method, device, equipment and storage medium for processing order

Similar Documents

Publication Publication Date Title
CN108536761B (en) Report data query method and server
CN112507029B (en) Data processing system and data real-time processing method
CN109063196B (en) Data processing method and device, electronic equipment and computer readable storage medium
CN106503276A (en) A kind of method and apparatus of the time series databases for real-time monitoring system
CN106487891B (en) A method of the processing message based on kafka
CN108762738B (en) Platformized data acquisition system
CN106997414B (en) Multidisciplinary collaborative simulation implementation method based on reflective memory network
CN105227367A (en) A kind of warning analysis processing method of low delay
CN106780149A (en) A kind of equipment real-time monitoring system based on timed task scheduling
CN111177237B (en) Data processing system, method and device
CN112559475A (en) Data real-time capturing and transmitting method and system
CN111754268A (en) OTT big data-based user label generation method, management system and storage medium
CN112328684A (en) Method for synchronizing time sequence data to Kafka in real time based on OpenTsdb
CN114090378A (en) Custom monitoring and alarming method based on Kapacitor
CN115391429A (en) Time sequence data processing method and device based on big data cloud computing
CN114265883B (en) Method, equipment and storage medium for real-time data management
CN115617849A (en) Data processing method and device, electronic equipment and storage medium
CN116610531B (en) Method for collecting data embedded points and requesting image uploading data based on code probe
CN110955757A (en) Photovoltaic power station equipment log retrieval method and system
CN116431429A (en) Unified monitoring platform and monitoring method for middleware
Wang et al. Block storage optimization and parallel data processing and analysis of product big data based on the hadoop platform
CN115422259A (en) Data processing method, system, equipment and storage medium of time sequence database
CN108959041B (en) Method for transmitting information, server and computer readable storage medium
Zhao et al. Collecting, managing and analyzing social networking data effectively
CN112256446A (en) Kafka message bus control method and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20210205