CN109861887A - A kind of OpenTsdb service indication polymerization based on Zookeeper - Google Patents
A kind of OpenTsdb service indication polymerization based on Zookeeper Download PDFInfo
- Publication number
- CN109861887A CN109861887A CN201910145249.4A CN201910145249A CN109861887A CN 109861887 A CN109861887 A CN 109861887A CN 201910145249 A CN201910145249 A CN 201910145249A CN 109861887 A CN109861887 A CN 109861887A
- Authority
- CN
- China
- Prior art keywords
- tsd
- worker
- tsd process
- service indication
- zookeeper
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The present invention discloses a kind of OpenTsdb service indication polymerization based on Zookeeper, is related to technical field of data processing；The TSD process of Alive is indicated by registering interim orderly znode node on Zookeeper, realize the election for possessing the TSD process of Worker role, the broadcast of Worker TSD process metadata information is realized by registering customized WorkerChangedWatcher on interim orderly znode node, and the TSD process for possessing Worker role is responsible for timing and polymerize the index inquiry request that index, persistence achievement data and customer in response end are initiated.
Description
Technical field
The present invention discloses a kind of OpenTsdb service indication polymerization based on Zookeeper, is related to data processing skill
Art field.
Background technique
Internet of Things IoT (Internet of Things) system is divided into sensing layer, transport layer, application layer, with Internet of Things
The development of technology, more and more objects (equipment or sensor etc.) are connected with each other, so that the time series data of magnanimity is produced,
In this context, it has been born and has met the professional domain type database i.e. time series database OpenTsdb of mass data quick storage.
Time series data refers to the series of values data point of some specific indexes amount in a period of time, OpenTsdb
By time series data by index and label and timestamp come one index amount of unique identification, these achievement datas have immutable
Property.TSD is the service processes of OpenTsdb, it is the stateless service based on HBase, and OpenTsdb generally passes through multiple TSD
Realize the extending transversely of cluster, each TSD example can receive write-in and the read requests of time series data, unit time
Write-in, each TSD example of read requests quantitative index can only individually count, without the statistical number of entire OpenTsdb cluster
According to, but the overall performance index of OpenTsdb cluster just can accurately reflect the handling capacity of cluster.And the invention proposes
A kind of OpenTsdb service indication polymerization based on Zookeeper, it is interim orderly and permanent in Zookeeper by utilizing
Two kinds of znode node realizes the registration of Alive TSD process, the election of Worker TSD and service indication number
The work of indicator polymerization and persistence is realized according to processes, Worker TSD such as interim storages, increases the multiple whole of cluster
Body performance indicator can accurately reflect the handling capacity of OpenTsdb cluster and the actual use situation of persistent storage, side
OpenTsdb cluster performance Optimization Work, while be also convenient for monitoring OpenTsdb cluster.In platform of internet of things for
User understands the hot spot period that time series data generates, further to analyze time series data.
Summary of the invention
The present invention is directed to problem of the prior art, provides a kind of OpenTsdb service indication polymerization based on Zookeeper
Method increases multiple overall performance indexs of cluster, can accurately reflect OpenTsdb cluster handling capacity and persistently
Change the actual use situation of storage.
Concrete scheme proposed by the present invention is:
A kind of OpenTsdb service indication polymerization based on Zookeeper, each TSD process registers of OpenTsdb
On the interim orderly znode node in Zookeeper, and attempt to be registered as Worker TSD process, while interim orderly
WorkerChangedWatcher monitor is registered on znode node, carries out the hot-swap of Worker TSD process,
Worker TSD process starts service indication data aggregate task according to the timer of configuration, by each TSD process
Service indication data aggregate phase adduction persistence, Worker TSD process also according to client query request, will take accordingly
Indicator polymerization data of being engaged in generate response and return to client.
After each TSD process of OpenTsdb normally starts in the method, the metadata information of TSD process is write
Enter in the interim orderly data field of znode node, is registered on the interim orderly znode node of Zookeeper.
Each TSD process registers pass through choosing on the interim orderly znode node in Zookeeper in the method
It lifts algorithm and elects polymerization and persistence that Worker TSD process carries out service indication data.
WorkerChangedWatcher monitor is registered on interim orderly znode node in the method, is subscribed to
NodeDataChanged, NodeChildrenChanged and NodeDeleted event, when such an event occurs, each
TSD process reattempts to the Worker TSD process that is registered as, and carries out the hot-swap of Worker TSD process.
The service indication data of each TSD process are stored in corresponding permanent znode node in the method, permanently
Service indication data persistence in znode node is into bottom HBase Cluster Database.
The service indication inquiry request of client is forwarded to a certain TSD process in the method, by this certain TSD into
The reference of the Worker TSD process access point of journey, sends inquiry request in Worker TSD process.
OpenTsdb before multiple TSD processes by disposing Load Balancer load balancer in the method
The extending transversely of cluster is carried out, client is sent in service indication inquiry request to Load Balancer load balancer, according to
The tactful forwarding inquiries request of configuration passes through drawing for the Worker TSD process access point of a certain TSD process to a certain TSD process
With sending inquiry request in Worker TSD process.
Worker TSD process is handled using the MetricQueryHandler index requestor of itself in the method
Inquiry request, according to service indication and querying condition by znode node serve achievement data corresponding in Zookeeper and
Response is generated after service indication data aggregate in HBase, and response is finally returned to client.
Usefulness of the present invention is:
The present invention provides a kind of OpenTsdb service indication polymerization based on Zookeeper, by Zookeeper
The interim orderly znode node of upper registration indicates the TSD process of Alive, realizes the choosing for possessing the TSD process of Worker role
It lifts, realizes Worker TSD by registering customized WorkerChangedWatcher on interim orderly znode node
The broadcast of process metadata information, possess Worker role TSD process be responsible for timing polymerize index, persistence achievement data with
And the index inquiry request that customer in response end is initiated, it can accurately reflect that the handling capacity of OpenTsdb cluster and persistence are deposited
The actual use situation of storage, facilitates the performance Optimization Work of OpenTsdb cluster, while being also convenient for monitoring OpenTsdb cluster.
The hot spot period that time series data generates is understood for user in platform of internet of things, further to analyze time series data.
Detailed description of the invention
Fig. 1 is OpenTsdb aggregated structure schematic diagram；
The flow chart of Worker TSD process is obtained in Fig. 2 present invention；
OpenTsdb service indication querying flow figure in Fig. 3 present invention.
Specific embodiment
The present invention provides a kind of OpenTsdb service indication polymerization based on Zookeeper, each of OpenTsdb
TSD process registers are attempted to be registered as Worker TSD process on the interim orderly znode node in Zookeeper, together
When WorkerChangedWatcher monitor is temporarily registered on orderly znode node, carry out the fervent of Worker TSD process
It changes,
Worker TSD process starts service indication data aggregate task according to the timer of configuration, by each TSD process
Service indication data aggregate phase adduction persistence, Worker TSD process also according to client query request, will take accordingly
Indicator polymerization data of being engaged in generate response and return to client.
The present invention will be further explained below with reference to the attached drawings and specific examples, so that those skilled in the art can be with
It more fully understands the present invention and can be practiced, but illustrated embodiment is not as a limitation of the invention.
Using the method for the present invention, service indication polymerization is carried out to the time series database OpenTsdb of certain Internet of Things IoT,
With reference to Fig. 1, TSD process is registered after normally starting in Zookeeper, by the ID mark of TSD process, HTTP URI
In the interim orderly znode node (/tsdb/tsd-0) data field of the metadata informations such as address and version write-in, trial will be certainly
Oneself is registered as Worker TSD process, reads the data of "/tsdb " node, and "/tsdb " node preserves Worker TSD's
Relevant information such as " { worker:tsd-0 } " elects Worker TSD process by election algorithm and carries out service indication data
Polymerization and persistence, such as choose smallest sequence number TSD process assign its Worker role,
WorkerChangedWatcher monitor is registered on "/tsdb " znode node simultaneously, is subscribed to
NodeDataChanged, NodeChildrenChanged and NodeDeleted event, when such an event occurs, each
TSD can be reattempted to and is registered as oneself to possess Worker role, to realize the hot-swap of Worker TSD；
Each TSD process can be incited somebody to action when receiving write-in or the request of reading data point, establishing connection with client
Service indication data are stored in corresponding permanent znode node, and the service indication data persistence in permanent znode node is extremely
In the index raw data table tsdb_raw_metrics of bottom HBase Cluster Database, the service in permanent znode node refers to
It, can be by this permanent znode knot removal, Worker TSD process after marking data and persistence to bottom HBase Cluster Database
Start indicator polymerization task according to the timer of configuration, the service indication data of each TSD process are permanent in Zookeeper
Numerical value in the data field of znode node "/tsdb/total_dps ", which polymerize, to be added, and "/tsdb/total_ is still stored in
In the permanent znode node of dps ", until persistence is into the index raw data table of bottom HBase Cluster Database,
With reference to Fig. 1 and Fig. 3, OpenTsdb before multiple TSD processes by disposing Load Balancer load balancer
The extending transversely of cluster is carried out, in client transmission service indication inquiry request to Load Balancer load balancer, root
It requests according to the tactful forwarding inquiries of configuration to a certain TSD process, passes through the Worker TSD process access point of a certain TSD process
(WorkerTSDEndpoint) reference sends inquiry request in Worker TSD process, and Worker TSD process utilizes
The MetricQueryHandler index requestor of itself handles inquiry request, is read according to service indication and querying condition
Service indication data in tsdb_raw_metrics table, by znode node serve achievement data corresponding in Zookeeper and
Response, and the tables of data that service indication aggregated data persistence is extremely responded are generated after service indication data aggregate in HBase
In, response is finally returned to client.
Embodiment described above is only to absolutely prove preferred embodiment that is of the invention and being lifted, protection model of the invention
It encloses without being limited thereto.Those skilled in the art's made equivalent substitute or transformation on the basis of the present invention, in the present invention
Protection scope within.Protection scope of the present invention is subject to claims.
Claims (8)
1. a kind of OpenTsdb service indication polymerization based on Zookeeper, it is characterized in that
Each TSD process registers of OpenTsdb are attempted to register on the interim orderly znode node in Zookeeper
For Worker TSD process, while WorkerChangedWatcher monitor is registered on interim orderly znode node, carried out
The hot-swap of Worker TSD process,
Worker TSD process starts service indication data aggregate task according to the timer of configuration, by the clothes of each TSD process
Business achievement data polymerize phase adduction persistence, and Worker TSD process is requested also according to client query, and corresponding service is referred to
It marks aggregated data and generates response back to client.
2. according to the method described in claim 1, it is characterized in that after each TSD process of OpenTsdb normally starts, by TSD
In the interim orderly data field of znode node of the metadata information write-in of process, it is registered in the interim orderly of Zookeeper
On znode node.
3. method according to claim 1 or 2, it is characterized in that each TSD process registers are in temporarily having in Zookeeper
On sequence znode node, the polymerization and persistence that Worker TSD process carries out service indication data are elected by election algorithm.
4. according to the method described in claim 3, it is characterized in that being registered on interim orderly znode node
WorkerChangedWatcher monitor, subscribe to NodeDataChanged, NodeChildrenChanged and
NodeDeleted event, when such an event occurs, each TSD process, which reattempts to, is registered as Worker TSD process, into
The hot-swap of row Worker TSD process.
5. according to the method described in claim 4, it is characterized in that the service indication data of each TSD process are stored in accordingly forever
In long znode node, the service indication data persistence in permanent znode node is into bottom HBase Cluster Database.
6. method according to claim 1 or 5, it is characterized in that the service indication inquiry request of client be forwarded to it is a certain
TSD process sends Worker for inquiry request by the reference of the Worker TSD process access point of this certain TSD process
In TSD process.
7. according to the method described in claim 6, it is characterized in that OpenTsdb is by disposing Load before multiple TSD processes
Balancer load balancer carries out the extending transversely of cluster, and client sends service indication inquiry request to Load
On Balancer load balancer, is requested according to the tactful forwarding inquiries of configuration to a certain TSD process, pass through a certain TSD process
Worker TSD process access point reference, send inquiry request in Worker TSD process.
8. method according to claim 1 or claim 7, it is characterized in that Worker TSD process utilizes itself
MetricQueryHandler index requestor handles inquiry request, according to service indication and querying condition by Zookeeper
In generate response after service indication data aggregate in corresponding znode node serve achievement data and HBase, and will respond most
Client is returned to eventually.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910145249.4A CN109861887B (en) | 2019-02-27 | 2019-02-27 | Zookeeper-based OpenTsdb service index aggregation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910145249.4A CN109861887B (en) | 2019-02-27 | 2019-02-27 | Zookeeper-based OpenTsdb service index aggregation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109861887A true CN109861887A (en) | 2019-06-07 |
| CN109861887B CN109861887B (en) | 2020-08-28 |
Family
ID=66899093
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910145249.4A Active CN109861887B (en) | 2019-02-27 | 2019-02-27 | Zookeeper-based OpenTsdb service index aggregation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109861887B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110532114A (en) * | 2019-09-04 | 2019-12-03 | 浪潮云信息技术有限公司 | Timing task management system and distribution method based on message queue |
| CN111309549A (en) * | 2020-02-03 | 2020-06-19 | 北京字节跳动网络技术有限公司 | Monitoring method, system, readable medium and electronic device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104376102A (en) * | 2014-11-26 | 2015-02-25 | 浪潮电子信息产业股份有限公司 | Connection method based on python and HBase jdbc |
| CN104734946A (en) * | 2015-04-09 | 2015-06-24 | 北京易掌云峰科技有限公司 | Multi-tenant high-concurrency instant messaging cloud platform |
| US20160301587A1 (en) * | 2015-04-09 | 2016-10-13 | Alibaba Group Holding Limited | Apparatus, system and method for fast leader election by coordination service |
| CN106161462A (en) * | 2016-08-29 | 2016-11-23 | 无锡华云数据技术服务有限公司 | A kind of network security certification method |
-
2019
- 2019-02-27 CN CN201910145249.4A patent/CN109861887B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104376102A (en) * | 2014-11-26 | 2015-02-25 | 浪潮电子信息产业股份有限公司 | Connection method based on python and HBase jdbc |
| CN104734946A (en) * | 2015-04-09 | 2015-06-24 | 北京易掌云峰科技有限公司 | Multi-tenant high-concurrency instant messaging cloud platform |
| US20160301587A1 (en) * | 2015-04-09 | 2016-10-13 | Alibaba Group Holding Limited | Apparatus, system and method for fast leader election by coordination service |
| CN106161462A (en) * | 2016-08-29 | 2016-11-23 | 无锡华云数据技术服务有限公司 | A kind of network security certification method |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110532114A (en) * | 2019-09-04 | 2019-12-03 | 浪潮云信息技术有限公司 | Timing task management system and distribution method based on message queue |
| CN111309549A (en) * | 2020-02-03 | 2020-06-19 | 北京字节跳动网络技术有限公司 | Monitoring method, system, readable medium and electronic device |
| CN111309549B (en) * | 2020-02-03 | 2023-04-21 | 北京字节跳动网络技术有限公司 | Monitoring method, monitoring system, readable medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109861887B (en) | 2020-08-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8301634B2 (en) | Storing and retrieving objects on a computer network in a distributed database | |
| CN103366015B (en) | A kind of OLAP data based on Hadoop stores and querying method | |
| CN104090962B (en) | Towards the nested query method of magnanimity distributed data base | |
| CN106528683B (en) | A kind of the big data cloud search system and its method balanced based on index burst | |
| RU2005130414A (en) | SYSTEM AND METHOD FOR DETERMINING SWITCHING UNDER OPTIMALITY OF TARGET OBJECT BACK AND PRIORITY OF TARGET OBJECT FOR DISTRIBUTED FILE SYSTEM | |
| CN109861887A (en) | A kind of OpenTsdb service indication polymerization based on Zookeeper | |
| CN105574334A (en) | Medical information processing method and system | |
| CN111400288A (en) | Data quality inspection method and system | |
| CN105488201A (en) | Log inquiry method and system | |
| US20140229521A1 (en) | Storing and retrieving objects on a computer network in a distributed database | |
| CN102169491B (en) | Dynamic detection method for multi-data concentrated and repeated records | |
| CN106202209B (en) | The storage of distributed structured data and querying method towards commodity screening application | |
| US20110055219A1 (en) | Database management device and method | |
| CN108734566A (en) | Collage-credit data querying method, terminal device and medium | |
| CN106909780A (en) | Medical image information processing method and processing device | |
| CN112699098A (en) | Index data migration method, device and equipment | |
| CN110019341B (en) | Data query method and device | |
| US10552406B2 (en) | Maintaining object and query result consistency in a triplestore database | |
| WO2000063798A1 (en) | A method and an apparatus for the processing of queries to a database | |
| CN104239447A (en) | Power-grid big time series data storage method | |
| WO2013069073A1 (en) | Time sequence data management system, apparatus and method | |
| CN105786990B (en) | The method and device of database data storage and quick search | |
| CN110008243A (en) | A kind of tables of data processing method and processing device | |
| CN107797924A (en) | A kind of method for detecting abnormality and its terminal of SQL scripts | |
| CN105447137A (en) | Algorithm for retrieving data with same master-slave relation from big data on the basis of relational database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 250100 No. 1036 Tidal Road, Jinan High-tech Zone, Shandong Province, S01 Building, Tidal Science Park Applicant after: Inspur cloud Information Technology Co., Ltd Address before: 250100 Ji'nan science and technology zone, Shandong high tide Road, No. 1036 wave of science and Technology Park, building S06 Applicant before: SHANDONG INSPUR CLOUD INFORMATION TECHNOLOGY Co.,Ltd. |
|
| CB03 | Change of inventor or designer information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Zhang Hui Inventor after: Sun Siqing Inventor after: Yuan Xiaolong Inventor before: Yuan Xiaolong |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |