CN110134511A - A kind of shared storage optimization method of OpenTSDB - Google Patents
A kind of shared storage optimization method of OpenTSDB Download PDFInfo
- Publication number
- CN110134511A CN110134511A CN201910293370.1A CN201910293370A CN110134511A CN 110134511 A CN110134511 A CN 110134511A CN 201910293370 A CN201910293370 A CN 201910293370A CN 110134511 A CN110134511 A CN 110134511A
- Authority
- CN
- China
- Prior art keywords
- tenant
- resource
- opentsdb
- data
- optimization method
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 36
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000005457 optimization Methods 0.000 title claims abstract description 23
- 230000008901 benefit Effects 0.000 claims abstract description 7
- 238000007726 management method Methods 0.000 claims description 30
- 238000012790 confirmation Methods 0.000 claims description 3
- 238000013523 data management Methods 0.000 claims description 3
- 238000004519 manufacturing process Methods 0.000 claims description 3
- 238000013500 data storage Methods 0.000 description 7
- 238000012544 monitoring process Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000004575 stone Substances 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention is more particularly directed to a kind of OpenTSDB to share storage optimization method.The OpenTSDB shares storage optimization method, when tenant creates OpenTSDB service, creates the tables of data of the exclusive user in OpenTSDB storage end；For the different business application example of different tenants or different tenants, the use resource of each tenant is managed using the quotas administered of HBase；Meanwhile tenant is counted using the resource utilization in service process, to the quota that cluster items resource uses carry out disclosure, accomplish it is shared storage application in resource using the transparency.The OpenTSDB shares storage optimization method, the expense undertaken in cluster resource according to tenant, tenant are divided using the hierarchical management and priority weighting of resource, the resource use of tenant is not only set to obtain controllable management, tenant is also set to have obtained the benefit that resource uses under limited investment, resource expense detail list simultaneously, make the expense of tenant using intuitively showing, to improve customer satisfaction.
Description
Technical field
The present invention relates to time series database and HBase technical field, in particular to a kind of OpenTSDB shares storage optimization
Method.
Background technique
Big data technology achieves good development at home in recent years, is related to data acquisition, data storage, number among these
According to technologies such as analysis and data minings.
Why big data has its name, exactly because associated mass data, and time series data is then among these
Account for very big specific gravity.Time series data, sees its name, then is the data closely related with the time, such as in Internet of Things service,
The data that every sensor generates；The each point of data etc. generated for each second in monitoring system.Meanwhile relevant database can not
Meet to time series data it is effective storage with handle, therefore there is an urgent need to one kind done specifically for time series data it is excellent
The Database Systems of change, i.e. time series database.
Time series database, full name are time series databases.Time series databases are mainly used for referring to that the processing band time marks
The data of label (changing according to the sequence of time, i.e. the time serializes), the data with time tag are also referred to as time series data.
Currently, time series data is mainly set by all types of real-time monitorings such as power industry, chemical industry, inspection and analysis
The standby data for acquiring, generating, the typical feature of these industrial datas is: generation frequency is fast (in each monitoring point one second
Can produce a plurality of data), depend critically upon acquisition time (each data is required to correspond to unique time), measuring point multi information
Amount is big, and (conventional real-time monitoring system has thousands of monitoring point, and monitoring point all generates data, generate daily several each second
The data volume of ten GB).
Time series database service memory is managed with O&M and is realized for convenience under the usage scenario of High Availabitity, shared to deposit
Storage is common framework mode.Nowadays widely applied time series databases are varied, also each advantageous.Time series number
It according to library OpenTSDB, is based on HBase (the non-relational distributed data base of open source), is the NoSQL data of distributed expandable
Library.
OpenTSDB designer has done many optimizations in data storage and data access side face, such as a dimension is same
The data of section time carry out cohesively managed not only makes data query more convenient, and subtract using data dictionary by data compression
The disk of small data occupies.Again because it relies on mature HBase data storage, either in Data safeguard and reading and writing data plan
There is the firm foundation stone of comparison on slightly, increases its availability.
HBase (Hadoop Database), i.e. hadoop database, be a high reliability, high-performance, towards column, can
Flexible distributed memory system can erect large-scale structureization storage collection using HBase technology on cheap PCServer
Group.
In OpenTSDB service application, resource management and utilization of resources limitation between each tenant how are effectively carried out,
Expense that how service side pays according to tenant carries out resource and reasonably divides, and become in shared data storage important asks
Topic.
Based on the above situation, the invention proposes a kind of OpenTSDB to share storage optimization method.
Summary of the invention
In order to compensate for the shortcomings of the prior art, the present invention provides a kind of OpenTSDB being simple and efficient to share storage optimization
Method.
The present invention is achieved through the following technical solutions:
A kind of shared storage optimization method of OpenTSDB, which comprises the following steps:
(1) when tenant creates OpenTSDB service, the tables of data of the exclusive user is created in OpenTSDB storage end；
(2) for the different business application example of different tenants or different tenants, using the quotas administered of HBase to each
The use resource of a tenant is managed；
(3) simultaneously, tenant is counted using the resource utilization in service process, cluster items resource is used
Quota carry out disclosure, accomplish it is shared storage application in resource using the transparency.
In the step (1), when tenant opens shared OpenTSDB service, according to tenant's name and service line in HBase
Create the data management table of hierarchical management.
In the step (1), when tenant creates affiliated time series database Service Instance, according to tenant's title and tenant's name
Lower items production line, creation includes the tables of data of two managerial classes in HBase management data table, uses the title of HBase
Space (Namespace) and tables of data (table) carry out the data storage resource management of tenant.
In the step (1), when same tenant holds the Service Instance of multiple business scenarios, then multiple tables of data are set
Under same life name space, the convenient resource classification management to different tenants.
When same tenant holds the Service Instance of multiple business scenarios, and tenant needs to carry out resource to one of them business
When inclination, tenant sets service priority weight, and service administrators carry out resource according to the priority weighting of tenant's prior confirmation
Quota management.
In the step (2), the expense that the service of OpenTSDB console is undertaken in cluster resource according to tenant, tenant makes
Resource constraint is carried out to the name space and tables of data of HBase with the hierarchical management of resource and service priority weight, makes tenant
The benefit that resource uses is obtained under limited investment.
In the step (3), service administrators are undertaken according to expense and formulate expense-the Resources list with cluster resource situation,
So that tenant is serviced undertaking for the clear general expenses of preceding energy in purchase, can accomplish resource expense transparent management.
The OpenTSDB shares storage optimization method, applies also for the quota management of Internet resources.
The beneficial effects of the present invention are: the OpenTSDB shares storage optimization method, carried on a shoulder pole in cluster resource according to tenant
Negative expense, tenant are divided using the hierarchical management and priority weighting of resource, obtain the resource use of tenant controllably
Management, also makes tenant obtain the benefit that resource uses, while resource expense detail list under limited investment, makes
The expense of tenant is using intuitively showing, to improve customer satisfaction.
Detailed description of the invention
Attached drawing 1 is that time series database of the present invention stores schematic diagram.
Attached drawing 2 is time series database example visioning procedure schematic diagram of the present invention.
Attached drawing 3 is accessing time sequence database flow diagram of the present invention.
Specific embodiment
In order to which technical problems, technical solutions and advantages to be solved are more clearly understood, tie below
Embodiment is closed, the present invention will be described in detail.It should be noted that specific embodiment described herein is only to explain
The present invention is not intended to limit the present invention.
The OpenTSDB shares storage optimization method, comprising the following steps:
(1) when tenant creates OpenTSDB service, the tables of data of the exclusive user is created in OpenTSDB storage end；
(2) for the different business application example of different tenants or different tenants, using the quotas administered of HBase to each
The use resource of a tenant is managed；
(3) simultaneously, tenant is counted using the resource utilization in service process, cluster items resource is used
Quota carry out disclosure, accomplish it is shared storage application in resource using the transparency.
In the step (1), when tenant opens shared OpenTSDB service, according to tenant's name and service line in HBase
Create the data management table of hierarchical management.
In the step (1), when tenant creates affiliated time series database Service Instance, according to tenant's title and tenant's name
Lower items production line, creation includes the tables of data of two managerial classes in HBase management data table, uses the title of HBase
Space (Namespace) and tables of data (table) carry out the data storage resource management of tenant.
Such as tenant A creates the used OpenTSDB example of IOT business, then claims in the name of HBase cluster creation A
Space, and creation stores the tables of data A:IOT of IOT business datum under this life name space.
The name space (Namespace) for logically by the relevant types of tissue of one group of function together, from
And avoid conflict caused by simple types name.For example, may be defined comprising the class row of " tree " in the class libraries of a biology,
And describe computer data structure application in also need to be defined " tree " structure.There is no namespaces concept at one
System in, due to the conflict of name, user cannot in the same application simultaneously use above-mentioned two typelib.
In the step (1), when same tenant holds the Service Instance of multiple business scenarios, then multiple tables of data are set
Under same life name space, the convenient resource classification management to different tenants.
Such as tenant U1 needs to create the Service Instance of business B1, then creates U1:B1-tsdb table in HBase and be used to deposit
Store up associated traffic data.If this business needs to carry out data storage management same U1 there are also B2, U1 is created in HBase:
The tables of data of B2-tsdb stores associated traffic data.
When same tenant holds the Service Instance of multiple business scenarios, and tenant needs to carry out resource to one of them business
When inclination, tenant sets service priority weight, and service administrators carry out resource according to the priority weighting of tenant's prior confirmation
Quota management.
In the step (2), the expense that the service of OpenTSDB console is undertaken in cluster resource according to tenant, tenant makes
Resource constraint is carried out to the name space and tables of data of HBase with the hierarchical management of resource and service priority weight, makes tenant
The benefit that resource uses is obtained under limited investment.
The expense of the U1 by taking storage resource as an example only allows it to be 100T to the utilization rate of disk, then U1:B1-tsdb adds U1:
The data of B1-tsdb storage, the occupancy disk total amount for other tables of data such as tsdb-uid for adding tsdb service to need is no more than
100T。
In the step (3), service administrators are undertaken according to expense and formulate expense-the Resources list with cluster resource situation,
So that tenant is serviced undertaking for the clear general expenses of preceding energy in purchase, can accomplish resource expense transparent management.
The OpenTSDB shares storage optimization method, applies also for the quota management of Internet resources.
Compared with prior art, which shares storage optimization method, can be under multi-tenant scene, according to tenant
The expense undertaken in cluster resource, tenant are divided using the hierarchical management and priority weighting of resource, not only make the money of tenant
Source use obtains controllable management, and tenant is also made to have obtained the benefit that resource uses, while resource under limited investment
Expense detail list makes the expense of tenant using intuitively showing, to improve customer satisfaction.
Claims (8)
1. a kind of OpenTSDB shares storage optimization method, it is characterised in that: the following steps are included:
(1) when tenant creates OpenTSDB service, the tables of data of the exclusive user is created in OpenTSDB storage end；
(2) for the different business application example of different tenants or different tenants, using the quotas administered of HBase to each rent
The use resource at family is managed；
(3) simultaneously, tenant is counted using the resource utilization in service process, is matched to what cluster items resource used
Volume carry out disclosure, accomplish it is shared storage application in resource using the transparency.
2. OpenTSDB according to claim 1 shares storage optimization method, it is characterised in that: in the step (1), when
When tenant opens shared OpenTSDB service, the data management table of hierarchical management is created in HBase according to tenant's name and service line.
3. OpenTSDB according to claim 2 shares storage optimization method, it is characterised in that: in the step (1),
Tenant create belonging to time series database Service Instance when, according to tenant's title and tenant every production line under one's name, in HBase data
Creation includes the tables of data of two managerial classes in table management, and the number of tenant is carried out using the name space and tables of data of HBase
According to storage resource management.
4. OpenTSDB according to claim 3 shares storage optimization method, it is characterised in that: in the step (1), when
When same tenant holds the Service Instance of multiple business scenarios, then multiple tables of data are placed under same life name space, it is convenient
To the resource classification management of different tenants.
5. OpenTSDB according to claim 4 shares storage optimization method, it is characterised in that: when same tenant hold it is more
The Service Instance of a business scenario, and tenant need to one of them business carry out resource inclination when, tenant set business it is preferential
Grade weight, service administrators carry out the quota management of resource according to the priority weighting of tenant's prior confirmation.
6. OpenTSDB shares storage optimization method according to claim 1 or 5, it is characterised in that: the step (2)
In, the expense that the service of OpenTSDB console is undertaken in cluster resource according to tenant, tenant using resource hierarchical management and
Service priority weight carries out resource constraint to the name space and tables of data of HBase, obtains tenant under limited investment
The benefit that resource uses.
7. OpenTSDB according to claim 1 shares storage optimization method, it is characterised in that: in the step (3), clothes
Business administrator undertakes according to expense and formulates expense-the Resources list with cluster resource situation, makes tenant can be bright before purchase services
Undertaking for true general expenses, can accomplish resource expense transparent management.
8. OpenTSDB according to claim 1 shares storage optimization method, it is characterised in that: apply also for Internet resources
Quota management.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910293370.1A CN110134511A (en) | 2019-04-12 | 2019-04-12 | A kind of shared storage optimization method of OpenTSDB |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910293370.1A CN110134511A (en) | 2019-04-12 | 2019-04-12 | A kind of shared storage optimization method of OpenTSDB |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110134511A true CN110134511A (en) | 2019-08-16 |
Family
ID=67569973
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910293370.1A Pending CN110134511A (en) | 2019-04-12 | 2019-04-12 | A kind of shared storage optimization method of OpenTSDB |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110134511A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110825581A (en) * | 2019-10-14 | 2020-02-21 | 广州力挚网络科技有限公司 | Data monitoring method and monitoring platform |
| CN112416593A (en) * | 2020-11-30 | 2021-02-26 | 北京百度网讯科技有限公司 | Resource management method and device, electronic equipment and computer readable medium |
| CN114978998A (en) * | 2021-02-26 | 2022-08-30 | 中移(苏州)软件技术有限公司 | Flow control method, device, terminal and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106559488A (en) * | 2016-11-24 | 2017-04-05 | 天津市普迅电力信息技术有限公司 | A kind of method of the electrical network geographical information space service for setting up tenant's driving |
| CN107360103A (en) * | 2016-05-09 | 2017-11-17 | 中国移动通信集团四川有限公司 | A kind of Operation ＆ Maintenance System and resource regulating method |
| CN107659450A (en) * | 2017-09-29 | 2018-02-02 | 深圳索信达数据技术股份有限公司 | Distribution method, distributor and the storage medium of big data cluster resource |
| CN107864211A (en) * | 2017-11-17 | 2018-03-30 | 中国联合网络通信集团有限公司 | Cluster resource dispatching method and system |
-
2019
- 2019-04-12 CN CN201910293370.1A patent/CN110134511A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107360103A (en) * | 2016-05-09 | 2017-11-17 | 中国移动通信集团四川有限公司 | A kind of Operation ＆ Maintenance System and resource regulating method |
| CN106559488A (en) * | 2016-11-24 | 2017-04-05 | 天津市普迅电力信息技术有限公司 | A kind of method of the electrical network geographical information space service for setting up tenant's driving |
| CN107659450A (en) * | 2017-09-29 | 2018-02-02 | 深圳索信达数据技术股份有限公司 | Distribution method, distributor and the storage medium of big data cluster resource |
| CN107864211A (en) * | 2017-11-17 | 2018-03-30 | 中国联合网络通信集团有限公司 | Cluster resource dispatching method and system |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110825581A (en) * | 2019-10-14 | 2020-02-21 | 广州力挚网络科技有限公司 | Data monitoring method and monitoring platform |
| CN112416593A (en) * | 2020-11-30 | 2021-02-26 | 北京百度网讯科技有限公司 | Resource management method and device, electronic equipment and computer readable medium |
| CN112416593B (en) * | 2020-11-30 | 2024-01-12 | 北京百度网讯科技有限公司 | Resource management method and device, electronic equipment and computer readable medium |
| CN114978998A (en) * | 2021-02-26 | 2022-08-30 | 中移(苏州)软件技术有限公司 | Flow control method, device, terminal and storage medium |
| CN114978998B (en) * | 2021-02-26 | 2023-12-12 | 中移(苏州)软件技术有限公司 | Flow control method, device, terminal and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110674228B (en) | Data warehouse model construction and data query method, device and equipment | |
| US10055426B2 (en) | System and method transforming source data into output data in big data environments | |
| CN106897322B (en) | A kind of access method and device of database and file system | |
| CN110674154B (en) | Spark-based method for inserting, updating and deleting data in Hive | |
| CN110647512B (en) | Data storage and analysis method, device, equipment and readable medium | |
| CN105183735A (en) | Data query method and query device | |
| CN109033113B (en) | Data warehouse and data mart management method and device | |
| CN105930446A (en) | Telecommunication customer tag generation method based on Hadoop distributed technology | |
| US10360394B2 (en) | System and method for creating, tracking, and maintaining big data use cases | |
| CN110134511A (en) | A kind of shared storage optimization method of OpenTSDB | |
| CN106649602B (en) | Business object data processing method, device and server | |
| CN102508919A (en) | Data processing method and system | |
| CN103455335A (en) | Multilevel classification Web implementation method | |
| Khan et al. | Efficient data access and performance improvement model for virtual data warehouse | |
| CN112632025A (en) | Power grid enterprise management decision support application system based on PAAS platform | |
| CN105095436A (en) | Automatic modeling method for data of data sources | |
| CN109150964A (en) | A kind of transportable data managing method and services migrating method | |
| CN116775605A (en) | Industrial data management and sharing platform based on artificial intelligence | |
| CN105824892A (en) | Method for synchronizing and processing data by data pool | |
| Suri et al. | A comparative study between the performance of relational & object oriented database in Data Warehousing | |
| CN116680090A (en) | Edge computing network management method and platform based on big data | |
| CN107291938A (en) | Order Query System and method | |
| US8229946B1 (en) | Business rules application parallel processing system | |
| Chereja et al. | Operationalizing analytics with NewSQL | |
| Ma et al. | Efficient attribute-based data access in astronomy analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190816 |