CN111625551A - Database monitoring data high-availability system based on ElasticSearch storage and implementation method thereof - Google Patents
Database monitoring data high-availability system based on ElasticSearch storage and implementation method thereof Download PDFInfo
- Publication number
- CN111625551A CN111625551A CN202010411300.4A CN202010411300A CN111625551A CN 111625551 A CN111625551 A CN 111625551A CN 202010411300 A CN202010411300 A CN 202010411300A CN 111625551 A CN111625551 A CN 111625551A
- Authority
- CN
- China
- Prior art keywords
- data
- elasticsearch
- prometheus
- monitoring data
- adapter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 81
- 238000000034 method Methods 0.000 title claims abstract description 31
- 238000005070 sampling Methods 0.000 claims abstract description 11
- JLQUFIHWVLZVTJ-UHFFFAOYSA-N carbosulfan Chemical group CCCCN(CCCC)SN(C)C(=O)OC1=CC=CC2=C1OC(C)(C)C2 JLQUFIHWVLZVTJ-UHFFFAOYSA-N 0.000 claims description 11
- 230000006835 compression Effects 0.000 claims description 9
- 238000007906 compression Methods 0.000 claims description 9
- 230000002085 persistent effect Effects 0.000 abstract description 6
- 238000006243 chemical reaction Methods 0.000 abstract description 3
- 238000011161 development Methods 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 7
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000002688 persistence Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Abstract
The invention discloses a database monitoring data high-availability system based on ElasticSearch storage and an implementation method thereof, which comprises the steps of reading historical data from ElasticSearch by Prometheus through an Adapter middleware, and performing remote reading configuration in Prometheus; after receiving the monitoring data request, Prometheus sends a query command to Adapter middleware in a specified format; the Adapter middleware searches data in the Elasticissearch through api, and searches the data in a manner of down-sampling and using a scroll query in the Elasticissearch; persistent storage of monitoring data in the ElasticSearch is completed through development of supported read-write Adapter middleware conversion, Prometheus can quickly acquire historical data from the ElasticSearch, the stability and the performance of the Prometheus for reading the data from the ElasticSearch can also be effectively improved, and the ElasticSearch is applied to other monitoring management services such as log collection and the like, so that persistent implementation of the same set of architecture is realized.
Description
Technical Field
The invention relates to a reading and writing method of database monitoring data, in particular to a high-availability system of database monitoring data based on elastic search storage and an implementation method thereof.
Background
Currently, the Prometheus officials do not support the reading and writing function of elastic search as remote storage, and currently, the mainstream Prometheus-based high-availability schemes include three types: the first method is realized by directly deploying two or more sets of Prometheus servers. This is the simplest to implement. And secondly, a Prometheus remote storage scheme is adopted, and monitoring data can be stored on a storage service of a third party. Currently Prometheus authorities support partial remote storage read-write schemes such as infiluxdb, PostgreSQL, etc. But most remote stores, including the ElasticSearch, only support writes and not reads. And thirdly, adding the characteristics of Prometheus cluster federation on the basis of the second method, and partitioning according to task level. But the remote storage scheme is consistent with the second.
If the first solution is adopted, although the implementation is simple, the disadvantages are also evident. The Prometheus under the scheme uses local storage, and data of a plurality of service ends are independent. Therefore, the same monitoring index needs to be collected for multiple times when multiple data are needed, the load of the system is increased, and the problems of data consistency and data persistence of multiple Prometous servers cannot be solved. It is also troublesome to perform server migration if necessary, and local monitoring data also needs to be moved. If the second or third scheme is adopted, although the persistence of data on the remote storage can be realized, the range of selectable schemes is less because the currently supported remote reading and writing scheme is limited. In order to realize high availability, new remote storage service is necessarily required to be introduced, so that the technical stack complexity of the whole database monitoring and management system is increased, and the later maintenance is not facilitated. Therefore, further research and improvement on the method and system for Prometheus with high data availability are needed.
Disclosure of Invention
One of the objectives of the present invention is to provide a database monitoring data high availability system based on elastic search storage and an implementation method thereof, so as to solve the data consistency problem and the data persistence problem of multiple Prometheus servers in similar Prometheus data high availability schemes in the prior art, increase the technical complexity of the whole database monitoring management system, and also be not beneficial to the later maintenance.
In order to solve the technical problems, the invention adopts the following technical scheme:
the invention provides a database monitoring data high-availability system based on ElasticSearch storage, which comprises Prometeus, ElasticSearch and Adapter middleware, wherein the Adapter middleware is respectively connected to the Prometeus and the ElasticSearch, and the method comprises the following steps:
the Prometheus is used for configuration for remote reading, and after receiving a monitoring data request, the Prometheus sends a query command to the Adapter middleware in a specified format.
The Adapter middleware is used for decoding the query command after receiving the query command from Prometheus, and the decoded data format is a ReadRequest structure defined in "githu. com/Prometheus/promppb".
The aforementioned Adapter middleware is also used for finding data in the Elasticsearch through api, and simultaneously finding data by means of downsampling and using a scroll query in the Elasticsearch. com/go/Protobuf/prototo, the data queried from the elastic search is re-encoded into the type of Protobuf and transferred to Prometheus using the Marshal function defined in "githu.
Preferably, the further technical scheme is as follows: the Prometheus is also used to send a query command to the Adapter middleware in a specified format after collecting new monitoring data. The Adapter middleware is used for decoding the monitoring data after receiving the monitoring data sent by Prometheus, and converting the monitoring data into data of a Sample structure type defined in githu, com/Prometheus/common/model after decoding. The Adapter middleware is also used for converting data of the Sample structure type into json format, wherein the json format contains the name, the tag information, the timestamp and the value of the monitoring data, and the Adapter middleware is written into the ElasticSearch through api.
The invention also provides a high availability implementation method of database monitoring data stored based on the ElasticSearch, which is characterized in that the method comprises the following steps of reading historical data from the ElasticSearch by Prometous through Adapter middleware:
configuration for remote reading is performed in Prometheus. Prometheus, upon receiving a monitoring data request, sends a query command to the Adapter middleware in a specified format.
After receiving the query command from Prometheus, the Adapter middleware decodes the query command, and the decoded data format is a ReadRequest structure defined in "githu.
The Adapter middleware searches data in the Elasticissearch through api, and searches data in a manner of down-sampling and using a scroll query in the Elasticissearch.
com/go/Protobuf/prototo, the data queried from the elastic search is re-encoded into the type of Protobuf and transferred to Prometheus using the Marshal function defined in "githu.
Preferably, the further technical scheme is as follows: the method for realizing high availability of database monitoring data stored based on the ElasticSearch according to claim 1, wherein the method further comprises the following steps of transcoding and writing Prometheus data into the ElasticSearch through Adapter middleware:
configuration for remote reading is performed in Prometheus. And after collecting new monitoring data, the Prometheus sends a query command to the Adapter middleware in a specified format.
After receiving the monitoring data sent by Prometheus, the Adapter middleware decodes the monitoring data, and converts the monitoring data into data of a Sample structure type defined in githu, com/Prometheus/common/model after decoding.
The Adapter middleware converts data of the Sample structure type into json format, wherein the json format contains the name, the tag information, the timestamp and the value of the monitoring data, and the Adapter middleware writes the monitoring data into the ElasticSearch through api.
Preferably, the further technical scheme is as follows: the Prometheus receives a monitoring data request by PromQL for monitoring data query. The Adapter middleware supports four operators, and the four operators are transferred to the boot query command in the elastic search, and then data is searched in the elastic search through api.
The further technical scheme is as follows: the down-sampling is that the Adapter middleware acquires the step size of the selected data from the query command with the specified format from Prometheus, and directly queries all data in a time period that Prometheus wants to acquire from the elastic search for the request with the data step size less than 60S. For the request with the data step length larger than 60S, using the pool query in the ElasticSearch to divide the query time by taking the step length as a reference, querying each data of the first 30S in the divided time range by taking the step length as the reference, and then merging.
The further technical scheme is as follows: the Adapter middleware converts the decoded data into data of a Sample structure type defined in "gitubb.com/prometheus/common/model" through a prototokamplesrotobuf function. The Adapter middleware converts the data of the Sample structure type into the json format through a Marshal function defined in a Golang standard library 'encoding/json'.
The further technical scheme is as follows: the Adapter middleware writes json format data into the elastic search by api using bulk operation in the elastic search.
The further technical scheme is as follows: after the Adapter middleware writes json format data into an ElasticSearch through api, the default compression mode of the Index storing Prometheus data is modified to best _ compression, then the Index of the field which does not need to be indexed is set to false, and all the tag fields of String type in the ElasticSearch are set to tag fields of keyword type.
Compared with the prior art, the invention has the following beneficial effects: the persistent storage of the monitoring data in the ElasticSearch is completed by developing the read-write Adapter middleware conversion supported, and during query, Prometheus can quickly acquire historical data from the ElasticSearch, and can also effectively improve the stability and performance of the Prometheus for reading the data from the ElasticSearch, and the ElasticSearch is applied to other monitoring management services such as log collection, so that the persistent realization of the same set of framework is realized.
Detailed Description
The invention will be further illustrated with reference to specific examples.
The following embodiments of the present invention refer to an elastic search which is a Lucene (full text search engine) encapsulated elastic search server, and can be regarded as a document type database, Prometheus (promamelis) is a set of open source monitoring system developed by using Go language, and Adapter middleware is a virtual module for executing a specified flow of a computer program.
One embodiment of the present invention is a method for implementing database monitoring data high availability based on elastic search storage, the method includes a step of implementing, by Adapter middleware, reading history data from elastic search by Prometheus, and specifically, the method is performed according to the following steps:
step 101, performing remote read configuration in Prometheus.
remote_read：
- url: "http://prometheus-remote-storage-adapter-svc:9201/read"
After the Prometheus receives the monitoring data request, when monitoring data query is performed by using PromQL or other services request monitoring data of the Prometheus, the Prometheus still sends a query command to the Adapter middleware in a specified format (Protobuf).
Step 102, after the Adapter middleware receives the query command of the Protobuf type from Prometous, decoding the query command data by using the Unmarshal function defined in the format of gitub.com/go/Protobuf/proto, wherein the decoded data format is the ReadRequest structure defined in the format of gitub.com/Prometheus/prompub.
Step 103, the Adapter middleware searches data in the elastic search through api, and searches data in a manner of downsampling and using a scroll query in the elastic search. Since Prometheus requires the Adapter to support the resolution of four operators "=", "| =", "= -", "| -, -", in this step, the four operators are transferred to boot query command in the Elasticsearch according to the query.
The data is searched in the Elasticissearch through the api, because the Elasticissearch queries and acquires the data through the api, the default setting can only return 10000 data points. Therefore, although the open-source elastic search-adapter supports prometheus to remotely read data from the elastic search, due to the limitation of returning data points, when data with a long time span needs to be queried, data which can only be queried for a small part of time can occur. Although the setting can be increased, more memory resources are consumed after the setting is increased, when a large batch of requests exist, the efficiency is low, and meanwhile, high requirements are put forward on computer hardware, so the problem is solved by down-sampling, using the scroll query in the Elasticsearch and matching with the characteristics of the go language protocol.
The down-sampling is a step size for acquiring the selected data from the data decoded by protobuf of the remote read request, wherein the step size is related to a time range of data requested by Prometheus, and the longer the time span required to be queried, the longer the step size for acquiring the data is, and vice versa. Therefore, for a request with the step size less than 60s, all data in the time period that Prometheus wants to acquire is directly queried from the ElasticSearch. For the request with the step length larger than 60 seconds, using the pool query in the ElasticSearch to divide the query time by taking the step length as a reference, querying the data of the first 30s in each time range divided by taking the step length as the reference, and then combining the data. Therefore, the down-sampling is realized, the number of data points acquired from the ElasticSearch can be generally ensured, the data points can not be increased rapidly along with the expansion of the query time range, and the data points are basically maintained in a reasonable range, so that the pressure of resources such as a memory and the like can be reduced, and the query efficiency is improved. Secondly, for some metrics, because there are too many tags, even if the query data is downsampled, the limit of 10000 points set by the ElasticSearch is exceeded, so the present invention solves the problem by using a scroll query in the ElasticSearch, which takes snapshot data at a certain time point, and any change in the index after the query is initialized is ignored by the scroll query. It achieves this feature by saving the old data files, and as a result, just like retaining the indexed view at initialization, it can search a batch of data first, then search a batch of data next time, and so on, until all data is searched out, the scroll query consumes less system resources than the from-size deep query. Meanwhile, the characteristic that the coroutine of the go language consumes less resources is utilized, large-batch query is concurrently processed, and the query efficiency is improved.
com/go/Protobuf/prototo, re-encode the data queried from the elastic search into a Protobuf type, and transmit the data to Prometheus, using a Marshal function defined in "githu.
According to another embodiment of the present invention, similar to the method described above, the present invention further provides steps of transcoding and writing Prometheus data into an ElasticSearch through Adapter middleware based on the ElasticSearch, specifically as follows:
step 201, configuration of remote reading is carried out in Prometheus. In the same manner as described above, remote _ write:
- url: "http://prometheus-remote-storage-adapter-svc:9201/write"
after the configuration of remote storage is performed, and new monitoring data is collected by promemeus, the data is sent to the Adapter middleware in a specified format (Protobuf).
Step 202, after the Adapter middleware receives the monitoring data of the Protobuf type sent by Prometheus, firstly, using the Unmarshal function defined in "gitub.com/go/Protobuf/proto" to decode the data, and the format of the decoded data is the WriteRequest structure defined in "gitub.com/Prometheus/prompsb". The decoded data is then converted into data of the Sample structure type defined in "gitubu. com/prometheus/common/model" using the prototokamplesrotobuf function.
Step 203, the Adapter middleware converts the data of the Sample structure type into a json format by using a Marshal function defined in a gold standard library 'encoding/json', wherein the data comprises the name, the tag information, the timestamp and the value of the monitoring data, and writes the data into an elastic search through the api. The writing mode uses bulk operation of bulk in the elastic search, and aims to reduce resource pressure such as bandwidth caused by frequent writing and improve the speed of writing data.
When data is written to an elastic search through api, the data occupies more disk space. Taking the test environment as an example, when there are three data nodes, the monitoring data stored in the ElasticSearch occupies about 1.3G of disk space each day. The invention therefore optimizes the Prometheus data stored in the ElasticSearch. First, the default compression mode of Index (similar to the concept of table in the conventional database) storing the Prometheus data is changed to best _ compression, so as to improve the compression efficiency when storing data. Next, the index of the field that does not need to be indexed is set to false. Finally, because there are many tags in Prometheus data, these tags take up more space. And we do not need to perform phrase queries on these tags at query time. Therefore, in the elastic search, tag fields of the String types are all set to be of the keyword type, so that the generation of unnecessary information is reduced. After the optimization, the same environment is used for testing, the size of the monitoring data stored in the ElasticSearch every day is reduced to 800M, and the occupation of the disk space is reduced by about 40%.
In the embodiment, persistent storage of monitoring data in the Elasticsearch is completed by development of supported read-write Adapter middleware conversion, and during query, Prometheus can quickly acquire historical data from the Elasticsearch, and also can effectively improve the stability and performance of reading data from the Elasticsearch by Prometheus, and the Elasticsearch is applied to other monitoring management services such as log collection, so that persistent implementation of the same set of architecture is realized.
Similar to the method described above, the present invention is also a system based on the functions of the above method and used for implementing the method, and the present invention refers to a database monitoring data high availability system based on the storage of the elastic search, the system includes Prometheus, elastic search and Adapter middleware, the Adapter middleware accesses the Prometheus and the elastic search respectively, wherein:
the Prometheus is used for remote read configuration, and after receiving a monitoring data request, the Prometheus sends a query command to the Adapter middleware in a specified format.
The Adapter middleware is used for decoding the query command after receiving the query command from Prometheus, and the decoded data format is a ReadRequest structure defined in "gitubb.
The Adapter middleware is also used for searching data in the elastic search through api, and searching data in a manner of downsampling and using a scroll query in the elastic search. com/go/Protobuf/prototo, the data queried from the elastic search is re-encoded into the type of Protobuf and transferred to Prometheus using the Marshal function defined in "githu.
Prometheus is also used to send query commands to Adapter middleware in a prescribed format after new monitoring data is collected. The Adapter middleware is used for decoding the monitoring data after receiving the monitoring data sent by Prometheus, and converting the monitoring data into data of a Sample structure type defined in githu, com/Prometheus/common/model after decoding. The Adapter middleware is also used for converting data of the Sample structure type into json format, wherein the json format contains the name, the tag information, the timestamp and the value of the monitoring data, and the Adapter middleware is written into the ElasticSearch through api. The write may use bulk operations in the Elasticsearch to write json formatted data to the Elasticsearch.
Similar to the method, the down-sampling is that the Adapter middleware acquires the step size of the selected data from the query command with the specified format from Prometheus, and for the request with the data step size smaller than 60S, the adaptor middleware directly queries all data in the time period that Prometheus wants to acquire from the elastic search. For the request with the data step length larger than 60S, using the pool query in the ElasticSearch to divide the query time by taking the step length as a reference, querying each data of the first 30S in the divided time range by taking the step length as the reference, and then merging.
The Adapter middleware converts the decoded data into data of a Sample structure type defined in "gitubb.com/prometheus/common/model" through a prototokamplesrotobuf function. The Adapter middleware converts the data of the Sample structure type into a json format through a Marshal function defined in a Golang standard library 'encoding/json'.
Similar to the method, after the Adapter middleware writes json format data into the ElasticSearch through api, the default compression mode of Index for storing Prometheus data is modified to best _ compression, then Index of fields which do not need to be indexed is set to false, and all label fields of String type in the ElasticSearch are set to label fields of keyword type.
And the Prometheus queries monitoring data by PromQL and receives a monitoring data request. The Adapter middleware supports four operators, and the four operators are transferred to Bool query commands in the Elasticissearch, and then data is searched in the Elasticissearch through api.
In addition to the foregoing, it should be noted that reference throughout this specification to "one embodiment," "another embodiment," "an embodiment," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment described generally throughout this application. The appearances of the same phrase in various places in the specification are not necessarily all referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with any embodiment, it is submitted that it is within the scope of the invention to effect such feature, structure, or characteristic in connection with other embodiments.
Although the invention has been described herein with reference to a number of illustrative embodiments thereof, it should be understood that numerous other modifications and embodiments can be devised by those skilled in the art that will fall within the spirit and scope of the principles of this disclosure. More specifically, various variations and modifications are possible in the component parts and/or arrangements of the subject combination arrangement within the scope of the disclosure and claims of this specification. In addition to variations and modifications in the component parts and/or arrangements, other uses will also be apparent to those skilled in the art.
Claims (10)
1. A high-availability implementation method for monitoring data of a database stored on the basis of an ElasticSearch is characterized by comprising the following steps of reading historical data from the ElasticSearch by Prometous through Adapter middleware:
configuration for remote reading in Prometheus; after receiving the monitoring data request, Prometheus sends a query command to Adapter middleware in a specified format;
after receiving the query command from Prometheus, the Adapter middleware decodes the query command, and the decoded data format is a ReadRequest structure defined in "githu. com/Prometheus/promppb";
the Adapter middleware searches data in the Elasticissearch through api, and searches the data in a manner of down-sampling and using a scroll query in the Elasticissearch;
com/go/Protobuf/prototo, the data queried from the elastic search is re-encoded into the type of Protobuf and transferred to Prometheus using the Marshal function defined in "githu.
2. The method for realizing high availability of database monitoring data stored on the basis of the ElasticSearch according to claim 1, wherein the method further comprises the following steps of transcoding and writing Prometous data into the ElasticSearch through Adapter middleware:
configuration for remote reading in Prometheus; after acquiring new monitoring data, Prometheus sends a query command to Adapter middleware in a specified format;
after receiving the monitoring data sent by Prometheus, the Adapter middleware decodes the monitoring data, and converts the monitoring data into data of a Sample structure type defined in githu, com/Prometheus/common/model after decoding;
the Adapter middleware converts data of the Sample structure type into json format, wherein the json format contains the name, the tag information, the timestamp and the value of the monitoring data, and the Adapter middleware writes the monitoring data into the ElasticSearch through api.
3. The database monitoring data high availability implementation method based on ElasticSearch storage according to claim 1 or 2, characterized in that: the Prometheus carries out monitoring data query by PromQL and receives a monitoring data request; the Adapter middleware supports four operators, and the four operators are transferred to Bool query commands in the Elasticissearch, and then data is searched in the Elasticissearch through api.
4. The database monitoring data high availability implementation method based on ElasticSearch storage according to claim 1 or 2, characterized in that: the down sampling is that the Adapter middleware acquires the step length of the selected data from a query command in a specified format from Prometheus, and directly queries all data in a time period which is required to be acquired by Prometheus from ElasticSearch for a request with the data step length smaller than 60S;
for the request with the data step length larger than 60S, using the pool query in the ElasticSearch to divide the query time by taking the step length as a reference, querying each data of the first 30S in the divided time range by taking the step length as the reference, and then merging.
5. The database monitoring data high availability implementation method based on ElasticSearch storage of claim 2, characterized in that: the Adapter middleware converts the decoded data into data of a Sample structure type defined in githu, com/prometheus/common/model through a protoToSamplesProtobuf function; the Adapter middleware converts the data of the Sample structure type into a json format through a Marshal function defined in a Golang standard library 'encoding/json'.
6. The database monitoring data high availability implementation method based on ElasticSearch storage of claim 2, characterized in that: the Adapter middleware writes json format data into the ElasticSearch by api using bulk operation in the ElasticSearch.
7. The database monitoring data high availability implementation method based on ElasticSearch storage according to claim 2 or 6, characterized in that: after the Adapter middleware writes json format data into an ElasticSearch through api, the default compression mode of the Index for storing Prometous data is modified to best _ compression, then the Index of the field which does not need to be indexed is set to false, and all the tag fields of String type in the ElasticSearch are set to tag fields of keyword type.
8. An ElasticSearch storage-based database monitoring data high availability system, which is characterized by comprising Prometous, ElasticSearch and Adapter middleware, wherein the Adapter middleware respectively accesses the Prometous and the ElasticSearch, and the ElasticSearch storage-based database monitoring data high availability system comprises:
the Prometheus is used for carrying out remote reading configuration, and after receiving a monitoring data request, the Prometheus sends a query command to Adapter middleware in a specified format;
the Adapter middleware is used for decoding the query command after receiving the query command from Prometheus, wherein the decoded data format is a ReadRequest structure defined in githu, com/Prometheus/prompb;
the Adapter middleware is also used for searching data in the Elasticissearch through api, and searching data in a manner of down-sampling and using a scroll query in the Elasticissearch; com/go/Protobuf/prototo, the data queried from the elastic search is re-encoded into the type of Protobuf and transferred to Prometheus using the Marshal function defined in "githu.
9. The database monitoring data high availability system based on ElasticSearch storage according to claim 8, characterized in that:
the Prometous is also used for sending a query command to the Adapter middleware in a specified format after new monitoring data are collected;
the Adapter middleware is used for decoding the monitoring data after receiving the monitoring data sent by Prometous, and converting the monitoring data into data of a Sample structure type defined in githu, com/Prometous/common/model after decoding;
the Adapter middleware is also used for converting data of the Sample structure type into json format, wherein the json format contains the name, the tag information, the timestamp and the value of the monitoring data, and the Adapter middleware is written into the ElasticSearch through api.
10. The database monitoring data high availability system based on ElasticSearch storage according to claim 9, characterized in that: the down sampling is that the Adapter middleware acquires the step length of the selected data from a query command in a specified format from Prometheus, and directly queries all data in a time period which is required to be acquired by Prometheus from ElasticSearch for a request with the data step length smaller than 60S;
for the request with the data step length larger than 60S, utilizing the pool query in the ElasticSearch to divide the query time by taking the step length as a reference, querying each data of the first 30S in the divided time range by taking the step length as the reference, and then merging;
the Adapter middleware converts the decoded data into data of a Sample structure type defined in githu, com/prometheus/common/model through a protoToSamplesProtobuf function; the Adapter middleware converts the data of the Sample structure type into a json format through a Marshal function defined in a Golang standard library 'encoding/json'.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010411300.4A CN111625551A (en) | 2020-05-15 | 2020-05-15 | Database monitoring data high-availability system based on ElasticSearch storage and implementation method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010411300.4A CN111625551A (en) | 2020-05-15 | 2020-05-15 | Database monitoring data high-availability system based on ElasticSearch storage and implementation method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111625551A true CN111625551A (en) | 2020-09-04 |
Family
ID=72260637
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010411300.4A Pending CN111625551A (en) | 2020-05-15 | 2020-05-15 | Database monitoring data high-availability system based on ElasticSearch storage and implementation method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111625551A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112231531A (en) * | 2020-09-15 | 2021-01-15 | 山东浪潮通软信息科技有限公司 | Data display method, equipment and medium based on openstb |
| CN112650779A (en) * | 2021-01-12 | 2021-04-13 | 浪潮云信息技术股份公司 | Cloud auditing method based on ElasticSearch supporting deep page jump query |
| CN113055490A (en) * | 2021-03-24 | 2021-06-29 | 杭州群核信息技术有限公司 | Data storage method and device |
| CN113377816A (en) * | 2021-06-11 | 2021-09-10 | 重庆农村商业银行股份有限公司 | Distributed data access system |
| CN114090505A (en) * | 2021-11-23 | 2022-02-25 | 成都深思科技有限公司 | Intelligent resource scheduling and efficient concurrent data classification method |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109245931A (en) * | 2018-09-19 | 2019-01-18 | 四川长虹电器股份有限公司 | The log management of container cloud platform based on kubernetes and the implementation method of monitoring alarm |
| CN110377489A (en) * | 2019-07-17 | 2019-10-25 | 国家电网有限公司 | The method and apparatus of log information processing |
| US20200137029A1 (en) * | 2018-10-29 | 2020-04-30 | Hewlett Packard Enterprise Development Lp | Secure channel for cloud deployment unit |
-
2020
- 2020-05-15 CN CN202010411300.4A patent/CN111625551A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109245931A (en) * | 2018-09-19 | 2019-01-18 | 四川长虹电器股份有限公司 | The log management of container cloud platform based on kubernetes and the implementation method of monitoring alarm |
| US20200137029A1 (en) * | 2018-10-29 | 2020-04-30 | Hewlett Packard Enterprise Development Lp | Secure channel for cloud deployment unit |
| CN110377489A (en) * | 2019-07-17 | 2019-10-25 | 国家电网有限公司 | The method and apparatus of log information processing |
Non-Patent Citations (4)
| Title |
|---|
| YURIADAMS: "prometheus-elasticsearch-adapter", 《HTTPS://GITHUB.COM/YURIADAMS/PROMETHEUS-ELASTICSEARCH-ADAPTER》 * |
| 张晓艺: "八大优秀IT基础设施监控工具", 《计算机与网络》 * |
| 方瑜庆等: "安徽高速收费稽查分析中Elasticsearch应用研究", 《中国交通信息化》 * |
| 王伟军: "基于Kubernetes的容器云平台建设", 《电脑知识与技术》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112231531A (en) * | 2020-09-15 | 2021-01-15 | 山东浪潮通软信息科技有限公司 | Data display method, equipment and medium based on openstb |
| CN112650779A (en) * | 2021-01-12 | 2021-04-13 | 浪潮云信息技术股份公司 | Cloud auditing method based on ElasticSearch supporting deep page jump query |
| CN113055490A (en) * | 2021-03-24 | 2021-06-29 | 杭州群核信息技术有限公司 | Data storage method and device |
| CN113055490B (en) * | 2021-03-24 | 2022-10-11 | 杭州群核信息技术有限公司 | Data storage method and device |
| CN113377816A (en) * | 2021-06-11 | 2021-09-10 | 重庆农村商业银行股份有限公司 | Distributed data access system |
| CN113377816B (en) * | 2021-06-11 | 2022-10-25 | 重庆农村商业银行股份有限公司 | Distributed data access system |
| CN114090505A (en) * | 2021-11-23 | 2022-02-25 | 成都深思科技有限公司 | Intelligent resource scheduling and efficient concurrent data classification method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111625551A (en) | Database monitoring data high-availability system based on ElasticSearch storage and implementation method thereof | |
| US11816083B2 (en) | Method and system for indexing of time-series data | |
| US9946735B2 (en) | Index structure navigation using page versions for read-only nodes | |
| US8812627B2 (en) | System and method for installation and management of cloud-independent multi-tenant applications | |
| EP2973018B1 (en) | A method to accelerate queries using dynamically generated alternate data formats in flash cache | |
| US20170206236A1 (en) | Replicated database startup for common database storage | |
| US9590915B2 (en) | Transmission of Map/Reduce data in a data center | |
| US8250590B2 (en) | Apparatus, system, and method for seamless multiple format metadata abstraction | |
| CN105144121A (en) | Caching content addressable data chunks for storage virtualization | |
| CN107818111B (en) | Method for caching file data, server and terminal | |
| CN114968953A (en) | Log storage and retrieval method, system, terminal equipment and medium | |
| US7743333B2 (en) | Suspending a result set and continuing from a suspended result set for scrollable cursors | |
| CN104408084B (en) | A kind of big data screening technique and device | |
| US20120054429A1 (en) | Method and apparatus for optimizing data allocation | |
| US9195682B2 (en) | Integrated analytics on multiple systems | |
| CN112988884B (en) | Big data platform data storage method and device | |
| CN113051221A (en) | Data storage method, device, medium, equipment and distributed file system | |
| US20220350786A1 (en) | Data transfer and management system for in-memory database | |
| CN113051244B (en) | Data access method and device, and data acquisition method and device | |
| US20130282756A1 (en) | Method of processing files in storage system and data server using the method | |
| CN116701387A (en) | Data segmentation writing method, data reading method and device | |
| US11341163B1 (en) | Multi-level replication filtering for a distributed database | |
| US20060059176A1 (en) | Suspending a result set and continuing from a suspended result set | |
| US10558618B1 (en) | Metadata compression | |
| CN117093158B (en) | Storage node, system and data processing method and device of distributed storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200904 |