CN113486002B - Data processing system and method - Google Patents
Data processing system and method Download PDFInfo
- Publication number
- CN113486002B CN113486002B CN202110577469.1A CN202110577469A CN113486002B CN 113486002 B CN113486002 B CN 113486002B CN 202110577469 A CN202110577469 A CN 202110577469A CN 113486002 B CN113486002 B CN 113486002B
- Authority
- CN
- China
- Prior art keywords
- data
- monitoring
- query request
- monitoring data
- fquery
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 33
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000012544 monitoring process Methods 0.000 claims abstract description 269
- 238000004806 packaging method and process Methods 0.000 claims description 8
- 238000003672 processing method Methods 0.000 claims description 4
- 230000007774 longterm Effects 0.000 abstract description 9
- 230000008569 process Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 239000012634 fragment Substances 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2219—Large Object storage; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Selective Calling Equipment (AREA)
- Testing And Monitoring For Control Systems (AREA)
Abstract
The invention provides a data processing system and a method, wherein the system comprises the following steps: prometaus is used for collecting monitoring data and slicing each collected monitoring data. Fsider obtains each piece of monitoring data of the monitoring software in the historical time period, and sends the monitoring data to a memory for storage. Fgraph is configured to send a data query request to a query component, fquery, after receiving the data query request, forwards the data query request to Fsider, fsider, queries each monitoring data corresponding to the data query request based on data information in the data query request, and returns the monitoring data according to an original request path. Fgraph, after receiving each monitoring data, integrating and drawing corresponding data drawings, and displaying by Fquery. By applying the system provided by the invention, long-term cold storage of the monitoring data can be realized, and preheating scheduling is performed when the monitoring data needs to be scheduled.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing system and method.
Background
Prometaus is an open source Prometaus, which is suitable for monitoring hardware indexes such as server-oriented and high-dynamic service-oriented architecture. The prior art Prometheus periodically captures monitoring data on the monitored object via PrometheusServer and pushes and stores the monitoring data in the time series database TSDB.
Although Prometaus well supports monitoring of monitoring objects such as servers and component services, with the progress of technology, the variety and quantity of monitoring data are increased along with the multiple growth of servers and components, and a time sequence database in Prometaus is difficult to support long-term storage of a large amount of data.
Disclosure of Invention
In view of the above, the present invention provides a monitoring data processing system and method, by which long-term cold storage of monitoring data can be achieved, and warm-up scheduling can be performed when dispatch is required.
A monitoring data processing system, comprising:
monitoring clusters, edge collectors Fsider, memory, query component Fquery, and drawing controls Fgragh;
the monitoring cluster comprises at least one monitoring software Prometaheus;
Each Prometaus is used for collecting each monitoring data of a corresponding monitoring object according to a preset time period and slicing each monitoring data according to a preset monitoring type;
the Fsider is configured to obtain each monitoring data collected by each promethaus in a historical time period, and send each monitoring data collected in the historical time period to the memory; when receiving the data query request sent by Fquery, acquiring data information corresponding to the data query request; based on the data information, acquiring each monitoring data corresponding to the data query request from each Prometaus, and sending each monitoring data corresponding to the data query request to the Fquery;
the memory is used for storing each monitoring data sent by the Fsider;
The Fquery is configured to receive a data query request issued by the Fgraph, and forward the data query request to the Fsider; when receiving each monitoring data corresponding to the data query request fed back by Fsider, sending each monitoring data corresponding to the data query request to Fgraph; when receiving the data drawing sent by each monitoring data corresponding to the Fgraph based on the data query request, displaying the data drawing;
The Fgraph is configured to issue a data query request to the Fquery; and when receiving the monitored data fed back by Fquery based on the data query request, integrating all the monitored data corresponding to the data query request, generating a corresponding data drawing, and sending the data drawing to Fquery.
The above data processing system, optionally, the promethaus includes:
data nodes, monitoring services and a slicing module;
The data node is used for collecting each monitoring data corresponding to each monitoring type of the monitoring object according to the time period;
The slicing module is used for slicing each monitoring data according to each monitoring type;
The monitoring service is used for creating TSDB storage files corresponding to the monitored data of each fragmented; acquiring time sequence information corresponding to the fragmented monitoring data; and setting each time sequence information as index information corresponding to the TSDB storage file.
The above data processing system, optionally, the Fsider includes:
a first query interface and collector component;
The first query interface is configured to obtain data information corresponding to a data query request sent by the Fquery when the data query request is received; based on the data information, acquiring each monitoring data corresponding to the data query request from each Prometaus, packaging each monitoring data corresponding to the data query request, and sending each packaged monitoring data to the Fquery;
the collector component is used for acquiring each monitoring data acquired by each Prometaus in a historical time period and sending each monitoring data acquired in the historical time period to the memory.
In the above data processing system, optionally, after the first query interface obtains each monitoring data corresponding to the data query request, the collector component collects each monitoring data corresponding to the data query request obtained by the first query interface, and sends each monitoring data corresponding to the data query request to the memory.
The above data processing system, optionally, further comprises:
store gateway control Fstoregateway;
The Fstoregateway is configured to receive a data query request sent by the Fquery, and obtain data information corresponding to the data query request; and inquiring each monitoring data corresponding to the data inquiry request from the memory based on the data information, and sending each monitoring data corresponding to the data inquiry request to Fquery.
The above data processing system, optionally, the Fquery includes:
the second query interface, the third query interface and the display interface;
The second query interface is configured to receive the data query request issued by the Fgraph, and send each monitoring data corresponding to the data query request to the Fgraph;
the third query interface is configured to forward the data query request to the Fsider or the Fstoregateway, and receive each monitoring data corresponding to the data query request fed back by the Fsider or the Fstoregateway;
and the display interface is used for displaying the data drawing sent by the Fgraph.
The above data processing system, optionally, the Fstoregateway includes:
A fourth query interface and a fifth query interface;
The fourth query interface is configured to receive the data query request sent by the Fquery, and send each monitoring data corresponding to the data query request to the Fquery;
The fifth query interface is configured to obtain data information in the data query request, and obtain each monitoring data corresponding to the data query request from the memory based on the data information.
In the above data processing system, optionally, when the Fquery receives the data query request sent by the Fgraph, it determines, based on data information corresponding to the data query request, whether each piece of monitoring data to be queried belongs to the monitoring data collected in the historical time period; if yes, forwarding the data query request to the Fstoregateway; and otherwise, forwarding the data query request to the Fsider.
A data processing method, the method being applied to Fsider, the method comprising:
acquiring each monitoring data acquired by each Prometheus in a historical time period, wherein each Prometheus is used for acquiring each monitoring data in a corresponding monitoring range in real time, and slicing each monitoring data according to a preset monitoring type;
Sending each monitoring data acquired in the historical time period to a memory;
when receiving the data query request sent by Fquery, acquiring data information corresponding to the data query request;
based on the data information, acquiring each monitoring data corresponding to the data query request from each Prometaheus;
And sending each monitoring data corresponding to the data query request to Fquery, so that the Fquery forwards each monitoring data to Fgraph to trigger the Fgraph to draw a corresponding data drawing based on each monitoring data corresponding to the data query request.
The method, optionally, further includes, after obtaining each monitoring data corresponding to the data query request from each promethaus:
and packaging each monitoring data corresponding to the data query request, and sending each packaged monitoring data to Fquery.
Compared with the prior art, the invention has the following advantages:
The invention provides a data processing system, comprising: monitoring clusters, edge collectors Fsider, memory, query component Fquery, and drawing controls Fgragh; the monitoring cluster comprises at least one monitoring software Prometaheus; each Prometaus is used for collecting each monitoring data of a corresponding monitoring object according to a preset time period and slicing each monitoring data according to a preset monitoring type; the Fsider is configured to obtain each monitoring data collected by each promethaus in a historical period, and send each monitoring data collected in the historical period to the memory; when receiving the data query request sent by Fquery, acquiring data information corresponding to the data query request; based on the data information, acquiring each monitoring data corresponding to the data query request from each Prometaus, and sending each monitoring data corresponding to the data query request to the Fquery; the memory is used for storing each monitoring data sent by the Fsider; the Fquery is configured to receive a data query request issued by the Fgraph, and forward the data query request to the Fsider; when receiving each monitoring data corresponding to the data query request fed back by Fsider, sending each monitoring data corresponding to the data query request to Fgraph; when receiving the data drawing sent by each monitoring data corresponding to the Fgraph based on the data query request, displaying the data drawing; the Fgraph is configured to issue a data query request to the Fquery; and when receiving the monitored data which is fed back by the Fquery based on the data query request, integrating the queried monitored data, generating data drawings corresponding to each monitored data, and sending the data drawings to the Fquery. By applying the system provided by the invention, long-term cold storage of the monitoring data can be realized, and preheating scheduling is performed when the monitoring data needs to be scheduled.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a system architecture diagram of a data processing system according to an embodiment of the present invention;
FIG. 2 is a further system architecture diagram of a data processing system according to an embodiment of the present invention;
FIG. 3 is a further system architecture diagram of a data processing system according to an embodiment of the present invention;
Fig. 4 is a flowchart of a method for processing data according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the present disclosure, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions, and the terms "comprise," "include," or any other variation thereof, are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The invention is operational with numerous general purpose or special purpose computing device environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet devices, multiprocessor devices, distributed computing environments that include any of the above devices or devices, and the like.
An embodiment of the present invention provides a monitoring data processing system, where a system structure diagram of the system is shown in fig. 1, and specifically includes:
Monitoring cluster 100, edge collector Fsider, memory 300, query component Fquery400, and drawing control Fgragh500;
The monitoring cluster comprises at least one monitoring software Prometaus 101;
each promethaus 101 is configured to collect each monitoring data of a corresponding monitored object according to a preset time period, and segment each monitoring data according to a preset monitoring type;
The Fsider200 is configured to acquire each monitoring data acquired by each promethaus 101 in a historical period, and send each monitoring data acquired in the historical period to the memory 300; when a data query request sent by Fquery400,400 is received, acquiring data information corresponding to the data query request; based on the data information, acquiring each monitoring data corresponding to the data query request from each Prometaus 101, and sending each monitoring data corresponding to the data query request to Fquery400,400;
The memory 300 is configured to store each monitoring data sent by the Fsider;
The Fquery unit 400 is configured to receive a data query request issued by the Fgraph unit 500, and forward the data query request to the Fsider unit 200; when receiving each monitoring data corresponding to the data query request fed back by Fsider200,200, sending each monitoring data corresponding to the data query request to Fgraph500,500; when receiving the data drawing sent by each monitoring data corresponding to the Fgraph based on the data query request, displaying the data drawing;
The Fgraph unit is configured to issue a data query request to the Fquery unit; when receiving the monitored data fed back by Fquery based on the data query request, integrating the monitored data corresponding to the data query request, generating a corresponding data drawing, and sending the data drawing to Fquery.
In the data processing system provided by the embodiment of the invention, the Prometaus 101 is an open-source monitoring software, and each Prometaus 101 in the monitoring cluster 100 is used for monitoring a corresponding monitoring object and collecting monitoring data of the corresponding monitoring object. Wherein, the monitoring object can be a server or a service component, etc.; the monitoring data may actually be time series data. The data type of the monitoring data is segmented according to the service type of the server or the component, IDCs and different private cloud clusters. To ensure long-term storage of the monitoring data collected in Prometaus 101, edge collector Fsider is used to implement data upload and query functions. In order to reduce the uploading pressure of the monitoring data to the memory and facilitate the storage of each monitoring data, the Prometaus 101 collects and segments the monitoring data in a small range and regionally according to different idc and different services, but the real-time monitoring data is queried, the service call checking can access a plurality of Prometaus nodes to complete the query, thus delaying time and increasing operability and complexity. Therefore, through Fgragh, the fragmented monitoring data and the monitoring data of different monitoring objects can be subjected to integrated query.
Specifically, fgraph when data needs to be integrated and a corresponding plot is generated, a corresponding data query request is sent to Fsider. Fquery after receiving the data query request, the data query request is forwarded by Fquery, fsider200 analyzes the data query request, obtains data information therein, the data information contains time sequence information of monitoring data to be queried, queries each monitoring data corresponding to the data query request in each Prometaus 101 based on the data information, sends each monitoring data to Fgraph500, fgraph500 via Fquery400, and after receiving each monitoring data, integrates each monitoring data according to each monitoring data fragment information, generates corresponding data drawing, and Fquery displays the data drawing. This is equivalent to Fgragh to complete the task of retrieving each promethaus 101, and for business, only one interface is needed to complete the integration of all region data.
Optionally, the promethaus 101 stores the fragmented individual monitoring data after fragmenting the individual monitoring data.
Further, after each monitoring data in the historical time period is obtained, the Fsider may encapsulate and compress each monitoring data and then send each monitoring data to the memory 300 for storage.
Note that, the promethaus is connected to the Fsider through the HTTP protocol, fquery is connected to the Fsider through the HTTP protocol, fgraph is connected to the Fquery through the HTTP protocol, and Fsider is connected to the memory through the gRPC protocol.
By applying the system provided by the embodiment of the invention, long-term cold storage of the monitoring data can be realized, and preheating scheduling is performed when the monitoring data needs to be scheduled. Meanwhile, the data are segmented through the data types of the monitoring data, and after the data are segmented according to the types, the monitoring data can be integrated together through data drawing for display although the data are scattered.
Referring to FIG. 2, the data processing system may specifically further include:
Store gateway control Fstoregateway600;
The Fstoregateway is configured to receive the data query request sent by the Fquery, and obtain data information corresponding to the data query request; and based on the data information, inquiring each monitoring data corresponding to the data inquiry request from the memory, and sending each monitoring data corresponding to the data inquiry request to Fquery.
It will be appreciated that Fstoregateway is used to retrieve the cold backup history monitoring data for querying of the history monitoring data. The memory 300 stores monitoring data collected by each promethaus 101 during a historical period, so when Fquery needs to send a data query request corresponding to the historical monitoring data, the data query request may be sent to the Fstoregateway 600. Fstoregateway600 upon receiving a data query request, parse the data query request to obtain data information, and retrieve historical monitoring data from the memory 300 based on the data information.
Specifically, when Fstoregateway600,600 is included in the data processing system, fquery400,400 forwards the data query to Fsider200,200 or Fstoregateway when it is required to forward the data query. If the data query request requests historical monitoring data, the data query request is forwarded to Fstoregateway600 or Fsider, fsider200 may obtain the historical monitoring data through Prometheus101 or from memory 300. If the data query request requests monitoring data collected by Prometaus 101 during the current time period, the data query request is sent Fsider to Fsider.
Wherein Fstoregateway600,600 is connected to Fquery400,400 via HTTP protocol, and Fstoregateway600,600 is connected to memory via gRPC protocol.
By applying the system provided by the embodiment of the invention, fstoregateway can acquire historical monitoring data so as to facilitate the inquiry of the historical data.
Referring to fig. 3, the promethaus 101 includes:
data node 1011, monitoring service 1012, and fragment module 1013;
The data node 1011 is configured to collect, according to the time period, each monitoring data corresponding to each monitoring type of the monitored object;
the slicing module 1012 is configured to slice each piece of the monitoring data according to each monitoring type;
The monitoring service 1013 is configured to create TSDB storage files corresponding to the monitored data of each fragmented; acquiring time sequence information corresponding to the fragmented monitoring data; and setting each time sequence information as index information corresponding to the TSDB storage file.
It will be appreciated that the data node SSD/HHD1011 collects each monitoring data corresponding to the monitoring required by the promethaus 101 to which it belongs according to a time period, and fragments the collected monitoring data through the slicing module Shard 1012. Finally, the monitoring service Prometheus server generates TSDB storage file corresponding to each time period, and the fragmented monitoring data collected in the same time period are stored in the TSDB storage file. By acquiring the time sequence information corresponding to each monitoring data as index information corresponding to the TSDB storage file, each monitoring data in the TSDB storage file can be retrieved through each index information.
Referring to fig. 3, the Fsider200,200 includes:
A first query interface 201 and a collector component 202;
The first query interface 201 is configured to obtain, when receiving a data query request sent by the Fquery400,400, data information corresponding to the data query request; based on the data information, acquiring each monitoring data corresponding to the data query request from each Prometaus 101, packaging each monitoring data corresponding to the data query request, and sending each packaged monitoring data to Fquery400,400;
The collector component 202 is configured to obtain each monitoring data collected by each of the promethaus 101 during a historical period of time, and send each monitoring data collected during the historical period of time to the memory 300.
Specifically, in the system provided by the embodiment of the present invention, the first query interface 201 may be StoreAPI, which is mainly used for packaging data. Likewise, fquery400,400 sends a data query request through the first query interface 201. The first query interface 201 obtains each monitoring data corresponding to the data query request from the data node 1011 of the promethaus 101, and feeds back the monitoring data to Fquery400,400 after packaging. The collector component 202 periodically obtains the various monitoring data collected by the Prometaus 101 over the historical period of time and sends it to the memory 300 for cold storage for subsequent querying and recall.
It may be appreciated that, because the first query interface 201 is configured to receive the data query request, after the first query interface 201 obtains the corresponding respective monitoring data according to the data query request, the collector component 202 may send the queried respective monitoring data to the memory for storing, and the specific process may include:
After the first query interface 201 obtains each monitoring data corresponding to the data query request, the collector component 202 collects each monitoring data corresponding to the data query request obtained by the first query interface 201, and sends each monitoring data corresponding to the data query request to the memory 300.
By applying the system provided by the embodiment of the invention, the Fsider200,200 can realize data query and cold standby storage.
Referring to fig. 3, the Fquery400,400 includes:
A second query interface 401, a third query interface 402, and a display interface 403;
The second query interface 401 is configured to receive a data query request issued by the Fgraph500,500, and send each monitoring data corresponding to the data query request to the Fgraph500,500;
the third query interface 402 is configured to forward the data query request to the Fsider or the Fstoregateway600, and receive each monitoring data corresponding to the data query request fed back by the Fsider or the Fstoregateway 600;
the display interface 403 is configured to display the data plot sent by the Fgraph.
Specifically, the data processing systems are provided with Fsider, fstoregateway, 600 and Fquery with a process of forwarding the data query request, specifically including:
When the Fquery400 receives the data query request sent by the Fgraph, determining whether each piece of monitoring data to be queried belongs to the monitoring data collected in the historical time period or not based on the data information corresponding to the data query request; if yes, forwarding the data query request to the Fstoregateway600,600; otherwise, the data query request is forwarded to the Fsider200,200.
The historical time period is the sum of the time periods before the current time period. That is, fsider200 acquires each monitor data within the historical period of time as all monitor data that has been acquired between the current time periods. When Prometaus 101 enters the next time period to collect monitoring data, the original current time period belongs to a historical time period, and Fsider200 collects the monitoring data of the current time period.
It can be understood that Fquery, after receiving the data query request sent by Fgraph500,500, determines, according to the data information of the data query request, whether the monitored data to be queried belongs to the monitored data collected in the historical time period, so as to determine whether the object for receiving each monitored data corresponding to the data query request is Fsider or Fstoregateway. The Fgraph is received through the second query interface 401 to issue a data query request, the third query interface 402 determines a forwarding path of the data query request according to the data information in the data query request, and after receiving each monitoring data corresponding to the data query request, each monitoring data is returned to Fquery according to the request path. When Fquery, according to each monitoring data, a corresponding data drawing is drawn, the display interface 403 displays the data drawing.
Alternatively, the second query interface may be StoreAPI and the third query interface may be Prometheus QueryAPI, and the display interface is WEBUI.
Referring to fig. 3, the Fstoregateway600,600 includes:
a fourth query interface 601 and a fifth query interface 602;
The fourth query interface 601 is configured to receive a data query request sent by the Fquery a, and send each monitoring data corresponding to the data query request to the Fquery a;
The fifth query interface 602 is configured to obtain data information in the data query request, and obtain, from the memory 300, each monitoring data corresponding to the data query request based on the data information.
Alternatively, the fourth query interface may be StoreAPI and the fifth query interface Prometheus QueryAPI.
It is understood that Fstoregateway600,600 is mainly used to implement the query and forwarding of history monitoring data of a cold backup. The fourth query interface 601 of Fstoregateway600 is configured to receive the data query request that the Fquery needs to query the historical monitoring data, and forward the monitoring data. And the fifth query interface 602 performs data query and retrieval according to the data query request.
According to the data processing system provided by the embodiment of the invention, long-term storage of the monitoring data is realized through the memory, meanwhile, the monitoring data is called through Fsider and Fstoregateway, and meanwhile, the Fquery display is performed after the Fgraph drawing.
The specific implementation process and derivative manner of the above embodiments are all within the protection scope of the present invention.
Corresponding to the system shown in fig. 1, the embodiment of the present invention further provides a data processing method, which is used for implementing the method in fig. 1, where the data processing method provided in the embodiment of the present invention may be applied to and Fsider, and a method flowchart of the method is shown in fig. 4, and specifically includes:
S401: and acquiring each monitoring data acquired by each Prometheus in the historical time period, wherein each Prometheus is used for acquiring each monitoring data in a corresponding monitoring range in real time, and slicing each monitoring data according to a preset monitoring type.
Specifically, prometheus collects monitoring data specifically according to a set time period.
S402: and sending each monitoring data acquired in the historical time period to a memory.
Specifically, the memory is used for storing the fragmented monitoring data for a long time, and when the monitoring data in the fragmented monitoring data needs to be called, the history monitoring data of the original cold backup can be switched into hot call monitoring data.
S403: and when the data query request sent by Fquery is received, acquiring data information corresponding to the data query request.
Specifically, the data information includes time sequence information of each monitoring data to be queried, after each monitoring data in the current time period is collected and segmented, each monitoring data in the same time period is stored in the same TSDB storage file, and index information corresponding to the TSDB storage file is constructed according to the time sequence information of each monitoring data. And searching monitoring data matched with the time sequence data in the index information through the time sequence data in the data information.
S404: and acquiring each monitoring data corresponding to the data query request from each Prometaheus based on the data information.
Specifically, in addition to each monitoring data corresponding to the data query request may be obtained from promethaus, each monitoring data corresponding to the data query request may also be obtained from the memory.
S405: and sending each monitoring data corresponding to the data query request to Fquery, so that the Fquery forwards each monitoring data to Fgraph to trigger the Fgraph to draw a corresponding data drawing based on each monitoring data corresponding to the data query request.
In the data processing system provided by the embodiment of the invention, fsider acquires each piece of monitoring data acquired by Prometaus according to a historical time period, and stores each piece of monitoring data into a memory to realize long-term storage of the data. Meanwhile, the Fsider receives a data query request sent by Fquery in the process of data transfer, obtains each monitoring data corresponding to the data query request from Prometaus according to the data information corresponding to the data query request, returns each queried monitoring data to Fquery and Fquery, sends each monitoring data to Fgraph, draws a data drawing corresponding to each monitoring data by Fgraph, and finally monitors the data drawing by Fquery.
By applying the method provided by the embodiment of the invention, the long-term cold storage of the monitoring data can be realized, and the preheating scheduling is carried out when the monitoring data needs to be scheduled.
Further, after each monitoring data corresponding to the data query request is obtained from each promethaus, the method further includes:
and packaging each monitoring data corresponding to the data query request, and sending each packaged monitoring data to Fquery.
It can be appreciated that each monitoring data is encapsulated to reduce the bandwidth occupied by data transmission before forwarding each monitoring data corresponding to the data query request to Fquery.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for a system or system embodiment, since it is substantially similar to a method embodiment, the description is relatively simple, with reference to the description of the method embodiment being made in part. The systems and system embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
Those of skill would further appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both.
To clearly illustrate this interchangeability of hardware and software, various illustrative components and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (8)
1. A data processing system, comprising:
monitoring clusters, edge collectors Fsider, memory, query component Fquery, drawing control Fgraph, and storage gateway control Fstoregateway;
The monitoring cluster comprises at least one monitoring software Prometaheus; the promethaus comprises: data nodes, monitoring services and a slicing module; the data node is used for collecting each monitoring data corresponding to each monitoring type of the monitoring object according to the time period; the slicing module is used for slicing each monitoring data according to each monitoring type; the monitoring service is used for creating TSDB storage files corresponding to the monitored data of each fragmented; acquiring time sequence information corresponding to the fragmented monitoring data; setting each time sequence information as index information corresponding to the TSDB storage files;
Each Prometaus is used for collecting each monitoring data of a corresponding monitoring object according to a preset time period and slicing each monitoring data according to a preset monitoring type;
the Fsider is configured to obtain each monitoring data collected by each promethaus in a historical time period, and send each monitoring data collected in the historical time period to the memory; when receiving the data query request sent by Fquery, acquiring data information corresponding to the data query request; based on the data information, acquiring each monitoring data corresponding to the data query request from each Prometaus, and sending each monitoring data corresponding to the data query request to the Fquery;
the memory is used for storing each monitoring data sent by the Fsider;
The Fquery is configured to receive a data query request issued by the Fgraph, and forward the data query request to the Fsider or the Fstoregateway; when receiving each monitoring data corresponding to the data query request fed back by Fsider or Fstoregateway, sending each monitoring data corresponding to the data query request to Fgraph; when receiving the data drawing sent by each monitoring data corresponding to the Fgraph based on the data query request, displaying the data drawing;
The Fgraph is configured to issue a data query request to the Fquery; when receiving the monitored data fed back by Fquery based on the data query request, integrating each monitored data corresponding to the data query request, generating a corresponding data drawing, and sending the data drawing to Fquery;
The Fstoregateway is configured to receive a data query request sent by the Fquery, and obtain data information corresponding to the data query request; and inquiring each monitoring data corresponding to the data inquiry request from the memory based on the data information, and sending each monitoring data corresponding to the data inquiry request to Fquery.
2. The data processing system of claim 1, wherein the Fsider comprises:
a first query interface and collector component;
The first query interface is configured to obtain data information corresponding to a data query request sent by the Fquery when the data query request is received; based on the data information, acquiring each monitoring data corresponding to the data query request from each Prometaus, packaging each monitoring data corresponding to the data query request, and sending each packaged monitoring data to the Fquery;
the collector component is used for acquiring each monitoring data acquired by each Prometaus in a historical time period and sending each monitoring data acquired in the historical time period to the memory.
3. The data processing system of claim 2, wherein the collector component collects each monitoring data corresponding to the data query request obtained by the first query interface and sends each monitoring data corresponding to the data query request to the memory after each monitoring data corresponding to the data query request is obtained by the first query interface.
4. The data processing system of claim 1, wherein the Fquery comprises:
the second query interface, the third query interface and the display interface;
The second query interface is configured to receive the data query request issued by the Fgraph, and send each monitoring data corresponding to the data query request to the Fgraph;
the third query interface is configured to forward the data query request to the Fsider or the Fstoregateway, and receive each monitoring data corresponding to the data query request fed back by the Fsider or the Fstoregateway;
and the display interface is used for displaying the data drawing sent by the Fgraph.
5. The data processing system of claim 1, wherein the Fstoregateway comprises:
A fourth query interface and a fifth query interface;
The fourth query interface is configured to receive the data query request sent by the Fquery, and send each monitoring data corresponding to the data query request to the Fquery;
The fifth query interface is configured to obtain data information in the data query request, and obtain each monitoring data corresponding to the data query request from the memory based on the data information.
6. The data processing system of claim 4, wherein when the Fquery receives the data query request sent by the Fgraph, it determines, based on data information corresponding to the data query request, whether each monitored data to be queried belongs to the monitored data collected in the historical time period; if yes, forwarding the data query request to the Fstoregateway; and otherwise, forwarding the data query request to the Fsider.
7. A data processing method, characterized in that the method is applied to Fsider in the data processing system according to any one of claims 1 to 6, the method comprising:
acquiring each monitoring data acquired by each Prometheus in a historical time period, wherein each Prometheus is used for acquiring each monitoring data in a corresponding monitoring range in real time, and slicing each monitoring data according to a preset monitoring type;
Sending each monitoring data acquired in the historical time period to a memory;
when receiving the data query request sent by Fquery, acquiring data information corresponding to the data query request;
based on the data information, acquiring each monitoring data corresponding to the data query request from each Prometaheus;
And sending each monitoring data corresponding to the data query request to Fquery, so that the Fquery forwards each monitoring data to Fgraph to trigger the Fgraph to draw a corresponding data drawing based on each monitoring data corresponding to the data query request.
8. The method of claim 7, further comprising, after obtaining each monitoring data corresponding to the data query request from each of the promethaus:
and packaging each monitoring data corresponding to the data query request, and sending each packaged monitoring data to Fquery.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110577469.1A CN113486002B (en) | 2021-05-26 | 2021-05-26 | Data processing system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110577469.1A CN113486002B (en) | 2021-05-26 | 2021-05-26 | Data processing system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113486002A CN113486002A (en) | 2021-10-08 |
| CN113486002B true CN113486002B (en) | 2024-04-30 |
Family
ID=77933237
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110577469.1A Active CN113486002B (en) | 2021-05-26 | 2021-05-26 | Data processing system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113486002B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111651318A (en) * | 2020-04-30 | 2020-09-11 | 中国平安财产保险股份有限公司 | Prometheus-based interface monitoring method and device, computer equipment and storage medium |
| CN111694708A (en) * | 2020-05-28 | 2020-09-22 | 新浪网技术(中国)有限公司 | Data query method and device, electronic equipment and storage medium |
-
2021
- 2021-05-26 CN CN202110577469.1A patent/CN113486002B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111651318A (en) * | 2020-04-30 | 2020-09-11 | 中国平安财产保险股份有限公司 | Prometheus-based interface monitoring method and device, computer equipment and storage medium |
| CN111694708A (en) * | 2020-05-28 | 2020-09-22 | 新浪网技术(中国)有限公司 | Data query method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113486002A (en) | 2021-10-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20230144450A1 (en) | Multi-partitioning data for combination operations | |
| CN106709003A (en) | Hadoop-based mass log data processing method | |
| CN108471366A (en) | A kind of stereoscopic monitoring system of facing cloud native applications | |
| US20030187868A1 (en) | Data acquisition system | |
| CN101751032B (en) | Method and system for managing automatic control system and video monitoring system | |
| CN103944830A (en) | Computer system | |
| WO2011144022A2 (en) | Method, system and apparatus for hybrid federated search | |
| CN102208991A (en) | Blog processing method, device and system | |
| CN112506870B (en) | Data warehouse increment updating method and device and computer equipment | |
| CN112181931A (en) | Big data system link tracking method and electronic equipment | |
| CN110377854A (en) | User access activity information monitoring method and device, computer equipment | |
| CN110928934A (en) | Data processing method and device for business analysis | |
| CN111209467A (en) | Data real-time query system under multi-concurrency multi-channel environment | |
| WO2011160552A1 (en) | Method and device for data storage in a statistical forms system | |
| CN112054923B (en) | Service request detection method, equipment and medium | |
| CN112632129A (en) | Code stream data management method, device and storage medium | |
| US10984444B2 (en) | Systems and methods for generating intelligent account reconfiguration offers | |
| CN112231296A (en) | Distributed log processing method, device, system, equipment and medium | |
| CN106326280B (en) | Data processing method, device and system | |
| CN108241725A (en) | A kind of data hot statistics system and method | |
| CN113190522B (en) | Log processing method and device, computing equipment and storage medium | |
| CN113486002B (en) | Data processing system and method | |
| CN112417050A (en) | Data synchronization method and device, system, storage medium and electronic device | |
| CN105956141A (en) | Medicine data storage method based on Internet of things | |
| CN115794924A (en) | Internet of things data management method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |