CN117827586A

CN117827586A - Index monitoring method and system for row protocol structure and storage library

Info

Publication number: CN117827586A
Application number: CN202311824407.1A
Authority: CN
Inventors: 胡麦芳; 谭锦志; 孙洋; 詹士潇; 张延楠
Original assignee: Hangzhou Qulian Technology Co Ltd
Current assignee: Hangzhou Qulian Technology Co Ltd
Priority date: 2023-12-26
Filing date: 2023-12-26
Publication date: 2024-04-05

Abstract

The application is applicable to the technical field of index monitoring, and provides a line protocol structure for index monitoring of a storage library, which is characterized in that the line protocol structure is used for grabbing time sequence data of an index to be monitored of the storage library, and the line protocol structure comprises: the time sequence measurement system comprises a time sequence name, a time sequence label set, a time sequence field set and a time sequence time value, wherein the time sequence label set is a time sequence label set of a current time sequence measurement index, and the time sequence field set is a time sequence field set of the current time sequence measurement index. And the index monitoring of the storage library is performed through the row protocol structure, so that the overhead of network transmission is reduced, and the efficiency of index data transmission is improved.

Description

Index monitoring method and system for row protocol structure and storage library

Technical Field

The application belongs to the technical field of index monitoring, and particularly relates to an index monitoring method and system of a row protocol structure and a storage library.

Background

Index monitoring is a fine process involving real-time monitoring and assessment of various parameters of a repository to ensure that its performance reaches a desired level. These parameters may include, but are not limited to, throughput, response time, error rate, availability, etc. By monitoring and analyzing the indexes, the problems of the storage library can be found and solved in time, and the performance of the storage library is optimized.

The traditional monitoring mode of the storage indexes in the prior art mainly comprises two modes: push mode and Pull mode. The Push mode is to take the time sequence monitoring data storage module as a server side, take the acquisition side as a client side and actively report the data. When the existing protocol is used for index monitoring in a Push mode, the problem of high base number labels exists, the index is slowly constructed due to the redundancy of invalid labels, meanwhile, the existing line protocol structure is limited in deployment environment in a weak network environment, network overhead is not facilitated to be reduced, and after the time sequence data quantity reaches a certain threshold value, query is slow, so that the efficiency of data transmission is not facilitated to be improved.

Another Pull mode is the active fetching of data from the memory bank by timing monitoring components such as promethaus, which requires that the memory bank must open the relevant interface to promethaus, the so-called exposer. The existing protocol only supports specific data types when used for index monitoring in Pull mode, so that an index model is single, and if the single index model needs to count a plurality of indexes under the same time sequence, a plurality of data are required to be transmitted through a network to realize statistics, which is not beneficial to reducing network overhead and improving data transmission efficiency.

Disclosure of Invention

The embodiment of the application provides an index monitoring method and system of a row protocol structure and a storage library, which are beneficial to reducing the overhead of network transmission and improving the efficiency of index data transmission.

A first aspect of embodiments of the present application provides a row protocol structure for index monitoring of a storage library, the row protocol structure being used to capture time-series data of an index to be monitored of the storage library, the row protocol structure comprising: the time sequence measurement system comprises a time sequence name, a time sequence label set, a time sequence field set and a time sequence time value, wherein the time sequence label set is a time sequence label set of a current time sequence measurement index, and the time sequence field set is a time sequence field set of the current time sequence measurement index.

A second aspect of an embodiment of the present application provides an index monitoring method for a repository, including:

based on the index to be monitored of the memory bank, constructing a row protocol for grabbing time sequence data of the memory bank, wherein the row protocol comprises a time sequence name, a time sequence label set, a time sequence field set and a time sequence time value, the time sequence label set is a label set of a current time sequence measurement index, and the time sequence field set is a field set of the current time sequence measurement index;

constructing a structure body which can be called by the storage library according to the row protocol;

And sending the structural body to a memory buffer area to capture time sequence data of the index to be monitored, and summarizing a plurality of pieces of time sequence data into a data block.

A third aspect of the embodiments of the present application provides an index monitoring system for a storage library, including:

the line protocol construction module is used for constructing a line protocol for grabbing time sequence data of the storage library based on the index to be monitored of the storage library, the line protocol comprises a time sequence name, a time sequence label set, a time sequence field set and a time sequence time value, the time sequence label set is a label set of the current time sequence measurement index, and the time sequence field set is a field set of the current time sequence measurement index;

the structure body construction module is used for constructing a structure body which can be called by the storage library according to the row protocol;

and the data block summarizing module is used for sending the structural body to a memory buffer area so as to grasp the time sequence data of the index to be monitored and summarizing a plurality of pieces of time sequence data into data blocks.

A fourth aspect of the embodiments of the present application provides a terminal device, including: the system comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor realizes the index monitoring method of the storage library according to the second aspect when executing the computer program.

A fifth aspect of the embodiments of the present application provides a computer-readable storage medium storing a computer program, where the computer program is executed by a processor to implement the method for monitoring an index of a storage library according to the second aspect.

A sixth aspect of the embodiments of the present application provides a computer program product, which when run on a terminal device, causes the terminal device to perform the method for monitoring an index of a repository according to the second aspect.

Compared with the prior art, the embodiment of the application has the beneficial effects that: the application provides a line protocol structure for index monitoring of a storage library, wherein the line protocol structure is used for capturing time sequence data of an index to be monitored of the storage library, the time sequence data with the line protocol structure is time sequence data supporting a line protocol, and is characterized in that the line protocol structure mainly uses line units for transmitting and storing the data, and the line protocol structure comprises: the time sequence measurement system comprises a time sequence name, a time sequence label set, a time sequence field set and a time sequence time value, wherein the time sequence label set is a time sequence label set of a current time sequence measurement index, and the time sequence field set is a time sequence field set of the current time sequence measurement index. Because the time sequence tag set comprises time sequence tags of a plurality of current time sequence measurement indexes, the time sequence field set comprises time sequence fields of a plurality of current time sequence measurement indexes, the index model based on the line protocol structure is more diversified, statistics is carried out on a plurality of indexes under the same time sequence without transmitting a plurality of data through a network, invalid tags are integrated by defining the time sequence tag set, slow construction of indexes caused by invalid tag redundancy is avoided, network overhead is reduced, and data transmission efficiency is improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following description will briefly introduce the drawings that are needed in the embodiments or the description of the prior art, it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a row protocol structure for index monitoring of a storage library according to an embodiment of the present application;

FIG. 2 is a flowchart of a method for monitoring an index of a storage library according to an embodiment of the present application;

fig. 3 is a schematic diagram of an index monitoring method based on a row protocol structure according to an embodiment of the present application;

FIG. 4 is a schematic structural diagram of an index monitoring system of a storage library according to an embodiment of the present application;

fig. 5 is a block diagram of a terminal device according to an embodiment of the present application.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It should also be understood that the term "and/or" as used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.

In addition, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used merely to distinguish between descriptions and are not to be construed as indicating or implying relative importance.

Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.

It should be understood that the sequence number of each step in this embodiment does not mean the sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

Index monitoring is a fine-grained process that involves real-time monitoring and assessment of various parameters of a repository to ensure that its performance reaches a desired level. These parameters may include, but are not limited to, throughput, response time, error rate, availability, etc. By monitoring and analyzing the indexes, possible problems are found and solved in time, and the performance of the storage library is optimized.

The storage schemes of the storage library mainly comprise two types, namely a single storage scheme suitable for occasions with smaller data volume or extremely high data security requirements, and a distributed storage scheme suitable for large-scale data storage requirements, wherein the single storage scheme or the distributed storage scheme is required to depend on an efficient index monitoring system to acquire accurate performance data so as to optimize the storage library.

The traditional monitoring mode of the storage indexes in the prior art mainly comprises two modes: push mode and Pull mode. The Push mode is to take the time sequence monitoring data storage module as a server side, take the acquisition side as a client side and actively report the data. However, when the existing protocol is used for index monitoring in the Push mode, on one hand, there is a problem of high base number of labels, for example, in the time sequence monitoring of Kubernetes, for the minimum scheduling unit Pod, we concern about indexes such as CPU/memory/disk IO/network IO/load of the current minimum scheduling unit Pod, and for the role of the index labels, it is only necessary to locate the current Pod, but in the time sequence, there are a large number of invalid labels in the existing line protocol structure, and the invalid labels are also involved in the construction of indexes, so that the invalid indexes are other index labels of Pod except the concerned indexes, and for this case, the construction of indexes is slow, and after the time sequence data amount reaches a certain threshold, the situation of slow query occurs, so that under this condition, we cannot use the existing protocol to support.

On the other hand, for the Push mode, only the Http protocol is supported, while the Http protocol supports transmission of bulk data, the single network overhead is still large for the time-series data volume, so that the Http is not suitable for the current data transmission mode. For the two protocol modes of TCP and UDP supported by InfluxDB, the effect is better than that of the normal environment in the weak network environment, but the limitation of the deployment environment is not applicable in the normal environment. On the other hand, the Push mode cannot perform distributed storage, the existing InfluxDB data expansion scheme does not open the source, the InfluxDB is protected by foreign laws, and the distributed data storage scheme cannot be used in a formal environment.

In the prior art Pull mode, i.e. the active fetching of data from a memory bank by a timing monitoring component such as promethaus, this requires that the memory bank has to open the relevant interface to promethaus, the so-called exposer. When the existing protocol is used for index monitoring in Pull mode, on one hand, the problem of index data type singleization exists, and only specific data types, such as floating point numbers, integers and unsigned integers, are supported, but character strings and Boolean values are not supported. Since existing protocols only support a particular index or data type, only the particular index value associated therewith is allowed to be recorded. For example, for counting the number of function calls, only the index related to the count is allowed to be recorded, and the data related to other indexes such as delay cannot be recorded, so that the index model is unified. And if the unified index model needs to count a plurality of indexes under the same time sequence, the statistics is realized by transmitting a plurality of data through a network, which is not beneficial to reducing network overhead and improving the efficiency of data transmission.

On the other hand, the conventional timing mode generally adopts a mode of exposing the timing data interface externally, and the mode needs to upgrade and reform the current memory bank to adapt to the progressive timing acquisition mode. However, this way of progressing is not secure for local storage. Local storage is usually cached as an object in our program, and is not suitable for such a progressive time sequence acquisition mode. Therefore, there is a need to find a way to progress more suitable for local storage to ensure more efficient and reliable acquisition and processing of time series data.

In yet another aspect, prometheus currently provides local storage, but Prometheus' storage performance is far from being available relative to other sequential storage banks. Because Prometaus is based on local storage, its storage capacity is limited by hardware resources. In a large data storage scenario, the storage capability of Prometheus may not be satisfactory. Furthermore, since the storage of Prometheus is optimized for time series, the storage and querying of non-time series data may be inefficient.

Compared with the traditional scheme, the invention provides a row protocol structure and an index monitoring method based on the row protocol structure, and provides time sequence data supporting the row protocol. In addition, the invalid labels can be integrated by defining the time sequence label set in the line protocol structure, so that slow construction of indexes caused by invalid label redundancy is avoided, network overhead is reduced, and data transmission efficiency is improved.

The greatest difference for the improvement of the monitoring method is that on one hand, the application adopts a row protocol to actively send data to the storage back end, so that the accuracy and timeliness of the data are ensured, and meanwhile, the related persistent process is not needed to be used for exposing the index itself, namely, the network request needing to be accessed is not needed. This non-process persistence approach can reduce the resource consumption and complexity of the system while improving the efficiency and security of data transmission. On the other hand, the network transmission protocol uses GRPC to carry out network transmission, so that the exposure risk is further reduced, and the safety performance is improved to a greater extent. The risk of hacking can be greatly reduced by adopting non-process persistence and GRPC protocol for data transmission, thereby improving the safety performance of the system.

In order to illustrate the technical solution of the present application, the following description is made by specific embodiments.

Referring to fig. 1, a schematic structural diagram of a row protocol structure for index monitoring of a storage library according to an embodiment of the present application is shown. As shown in fig. 1, the row protocol structure is used to capture time sequence data of the index to be monitored of the storage library, and the row protocol structure 100 includes the following structures: the time sequence label set is a time sequence label set of the current time sequence measurement index, and the time sequence field set is a time sequence field set of the current time sequence measurement index.

In this application, a Line Time Data (LTD) Line protocol is a Data exchange protocol, and in this application, the Line protocol is combined with a memory bank Time sequence, so that the Line protocol uses the protocol characteristics of Data transmission and storage in Line units, and more efficient Data capture and analysis are realized by the Line protocol Time Data. The row protocol timing data has a row protocol structure whose composition includes four parts: a timing name, a set of timing labels, a set of timing fields, and a timing time value. Space is used for distinguishing four parts in the row protocol structure, and each element in each part is used for distinguishing, for example, row protocol time sequence is as follows: func_call_num method=write, up stream=main count=1, lay= 101698805690957149000.

The timing name 101 is the name of the current timing metric, for example: func_call_num (number of calls to method).

The time series tag set 102 is a tag set of a current time series metric index, for example: method=write, up stream=main (i.e., indicates that the current method is Write, and that the current method is upstream of Main).

The time sequence field set 103 is a field set of the current time sequence metric index, such as: count=10, and lay=10 (i.e. the current number of times is 10 times, and the delay is 10ns, where ns is nanosecond, and is the default time unit of the current time, and in the actual use process, the corresponding unit value can be obtained by calculation according to the specific time).

The time sequence time value 104 is the acquisition time of the current time sequence metric index. Such as: 1698805690957149000 (default nanoseconds: ns units).

Further, the time sequence field set comprises a plurality of pairs of first key values formed by time sequence fields and time sequence field values, and each first key value corresponds to a different index model.

In the embodiment of the present application, for example, the row protocol timing is: func_call_num method=write, up stream=main count=1, lay= 101698805690957149000. The time sequence field set includes a plurality of pairs of time sequence fields and time sequence field values, that is, first key value pairs formed by count=1 and lay=1, wherein count=1 and lay=1 are a pair of time sequence fields and time sequence field values respectively, count and lay are time sequence fields, and are later time sequence field values corresponding to each time sequence field, if the count and lay can also be replaced by other indexes, such as time stamps, different first key value pairs and corresponding index models are also different. That is, the first key pair in the present application includes two fields, namely a count field and a lay field, which are equivalent to merging the two fields, and the row protocol in the prior art is the following two rows of data if it is to be expressed by time sequence:

func_call_num_count{method＝write,upstream＝main}1698805690 1

func_call_num_lay{method＝write,upstream＝main}1698805690 10

in the embodiment of the application, only one line of data is needed to express at the 1698805690957149000 time point, the current method is Write, the upstream of the current method is main, the Write method is called 1 time by the upstream, the delay is 10ns, a line of protocol comprises two field labels of count and lay, the two indexes are combined into a field set, statistics is carried out on a plurality of indexes under the same time sequence, the statistics is carried out without transmitting a plurality of data through a network, and the statistics can be carried out on a plurality of index values only by one line of protocol. In addition, the invalid labels can be integrated by defining the time sequence label set in the line protocol structure, so that slow construction of indexes caused by invalid label redundancy is avoided, network overhead is reduced, and data transmission efficiency is improved.

Further, the time sequence field set comprises a plurality of first key value pairs formed by time sequence fields and time sequence field values, and the data types of the time sequence field values in each first key value pair are different.

In the embodiment of the present application, the data type of the timing field value in each first key value pair may be a floating point number, an integer, or an unsigned integer, and the data types may be different, for example, an integer lay=10 or a floating point number lay= 352.47, which is not specifically limited herein.

By using the above-mentioned line protocol structure, any number of labels can be defined according to the need, and the labels concerned can be flexibly combined according to the service requirement for analysis by integrating some invalid labels. So that the call situation of each method, the performance of each storage, etc. can be analyzed more finely. For example, in analyzing system performance, the row protocol structure may flexibly combine multiple tags to further profile system resource usage, application performance, and the like.

In the embodiment of the application, the row protocol structure supports any type of index model, and different index models can be designed for different measurement indexes. For example, a simple counter model may be used for a counter type indicator, and a time stamp and value may be used for a time series type indicator. The diversity enables us to select the most suitable index model according to actual demands, thereby improving the accuracy and efficiency of data analysis.

The present application provides a row protocol structure applied to index monitoring of a storage library, and a flow chart of an index monitoring method of a storage library is shown with reference to fig. 2. As shown in fig. 2, the index monitoring method 200 may include the following steps:

step 201, based on the index to be monitored of the memory bank, a row protocol for capturing the time sequence data of the memory bank is constructed, wherein the row protocol comprises a time sequence name, a time sequence label set, a time sequence field set and a time sequence time value, the time sequence label set is a label set of the current time sequence measurement index, and the time sequence field set is a field set of the current time sequence measurement index.

Included in the repository are a number of metrics to be monitored, which may include, but are not limited to, throughput, response time, error rate, availability, etc. In the method for monitoring the index in the embodiment of the application, each index to be detected in the memory bank needs to be analyzed and monitored through the time sequence data of the index to be detected in the memory bank, so that a line protocol for capturing the time sequence data needs to be constructed first. In the embodiment of the application, the time sequence data supporting the row protocol is provided, the LTD row protocol is a data exchange protocol, the row protocol is combined with the time sequence of the memory bank, the protocol characteristics that the row protocol performs data transmission and storage in row units are utilized, the row protocol time sequence data is the time sequence data of a row unit, and more efficient data capture and analysis are realized through the row protocol time sequence data.

In the embodiment of the present application, the row protocol includes four parts: a timing name, a set of timing labels, a set of timing fields, and a timing time value, the set of timing labels being a set of labels of a current timing metric indicator, and thus may include a plurality of timing labels, for example, method=write, upstream=main. The time sequence field set is a field set of the current time sequence metric index, and thus may include a plurality of time sequence fields, where each time sequence field corresponds to a time sequence field value, for example, count=1 and lay=10, where count and lay are time sequence fields, and 1 and 10 are corresponding time sequence field values, respectively.

The four parts in the row protocol are distinguished by spaces, and each element in each part is distinguished by a (a) and a (b) such as the row protocol is written as follows: func_call_number=write, up stream=main count=1, lay= 101698805690957149000. Then func_call_num is the timing name, method=write, up stream=main is the timing tag set, count=1, lay=10 is the timing field set, 1698805690957149000 is the timing time value. The Write is the current method, main is the upstream of the current method, count is the number of calls, and delay is the delay time, and the above-mentioned row protocol indicates that the Write method is called 1 time by the upstream at 1698805690957149000 time point under num time sequence, and the delay is 10ns.

Since the time sequence data of the row protocol has the characteristic of carrying out data transmission and storage in row units, that is to say, a plurality of index values can be included in one row of time sequence data, the constructed row protocol comprises a plurality of fields in one row of protocol, the two fields are combined into one field set, the statistics is carried out on the plurality of indexes under the same time sequence, the statistics is carried out without transmitting a plurality of data through a network, and the statistics can be carried out on the plurality of index values only by one row of protocol. In addition, the invalid labels can be integrated by defining the time sequence label set in the line protocol structure, so that slow construction of indexes caused by invalid label redundancy is avoided, network overhead is reduced, and data transmission efficiency is improved.

Step 202, constructing a structure body which can be called by the storage library according to the row protocol.

In this embodiment of the present application, according to the row protocol constructed in step 201, a structure body that can be called by a repository is continuously constructed, and first, a field that needs to be included in the structure body is constructed as shown in table 1, where a metric value name corresponds to a time sequence name, a metric tag value corresponds to a tag value in a time sequence tag set, a metric field value corresponds to a field value in a time sequence field set, and a metric time corresponds to a time sequence time value.

Field name	Field type	Field description
			Name	string	Metric name
Tag	map[string]string	Metric tag value
			Field	map[string]interface	Metric field value
Timestamp	int64	Measuring time

TABLE 1

S1: and constructing a time sequence name of the row protocol structure.

In the embodiment of the present application, the Name is a Name in the func_call_name, and the Name is taken as a time sequence Name of the time sequence, that is, a Name of the data object.

S2: and constructing a time sequence label set of the row protocol structure.

Timing labels are generally used in embodiments of the present application to describe features and attributes of data objects. For example, tag= [ method=write ] is used as a Tag, which means that the data object is a writing method, and other target methods such as a reading method, a calculating method, and the like may be replaced as needed. Another possibility is that in an implementation, for complex tags, such as the invalid tags mentioned above, data transmission is performed by packing and merging the invalid tags. Such as: the expression "invalid_tag" = [ system=centos, arch=arm 64] is constructed by: func_call_num method=write, upstrea m=main, invalid_tag= [ system=centros, arch=arm 64] count=1, lay= 10 1698805690957149000. Meanwhile, in the process of storing data, only the invalid_tag label is concerned, the system label is not concerned, and meanwhile, if analysis is needed for the content of the system label, the analysis is needed in an invalid_tag.system mode. In the line protocol expression, three elements are present in the time sequence label set, namely the method, the up stream and the invalid_tag, and the three time sequence labels are combined in the line time sequence label set.

S3: a set of timing fields of a row protocol structure is constructed.

In the embodiments of the present application, field values are typically used to store attributes and statistics of data objects. For example, [ count=1, lay=10 ] is used as the set of timing fields, where count and lay are timing fields, 1 and 10 are timing field values, which means that the data object has an atomically accumulated number of 1 (i.e. call number) and an execution time of 10 nanoseconds when executing one operation. Other field values, such as the number of errors, the number of successes, etc., may also be added as desired. By the method, the time sequence field set comprises a plurality of pairs of first key values formed by the time sequence field and the time sequence field values, the index model corresponding to each first key value is different, different index models are formed by flexibly defining the time sequence field set, and the line protocol structure supports any type of index model, so that different index models can be designed according to different measurement indexes. For example, a simple counter model may be used for a counter type indicator, and a time stamp and value may be used for a time series type indicator. The diversity of the index model enables us to select the most suitable index model according to actual demands, thereby improving the accuracy and efficiency of data analysis.

And 203, sending the structural body to a memory buffer area to capture time sequence data of the index to be monitored, and summarizing a plurality of pieces of time sequence data into a data block.

In this embodiment of the present application, a structure of a row protocol is constructed in step 202, the constructed structure of the row protocol is sent to a memory buffer of a storage library to capture time sequence data of an index to be monitored, the captured time sequence data of the index is subjected to Batch operation, that is, a block operation is performed, and a plurality of pieces of obtained time sequence data are summarized into a data block.

After the time series data are blocked, all the data blocks are transmitted to the storage back-end server, referring to fig. 3, a schematic diagram of an index monitoring method based on a row protocol structure provided in an embodiment of the present application is shown, and in this embodiment of the present application, there are two transmission modes, where the index monitoring method further includes: and transmitting all the data blocks in the current memory buffer area to a storage back-end server at regular time or every preset time.

In the embodiment of the application, whether the data block should be actively transmitted to the storage backend server is determined by determining a time threshold of the memory buffer after the index data is blocked, in one possible embodiment, all data in the memory buffer is quickly transmitted to the storage backend server by timing, and in another possible embodiment, all data blocks in the current memory buffer are transmitted to the storage backend server by every preset time period. Through the two modes, all data blocks in the memory buffer area are actively transmitted to the storage back-end server through the GRPC protocol, data is actively sent to the storage back-end, accuracy and timeliness of the data are ensured, the corresponding exposer is not needed to be provided for exposing indexes locally as in the prior art, the related persistence process is used for exposing the indexes themselves, namely network requests needing to be accessed are not needed, the non-process persistence mode can reduce the resource consumption and complexity of a system, meanwhile, the efficiency and safety of data transmission are improved, and in some scenes needing to protect sensitive data, the non-process persistence can avoid the risk of data leakage.

The index monitoring method further comprises the following steps: and after the number of the data blocks in the memory buffer area reaches a set threshold value, transmitting all the data blocks in the current memory buffer area to a storage back-end server.

In one possible embodiment of the present application, whether the data block should be actively transmitted to the storage backend server is determined by the buffer size, by adopting the above manner, all the data blocks in the memory buffer are actively transmitted to the storage backend server through the GRPC protocol, the data is actively sent to the storage backend, the corresponding exposer is not required to be provided locally to perform index exposure, the network access request is avoided, the risk of external exposure of our service is avoided, meanwhile, the network transmission protocol uses the GRPC to perform network transmission, the exposure risk is further reduced, and the security performance is improved to a greater extent. For example, in some internet applications, hackers often make use of network requests to attack. The risk of hacking can be greatly reduced by adopting non-process persistence and GRPC protocol for data transmission, thereby improving the safety performance of the system.

Further, transmitting all data blocks in the current memory buffer to the storage backend server, including: and compressing all the data blocks by adopting a GRPC protocol through a Protobuf algorithm and transmitting the compressed data blocks to the storage back-end server.

In the embodiment of the application, the whole data transmission adopts GRPC protocol, and the Protobuf data compression capacity is utilized, so that the maximum packet size in the data transmission process is achieved, and the network overhead and the transmission cost are reduced.

In the embodiment of the application, after all the data blocks are actively transmitted to the storage back-end server through the GRPC protocol, according to the index data to be analyzed during the index, the row protocol index data can Push the data to the corresponding data storage position through an active pushing mode, so that the efficiency and accuracy of data transmission can be effectively improved, the pressure of data acquisition and storage can be relieved, and through the active pushing of the row protocol index data, the real-time monitoring and early warning of the data can be realized, so that the service requirements can be better met.

In the embodiment of the application, large screen display is performed according to the index analysis result: executing accurate data query SQL sentences, acquiring and displaying corresponding time sequence curves from a mass data storage library, wherein the display mode enables a user to more intuitively know the trend and change trend of data, provides powerful support for decision making, enables the data to be clearer and more intuitive through large-screen display, enables the user to more conveniently check and analyze the data, and accordingly better utilizes data resources and improves working efficiency.

Furthermore, to further illustrate the core concept of the present application, to facilitate the salient comparison, the specific case in the above embodiment is taken as an example to illustrate that the currently existing row protocol a adopts, for example: func_call_num method=write=write, up=main, invalid_tag_system=centos, invalid_tag_arch=arm 64 count=1, lay= 10 1698805690957149000.

The row protocol B provided in the present application adopts, for example: func_call_num method=write, up=main, invalid_tag= [ system=centos, arch=arm 64] count=1, lay= 10 1698805690957149000.

For row protocol a

Format: func_call_num method=write, up flow=main, invalid_tag_system=centos, invalid_tag_arch=arm 64 count=1, lay= 10 1698805690957149000

Index: func_call_num_method_up_stream_invalid_tag_system_invalid_tag_arc h

The characteristics are as follows: in this design, each tag (e.g., system and arch) is treated as an independent field.

Influence:

index length: since each tag is a separate field, the index becomes longer.

Query performance: longer indexes may result in reduced query performance, particularly when complex queries involving multiple tags.

Flexibility: modifying or adding new labels may require adjusting the structure of the entire row protocol.

Whereas the line protocol B of the present application

Format: func_call_num method=writeupstream=main, invalid_tag= [ system=ce ntos, arch=arm 64] count=1, lay= 10 1698805690957149000

Index: func_call_num_method_up_stream_invalid_tag

The characteristics are as follows: one map (map) is used to contain all the invalid_tags.

Influence:

index length: the index length is significantly reduced because all the invalid_tags are merged into one field.

Query performance: shorter indexes may improve query efficiency, especially in queries involving multiple low frequency tags.

Flexibility: it is easier to add or modify labels in the invalid tag without adjusting the structure of the whole row protocol.

In combination, the design of row protocol B is more efficient in handling a large number of low frequency tags because it can reduce the index length, thereby improving query performance. This approach is particularly useful in those scenarios where there are a large number of tags that may be used but less frequently. Row protocol a is better suited for those scenarios that require frequent and detailed queries for each tag, although it may result in longer index and lower query performance.

It should be understood that, the technical concept of the present application is not to configure the row protocol a into the row protocol B, but rather, a way of how to solve the efficiency problem in the large-scale data monitoring analysis is found, in the prior art, the row protocol a is a general protocol, the present application is to modify the protocol form in a breakthrough manner, and all the invalid_tags are combined into one field, so that the index length is significantly reduced, and the query efficiency can be improved, so that no description of the technical concept as the present application, especially the technical concept of combining all the invalid_tags into one field in a breakthrough manner, has not yet appeared, and meanwhile, the prior art does not disclose how to solve the problem, so that the inventors of the present application face an unoriented technical barrier in the process of technical development.

The advantages of the present application, particularly in the case of handling numerous and mostly low frequency uses of tags, are summarized below, with the following properties:

1. higher data query efficiency

Reducing the number of indexes: by combining a plurality of low-frequency tags into one index_tag field, the number of indexes is greatly reduced, and thus the query efficiency is improved.

Optimized data storage structure: the method makes the data storage structure more compact, reduces the storage space occupied by the index, and improves the overall performance of the database.

2. Enhanced scalability and flexibility

New tags are easily added: in the row protocol B, it becomes very easy to add a new low frequency tag, without reconstructing the entire index structure, only by adding the new tag to the invalid_tag map.

Flexible data model: the method provides a more flexible way to handle dynamically changing tags, especially in case of frequent changes of the tag set.

3. Improving maintainability of system

Simplified data model: since the index structure is simpler, maintenance is easier, and the possibility of errors is reduced.

Clear data hierarchy: the primary tag and the secondary (low frequency) tag are clearly distinguished, so that a data model is clearer and is convenient to understand and operate.

4. Optimized resource utilization

Memory occupation is reduced: because of the shorter index, memory occupancy is correspondingly reduced, which is particularly important for memory-sensitive applications.

The cache efficiency is improved: shorter indexes may increase the efficiency of database caching because the cache may store more index entries.

5. Better inquiry performance

Fast index scan: the shorter index makes the index scanning process faster, especially in complex queries involving large amounts of data.

Optimized query plan: the database may generate the query plan more efficiently because it does not have to handle lengthy indexes.

6. Improved scalability

Easy horizontal expansion: due to the simplification of the data model, horizontal expansion of the database is easier to achieve, especially in a distributed database system.

The adaptability is strong: the method is more suitable for the rapidly changing business requirement, and can rapidly adapt to the new data mode and the query requirement.

In general, row protocol B provides an optimized solution for handling complex data environments with a large number of low frequency tags by its efficient, flexible and scalable nature. This approach has significant advantages in terms of improving query performance, simplifying data structure management, and enhancing the overall flexibility and maintainability of the system.

To further illustrate the advantages of row protocol B, the present application provides a typical scenario: large-scale log analysis and long-term trend monitoring. Such a scenario typically involves processing large amounts of log data that contain a variety of unusual tags, but with high demands on overall trend analysis and query efficiency of historical data.

Scene description

Consider a large cloud computing platform that deploys thousands of service instances, each of which generates log data that contains information about various operations and events. These logs may be used to monitor system health, user activity, performance metrics, and the like.

Critical requirements

Long-term data storage: a large amount of history log data needs to be stored for history trend analysis and data mining.

Multidimensional analysis: multidimensional analysis is performed on log data, but most dimension labels are not used frequently.

Optimized query efficiency: for queries of historical data, efficient retrieval capabilities are needed, especially when tags are involved that are used at low frequencies.

The row protocol of the application has the advantages compared with row protocol A:

index overhead is reduced: because the row protocol B combines a plurality of labels used by low frequency into one index_tag field, the number of indexes is reduced, and therefore, the storage and maintenance cost of the indexes is reduced.

The query efficiency is improved: the structure of row protocol B can significantly improve query efficiency when conducting historical data queries, especially queries spanning long time ranges, because it simplifies the index structure.

Reducing data redundancy: in the row protocol B, the unusual tags are combined into one field, which can reduce redundancy of data, making data storage more efficient.

Better scalability: as systems evolve, more low frequency tags may be introduced. In row protocol B, this processing of the newly added tag is more flexible and efficient.

Is suitable for long-term trend analysis: row protocol B is better suited for long-term trend analysis and large data volume processing because it optimizes query performance for large data sets by reducing index length.

The occupation of resources is reduced: because of the more compact index, row protocol B may reduce storage and memory overhead, which is particularly important for large-scale data sets.

In summary, the row protocol structure of the present application provides significant advantages for such applications through its reduced index length and higher query efficiency in scenarios requiring processing large amounts of historical log data and long-term trend monitoring. Particularly, under the condition that a large data set is required to be frequently inquired and a large number of low-frequency use labels are involved, the line protocol structure of the line protocol structure can effectively improve the overall performance, reduce the resource consumption and improve the expandability. Referring to fig. 4, a schematic structural diagram of an index monitoring system of a storage library according to an embodiment of the present application is shown, and for convenience of explanation, only a portion related to the embodiment of the present application is shown.

The index monitoring system 300 of the storage library specifically may include the following modules:

a row protocol construction module 310, configured to construct a row protocol for grabbing time sequence data of a storage library based on an index to be monitored of the storage library, where the row protocol includes a time sequence name, a time sequence tag set, a time sequence field set and a time sequence time value, the time sequence tag set is a tag set of a current time sequence measurement index, and the time sequence field set is a field set of the current time sequence measurement index;

A structure building module 320, configured to build a structure that can be called by the repository according to the row protocol;

and the data block summarizing module 330 is used for sending the structural body to a memory buffer area so as to grasp the time sequence data of the index to be monitored and summarizing a plurality of pieces of time sequence data into data blocks.

In the embodiment of the present application, the index monitoring system 300 of the repository further includes the following modules:

the data block transmission module is used for transmitting all the data blocks in the current memory buffer area to the storage back-end server at regular time or every preset time;

when the number of the data blocks in the memory buffer area reaches a set threshold value, transmitting all the data blocks in the current memory buffer area to a storage back-end server;

and compressing all the data blocks by adopting a GRPC protocol through a Protobuf algorithm and transmitting the compressed data blocks to the storage back-end server.

The index monitoring of the temporary data repository provided in the embodiment of the present application may be applied in the foregoing method embodiment, and details refer to the description of the foregoing method embodiment, which is not repeated herein.

Fig. 5 is a schematic structural diagram of a terminal device provided in an embodiment of the present application. As shown in fig. 5, the terminal device 700 of this embodiment includes: at least one processor 710 (only one is shown in fig. 5), a memory 720 and a computer program 721 stored in the memory 720 and executable on the at least one processor 710, the steps in the index monitoring method embodiments of the above-mentioned memory bank being implemented when the processor 710 executes the computer program 721.

The terminal device 700 may be a computing device such as a desktop computer, a notebook computer, a palm computer, a cloud server, etc. The terminal device may include, but is not limited to, a processor 710, a memory 720. It will be appreciated by those skilled in the art that fig. 5 is merely an example of a terminal device 700 and is not limiting of the terminal device 700, and may include more or fewer components than shown, or may combine certain components, or different components, such as may also include input-output devices, network access devices, etc.

The processor 710 may be a central processing unit (Central Processing Unit, CPU), the processor 710 may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 720 may in some embodiments be an internal storage unit of the terminal device 700, such as a hard disk or a memory of the terminal device 700. The memory 720 may also be an external storage device of the terminal device 700 in other embodiments, for example, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the terminal device 700. Further, the memory 720 may also include both an internal storage unit and an external storage device of the terminal device 700. The memory 720 is used to store an operating system, application programs, boot Loader (Boot Loader), data, other programs, etc., such as program codes of the computer program. The memory 720 may also be used to temporarily store data that has been output or is to be output.

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other manners. For example, the apparatus/terminal device embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical function division, and there may be additional divisions in actual implementation, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each method embodiment described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

The implementation of all or part of the flow of the method of the above embodiment may also be accomplished by a computer program product, which when run on a terminal device, causes the terminal device to perform the steps of the method embodiments described above.

The above embodiments are only for illustrating the technical solution of the present application, and are not limiting. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.

Claims

1. A row protocol structure for index monitoring of a storage library, the row protocol structure being for capturing time series data of an index to be monitored of the storage library, the row protocol structure comprising: the time sequence measurement system comprises a time sequence name, a time sequence label set, a time sequence field set and a time sequence time value, wherein the time sequence label set is a time sequence label set of a current time sequence measurement index, and the time sequence field set is a time sequence field set of the current time sequence measurement index.

2. The row protocol structure of claim 1, wherein the set of timing fields includes a plurality of pairs of first key values of timing fields and timing field values, each first key value pair having a different corresponding index model.

3. The row protocol structure of claim 1, wherein the set of timing fields includes a plurality of pairs of first key values of timing fields and timing field values, each of the first key value pairs having a different index type of timing field value.

4. A method for monitoring an index of a repository, comprising:

5. The method of index monitoring of a storage library of claim 4, wherein said method of index monitoring further comprises:

And transmitting all the data blocks in the current memory buffer area to a storage back-end server at regular time or every preset time.

6. The method of index monitoring of a storage library of claim 4, wherein said method of index monitoring further comprises:

and after the number of the data blocks in the memory buffer area reaches a set threshold value, transmitting all the data blocks in the current memory buffer area to a storage back-end server.

7. The method of claim 5 or 6, wherein transmitting all data blocks in the current memory buffer to the storage backend server comprises:

8. An index monitoring system for a storage library, comprising:

9. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 4 to 7 when executing the computer program.

10. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the method according to any one of claims 4 to 7.