CN111459980A

CN111459980A - Monitoring data storage and query method and device

Info

Publication number: CN111459980A
Application number: CN201910052587.3A
Authority: CN
Inventors: 张晓宁; 张运鹏; 何小锋
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Priority date: 2019-01-21
Filing date: 2019-01-21
Publication date: 2020-07-28

Abstract

The invention discloses a method and a device for storing and querying monitoring data, and relates to the technical field of computers. One specific implementation of the storage method comprises the following steps: coding the monitoring data to be stored to obtain the coded monitoring data to be stored; when the time dimension information of the monitoring data to be stored is in the second level, storing the monitoring data to be stored after the coding processing into a cache and a database; and when the time dimension information of the monitoring data to be stored is not in the second level, storing the monitoring data to be stored after the encoding processing into a database. The embodiment encodes the monitoring data, can remarkably reduce the disk space of data storage, stores the second-level monitoring data in the cache and the database respectively, can meet the efficiency of different types of query, and ensures the query speed of recent data in the second-level monitoring data.

Description

Monitoring data storage and query method and device

Technical Field

The invention relates to the technical field of computers, in particular to a method and a device for storing and querying monitoring data.

Background

In large-scale highly concurrent internet applications, when a system has a functional or performance problem, it is necessary to be able to quickly and accurately locate the problem. An efficient monitoring system plays a key role in problem location. Currently, system monitoring has stepped from the past minute level monitoring to the second level monitoring era. The near hundreds of times of data volume and concurrency presents an unprecedented challenge for overall monitoring systems, particularly data storage systems at core positions.

At present, for time series-based data storage such as monitoring data, common solutions are (1) using a non-relational database (i.e., a NoSQ L database such as Hbase, casandra), concatenating a metric (metric) and a timestamp (timestamp) as a row master key (rowkey), and storing data at each time point as a value (value) as one record, (2) using a development time series database (i.e., OpenTSDB) designed to merge the rowkey as the metric _ timestamp _ [ tag1_ value 1 … _ tag _ value ] or multiple rows of data as single rows and then further merge multiple columns into a single column, and (3) a scheme based on native storage (e.g., Prometheus) redesigned from a storage layer.

In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art: the existing non-relational database is a monitoring data storage scheme aiming at the minute level, and the data writing or data query performance of the existing non-relational database can not meet the service requirements in the face of the magnitude and the concurrency of second-level data; secondly, OpenTSDB needs to put source data into a memory, and when a range of data is queried each time, a large range of data needs to be fetched into the memory for filtering, so that the memory is in high demand, and the memory overflow is easily caused; and thirdly, the scheme based on the storage of the native time series data is weak in data copying and linear expansion support in design, which is unacceptable for large-scale monitoring application with rapidly growing data.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method and an apparatus for storing and querying monitored data, which can significantly reduce disk space for data storage, meet different types of query efficiency, and ensure a query speed of recent data in second-level monitored data.

To achieve the above object, according to a first aspect of the embodiments of the present invention, a storage method of monitoring data is provided.

The storage method of the monitoring data of the embodiment of the invention comprises the following steps: coding the monitoring data to be stored to obtain the coded monitoring data to be stored; when the time dimension information of the monitoring data to be stored is in the second level, storing the monitoring data to be stored after the coding processing into a cache and a database; and when the time dimension information of the monitoring data to be stored is in a non-second level, storing the monitoring data to be stored after the encoding processing into a database.

Optionally, the encoding processing of the monitoring data to be stored includes: acquiring character string data in the monitoring data to be stored; judging whether a character string identifier corresponding to the character string data exists in a memory, a cache or a database; if so, acquiring the character string identification, and replacing the character string data in the monitoring data to be stored with the character string identification; if not, calling coding service, generating a character string identifier corresponding to the character string data, and replacing the character string data in the monitoring data to be stored with the generated character string identifier.

Optionally, the storing the monitoring data to be stored, which is subjected to encoding processing, in a cache includes: and storing the monitoring data to be stored after the encoding processing into a cache according to a cache storage cycle and a cache storage format.

Optionally, the storing the monitoring data to be stored, which is subjected to encoding processing, in a database includes: determining a first storage table corresponding to the monitoring data to be stored according to the time dimension information and the condition information of the monitoring data to be stored, wherein the time dimension information comprises: second level, grading, time level and day level, the condition information comprises: monitoring index conditions and multidimensional combination conditions, wherein the first storage meter is positioned in a database; and storing the monitoring data to be stored after the coding processing into the first storage table according to the storage cycle and the storage format corresponding to the first storage table.

Optionally, the method further comprises: and if the condition information of the monitoring data to be stored is a multi-dimensional combination condition, calling a coding service to generate a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be stored.

To achieve the above object, according to a second aspect of the embodiments of the present invention, a query method of monitoring data is provided.

The query method of the monitoring data of the embodiment of the invention comprises the following steps: according to the received query request, determining time dimension information and condition information of the monitoring data to be queried, wherein the time dimension information comprises: second level, grading, time level and day level, the condition information comprises: monitoring index conditions and multidimensional combination conditions; judging whether the time dimension information of the monitoring data to be inquired is in the second level or not and whether the condition information of the monitoring data to be inquired is a monitoring index condition or not; and if so, inquiring the data to be inquired in a cache or a database, and if not, inquiring the data to be inquired in the database.

Optionally, the querying the data to be queried in the cache or the database includes: judging whether the data to be inquired is stored in the cache or not according to the inquiry time information in the inquiry request; and if so, inquiring the data to be inquired in the cache, otherwise, inquiring the data to be inquired in the database.

Optionally, the querying the data to be queried in the database includes: and determining a second storage table corresponding to the monitoring data to be inquired according to the time dimension information and the condition information of the monitoring data to be inquired, and inquiring the monitoring data to be inquired from the second storage table, wherein the second storage table is in a database.

Optionally, after determining the time dimension information and the condition information of the monitoring data to be queried, the method further includes: and if the condition information of the monitoring data to be inquired is a multi-dimensional combination condition, acquiring a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be inquired.

To achieve the above object, according to a third aspect of the embodiments of the present invention, there is provided a storage apparatus for monitoring data.

The storage device of the monitoring data of the embodiment of the invention is characterized by comprising the following components: the encoding module is used for encoding the monitoring data to be stored and acquiring the encoded monitoring data to be stored; the first storage module is used for storing the monitoring data to be stored after the coding processing into a cache and a database when the time dimension information of the monitoring data to be stored is in the second level; and the second storage module is used for storing the monitoring data to be stored after the coding processing into a database when the time dimension information of the monitoring data to be stored is in a non-second level.

Optionally, the encoding module is further configured to: acquiring character string data in the monitoring data to be stored; judging whether a character string identifier corresponding to the character string data exists in a memory, a cache or a database; if so, acquiring the character string identification, and replacing the character string data in the monitoring data to be stored with the character string identification; if not, calling coding service, generating a character string identifier corresponding to the character string data, and replacing the character string data in the monitoring data to be stored with the generated character string identifier.

Optionally, the first storage module is further configured to: and storing the monitoring data to be stored after the encoding processing into a cache according to a cache storage cycle and a cache storage format.

Optionally, the second storage module is further configured to: determining a first storage table corresponding to the monitoring data to be stored according to the time dimension information and the condition information of the monitoring data to be stored, wherein the time dimension information comprises: second level, grading, time level and day level, the condition information comprises: monitoring index conditions and multidimensional combination conditions, wherein the first storage meter is positioned in a database; and storing the monitoring data to be stored after the coding processing into the first storage table according to the storage cycle and the storage format corresponding to the first storage table.

Optionally, the encoding module is further configured to: and if the condition information of the monitoring data to be stored is a multi-dimensional combination condition, calling a coding service to generate a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be stored.

To achieve the above object, according to a fourth aspect of the embodiments of the present invention, there is provided a query device for monitoring data.

The query device of the monitoring data of the embodiment of the invention comprises: the determining module is used for determining time dimension information and condition information of the monitoring data to be queried according to the received query request, wherein the time dimension information comprises: second level, grading, time level and day level, the condition information comprises: monitoring index conditions and multidimensional combination conditions; the judging module is used for judging whether the time dimension information of the monitoring data to be inquired is in the second level or not and whether the condition information of the monitoring data to be inquired is a monitoring index condition or not; and the query module is used for querying the data to be queried in the cache or the database if the data to be queried is in the cache or the database, and querying the data to be queried in the database if the data to be queried is not in the database.

Optionally, the query module is further configured to: judging whether the data to be inquired is stored in the cache or not according to the inquiry time information in the inquiry request; and if so, inquiring the data to be inquired in the cache, otherwise, inquiring the data to be inquired in the database.

Optionally, the query module is further configured to: and determining a second storage table corresponding to the monitoring data to be inquired according to the time dimension information and the condition information of the monitoring data to be inquired, and inquiring the monitoring data to be inquired from the second storage table, wherein the second storage table is in a database.

Optionally, the determining module is further configured to: and if the condition information of the monitoring data to be inquired is a multi-dimensional combination condition, acquiring a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be inquired.

To achieve the above object, according to a fifth aspect of embodiments of the present invention, there is provided an electronic apparatus.

An electronic device of an embodiment of the present invention includes: one or more processors; the storage device is used for storing one or more programs, and when the one or more programs are executed by one or more processors, the one or more processors realize the storage and/or query method of the monitoring data of the embodiment of the invention.

To achieve the above object, according to a sixth aspect of embodiments of the present invention, there is provided a computer-readable medium.

A computer-readable medium of an embodiment of the present invention stores thereon a computer program, and when the program is executed by a processor, the computer program implements a storage and/or query method of monitoring data of an embodiment of the present invention.

One embodiment of the above invention has the following advantages or benefits:

(1) in the aspect of space dimension design, the character strings in the monitoring data are converted into character string identifiers, so that the disk space for data storage is remarkably reduced; the second-level monitoring data are respectively stored in the cache and the database, so that the efficiency of different types of query can be met; converting the multi-dimensional combined query condition into an identifier, reducing the storage space and greatly improving the data query speed;

(2) in the aspect of time dimension design, when monitoring data are stored in a database, the monitoring data with different time granularities are stored in different storage tables, and each data table is specially designed for rows and columns, so that the data volume of a single table can be controlled, and the speed of storing and retrieving the single table is ensured; storing the data into a cache, wherein the second-level data records take minutes as granularity, so that the query speed of the second-level monitoring latest data is ensured;

(3) on the aspect of overall dimension design, the coding service is deployed independently, and can be linearly expanded according to concurrency and data volume, so that millisecond-level response of the coding service is ensured; the monitoring data are shunted according to the time dimension, and data flow is processed in a flow type batch mode, so that the high speed of data processing is ensured, and data backlog is avoided.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

fig. 1 is a schematic diagram of the main steps of a storage method of monitoring data according to an embodiment of the present invention;

FIG. 2 is a message format diagram of a KAFKA queue according to an embodiment of the invention;

FIG. 3 is a schematic diagram of a Redis cache data format design according to an embodiment of the present invention;

FIG. 4 is a schematic design diagram of an HBase database table according to an embodiment of the invention;

fig. 5 is a main flow diagram of a storage method of monitoring data according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of the main steps of a query method of monitoring data according to an embodiment of the present invention;

FIG. 7 is a schematic flow chart of a query method for monitoring data according to an embodiment of the present invention;

FIG. 8 is an architectural diagram of a storage and monitoring system for monitoring data according to an embodiment of the invention;

FIG. 9 is a schematic diagram of an architecture of a coding service according to an embodiment of the present invention;

FIG. 10 is a schematic diagram of an architecture of a storage service according to an embodiment of the invention;

FIG. 11 is an architectural diagram of a query service according to an embodiment of the invention;

FIG. 12 is a schematic diagram of the main modules of a storage device for monitoring data according to an embodiment of the present invention;

FIG. 13 is a schematic diagram of the main blocks of a query device for monitoring data according to an embodiment of the present invention;

FIG. 14 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;

fig. 15 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Currently, system monitoring has advanced from past minute level monitoring to second level monitoring, that is, 150 ten thousand records per second in the original minute level becomes 9000 ten thousand records per second at present, but the existing storage technology means has no capacity for monitoring data in such magnitude and concurrency. Therefore, the invention provides a monitoring data storage and query method. Fig. 1 is a schematic diagram of main steps of a storage method of monitoring data according to an embodiment of the present invention. As a reference embodiment of the present invention, as shown in fig. 1, the main steps of the storage method of monitoring data according to the embodiment of the present invention may include:

step S101: coding the monitoring data to be stored to obtain the coded monitoring data to be stored;

step S102: when the time dimension information of the monitoring data to be stored is in the second level, storing the monitoring data to be stored after the coding processing into a cache and a database;

step S103: and when the time dimension information of the monitoring data to be stored is not in the second level, storing the monitoring data to be stored after the encoding processing into a database.

Because the monitoring data contains a large amount of character string data, such as Metric key (monitoring index key, which is used to indicate a certain index of a certain service, such as the number of records written in a log database per second, or the memory occupancy rate of a virtual machine, etc.), Host (Host key, which is used to indicate which Host a certain index value comes from at a certain time), Group (grouping key, which is used to indicate which Group comes from, and a grouping refers to a logical grouping of services, for example, terminal services may be classified as mobile phones, desktop computers, etc.), etc., these character strings occupy a large amount of disk or memory space when being stored, and the efficiency of matching and retrieving character strings is not high. Therefore, the character strings can be coded to obtain the unique identification of the character strings, for example, the character strings are converted into a long shaped numerical value.

In summary, the invention encodes the monitoring data to be stored, which not only can greatly reduce the storage space, but also can improve the query speed. As another reference embodiment of the present invention, a specific method for programming monitoring data to be stored may include:

step S1011: acquiring character string data in monitoring data to be stored;

step S1012: judging whether a character string identifier corresponding to the character string data exists in a memory, a cache or a database;

step S1013: if so, acquiring a character string identifier, and replacing character string data in the monitoring data to be stored with the character string identifier;

step S1014: if not, calling coding service, generating a character string identifier corresponding to the character string data, and replacing the character string data in the monitoring data to be stored with the generated character string identifier.

It is worth noting that the character string data is obtained in step S1011, so that the present invention can set which character strings need to be encoded and which do not need to be encoded according to the service requirement, thereby improving the practicability of the present scheme. In consideration of the magnitude and the concurrency of monitoring data, the response time of the codes in the invention is in millisecond level, and in practical application, the query speed in a memory, a cache and a database is the fastest, namely the query speed is in the memory, then the cache and then the database. Therefore, in step S1012, the optimal determination sequence is to first query whether a character string identifier corresponding to the character string data exists in the memory, if not, query whether a required character string identifier exists in the cache, and if not, query whether to store the required character string identifier in the database. Of course, the present invention may also follow other determination orders, which are not limited thereto. In addition, in step S1014, if there is no required string identifier in the memory, the cache, and the database, the encoding service needs to be invoked to generate the string identifier, thereby completing the encoding of the data to be queried.

After the monitoring data to be stored is encoded in step S101, the present invention classifies the monitoring data into two categories for storage according to the time dimension information of the monitoring data, and in step S102, it is mentioned that if the time dimension information of the monitoring data to be stored is of the second level, the monitoring data needs to be stored in both the cache and the database. In the invention, in order to meet the query of the second-level monitoring data, the second-level monitoring data is simultaneously put into a cache and a database, and data of other time dimensions are directly put into the database. In order to coordinate the speed between the inflow of monitoring data and the data storage (buffer and database), two thread groups and two data buffer queues may be designed for the second level monitoring data, one for processing data written to the buffer and the other for processing data written to the database. The design ensures that each thread can process single type data stream in batch, and greatly accelerates the speed of data storage.

In contrast, in step S103, if the time dimension information of the monitoring data to be stored is on the non-second level, the monitoring data is directly stored in the database, the time dimension information of the present invention refers to the time dimension of the inflow of the monitoring data, and may include, for example, the second level, the classification (i.e., the minute level), the hour (i.e., the hour level), and the day (i.e., the day level), in the present invention, the storage period and the storage format requirements of the cache storage need to be satisfied when the second level monitoring data is stored in the cache, in the present invention, the monitoring data may be KAFKA queue (i.e., a high throughput message queue) or monitoring data from a third party application, and then the encoded monitoring data is stored in the cache or the database according to rules, in the present invention, the cache may be Redis (i.e., a key-value based distributed cache database, which is suitable for fast retrieval of small-scale data), the database may be HBase (i.e., a key-value family based on data, a family design, and a storage attribute of the key-value of the host may be expressed as a flag indicating that the actual time dimension of the host node, the host node b — a, the node b — c — a, the graph may be expressed as a flag, or a flag indicating that the actual time dimension of the information of the host.

Fig. 3 is a schematic diagram of a Redis cache data format design according to an embodiment of the present invention. As can be seen from fig. 3, one piece of data is stored every minute, and the value of each piece of data is a hash table, containing 60 pieces of data from 0 to 59 seconds. The purpose of this design is to reduce the number of keys in the cache and facilitate data queries on the order of minutes, for example, the most common query requirement is to look at the application performance statistics of the last 10 minutes. The design enables the cache data to be accessed most efficiently.

The above embodiments explain how to store the monitoring data into the cache, and here, a specific process of storing the monitoring data to be stored, which is subjected to encoding processing, into the database is described. The storing of the monitoring data in the database may include: determining a first storage table corresponding to the monitoring data to be stored according to the time dimension information and the condition information of the monitoring data to be stored; and storing the monitoring data to be stored after the coding processing into the first storage table according to the storage period and the storage format corresponding to the first storage table.

Wherein, the time dimension information is explained above, and refers to the time dimension of monitoring data inflow; the condition information refers to storage dimension conditions when the monitoring data is stored, and may include monitoring index conditions (i.e., storage is performed only based on the monitoring index conditions) and multidimensional combination conditions (i.e., storage is performed based on a combination of other dimension information of the monitoring data, such as combination query based on multiple conditions of a host, a group, a machine room, and the like). Fig. 4 is a schematic design diagram of an HBase database table according to an embodiment of the present invention. As seen in FIG. 4, the data in the second-level data table and the data in the non-second-level data table are designed differently, and the data in each time dimension is divided into a data table based on monitoring indexes (ump _ xxx) and a data table based on multi-dimensional combination (ump _ tags _ xxx).

The database storage in the invention is comprehensively considered from the time dimension, the space dimension and the data processing dimension, the data processing pressure is dispersed, and the data storage capacity under the condition of large-scale high concurrency is integrally ensured.

(1) In the aspect of time dimension, monitoring data streams of different levels such as second level, classification level, time level and day level are processed respectively, monitoring data are divided into different queues to flow in, processing logic is processed by different threads respectively, and finally the monitoring data streams are written into storage tables of different databases. The purpose of the design can ensure that each link of monitoring data processing can be processed in a batch streaming mode, and the speed is very high.

(2) From the space dimension, different types of monitoring data are written into different tables, so that the data volume of each table is reduced, the format of the table can be designed according to different types of data, and the access speed of a single table is improved. For the minute and hour granularity of the monitored data, since the total amount is in a range that is acceptable for the single table, the single table mode can be adopted, and if the single table efficiency is reduced along with the rapid increase of the monitored data amount, the weekly or daily table mode can also be adopted. For monitoring data with second-level granularity, a daily tabulation method is needed due to the fact that the data volume is too large (in the order of billions of days). In addition, the second-level monitoring data can adopt a design idea of one data per hour, the number of records of each table is further reduced, the data access pressure of a single table is greatly relieved, and the data writing and retrieving speed is remarkably improved.

(3) And in the data processing dimension, different thread pools and local cache queues are adopted for processing different types of monitoring data. For example, the data written into the cache and the data written into the database are respectively and independently processed, so that the delay caused by the difference of the data processing speed is avoided.

It should be noted that, from the perspective of large-scale data processing, the monitoring data storage and monitoring data query belong to a system concept, and therefore, the storage method of the monitoring data may further include: and if the condition information of the monitoring data to be stored is the multidimensional combination condition, calling the coding service to generate a multidimensional combination identifier corresponding to the multidimensional combination condition of the monitoring data to be stored. The method has the advantages that for the query based on the multi-dimensional combination condition, such as the basic Metric key, the Host or the combination query of a plurality of conditions, the encoding service can be called to encode the combination condition, so that the combination of all the query conditions can be prevented from being written into the rowkey, and the query efficiency is greatly improved.

Fig. 5 is a main flow diagram of a storage method of monitoring data according to an embodiment of the present invention. As shown in fig. 5, the main process of the storage method of monitoring data of the present invention may include:

step S501: acquiring character string data in monitoring data to be stored;

step S502: judging whether a character string identifier corresponding to the character string data exists in a memory, a cache or a database, if so, executing a step S503, otherwise, executing a step S504;

step S503: acquiring a character string identifier, and replacing character string data in the monitoring data to be stored with the character string identifier;

step S504: calling coding service, generating a character string identifier corresponding to the character string data, and replacing the character string data in the monitoring data to be stored with the generated character string identifier;

step S505: judging whether the time dimension information of the monitoring data to be stored is in second degree, if so, executing a step S506, otherwise, executing a step S507;

step S506: storing the monitoring data to be stored after the coding processing into a cache according to a cache storage cycle and a cache storage format;

step S507: acquiring time dimension information and condition information of monitoring data to be stored, wherein the time dimension information may include: second level, ranking level, time level and day level, the condition information may include: monitoring index conditions and multidimensional combination conditions;

step S508: judging whether the condition information of the monitoring data to be stored is a multidimensional combination condition, if so, executing a step S509;

step S509: calling coding service to generate a multidimensional combination identifier corresponding to a multidimensional combination condition of the monitoring data to be stored;

step S510: determining a first storage table corresponding to the monitoring data to be stored according to the time dimension information and the condition information of the monitoring data to be stored, wherein the first storage table is positioned in a database;

step S511: and storing the monitoring data to be stored after the coding processing into the first storage table according to the storage period and the storage format corresponding to the first storage table.

It is to be noted that the determination sequence of step S502 is specifically described in step S1012, and will not be described in detail here. The execution order of steps S508, S509, S510 and S511 may be adjusted according to actual circumstances as long as it is ensured that step S508 is executed before step S509 and step S510 is executed before step S511.

Query requests are not as heavily stressed of high concurrency as compared to storage. For the second-level monitoring data, the main pressure of inquiry is how to quickly find the monitoring data of a period of time (for example, 20 minutes) near a certain time point in the trillion-level monitoring data. Since the data storage is monitored and distributed evenly from multiple dimensions, different query requests may fall on different storage systems (caches or databases) or storage tables. Fig. 6 is a schematic diagram of main steps of a query method of monitoring data according to an embodiment of the present invention. As a reference embodiment of the present invention, as shown in fig. 6, the main steps of the query method for monitoring data according to the embodiment of the present invention may include:

step S601: determining time dimension information and condition information of the monitoring data to be queried according to the received query request, wherein the time dimension information and the condition information are explained in the above embodiment and are not described again;

step S602: judging whether the time dimension information of the monitoring data to be inquired is in the second level or not and whether the condition information of the monitoring data to be inquired is a monitoring index condition or not;

step S603: if yes, inquiring data to be inquired in the cache or the database, and if not, inquiring data to be inquired in the database.

The query request comprises the time dimension level and the condition information of the monitoring data to be queried, if the monitoring data to be queried meets the following two conditions: if the time dimension information is the second level and the condition information is the monitoring index condition, it indicates that the data to be queried may be stored in the cache or in the database, and therefore, it is necessary to determine whether the monitoring data to be queried exists in the cache according to the query time information in the query request. The query time information refers to the generation time of the monitoring data, and the recent monitoring data is stored in the cache, for example, only the second-level monitoring data of about half an hour is stored in the cache, and the historical monitoring data can be stored in the database.

Therefore, as another referential embodiment of the present invention, querying the data to be queried in the cache or the database may include: judging whether to store the data to be inquired in the cache or not according to the inquiry time information in the inquiry request; and if so, inquiring the data to be inquired in the cache, otherwise, inquiring the data to be inquired in the database. For example, for second-level monitoring data at the latest time (e.g., within 20 minutes of the current time), it is necessary to ensure real-time and fast response of queries, so such queries retrieve data from cache. For historical data, the query is retrieved from a database. Although the data volume of second-level monitoring data is huge, the invention can adopt a mechanism of building the table according to the day, so that the data volume of the single table is controlled below one billion level, and the retrieval speed of the single table is ensured. In addition, the second-level monitoring data is stored according to the granularity of hours, one in each hour, so that each query request only needs to locate one record (query within a single hour) or two records (query within an hour-crossing time range), and the query speed is very high.

The above embodiment describes how to query data in the cache, and as another reference embodiment of the present invention, querying data to be queried in the database may include: and determining a second storage table corresponding to the monitoring data to be inquired according to the time dimension information and the condition information of the monitoring data to be inquired, and inquiring the data to be inquired from the second storage table, wherein the second storage table is in the database. The requests of the monitoring data query and the multi-dimensional combined query based on the non-second-level time dimension are executed in the database because the data volume of the data is overlarge and is not suitable for being stored in the cache.

Besides, for the request of the multidimensional combination query, the query speed is almost the same as that of the single-dimensional condition query because the multidimensional combination is converted into an identifier by the encoding service when the data storage is monitored (the only time difference is that the identifier is queried from the encoding service, but the difference is negligible because the response time of the encoding service is millisecond-level). Moreover, since the multidimensional combination is converted into an identifier by the encoding service when the monitoring data is stored, after determining the condition information of the monitoring data to be queried, the monitoring data querying method of the present invention may further include: and if the condition information of the monitoring data to be inquired is the multi-dimensional combination condition, acquiring a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be inquired.

Fig. 7 is a main flow diagram of a query method of monitoring data according to an embodiment of the present invention. As shown in fig. 7, the main process of the query method of monitoring data of the present invention may include:

step 701: determining time dimension information and condition information of the monitoring data to be queried according to the received query request;

step 702: judging whether the condition information of the monitoring data to be inquired is a multi-dimensional combination condition, if so, executing step S703;

step S703: acquiring a multidimensional combination identifier corresponding to a multidimensional combination condition of monitoring data to be inquired;

step S704: judging whether the time dimension information of the monitoring data to be inquired is in the second level, if so, executing a step S705, otherwise, executing a step S708;

step S705: judging whether the condition information of the monitoring data to be inquired is a monitoring index condition, if so, executing a step S706, otherwise, executing a step S708;

step S706: judging whether the data to be inquired is stored in the cache or not according to the inquiry time information in the inquiry request, if so, executing a step S707, otherwise, executing a step S708;

step S707: inquiring data to be inquired in a cache;

step S708: and determining a second storage table corresponding to the monitoring data to be inquired according to the time dimension information and the condition information of the monitoring data to be inquired.

It should be noted that the determination sequence of steps S704, S705 and S706 may be adjusted according to actual requirements, and will not be specifically described here.

According to the technical scheme for storing and querying the monitoring data, the space dimension, the time dimension and the whole dimension are comprehensively designed, so that the rapid storage of large-scale data is ensured, the data query speed and the query flexibility are ensured, the linear expansion capability of a system is also supported, and the method is specifically represented as follows:

For convenience of understanding, in the following description of the embodiment of the present invention, a value of "monitoring data" is "KAFKA queue", a value of "cache" is "Redis", a value of "database" is "HBase", and it is needless to say that "KAFKA queue", "Redis", and "HBase" are not used to limit the protection range of the technical solution of the present invention, and a specific value in the present invention may be adaptively adjusted according to a specific Service scenario, fig. 8 is a schematic diagram of an architecture of a monitoring and Storage system according to the embodiment of the present invention, as shown in fig. 8, the present invention adopts an architecture design based on distributed services, which mainly includes three services, a Code Service (Code Service), a Storage Service (Storage Service), and a Query Service (Query Service), each Service is respectively deployed as an independent cluster, may be linearly extended as a function of a Service requirement, a Query is linearly extended as a function, a Query is invoked by a Remote Procedure Call RPC (i.e., Remote Procedure Call) between services, a Query is a User, a Query request refers to a Computation result of a Computation by a Storage and a collaborative index of a Storage and a Computation result, such as avva, a Computation by a computing a client.

Fig. 9 is a schematic diagram of an architecture of an encoding service according to an embodiment of the present invention. Both the compute service (e.g., a flink streaming compute cluster) and the data store service invoke the encoding service to convert the string process that needs to be converted. The query service will invoke the encoding service, first convert the character string in the user query request, and then query the cache or database according to the conversion result. After receiving the transcoding request, the encoding service writes the newly generated code to both Redis and HBase, and two special specifications are needed: firstly, when the coded data is stored, the coded data is directly stored into a Redis cache, then a result is returned immediately, and the operation of writing into the HBase is asynchronous operation; in addition, the query can directly check the cache data, and the HBase can be checked only when the cache fails. These two points ensure that the coded service can complete (1 millisecond) the response instantaneously. In a second-level monitoring system, the response time of the system can be rapidly amplified by the ultra-large-scale data volume, so that the instant response of the coding service is one of the key links for ensuring the high performance of the whole data storage system. The asynchronous operation means that the main thread of the system does not wait for the data to be returned after being stored in the HBase, but sends the data to a queue, the data is read by another thread and then is stored, and the main thread directly returns the coded identifier to a calling end so as to reduce the response speed.

Currently, the idea of converting a character string into an encoding is adopted in OpenTSDB, but these databases use the encoding as an internal implementation, and both transcoding and data storage query are required. Thus, the database itself is highly stressed in computation and storage, and is inefficient. However, the scheme provided by the invention extracts the coding function to be used as a service independently, so that the storage pressure of the database can be relieved, and the content can be coded according to the service requirement. In addition, the single deployment service can be flexibly expanded according to the actual data volume and the concurrency, and powerful support is provided for large-scale data storage.

FIG. 10 is a schematic diagram of an architecture of a storage service according to an embodiment of the present invention. The storage service is a core service of the system, and the basic task of the storage service is to receive monitoring data sent by a KAFKA queue or a third-party application and store the monitoring data into a Redis cache or an HBase database according to rules. The storage module disperses the pressure of data processing from multiple dimensions, and the capacity of data storage under the condition of large-scale high concurrency is guaranteed on the whole. As can be seen from fig. 10, the storage service receives the KAFKA message queues according to different time dimensions (seconds, minutes, hours, etc.), and then processes the data for the different dimensions using corresponding logic, respectively. The storage service then calls the getOrCreateXXX interface of the encoding service to generate the corresponding data record. And finally, storing the data into a cache and a database according to the processing rules of the data with different dimensions. FIG. 11 is an architectural diagram of a query service according to an embodiment of the invention. As can be seen from the figure, for the second-level monitoring data, if the second-level monitoring data of nearly 30 minutes is queried, the second-level monitoring data is queried from Redis; if more than 30 minutes of second-level monitoring data is queried, then query from HBase.

Fig. 12 is a schematic diagram of main blocks of a storage device for monitoring data according to an embodiment of the present invention. As shown in fig. 12, the storage apparatus 1200 for monitoring data according to the embodiment of the present invention mainly includes the following modules: an encoding module 1201, a first storage module 1202 and a second storage module 1203. Wherein,

the encoding module 1201 may be configured to: coding the monitoring data to be stored to obtain the coded monitoring data to be stored;

the first storage module 1202 may be configured to: when the time dimension information of the monitoring data to be stored is in the second level, storing the monitoring data to be stored after the coding processing into a cache and a database;

the second storage module 1203 is operable to: and when the time dimension information of the monitoring data to be stored is not in the second level, storing the monitoring data to be stored after the encoding processing into a database.

In this embodiment of the present invention, the encoding module 1201 may further be configured to: acquiring character string data in monitoring data to be stored; judging whether a character string identifier corresponding to the character string data exists in a memory, a cache or a database; if so, acquiring a character string identifier, and replacing character string data in the monitoring data to be stored with the character string identifier; if not, calling coding service, generating a character string identifier corresponding to the character string data, and replacing the character string data in the monitoring data to be stored with the generated character string identifier.

In this embodiment of the present invention, the first storage module 1202 may further be configured to: and storing the monitoring data to be stored after the encoding processing into a cache according to a cache storage cycle and a cache storage format.

In this embodiment of the present invention, the second storage module 1203 may further be configured to: determining a first storage table corresponding to the monitoring data to be stored according to the time dimension information and the condition information of the monitoring data to be stored, wherein the time dimension information comprises: second level, grading, time level and day level, and the condition information comprises: monitoring index conditions and multidimensional combination conditions, wherein a first storage meter is positioned in a database; and storing the monitoring data to be stored after the coding processing into the first storage table according to the storage period and the storage format corresponding to the first storage table.

In this embodiment of the present invention, the encoding module 1201 may further be configured to: and if the condition information of the monitoring data to be stored is the multidimensional combination condition, calling the coding service to generate a multidimensional combination identifier corresponding to the multidimensional combination condition of the monitoring data to be stored.

Fig. 13 is a schematic diagram of main blocks of an inquiry apparatus for monitoring data according to an embodiment of the present invention. As shown in fig. 13, the query device 1300 for monitoring data according to the embodiment of the present invention mainly includes the following modules: a determination module 1301, a judgment module 1302 and an inquiry module 1303. Wherein,

the determination module 1301 may be used to: according to the received query request, determining time dimension information and condition information of the monitoring data to be queried, wherein the time dimension information comprises: second level, grading, time level and day level, and the condition information comprises: monitoring index conditions and multidimensional combination conditions;

the determining module 1302 may be configured to: judging whether the time dimension information of the monitoring data to be inquired is in the second level or not and whether the condition information of the monitoring data to be inquired is a monitoring index condition or not;

the query module 1303 may be configured to: if yes, inquiring data to be inquired in the cache or the database, and if not, inquiring data to be inquired in the database.

In this embodiment of the present invention, the query module 1303 may further be configured to: judging whether to store the data to be inquired in the cache or not according to the inquiry time information in the inquiry request; and if so, inquiring the data to be inquired in the cache, otherwise, inquiring the data to be inquired in the database.

In this embodiment of the present invention, the query module 1303 may further be configured to: and determining a second storage table corresponding to the monitoring data to be inquired according to the time dimension information and the condition information of the monitoring data to be inquired, and inquiring the data to be inquired from the second storage table, wherein the second storage table is in the database.

In this embodiment of the present invention, the determining module 1301 may further be configured to: and if the condition information of the monitoring data to be inquired is the multi-dimensional combination condition, acquiring a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be inquired.

From the above description, it can be seen that the storage and query device for monitoring data of the present invention is designed comprehensively from space dimension, time dimension and overall dimension, which not only ensures rapid storage of large-scale data, but also ensures data query speed and query flexibility, and also supports linear expansion capability of the system, and is specifically represented as:

Fig. 14 shows an exemplary system architecture 1400 of a monitoring data storage and query method or a monitoring data storage and query apparatus to which an embodiment of the present invention may be applied.

As shown in fig. 14, the system architecture 1400 may include

terminal devices

1401, 1402, 1403, a network 1404, and a server 1405. The network 1404 serves to provide a medium for communication links between the

terminal devices

1401, 1402, 1403 and the server 1405. The network 1404 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

A user may use

terminal devices

1401, 1402, 1403 to interact with a server 1405 via a network 1404, to receive or send messages or the like. The

terminal devices

1401, 1402, 1403 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).

The

terminal devices

1401, 1402, 1403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 1405 may be a server providing various services, such as a backend management server (for example only) providing support for shopping websites browsed by users using the

terminal devices

1401, 1402, 1403. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, target push information, product information — just an example) to the terminal device.

It should be noted that the storage and query method of the monitoring data provided by the embodiment of the present invention is generally executed by the server 1405, and accordingly, the storage and query device of the monitoring data is generally disposed in the server 1405.

It should be understood that the number of terminal devices, networks, and servers in fig. 14 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 15, shown is a block diagram of a computer system 1500 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 15 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.

As shown in fig. 15, the computer system 1500 includes a Central Processing Unit (CPU)1501 which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)1502 or a program loaded from a storage section 1508 into a Random Access Memory (RAM) 1503. In the RAM 1503, various programs and data necessary for the operation of the system 1500 are also stored. The CPU 1501, the ROM 1502, and the RAM 1503 are connected to each other by a bus 1504. An input/output (I/O) interface 1505 is also connected to bus 1504.

To the I/O interface 1505, AN input section 1506 including a keyboard, a mouse, and the like, AN output section 1507 including a keyboard, a liquid crystal display (L CD), and the like, a speaker, and the like, a storage section 1508 including a hard disk, and the like, and a communication section 1509 including a network interface card such as L AN card, a modem, and the like, the communication section 1509 performs communication processing via a network such as the internet, a drive 1510 is also connected as necessary to the I/O interface 1505, a removable medium 1511 such as a magnetic disk, AN optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted as necessary on the drive 1510 so that a computer program read out therefrom is mounted as necessary into the storage section 1508.

In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 1509, and/or installed from the removable medium 1511. The computer program executes the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 1501.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes an encoding module, a first storage module, and a second storage module. The names of the modules do not limit the modules themselves under certain conditions, for example, the encoding module may also be described as a "module for encoding the monitoring data to be stored and acquiring the encoded monitoring data to be stored".

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: coding the monitoring data to be stored to obtain the coded monitoring data to be stored; when the time dimension information of the monitoring data to be stored is in the second level, storing the monitoring data to be stored after the coding processing into a cache and a database; and when the time dimension information of the monitoring data to be stored is not in the second level, storing the monitoring data to be stored after the encoding processing into a database.

According to the technical scheme of the embodiment of the invention, the space dimension, the time dimension and the integral dimension are comprehensively designed, so that the rapid storage of large-scale data is ensured, the data query speed and the query flexibility are ensured, and the linear expansion capability of the system is supported, which is specifically represented as follows:

The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for storing monitored data, comprising:

coding the monitoring data to be stored to obtain the coded monitoring data to be stored;

when the time dimension information of the monitoring data to be stored is in the second level, storing the monitoring data to be stored after the coding processing into a cache and a database;

and when the time dimension information of the monitoring data to be stored is in a non-second level, storing the monitoring data to be stored after the encoding processing into a database.

2. The storage method according to claim 1, wherein the encoding processing of the monitoring data to be stored includes:

acquiring character string data in the monitoring data to be stored;

judging whether a character string identifier corresponding to the character string data exists in a memory, a cache or a database;

if so, acquiring the character string identification, and replacing the character string data in the monitoring data to be stored with the character string identification;

if not, calling coding service, generating a character string identifier corresponding to the character string data, and replacing the character string data in the monitoring data to be stored with the generated character string identifier.

3. The storage method according to claim 1, wherein the storing the monitoring data to be stored after the encoding processing into a cache comprises:

and storing the monitoring data to be stored after the encoding processing into a cache according to a cache storage cycle and a cache storage format.

4. The storage method according to claim 1, wherein the storing the monitoring data to be stored after the encoding processing into a database comprises:

determining a first storage table corresponding to the monitoring data to be stored according to the time dimension information and the condition information of the monitoring data to be stored, wherein the time dimension information comprises: second level, grading, time level and day level, the condition information comprises: monitoring index conditions and multidimensional combination conditions, wherein the first storage meter is positioned in a database;

and storing the monitoring data to be stored after the coding processing into the first storage table according to the storage cycle and the storage format corresponding to the first storage table.

5. The storage method according to claim 4, wherein the method further comprises:

and if the condition information of the monitoring data to be stored is a multi-dimensional combination condition, calling a coding service to generate a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be stored.

6. A query method for monitoring data is characterized by comprising the following steps:

according to the received query request, determining time dimension information and condition information of the monitoring data to be queried, wherein the time dimension information comprises: second level, grading, time level and day level, the condition information comprises: monitoring index conditions and multidimensional combination conditions;

judging whether the time dimension information of the monitoring data to be inquired is in the second level or not and whether the condition information of the monitoring data to be inquired is a monitoring index condition or not;

and if so, inquiring the data to be inquired in a cache or a database, and if not, inquiring the data to be inquired in the database.

7. The method according to claim 6, wherein the querying the data to be queried in the cache or the database comprises:

judging whether the data to be inquired is stored in the cache or not according to the inquiry time information in the inquiry request;

and if so, inquiring the data to be inquired in the cache, otherwise, inquiring the data to be inquired in the database.

8. The query method according to claim 6 or 7, wherein the querying the data to be queried in the database comprises:

and determining a second storage table corresponding to the monitoring data to be inquired according to the time dimension information and the condition information of the monitoring data to be inquired, and inquiring the monitoring data to be inquired from the second storage table, wherein the second storage table is in a database.

9. The query method according to claim 6, wherein after determining the time dimension information and the condition information of the monitoring data to be queried, the method further comprises:

and if the condition information of the monitoring data to be inquired is a multi-dimensional combination condition, acquiring a multi-dimensional combination identifier corresponding to the multi-dimensional combination condition of the monitoring data to be inquired.

10. A storage device for monitoring data, comprising:

the encoding module is used for encoding the monitoring data to be stored and acquiring the encoded monitoring data to be stored;

the first storage module is used for storing the monitoring data to be stored after the coding processing into a cache and a database when the time dimension information of the monitoring data to be stored is in the second level;

and the second storage module is used for storing the monitoring data to be stored after the coding processing into a database when the time dimension information of the monitoring data to be stored is in a non-second level.

11. The storage device of claim 10, wherein the encoding module is further configured to:

acquiring character string data in the monitoring data to be stored;

12. The storage device of claim 10, wherein the first storage module is further configured to:

13. The storage device of claim 10, wherein the second storage module is further configured to:

14. The storage device of claim 13, wherein the encoding module is further configured to:

15. An apparatus for querying monitoring data, comprising:

the determining module is used for determining time dimension information and condition information of the monitoring data to be queried according to the received query request, wherein the time dimension information comprises: second level, grading, time level and day level, the condition information comprises: monitoring index conditions and multidimensional combination conditions;

the judging module is used for judging whether the time dimension information of the monitoring data to be inquired is in the second level or not and whether the condition information of the monitoring data to be inquired is a monitoring index condition or not;

and the query module is used for querying the data to be queried in the cache or the database if the data to be queried is in the cache or the database, and querying the data to be queried in the database if the data to be queried is not in the database.

16. The query device of claim 15, wherein the query module is further configured to:

17. The query device of claim 15 or 16, wherein the query module is further configured to:

18. The querying device of claim 15, wherein the determining module is further configured to:

19. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-9.

20. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-9.