CN116126622A - Data monitoring method and device, electronic equipment and storage medium - Google Patents

Abstract

The embodiment of the disclosure provides a data monitoring method, a device, an electronic device and a storage medium, wherein the data monitoring method comprises the following steps: under the condition that the updating of the target data table is detected, traversing a monitoring rule expression associated with the target data table, and determining a plurality of first monitoring objects corresponding to the target data table in the monitoring rule expression; acquiring first object parameters of each first monitoring object according to the latest data of the target data table and storing the first object parameters in a preset storage area; for each first monitoring object, if the monitoring rule expression is determined to comprise the first monitoring object and at least one second monitoring object and the second object parameter of each second monitoring object is stored in the preset storage area, generating monitoring information according to the monitoring rule expression, the first object parameter and each second object parameter; according to the monitoring information, whether an alarm is triggered or not is determined, so that object parameters stored in a preset storage area can be reused, and the calculation workload is reduced.

Description

Data monitoring method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of data processing technologies, and in particular, to a data monitoring method, a data monitoring device, an electronic device, and a storage medium.

Background

With the development of electronic technology, the application of big data is becoming more and more widespread. In large data platforms, data is often stored in units of data tables, and the data tables are updated periodically. To ensure that data is anomalous, it is often necessary to configure some monitoring rules for one or more data tables. The monitoring system responsible for data quality monitoring generally needs to execute the monitoring task corresponding to each monitoring rule according to the preset monitoring rule.

In practical application, under the condition that the monitoring rule is complex, a large number of repeated calculations are often needed to execute the monitoring task, and the efficiency is low.

Disclosure of Invention

The embodiment of the application provides a data monitoring method, a data monitoring device, electronic equipment and a storage medium, so as to reduce the repeated calculation amount in a monitoring task.

In a first aspect, an embodiment of the present application provides a data monitoring method, including:

under the condition that updating of a target data table is detected, traversing a monitoring rule expression associated with the target data table, and determining a plurality of first monitoring objects corresponding to the target data table in the monitoring rule expression; the monitoring rule expression is used for generating monitoring information according to the latest data of at least one data table; the at least one data table includes the target data table;

Acquiring a first object parameter of each first monitoring object according to the latest data of the target data table and storing the first object parameter in a preset storage area;

for each first monitoring object, if it is determined that the monitoring rule expression includes the first monitoring object and at least one second monitoring object, and second object parameters of each second monitoring object are stored in the preset storage area, monitoring information corresponding to the monitoring rule expression is generated according to the monitoring rule expression, the first object parameters and each second object parameter; the second monitoring object corresponds to other data tables except the target data table;

and determining whether to trigger an alarm according to the monitoring information.

In a second aspect, an embodiment of the present application provides a data monitoring apparatus, including:

the traversing unit is used for traversing the monitoring rule expression associated with the target data table under the condition that the updating of the target data table is detected, and determining a plurality of first monitoring objects corresponding to the target data table in the monitoring rule expression; the monitoring rule expression is used for generating monitoring information according to the latest data of at least one data table; the at least one data table includes the target data table;

The acquisition unit is used for acquiring the first object parameters of each first monitoring object according to the latest data of the target data table and storing the first object parameters in a preset storage area;

the generation unit is used for generating monitoring information corresponding to the monitoring rule expression according to the monitoring rule expression, the first object parameters and the second object parameters if the monitoring rule expression comprises the first monitoring object and at least one second monitoring object and the second object parameters of each second monitoring object are stored in the preset storage area; the second monitoring object corresponds to other data tables except the target data table;

and the first determining unit is used for determining whether to trigger an alarm according to the monitoring information.

In a third aspect, an embodiment of the present application provides an electronic device, including: a processor; and a memory configured to store computer-executable instructions that, when executed, cause the processor to perform the data monitoring method of the first aspect.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the data monitoring method according to the first aspect.

It can be seen that, in the embodiment of the present application, in the case that the update of the target data table is detected, by determining a plurality of first monitoring objects corresponding to the target data table, and acquiring, according to the latest data of the target data table, first object parameters of each first monitoring object and storing the first object parameters in the preset storage area, it is possible that object parameters of each monitoring object corresponding to each data table may be stored in the preset storage area, where the object parameters are generated based on the latest data after the update of each data table; further, by generating the monitoring information according to the monitoring rule expression and each object parameter read from the preset storage area in the case where each object parameter required for determining the monitoring rule expression associated with the target data table is stored in the preset storage area, so as to determine whether or not to alarm according to the monitoring information, each object parameter required for generating the monitoring information according to each monitoring rule expression can be acquired from the preset storage area or newly generated, and the newly generated object parameter is also stored in the preset storage area and can be reused by other monitoring rule expressions, therefore, different monitoring rule expressions can reuse the same object parameter stored in the preset storage area, the amount of repeated calculation in the monitoring task is reduced, and each object parameter is not independently calculated from the beginning according to the monitoring rule expression corresponding to each monitoring rule when the monitoring information corresponding to each monitoring rule expression is generated.

Drawings

For a clearer description of embodiments of the present application or of the solutions of the prior art, the drawings that are required to be used in the description of the embodiments or of the prior art will be briefly described, it being obvious that the drawings in the description below are only some of the embodiments described in the present specification, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art;

fig. 1 is a process flow diagram of a data monitoring method according to an embodiment of the present application;

FIG. 2 is a process flow diagram of a configuration method of a monitoring system for performing a data monitoring method according to an embodiment of the present application;

FIG. 3 is a process flow diagram of yet another data monitoring method according to an embodiment of the present disclosure;

fig. 4 is a schematic diagram of a data monitoring device according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

In order to better understand the technical solutions in the embodiments of the present application, the following description will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present specification, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.

In practical application of the data monitoring scene, a plurality of monitoring rules with huge quantity and complex logic relationship may need to be configured, each monitoring rule corresponds to one monitoring task respectively, the monitoring system responsible for monitoring the quality of the data performs the monitoring tasks respectively, in the process of performing the monitoring tasks, different monitoring rules may be related to the same data table and share the same part of data, but the data among different monitoring tasks are not communicated, and a large number of repeated operations are inevitably needed. In order to solve the above problems, the embodiment of the present application provides a data monitoring method.

Fig. 1 is a process flow diagram of a data monitoring method according to an embodiment of the present application. The data monitoring method of fig. 1 may be performed by an electronic device, which may be a terminal device, such as a mobile phone, a notebook computer, an intelligent interaction device, etc.; alternatively, the electronic device may be a server, such as a stand-alone physical server, a server cluster, or a cloud server capable of cloud computing. Referring to fig. 1, the data monitoring method provided in this embodiment specifically includes steps S102 to S108.

Step S102, traversing a monitoring rule expression associated with a target data table under the condition that the updating of the target data table is detected, and determining a plurality of first monitoring objects corresponding to the target data table in the monitoring rule expression; the monitoring rule expression is used for generating monitoring information according to the latest data of at least one data table; the at least one data table includes a target data table.

In particular, a platform or system requiring data monitoring may store data in the form of a data table. The number of data tables that the same platform or system has may be multiple. In the plurality of data tables, each data table may have a plurality of metadata. Metadata (Metadata), also called intermediate data and relay data, is data (data about data) describing data, mainly describing data attribute (property) information, and is used to support functions such as indicating storage location, history data, resource searching, file recording, and the like.

Illustratively, the metadata may include: table name, whether it is a partition table, the type of each field, etc.

The update period of each data table may be the same or different. Types of update periods include, but are not limited to: daily updates, weekly updates, monthly updates, annual updates, etc. The data table will generate new data every new update period.

For each data table, the update period may be fixed or may be switched freely, for example, the service data corresponding to the data table is in a service rich season in 3 months to 9 months, the data update is frequent, the service is in a service light season in 10 months to 2 months, and the data update is less, so that the type of the update period of the data table is daily update in 3 months to 9 months, and the type of the update period of the data table is weekly update in 10 months to 2 months.

The target data table may be any one of a plurality of data tables that are present in a platform or system where data monitoring needs exist.

When a target data table update is detected, it may be when a new update period of the target data table comes.

And under the condition that the updating of the target data table is detected, traversing the monitoring rule expression associated with the target data table, and determining a plurality of first monitoring objects corresponding to the target data table in the monitoring rule expression.

The target data table may be associated with one or more monitoring rule expressions.

The monitoring rule expression may include at least one monitoring object in which a monitoring object corresponding to the target data table may be determined as a first monitoring object and monitoring objects corresponding to other data tables than the target data table may be determined as second monitoring objects.

The monitoring rule expressions, which may be expressions written by a user in accordance with a preconfigured monitoring rule syntax, each correspond to a monitoring rule for generating monitoring information based on the latest data in one or more data tables. Further, after the monitoring information is generated, it may be determined whether to trigger an alarm corresponding to the monitoring rule based on the monitoring information.

In the monitoring rule expression associated with the target data table, the monitoring rule corresponding to the monitoring rule expression is used for generating monitoring information based on the latest data in one or more data tables, wherein the one or more data tables comprise the target data table.

The one or more monitoring rule expressions associated with the target data table may be determined as follows: if any monitoring rule expression comprises at least one first monitoring object corresponding to the target data table, determining the monitoring rule expression as one monitoring rule expression associated with the target data table.

Traversing the monitoring rule expression associated with the target data table, determining a plurality of first monitoring objects corresponding to the target data table in the monitoring rule expression, which can be one or more monitoring rule expressions associated with the traversing target data table, and determining at least one first monitoring object corresponding to the target data table in the monitoring rule expression for each monitoring rule expression.

In the predetermined monitoring rule grammar, a monitoring object, a comparison operator, and an object parameter of the monitoring object may be configured.

The monitoring object may be a monitoring object of data quality, may be a metadata in a data table, or may be a parameter type determined based on the data table. Illustratively, the monitoring object may be the number of rows of a new partition of the data table, the maximum value of a certain field, the average value of a certain field, the past N days of fluctuation rate, or the like.

In the monitoring system for executing the data monitoring method, each monitoring object related to the data table may be configured in advance, and a corresponding object identifier is configured for each monitoring object.

In the monitoring rule grammar, a representation mode of a monitoring object in the monitoring rule expression can be configured, and in particular, the monitoring object can be represented by an object identifier and a first preset symbol. The first preset symbol may be used for the electronic device to identify the object identification if the monitoring rule expression is parsed based on the monitoring rule grammar.

Illustratively, in the monitoring rule expression, the monitoring object may be represented by $ { id }, for example, an object identification of 100 of the past N days of fluctuation rate of the average value of the field amountis represented by $ {100 }.

The comparison operators include, but are not limited to: greater than, greater than or equal to, less than or equal to, and unequal to. In the monitoring rule grammar, the representation of the comparison operators in the monitoring rule expression may be configured as shown in table 1:

comparing operator names	Monitoring symbols in regular expressions
		Greater than	>
Greater than or equal to	>＝
		Less than	<
Less than or equal to	<＝
		Equal to	=or= =
Not equal to	The following is carried out =or =<>

TABLE 1

Table 1 shows the representation of the various comparison operators in the monitoring rule expression in one embodiment.

Each monitored object has a corresponding object parameter at each update period. For each monitored object, its object parameter may be a monitored value obtained after completion of the monitoring task in the new update cycle execution. The monitored value may be a decimal or an integer, and may be represented by a numerical value.

For example, if the monitored object 1 is the average value of the field amountfor 7 days, the calculated result is 5.8%, and the object parameter of the monitored object 1 is 0.058.

The monitoring rule expression may be a simple expression composed of a monitoring object, a comparison operator, and an object parameter of the monitoring object, for example, $ {100} > =102 8. Wherein, $ {100} is used to represent a monitored object whose object identifier is "100", "> =" is a comparison operator "equal to or greater than", and 102.8 is a preset value, which is used to compare with an object parameter of the monitored object.

In the implementation, after the object parameters of the monitored object are obtained, the object parameters can be substituted into the monitored object, so that the monitored object parameters are compared with preset values to obtain a comparison result, and the comparison result can be determined as the monitoring information corresponding to the simple expression.

The monitoring rule expression may be a composite expression composed of a plurality of simple expressions. In the monitoring rule grammar, a construction mode of the compound expression in the monitoring rule expression can be configured. In this construction, a plurality of simple expressions may be connected by a logical AND, a logical OR.

The logical and may use the & & symbol, the logical or may use the ||symbol, for example, $ {100} > = 102 & $ {200} <500. The complex expression is represented by simple expression 1: $ {100} > = 102.8, logical AND symbol "& }, and, simple expression 2: and {200} <500, all three of which are formed together.

The monitoring rule expression may also be a nested expression. Nested expressions may be constructed based on simple expressions and compound expressions. Specifically, the nested relationship may be indicated by brackets. Based on the nesting relationship, the order of priority of the various calculation steps in the monitoring rule expression may be determined.

For example, $ {100} <80& ($ {200} >200 } <500 }). The nested expression is represented by simple expression 1: $ {100} <80, logical AND symbol "& }, bracket" () ", and compound expression 1: and $ {200} >200& $ {200} <500, the four are formed together.

In a specific implementation manner, the number of the monitoring rule expressions associated with the target data table is a plurality of; traversing the monitoring rule expression associated with the target data table, and determining a plurality of first monitoring objects corresponding to the target data table in the monitoring rule expression, wherein the method comprises the following steps: traversing the monitoring rule expression aiming at each monitoring rule expression, and extracting to obtain at least one object identifier; determining an object identifier associated with a target data table as a first object identifier in at least one object identifier; and determining the monitoring object corresponding to each first object identifier as a first monitoring object.

Each monitoring rule may have one or more monitoring objects, and further, the monitoring rule expression corresponding to each monitoring rule may include object identifiers of one or more monitoring objects, and at least one object identifier may be extracted from the monitoring rule expression by traversing the monitoring rule expression.

The at least one object identifier may include only the first object identifier, or may include both the first object identifier and other object identifiers other than the first object identifier. According to the target data table, the object identifier associated with the target data table can be determined as the first object identifier in the at least one object identifier.

After determining the at least one first object identification, a monitoring object corresponding to each first object identification may be determined as the first monitoring object.

In a specific implementation, traversing the monitoring rule expression and extracting to obtain at least one object identifier includes: traversing the monitoring rule expression and extracting to obtain at least one object identifier by running a first sub-class file of the inheritance basic class file; the basic class file is obtained by converting a pre-written expression grammar file; the first sub-class file is used for extracting the object identification from the monitoring rule expression according to the grammar rule corresponding to the expression grammar file.

After the monitoring rule grammar is predetermined, a grammar file of the monitoring rule grammar can be written by using an anltr tool or other tools to obtain an expression grammar file. The expression syntax file may be shared by a plurality of monitoring rules, and thus, for a platform or system requiring data monitoring, only one pre-written expression syntax file may be required.

Illustratively, in the expression syntax file, $ { id } may be used to represent the monitored object, and id may be a number used to represent the object identification.

Illustratively, in the expression syntax file, the compare operator may be represented by COMP, including, but not limited to: more than, equal to or greater than, less than, equal to or not equal to the corresponding symbols respectively.

Illustratively, in the expression syntax file, the object parameter of the monitored object may be represented by NUMBER, which may be an integer, defined by INT, or may be a fraction, defined by flow.

For example, in the expression syntax file, a conforming expression connection symbol may be defined, connected using a logical AND, a logical OR, the logical AND using the & & symbol, the logical OR using the ||symbol.

For example, in the expression syntax file, it is possible to define whether a simple expression is made up of a monitoring object, a comparison operator, and object parameters of the monitoring object, and whether the simple expression is enclosed by a bracket or a simple expression.

Illustratively, in an expression syntax file, a composite expression may be defined that is made up of simple expressions and logical and/or concatenation.

After the above-described expression grammar file is written, an anltr tool or other tool may be employed to automatically generate a base class file based on the expression grammar file, the base class file including grammar default parsing code. In the syntax default parsing code, the # remarks following the simple expression and the compound expression syntax are used to represent the method description in the generated syntax default parsing code.

Before traversing the monitoring rule expression, a first sub-class file inheriting the basic class file is also required to be written, and the first sub-class file can be used for extracting the object identification from the monitoring rule expression according to the grammar rule corresponding to the expression grammar file.

Illustratively, the base class file may be a DqcAlarmWhereBaseListener file, the first subclass file requiring an enterExprComp method therein to be rewritten to enable extraction of the value of "id" from within $ { id }.

The pre-written first subclass file can be used for expanding codes of the basic class file to achieve extraction of object identifiers.

By running the first sub-class file of the inheritance base class file, the monitoring rule expression may be traversed and at least one object identification extracted.

Step S104, according to the latest data of the target data table, acquiring the first object parameters of each first monitoring object and storing the first object parameters in a preset storage area.

The preset storage area may be a cache area. Illustratively, the preset storage area may be a dis (Remote Dictionary Server, remote dictionary service) cache area. The storage format may be $ { id } - > value.

The latest data of the target data table may be data of the target data table after the update of the target data table is detected.

The first monitoring object may be a metadata in the target data table, or may be a parameter type determined based on the target data table. Illustratively, the first monitoring object may be the number of rows of a new partition of the data table, the maximum value of a certain field, the average value of a certain field, the past N days of fluctuation rate, or the like.

In the latest data of the target data table, the object parameter of the first monitoring object may be acquired, if the first monitoring object is one metadata in the target data table, the acquiring mode may be acquiring the object parameter, and if the first monitoring object is one parameter type determined based on the target data table, the acquiring mode may be calculating the object parameter.

In a specific implementation manner, obtaining the first object parameter of each first monitoring object according to the latest data of the target data table includes: and executing the monitoring value calculation task corresponding to each first monitoring object according to the latest data of the target data table to obtain the first object parameter of each first monitoring object.

In the case where the first monitoring objects are of a type of parameter determined based on the target data table, each of the first monitoring objects corresponds to a monitoring value calculation task.

According to the latest data of the target data table, a monitoring value calculation task corresponding to each first monitoring object can be executed to obtain first object parameters of each first monitoring object, for example, the first monitoring object is an average value of a field 1, a field 2 and a field 3, field values of the field 1, the field 2 and the field 3 are read from the latest data of the target data table, and the monitoring value calculation task is executed to obtain the average value of the three field values to obtain first object parameters of the first monitoring object.

In another embodiment, the task of calculating the monitored value corresponding to the first monitored object may be executed according to the latest data and the historical data of the target data table, so as to obtain the first object parameter of the first monitored object.

Step S106, if it is determined that the monitoring rule expression includes a first monitoring object and at least one second monitoring object for each first monitoring object, and second object parameters of each second monitoring object are stored in the preset storage area, monitoring information corresponding to the monitoring rule expression is generated according to the monitoring rule expression, the first object parameters and each second object parameter; the second monitoring object corresponds to a data table other than the target data table.

And if the monitoring rule expression comprises only the first monitoring object, generating monitoring information corresponding to the monitoring rule expression according to the monitoring rule expression and the first object parameters.

For each first monitoring object, if the monitoring rule expression is determined to comprise both the first monitoring object and one or more second monitoring objects, inquiring whether second object parameters of each second monitoring object are stored in the preset storage area.

If the monitoring rule expression is determined to comprise a first monitoring object and at least one second monitoring object, the monitoring rule expression belongs to the monitoring rule expression associated with the target data table and belongs to the monitoring rule expressions of other data tables.

The other data table than the target data table may be the other data table than the target data table in the "at least one data table" as described in the foregoing step S102. That is, the monitoring rule expression is used to generate monitoring information based on the latest data of at least one data table including the target data table and other data tables other than the target data table.

For example, the target data table has associated with it 2 monitoring rule expressions: the monitoring rule expression 1 is used for generating monitoring information according to the latest data of the target data table, the data table 1 and the data table 2, wherein the first monitoring object is a monitoring object corresponding to the target data table, and the second monitoring object comprises a monitoring object corresponding to the data table 1 and a monitoring object corresponding to the data table 2; the monitoring rule expression 2 is used for generating monitoring information according to the target data table and the latest data of the data table 3, and the second monitoring object is the monitoring object corresponding to the data table 3.

It should be noted that for other data tables, similar to the target data table, the following steps are performed:

under the condition that the updating of other data tables is detected, traversing the monitoring rule expressions associated with the other data tables, and determining a plurality of second monitoring objects corresponding to the other data tables in the monitoring rule expressions; the monitoring rule expression is used for generating monitoring information according to the latest data of at least one data table; at least one data table includes the other data table; and acquiring second object parameters of each second monitoring object according to the latest data of the other data tables and storing the second object parameters in a preset storage area.

Therefore, if the update of the target data table is detected, the other data tables are not updated, and in this case, the second object parameters of each second monitoring object are not stored in the preset storage area.

And inquiring whether second object parameters of each second monitoring object are stored in the preset storage area, if the inquiring result is negative, continuing to wait for the next data table update, and when the next data table update is detected, determining the data table as a target data table, and returning to execute the step S102 and the subsequent steps.

If the query result is yes, generating monitoring information corresponding to the monitoring rule expression according to the monitoring rule expression, the first object parameters and each second object parameter.

In a specific implementation manner, after the first object parameter of each first monitoring object is obtained according to the latest data of the target data table and stored in the preset storage area, before the monitoring information corresponding to the monitoring rule expression is generated according to the monitoring rule expression, the first object parameter and each second object parameter, the data monitoring method further comprises: determining object identifications other than the first object identification as second object identifications in at least one object identification; and according to the second object identification, inquiring whether corresponding second object parameters exist in a preset storage area.

In the at least one object identifier extracted in step S102, object identifiers other than the first object identifier may be determined as second object identifiers, each of which corresponds to one second monitoring object.

According to the second object identifier, whether second object parameters of a second monitoring object corresponding to the second object identifier are stored or not can be inquired in the preset storage area.

In a specific implementation manner, according to a monitoring rule expression, a first object parameter and each second object parameter, monitoring information corresponding to the monitoring rule expression is generated, including: traversing the monitoring rule expression and analyzing the relation symbol; and generating monitoring information corresponding to the monitoring rule expression according to the first monitoring object, the first object parameters, each second monitoring object, each second object parameter and the analysis result of the relation symbol.

Traversing the monitoring rule expression and analyzing the relation symbol, and obtaining an analysis result of the relation symbol. And generating monitoring information corresponding to the monitoring rule expression according to the first monitoring object, the first object parameters, each second monitoring object, each second object parameter and the analysis result of the relation symbol.

For example, in the monitoring rule expression $ {100} = 102.8 $ {200} <500, the first object parameter is substituted into the corresponding position of the first monitoring object $ {100} in the monitoring rule expression, and compared with the preset value 102.8, if the first object parameter is determined to be greater than or equal to 102.8 based on the comparison result, the monitoring sub information corresponding to the simple expression $ {100} = 102 8 is true, otherwise, false is substituted into the corresponding position of the second monitoring object $ {200} in the monitoring rule expression, comparing the magnitude with a preset value 500, if the second object parameter is determined to be smaller than 500 based on the comparison result, the monitoring sub-information corresponding to the simple expression $ {200} <500 is true, otherwise, false, and determining that the monitoring information corresponding to the monitoring rule expression is true when the monitoring sub-information corresponding to the two simple expressions is true and the monitoring information corresponding to the monitoring rule expression is false when the monitoring sub-information corresponding to the two simple expressions is true.

In one specific implementation, traversing the monitoring rule expression and parsing the relational symbols includes: traversing the monitoring rule expression by operating the second sub-class file of the inheritance basic class file, extracting the relation symbol, analyzing the calculation mode corresponding to the relation symbol, and analyzing the monitoring object corresponding to the relation symbol; the basic class file is obtained by converting a pre-written expression grammar file; the second sub-class file is used for analyzing the relation type symbol from the monitoring rule expression according to the grammar rule corresponding to the expression grammar file.

Before traversing the monitoring rule expression, a second sub-class file inheriting the basic class file is also required to be written, and the second sub-class file can be used for analyzing the relation symbol from the monitoring rule expression according to the grammar rule corresponding to the expression grammar file.

Illustratively, the base class file may be a dqcaalarmlowerebaselistener file, where the second sub-class file requires a visitWhereBracket, visitExprBracket, visitWhereAnd, visitWhereOr, visitWhereExpr, visitExprComp method of overwriting.

The pre-written second subclass file can be used for expanding codes of the basic class file to realize analysis of relational symbols.

By running the second sub-class file of the inheritance basic class file, the monitoring rule expression can be traversed, the relation symbol can be extracted, the calculation mode corresponding to the relation symbol can be analyzed, and the monitoring object corresponding to the relation symbol can be analyzed.

For example, the calculation mode corresponding to the analysis relation symbol is ">", the calculation mode corresponding to the analysis relation symbol is to compare the values of the left side and the right side, if the left side value is larger than the right side value, the monitoring sub-information corresponding to the simple expression where the relation symbol is ">", and otherwise, the monitoring sub-information corresponding to the simple expression where the relation symbol is ">", is false.

The monitored object corresponding to the relational expression symbol is analyzed, for example, the relational expression symbol is ">", and the numerical value on the left side is the first monitored object a1. In addition, the relation symbol is analyzed, and a preset numerical value on the right side of the relation symbol can be extracted.

In this embodiment, different monitoring rules may share the same expression syntax file, the basic class file, the first sub-class file and the second sub-class file, and further, when a user wants to configure a new rule or modify an old rule under the condition that the monitoring rules are relatively complex, only the monitoring rule expression needs to be written or modified, no code modification is needed, and no complicated nested configuration operation is needed, so that rule configuration efficiency is improved, and dependency of rule configuration operation on professional capability of configuration personnel is greatly reduced in a scene that the monitoring rules need to be frequently adjusted.

Step S108, determining whether to trigger an alarm according to the monitoring information.

The monitoring information may include first monitoring information indicating that there is an abnormality in data quality, and second monitoring information indicating that there is no abnormality in data quality.

If the monitoring information is the first monitoring information, determining to trigger an alarm; and if the monitoring information is the second monitoring information, determining that the alarm is not triggered.

The following may be a specific example to assist in explaining steps S102 to S108:

for example, when a new update period of the table a arrives, the first monitoring objects corresponding to the table a may be obtained, and the computing tasks are executed one by one to collect the first object parameters of each first monitoring object, and the first object parameters are stored in the Redis cache area. Because the object parameters of one monitoring object may be used by a plurality of monitoring rule expressions, the object parameters stored in the Redis cache area can be reused by other monitoring rule expressions, so that the object parameters of each monitoring object in the monitoring object are not required to be repeatedly calculated when the analysis of the other monitoring rule expressions is executed.

After the first object parameter is stored in the Redis cache area, all monitoring rule expressions related to the A table are tried to be analyzed, and analysis is carried out on the monitoring rule expressions one by one.

When analyzing the monitoring rule expression, firstly acquiring whether all object parameters required by the monitoring rule expression can be acquired in the Redis, if so, analyzing the monitoring rule expression, automatically analyzing to obtain monitoring information after all object parameters are acquired from the Redis, determining whether to trigger an alarm according to the monitoring information, and if so, sending out alarm information. If at least one object parameter which is not calculated exists in each object parameter required by the monitoring rule expression, waiting for the arrival of a new data period of other tables, and repeating the steps.

In the embodiment shown in fig. 1, in the case that the update of the target data table is detected, by determining a plurality of first monitoring objects corresponding to the target data table, and acquiring first object parameters of each first monitoring object according to the latest data of the target data table and storing the first object parameters in a preset storage area, it is possible to store object parameters of each monitoring object corresponding to each data table in the preset storage area, where the object parameters are generated based on the latest data after the update of each data table; further, by generating the monitoring information according to the monitoring rule expression and each object parameter read from the preset storage area in the case where each object parameter required for determining the monitoring rule expression associated with the target data table is stored in the preset storage area, so as to determine whether or not to alarm according to the monitoring information, each object parameter required for generating the monitoring information according to each monitoring rule expression can be acquired from the preset storage area or newly generated, and the newly generated object parameter is also stored in the preset storage area and can be reused by other monitoring rule expressions, therefore, different monitoring rule expressions can reuse the same object parameter stored in the preset storage area, the amount of repeated calculation in the monitoring task is reduced, and each object parameter is not independently calculated from the beginning according to the monitoring rule expression corresponding to each monitoring rule when the monitoring information corresponding to each monitoring rule expression is generated. The embodiments of the present application also provide an embodiment of a monitoring system, for the same technical concept as the foregoing method embodiment. Fig. 2 is a process flow diagram of a configuration method of a monitoring system for performing a data monitoring method according to an embodiment of the present application. Referring to fig. 2, the configuration method of the monitoring system specifically includes steps S202 to S210.

The monitoring system is used for monitoring the data quality of each data table according to a pre-configured monitoring rule expression.

Step S202, determining a monitoring rule grammar.

The monitoring rule grammar can refer to the corresponding description in step S102 in the embodiment of fig. 1.

Step S204, designing an antlr grammar file.

The antlr syntax file may refer to the corresponding description part of the expression syntax file in the embodiment of fig. 1.

Step S206, generating a default grammar parsing code.

The default syntax parsing code may refer to the corresponding description portion of the base class file in the embodiment of fig. 1.

Step S208, inherit and extend the default syntax parsing code.

Inheritance and extension default syntax parsing code may refer to corresponding description portions of the first sub-class file and the second sub-class file in the embodiment of fig. 1.

Step S210, embedding a monitoring system.

Since the technical conception is the same, the description in this embodiment is relatively simple, and the relevant parts only need to refer to the corresponding descriptions of the method embodiments provided above.

The embodiments of the present application also provide another embodiment of the data monitoring method, for the same technical concept as the foregoing method embodiment. Fig. 3 is a process flow chart of another data monitoring processing method according to an embodiment of the present application. Referring to fig. 3, the data monitoring method specifically includes steps S302 to S322.

In step S302, a new period of the data table is generated.

For example, a new update period of table a comes.

Step S304, traversing all monitoring rules related to the data table to obtain a monitoring object list.

For example, the monitoring rule 1 includes the number of rows $ {1} of the data table a table and the past 7 days of fluctuation rate $ {2} of the data table B table, and the monitoring object list may be determined to include the number of rows $ {1} of the a table.

Step S306, the monitoring tasks are respectively executed to calculate object parameters of the monitoring object.

Step S308, storing the object parameters into the Redis cache.

For example, when a new update period of the table a comes, a monitor value calculation task of monitoring the object $ {1} is executed, and assuming that the collected object parameter is 3 hundred million, the object parameter is stored in the Redis cache area in the following storage format: 1- >300000000.

Step S310, traversing all monitoring rules related to the data table, and analyzing each monitoring rule one by one.

In step S312, the monitoring rule expression is parsed.

Step S314, each object parameter required for monitoring the rule expression is acquired from the Redis.

In step S316, whether the calculation of each object parameter is completed.

If yes, go to step S318; if not, return to step S302.

For example, an attempt is made to parse this monitoring rule, find that only the object parameters of $ {1} and the object parameters of $ {2} in the Redis cache area do not exist, and then do nothing to wait. When a new update period of the B table comes, executing a monitoring value calculation task of a monitoring object $ {2}, and storing the object parameters into a Redis cache area under the assumption that the acquired object parameters are 0.4, wherein the storage format is as follows: 2- >0.4. The monitoring rule expression is tried to be analyzed, and the two object parameters are obtained from the Redis cache area, so that $ {1} is 3 hundred million, and $ {2} is 0.4.

In step S318, monitoring information is generated.

For example, parse monitoring rules: ($ {1} > 200000000|$ {1} < 10000000) & gt $ {2} >0.3, the left expression is true, the right expression is true, and finally the left &rightis true.

In step S320, whether the monitoring information is true is monitored.

If yes, go to step S322; if not, the process is ended.

Step S322, an alarm is triggered.

Since the technical conception is the same, the description in this embodiment is relatively simple, and the relevant parts only need to refer to the corresponding descriptions of the method embodiments provided above.

In the foregoing embodiments, a data monitoring method is provided, and correspondingly, based on the same technical concept, the embodiments of the present application further provide a data monitoring device, which is described below with reference to the accompanying drawings.

Fig. 4 is a schematic diagram of a data monitoring device according to an embodiment of the present application.

The present embodiment provides a data monitoring apparatus 400, including:

a traversing unit 401, configured to traverse a monitoring rule expression associated with a target data table and determine a plurality of first monitoring objects corresponding to the target data table in the monitoring rule expression when an update of the target data table is detected; the monitoring rule expression is used for generating monitoring information according to the latest data of at least one data table; the at least one data table includes the target data table;

An obtaining unit 402, configured to obtain, according to the latest data of the target data table, a first object parameter of each first monitoring object, and store the first object parameter in a preset storage area;

a generating unit 403, configured to, for each first monitoring object, if it is determined that the monitoring rule expression includes the first monitoring object and at least one second monitoring object, and the second object parameter of each second monitoring object is stored in the preset storage area, generate monitoring information corresponding to the monitoring rule expression according to the monitoring rule expression, the first object parameter, and each second object parameter; the second monitoring object corresponds to other data tables except the target data table;

a first determining unit 404, configured to determine whether to trigger an alarm according to the monitoring information.

Optionally, the number of the monitoring rule expressions associated with the target data table is a plurality of; the traversing unit 401 includes:

the extraction subunit is used for traversing the monitoring rule expression and extracting at least one object identifier aiming at each monitoring rule expression;

a first determining subunit, configured to determine, from the at least one object identifier, an object identifier associated with the target data table as a first object identifier;

And the second determining subunit is used for determining the monitoring object corresponding to each first object identifier as the first monitoring object.

Optionally, the data monitoring device further includes:

a second determining unit, configured to determine, in the at least one object identifier, an object identifier other than the first object identifier as a second object identifier;

and the inquiring unit is used for inquiring whether corresponding second object parameters exist in the preset storage area according to the second object identification.

Optionally, the generating unit 403 includes:

the analysis subunit is used for traversing the monitoring rule expression and analyzing the relation symbol;

and the generation subunit is used for generating the monitoring information corresponding to the monitoring rule expression according to the first monitoring object, the first object parameters, each second monitoring object, each second object parameter and the analysis result of the relational expression symbol.

Optionally, the extraction subunit is specifically configured to:

traversing the monitoring rule expression and extracting to obtain at least one object identifier by running a first sub-class file of the inheritance basic class file; the basic class file is obtained by converting a pre-written expression grammar file; and the first sub-class file is used for extracting the object identification from the monitoring rule expression according to the grammar rule corresponding to the expression grammar file.

Optionally, the parsing subunit is specifically configured to:

traversing the monitoring rule expression by operating a second sub-class file of the inheritance basic class file, extracting the relation symbol, analyzing a calculation mode corresponding to the relation symbol, and analyzing a monitoring object corresponding to the relation symbol; the basic class file is obtained by converting a pre-written expression grammar file; and the second sub-class file is used for analyzing the relation symbol from the monitoring rule expression according to the grammar rule corresponding to the expression grammar file.

Optionally, the acquiring unit 402 is specifically configured to:

and executing a monitoring value calculation task corresponding to each first monitoring object according to the latest data of the target data table to obtain first object parameters of each first monitoring object.

The data monitoring device provided by the embodiment of the application comprises: the traversing unit is used for traversing the monitoring rule expression associated with the target data table under the condition that the updating of the target data table is detected, and determining a plurality of first monitoring objects corresponding to the target data table in the monitoring rule expression; the monitoring rule expression is used for generating monitoring information according to the latest data of at least one data table; the at least one data table includes the target data table; the acquisition unit is used for acquiring the first object parameters of each first monitoring object according to the latest data of the target data table and storing the first object parameters in a preset storage area; the generation unit is used for generating monitoring information corresponding to the monitoring rule expression according to the monitoring rule expression, the first object parameters and the second object parameters if the monitoring rule expression comprises the first monitoring object and at least one second monitoring object and the second object parameters of each second monitoring object are stored in the preset storage area; the second monitoring object corresponds to other data tables except the target data table; the first determining unit is configured to determine whether to trigger an alarm according to the monitoring information, so that, in a case where one monitoring rule may relate to a plurality of monitoring objects in a plurality of data tables and each monitoring object in each data table may be used by a plurality of monitoring rules, by determining a plurality of first monitoring objects in a case where the target data table detects an update of the target data table, and acquiring first object parameters of each first monitoring object according to the latest data of the target data table and storing the first object parameters in a preset storage area, the first object parameters can be reused by different monitoring rules, thereby reducing the repeated calculation amount in a monitoring task.

Corresponding to the above-described data monitoring method, based on the same technical concept, the embodiment of the present application further provides an electronic device, where the electronic device is configured to execute the above-provided data monitoring method, and fig. 5 is a schematic structural diagram of an electronic device provided in the embodiment of the present application.

As shown in fig. 5, the electronic device may have a relatively large difference due to different configurations or performances, and may include one or more processors 501 and a memory 502, where the memory 502 may store one or more storage applications or data. Wherein the memory 502 may be transient storage or persistent storage. The application programs stored in memory 502 may include one or more modules (not shown), each of which may include a series of computer-executable instructions in the electronic device. Still further, the processor 501 may be configured to communicate with the memory 502 and execute a series of computer executable instructions in the memory 502 on an electronic device. The electronic device may also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input/output interfaces 505, one or more keyboards 506, and the like.

In one particular embodiment, an electronic device includes a memory, and one or more programs, where the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the electronic device, and execution of the one or more programs by one or more processors includes instructions for:

under the condition that updating of a target data table is detected, traversing a monitoring rule expression associated with the target data table, and determining a plurality of first monitoring objects corresponding to the target data table in the monitoring rule expression; the monitoring rule expression is used for generating monitoring information according to the latest data of at least one data table; the at least one data table includes the target data table;

acquiring a first object parameter of each first monitoring object according to the latest data of the target data table and storing the first object parameter in a preset storage area;

for each first monitoring object, if it is determined that the monitoring rule expression includes the first monitoring object and at least one second monitoring object, and second object parameters of each second monitoring object are stored in the preset storage area, monitoring information corresponding to the monitoring rule expression is generated according to the monitoring rule expression, the first object parameters and each second object parameter; the second monitoring object corresponds to other data tables except the target data table;

And determining whether to trigger an alarm according to the monitoring information.

An embodiment of a computer-readable storage medium provided in the present specification is as follows:

corresponding to the data monitoring method described above, the embodiments of the present application further provide a computer readable storage medium based on the same technical concept.

The computer readable storage medium provided in this embodiment is configured to store computer executable instructions, where the computer executable instructions when executed by a processor implement the following procedures:

under the condition that updating of a target data table is detected, traversing a monitoring rule expression associated with the target data table, and determining a plurality of first monitoring objects corresponding to the target data table in the monitoring rule expression; the monitoring rule expression is used for generating monitoring information according to the latest data of at least one data table; the at least one data table includes the target data table;

acquiring a first object parameter of each first monitoring object according to the latest data of the target data table and storing the first object parameter in a preset storage area;

for each first monitoring object, if it is determined that the monitoring rule expression includes the first monitoring object and at least one second monitoring object, and second object parameters of each second monitoring object are stored in the preset storage area, monitoring information corresponding to the monitoring rule expression is generated according to the monitoring rule expression, the first object parameters and each second object parameter; the second monitoring object corresponds to other data tables except the target data table;

And determining whether to trigger an alarm according to the monitoring information.

It should be noted that, in the present specification, the embodiments related to the computer readable storage medium and the embodiments related to the data monitoring method in the present specification are based on the same inventive concept, so that the specific implementation of the embodiments may refer to the implementation of the corresponding method, and the repetition is omitted.

The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present description can take the form of a computer program product on one or more computer-readable storage media (including, but not limited to, magnetic disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present description is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

Embodiments of the application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. One or more embodiments of the specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

The foregoing description is by way of example only and is not intended to limit the present disclosure. Various modifications and changes may occur to those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. that fall within the spirit and principles of the present document are intended to be included within the scope of the claims of the present document.

Claims (10)

1. A method of data monitoring, comprising:

under the condition that updating of a target data table is detected, traversing a monitoring rule expression associated with the target data table, and determining a plurality of first monitoring objects corresponding to the target data table in the monitoring rule expression; the monitoring rule expression is used for generating monitoring information according to the latest data of at least one data table; the at least one data table includes the target data table;

Acquiring a first object parameter of each first monitoring object according to the latest data of the target data table and storing the first object parameter in a preset storage area;

for each first monitoring object, if it is determined that the monitoring rule expression includes the first monitoring object and at least one second monitoring object, and second object parameters of each second monitoring object are stored in the preset storage area, monitoring information corresponding to the monitoring rule expression is generated according to the monitoring rule expression, the first object parameters and each second object parameter; the second monitoring object corresponds to other data tables except the target data table;

and determining whether to trigger an alarm according to the monitoring information.

2. The method of claim 1, wherein the number of monitoring rule expressions associated with the target data table is a plurality; traversing the monitoring rule expression associated with the target data table, and determining a plurality of first monitoring objects corresponding to the target data table in the monitoring rule expression, wherein the method comprises the following steps:

traversing the monitoring rule expressions aiming at each monitoring rule expression, and extracting to obtain at least one object identifier;

Determining an object identifier associated with the target data table as a first object identifier in the at least one object identifier;

and determining the monitoring object corresponding to each first object identifier as the first monitoring object.

3. The method according to claim 2, wherein after obtaining the first object parameter of each first monitored object according to the latest data of the target data table and storing the first object parameter in a preset storage area, before generating the monitoring information corresponding to the monitoring rule expression according to the monitoring rule expression, the first object parameter and each second object parameter, the method further comprises:

determining object identifications other than the first object identification as second object identifications in the at least one object identification;

and inquiring whether corresponding second object parameters exist in the preset storage area according to the second object identification.

4. The method according to claim 1, wherein the generating the monitoring information corresponding to the monitoring rule expression according to the monitoring rule expression, the first object parameter and each of the second object parameters includes:

Traversing the monitoring rule expression and analyzing a relational symbol;

and generating monitoring information corresponding to the monitoring rule expression according to the first monitoring object, the first object parameters, each second monitoring object, each second object parameter and the analysis result of the relational expression symbol.

5. The method of claim 2, wherein traversing the monitoring rule expression and extracting at least one object identification comprises:

traversing the monitoring rule expression and extracting to obtain at least one object identifier by running a first sub-class file of the inheritance basic class file; the basic class file is obtained by converting a pre-written expression grammar file; and the first sub-class file is used for extracting the object identification from the monitoring rule expression according to the grammar rule corresponding to the expression grammar file.

6. The method of claim 4, wherein traversing the monitoring rule expression and parsing a relational symbol comprises:

traversing the monitoring rule expression by operating a second sub-class file of the inheritance basic class file, extracting the relation symbol, analyzing a calculation mode corresponding to the relation symbol, and analyzing a monitoring object corresponding to the relation symbol; the basic class file is obtained by converting a pre-written expression grammar file; and the second sub-class file is used for analyzing the relation symbol from the monitoring rule expression according to the grammar rule corresponding to the expression grammar file.

7. The method according to claim 1, wherein the obtaining the first object parameter of each of the first monitoring objects according to the latest data of the target data table includes:

and executing a monitoring value calculation task corresponding to each first monitoring object according to the latest data of the target data table to obtain first object parameters of each first monitoring object.

8. A data monitoring device, the device comprising:

the traversing unit is used for traversing the monitoring rule expression associated with the target data table under the condition that the updating of the target data table is detected, and determining a plurality of first monitoring objects corresponding to the target data table in the monitoring rule expression; the monitoring rule expression is used for generating monitoring information according to the latest data of at least one data table; the at least one data table includes the target data table;

the acquisition unit is used for acquiring the first object parameters of each first monitoring object according to the latest data of the target data table and storing the first object parameters in a preset storage area;

the generation unit is used for generating monitoring information corresponding to the monitoring rule expression according to the monitoring rule expression, the first object parameters and the second object parameters if the monitoring rule expression comprises the first monitoring object and at least one second monitoring object and the second object parameters of each second monitoring object are stored in the preset storage area; the second monitoring object corresponds to other data tables except the target data table;

And the first determining unit is used for determining whether to trigger an alarm according to the monitoring information.

9. An electronic device, the device comprising:

a processor; and a memory configured to store computer-executable instructions that, when executed, cause the processor to perform the data monitoring method of any of claims 1-7.

10. A computer readable storage medium for storing computer executable instructions which, when executed by a processor, implement a data monitoring method as claimed in any one of claims 1 to 7.

Images