CN115129498A

CN115129498A - Monitoring method, monitoring equipment and storage medium

Info

Publication number: CN115129498A
Application number: CN202210731206.6A
Authority: CN
Inventors: 董琼; 徐山凌; 张晶; 江旻; 杨杨
Original assignee: WeBank Co Ltd
Current assignee: WeBank Co Ltd
Priority date: 2022-06-24
Filing date: 2022-06-24
Publication date: 2022-09-30
Also published as: WO2023245893A1

Abstract

The embodiment of the application discloses a monitoring method, which comprises the following steps: determining a rule to be operated for monitoring the abnormity of a target product; analyzing the rule to be operated to obtain an identifier of the database to be accessed and an identifier of the data table to be accessed; determining the type of the rule to be operated based on the identifier of the database to be accessed and the identifier of the data table to be accessed; if the type is a target type, acquiring database attribute information of at least one corresponding target database based on the identifier of the database to be accessed and the identifier of the data table to be accessed; generating a target engine based on the database attribute information; and executing the rules to be operated aiming at least one target database through the target engine so as to realize the monitoring of the target product. The embodiment of the application also discloses monitoring equipment and a storage medium.

Description

Monitoring method, monitoring equipment and storage medium

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a monitoring method, a monitoring device, and a storage medium.

Background

With the rapid development of computer technology, more and more technologies are applied in the financial field, and the traditional financial industry is gradually changing to financial technology (Fintech), but higher requirements are also put forward on the technologies due to the requirements of the financial industry on safety and real-time performance. In the field of data processing, the reliability of a product full-link application system is generally determined by whether an anomaly exists in data in a database corresponding to different application systems. At present, different product full-link application systems can correspond to the same database or different databases in the same server in the same area, or different databases in different servers in different areas, so the abnormal data monitoring rules are often classified into the following categories: a. rules within a single table; b. the same-library cross-table rule; c. the same server in the same area is used for cross-library and cross-table rules; d. cross-region, cross-server, cross-library table rules. When monitoring is performed on the rules of the two types a and b, Structured Query Language (SQL) rules can be directly executed. When monitoring is carried out aiming at the rule of type c, a plurality of libraries can be accessed simultaneously by applying for the user right of the database, and the essence of the execution is the same as that of the rule monitoring of the type a and the type b. When monitoring is performed on the d-type rule, because the database is deployed in different server clusters and cannot be directly linked to table for Query, the technology in production at present is to extract data in the database to be accessed under different server clusters into a data warehouse tool (hive), and then execute a network object-oriented Query Language (HQL) to perform linked table Query monitoring.

However, when monitoring is performed on the d-type rule, data analysis can be performed only by extracting data from a plurality of different databases to hive, so that the real-time performance of data analysis is poor, and the computational resources consumed in the data extraction process are high, so that the efficiency of the monitoring and analyzing process is low.

Content of application

In order to solve the above technical problems, embodiments of the present application desirably provide a monitoring method, a monitoring device, and a storage medium, so as to solve the problem that the monitoring analysis efficiency of the present monitoring process for crossing servers and databases is low, and implement a real-time monitoring method for crossing servers and databases, without extracting data from different databases to hive, thereby reducing consumption of computational resources and improving efficiency of the monitoring analysis process.

The technical scheme of the application is realized as follows:

in a first aspect, a method of monitoring, the method comprising:

determining a rule to be operated for monitoring the abnormity of a target product;

analyzing the rule to be operated to obtain an identifier of the database to be accessed and an identifier of the data table to be accessed;

determining the type of the rule to be operated based on the identifier of the database to be accessed and the identifier of the data table to be accessed;

if the type is a target type, acquiring database attribute information of at least one corresponding target database based on the identifier of the database to be accessed and the identifier of the data table to be accessed;

generating a target engine based on the database attribute information;

and executing the rules to be operated aiming at least one target database through the target engine so as to realize the monitoring of the target product.

In a second aspect, a monitoring device, the device comprising: a memory, a processor, and a communication bus; wherein:

the memory to store executable instructions;

the communication bus is used for realizing communication connection between the processor and the memory;

the processor is configured to execute the monitoring program stored in the memory to implement the steps of the monitoring method according to any one of the above descriptions.

In a third aspect, a storage medium has a monitoring program stored thereon, which when executed by a processor implements the steps of the monitoring method according to any one of the above.

In the embodiment of the application, after a to-be-run rule for performing exception monitoring on a target product is determined, the to-be-run rule is analyzed to obtain a to-be-accessed database identifier and a to-be-accessed data table identifier, then the type of the to-be-run rule is determined based on the to-be-accessed database identifier and the to-be-accessed data table identifier, if the type of the to-be-run rule is the target type, database attribute information of at least one corresponding target database is obtained based on the to-be-accessed database identifier and the to-be-accessed data table identifier, a target engine is generated based on the database attribute information, and the to-be-run rule is executed aiming at the at least one target database through the target engine to realize monitoring on the target product. Therefore, the type of the to-be-operated rule is determined after the to-be-operated rule for monitoring is analyzed, and when the type of the to-be-operated rule is the target type, the target engine is generated according to the database attribute information of at least one target database related to the to-be-operated rule, so that the to-be-operated rule is executed through the target engine, monitoring operation is achieved, the problem that monitoring analysis efficiency is low in the monitoring process of crossing a server and a database at present is solved, a method for monitoring crossing the server and the database in real time is achieved, data does not need to be extracted from the database to the hive, consumption of operation resources is reduced, and efficiency of the monitoring analysis process is improved.

Drawings

Fig. 1 is a schematic flowchart of a monitoring method according to an embodiment of the present disclosure;

fig. 2 is a schematic flow chart of another monitoring method provided in the embodiment of the present application;

fig. 3 is a schematic flowchart of another monitoring method provided in the embodiment of the present application;

fig. 4 is a schematic flowchart of a monitoring method according to another embodiment of the present application;

fig. 5 is a schematic flow chart of another monitoring method according to another embodiment of the present application;

fig. 6 is a schematic flow chart of another monitoring method according to another embodiment of the present application;

fig. 7 is a schematic view of an application scenario provided in an embodiment of the present application;

fig. 8 is a schematic structural diagram of a monitoring device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

An embodiment of the present application provides a monitoring method, as shown in fig. 1, where the method is applied to monitoring equipment, and the method includes the following steps:

step 101, determining a rule to be run for monitoring the abnormity of a target product.

In this embodiment of the present application, the monitoring device may be a server device or a service cluster. The target product may be a business product, such as various business products in the financial field, and may be, for example, a loan business product in the financial field. The to-be-run rule is an existing rule for monitoring the target product, and may be called an inventory rule in some application scenarios.

And 102, analyzing the rule to be operated to obtain the identifier of the database to be accessed and the identifier of the data table to be accessed.

In the embodiment of the application, the rule to be run is analyzed, the database to be accessed related to the rule to be run is determined, and the identifier of the database to be accessed and the identifier of the data table to be accessed, which is specifically related to the rule to be run, are obtained.

And 103, determining the type of the rule to be operated based on the identifier of the database to be accessed and the identifier of the data table to be accessed.

In the embodiment of the application, the identifier of the database to be accessed and the identifier of the data table to be accessed are analyzed, and the type of the rule to be operated is determined. The rules to be run may be divided according to whether the accessed database spans a service area, and specifically may be divided into two types: one type is a non-cross service area type and the other type is a cross service area type.

And step 104, if the type is the target type, acquiring database attribute information of at least one corresponding target database based on the identifier of the database to be accessed and the identifier of the data table to be accessed.

In the embodiment of the present application, the target type is a cross-service area type. And when the type of the rule to be operated is a cross-service area type, determining at least one corresponding target database according to the identifier of the database to be accessed and the identifier of the data table to be accessed, and acquiring the database attribute information of the at least one corresponding target database. The database attribute information of the target database may be information required for accessing the database, and environmental information to which the target database belongs, and the like.

And 105, generating a target engine based on the database attribute information.

In the embodiment of the application, the target engine is generated according to the database attribute information of at least one target database corresponding to the rule to be run, so that the rule to be run is executed through the target engine subsequently, and real-time monitoring is realized.

And 106, executing rules to be run aiming at least one target database through the target engine.

The target engine executes the rules to be run aiming at least one target database so as to realize the monitoring of the target product.

In the embodiment of the application, the target engine is generated, the rule to be run is converted into the rule which can be run, the rule which can be run is executed, and the rule to be run is executed, so that the target product is monitored.

In the embodiment of the application, after a to-be-run rule for performing exception monitoring on a target product is determined, the to-be-run rule is analyzed to obtain a to-be-accessed database identifier and a to-be-accessed data table identifier, then the type of the to-be-run rule is determined based on the to-be-accessed database identifier and the to-be-accessed data table identifier, if the type of the to-be-run rule is the target type, database attribute information of at least one corresponding target database is obtained based on the to-be-accessed database identifier and the to-be-accessed data table identifier, a target engine is generated based on the database attribute information, and the to-be-run rule is executed aiming at the at least one target database through the target engine to achieve monitoring on the target product. Therefore, the type of the to-be-operated rule is determined after the to-be-operated rule for monitoring is analyzed, and when the type of the to-be-operated rule is the target type, the target engine is generated according to the database attribute information of at least one target database related to the to-be-operated rule, so that the to-be-operated rule is executed through the target engine, monitoring operation is achieved, the problem that monitoring analysis efficiency is low in the monitoring process of crossing a server and a database at present is solved, a method for monitoring crossing the server and the database in real time is achieved, data does not need to be extracted from different databases to live, consumption of operation resources is reduced, and efficiency of the monitoring analysis process is improved.

Based on the foregoing embodiments, an embodiment of the present application provides a monitoring method, as shown in fig. 2, the method is applied to a monitoring device, and the method includes the following steps:

step 201, determining a rule to be run for monitoring the target product for abnormity.

In the embodiment of the application, the monitoring device obtains the existing rules to be operated for anomaly monitoring aiming at the target product.

Step 202, analyzing the rule to be run to obtain the identifier of the database to be accessed and the identifier of the data table to be accessed.

In the embodiment of the application, the monitoring device analyzes the rule to be operated, and determines to obtain the identifier of the database to be accessed and the identifier of the data table to be accessed, which are included in the rule to be operated. The identifier of the database to be accessed is identifier information for uniquely identifying the corresponding database, and may be, for example, a database name, a database number, or the like; the identifier of the data table to be accessed is identification information for uniquely identifying the corresponding data table, and may be, for example, a name of the data table, a number of the data table, or the like.

And 203, determining the type of the rule to be operated based on the identifier of the database to be accessed and the identifier of the data table to be accessed.

In the embodiment of the application, the identifier of the database to be accessed and the identifier of the data table to be accessed are analyzed, and the service areas to which the corresponding database and the corresponding data table belong are determined. If the database corresponding to the identifier of the database to be accessed and the data table corresponding to the identifier of the data table to be accessed are both in the same service area, determining the type of the rule to be operated as a non-cross service area type; and if the database corresponding to the identifier of the database to be accessed and the data table corresponding to the identifier of the data table to be accessed are in a plurality of service areas, determining the type of the rule to be operated as a cross-service area type.

And 204, if the type is the target type, determining at least one target database to be accessed based on the identifier of the database to be accessed and the identifier of the data table to be accessed.

In the embodiment of the application, when the type of the rule to be run is a non-cross service area type, the rule to be run is directly executed. And when the type of the database to be accessed is the target type, namely the cross-service area type, determining the database to which the identifier of the data table to be accessed belongs and the database corresponding to the identifier of the database to be accessed, and then performing deduplication processing on the determined database to obtain at least one target database to be accessed.

Step 205, traversing the test environment corresponding to at least one target database to obtain database attribute information.

In the embodiment of the application, traversal analysis is performed on the test environment corresponding to at least one target database, and database attribute information of each target database is obtained. The database attribute information of each target database may be, for example, a test environment, a Data Center Node (DCN) area, an Internet Protocol (IP), a Port (Port), a User name (User), a Password (Password), and the like corresponding to each target database.

And step 206, generating a target engine based on the database attribute information.

In an embodiment of the present application, a target engine is generated based on database attribute information of at least one target database, and the target engine may be a federated (Federated) engine.

And step 207, executing the rules to be run aiming at the at least one target database through the target engine so as to realize the monitoring of the target product.

In the embodiment of the application, the target engine is used for converting the rule to be operated into the executable rule code for operation, so that the target product is monitored.

Based on the foregoing embodiment, in another embodiment of the present application, referring to fig. 3, after the monitoring device performs step 206, it is further configured to perform step 208:

and step 208, generating an engine data table corresponding to the target engine based on the database attribute information.

The engine data table is used for recording the characteristic information of the data table corresponding to the rule to be operated.

In the embodiment of the application, an engine data table corresponding to the target engine is generated based on the database attribute information of at least one target database, and is used for recording the characteristic information of all data corresponding to the rule to be run, so as to ensure the source data which can be accessed by the target engine.

Correspondingly, step 207 is implemented by step 207 a:

and step 207a, executing the rule to be run by the target engine based on the engine data table.

In the embodiment of the application, the target engine accesses the access data source recorded in the engine data table, and executes the rule to be run, so that the monitoring operation is realized.

Based on the foregoing embodiment, in other embodiments of the present application, referring to fig. 4, after the monitoring device performs step 207, the monitoring device is further configured to perform steps 209 to 210:

step 209, if it is detected that the rule to be run is updated according to the preset time interval, updating the engine data table by the target engine based on the updated rule to be run.

In the embodiment of the application, the monitoring device detects the rule to be operated according to the preset time interval, and detects whether the update exists, wherein the update comprises the conditions of rule increase, rule decrease, rule modification and the like. The target engine updates the engine data table based on the updated rule to be run, and the process of updating the engine data table may specifically be: and the target engine analyzes the updated rule to be operated, determines to obtain at least one reference database corresponding to the updated rule to be operated, and determines database attribute information of the at least one reference database to update the engine data table.

Step 210, recording the update time of the update engine data table through the target engine.

In the embodiment of the application, the target engine records the update time of the update engine data table, so that whether the rule to be run is updated or not can be detected through the update time of the engine data table.

Based on the foregoing embodiment, in another embodiment of the present application, as shown in fig. 5, after the monitoring device performs the capturing 209, the monitoring device is further configured to perform step 211:

and step 211, executing the updated rule to be run by the target engine based on the updated engine data table.

In this embodiment of the application, the implementation process of step 211 may refer to the specific implementation process of step 207a, and details are not described here again.

Based on the foregoing embodiments, in other embodiments of the present application, step 207a may be implemented by steps a 11-a 13:

step a11, generating an operation strategy by the target engine based on the rule to be operated and the engine data table.

In the embodiment of the application, the target engine analyzes the rule to be operated and the engine data sheet according to the requirement of the operation strategy generation, and further generates and obtains the operation strategy.

Step a12, generating a target execution statement by the target engine based on the operation strategy.

In the embodiment of the application, the target engine generates a corresponding target execution statement according to the generated operation strategy. It should be noted that the operation policy is generated for different service areas, and therefore, the generated target execution statement includes execution statements for different service areas.

Step a13, executing the target execution statement through the target engine to realize the execution of the rule to be executed.

In the embodiment of the application, when the target engine executes the target execution statement, the execution statements in different service areas are distributed to nodes corresponding to different service areas, and the corresponding execution statements are executed through the corresponding nodes, so that the execution of the rule to be executed is realized.

Based on the foregoing embodiments, in other embodiments of the present application, step a11 may be implemented by steps a 111-a 116:

step a111, determining at least one first data table of at least one database corresponding to the same virtual area under the same service area from the engine data table.

In the embodiment of the present application, the data tables recorded in the engine data table are grouped, and one grouping manner is to determine at least one first data table in at least one database corresponding to the same virtual area under the same service area.

Step a112, combining at least one first data table based on the rule to be operated to obtain a first strategy.

In the embodiment of the application, at least one first data table is combined according to the relation in at least one first data table in the rule to be operated to obtain the first strategy.

Step a113, determining at least one second data table of at least one database corresponding to different virtual areas under different service areas from the engine data table.

In the embodiment of the present application, the data tables recorded in the engine data table are grouped, and another grouping manner is to determine at least one second data table in at least one database corresponding to different virtual areas belonging to different service areas.

And a114, combining at least one second data table based on the rule to be operated to obtain a second strategy.

In the embodiment of the application, the at least one second data table is combined according to the relationship in the at least one second data table in the rule to be run, so that the second policy is obtained.

Step a115, determining that the service area corresponding to the virtual area only has 1 third data table from the engine data table.

In the embodiment of the present application, a third data table under a different virtual area belonging to only 1 service area is determined.

And a step a116, determining the third data table as a third strategy based on the rule to be operated.

The operation strategy comprises a first strategy, a second strategy and a third strategy.

In the embodiment of the present application, the third data table is directly determined as the third policy, and no combination processing is required.

Based on the foregoing embodiments, in other embodiments of the present application, referring to fig. 6, after the monitoring device performs step 207, the monitoring device is further configured to perform steps 212 to 214:

step 212, the target engine outputs the execution result.

And the execution result is obtained by executing the rule to be operated by the target engine.

In the embodiment of the application, after the target engine executes the target execution statement, a corresponding execution result is obtained, and the target engine outputs the execution result, which can be displayed.

And step 213, if the target product is determined to have an exception based on the execution result, determining at least one exception data rule through the target engine.

In the embodiment of the application, the execution result is analyzed, the execution result is determined to indicate that the target product is abnormal or the execution result indicates that the target product is abnormal, and the target engine is used for determining the rule to be operated, wherein the rule is abnormal, so that at least one abnormal data rule is obtained.

And step 214, executing an exception recovery operation by the target engine based on at least one exception data rule and a preset exception recovery mode.

In the embodiment of the present application, the preset abnormal data recovery method is a recovery method used for presetting data, and may be, for example, a number recovery method, a cleaning method, or a number recovery method and a cleaning method, where the number recovery method is a method of correcting data, and the cleaning method is a method of cleaning abnormal data. In this way, after the target engine determines that the at least one abnormal data rule is obtained, the data corresponding to the at least one abnormal data rule is processed in a preset abnormal repairing mode, and abnormal repairing operation is achieved.

Based on the foregoing embodiments, in other embodiments of the present application, step 214 can be implemented by steps 214a to 214 d:

step 214a, determining, by the target engine, at least one list object to which each abnormal data rule relates.

In the embodiment of the application, each abnormal data rule is analyzed, and at least one list object involved in each abnormal data rule is determined. For example, the list object may be list identification information of the list.

Step 214b, determining, by the target engine, an association rule for which each list object has an association relationship from at least one target database, so as to obtain a target association rule.

In the embodiment of the application, at least one target database is analyzed, the associated object having the association relation with each list object is determined, and the corresponding association rule is determined according to the associated object, so that all the association rules corresponding to at least one abnormal data rule can be determined and obtained and recorded as the target association rules.

And 214c, obtaining the rule to be repaired through the target engine based on the at least one abnormal data rule and the target association rule.

In the embodiment of the application, at least one abnormal data rule and a target association rule are analyzed and processed, and the rule to be repaired is determined.

And step 214d, executing the abnormal repairing operation by the target engine based on the rule to be repaired and the preset abnormal repairing mode.

In the embodiment of the application, the target engine processes the data corresponding to the rule to be repaired by adopting the repairing mode corresponding to the preset abnormal repairing mode, so as to realize the abnormal repairing operation.

It should be noted that, in some application scenarios, before executing step 214d, backup processing may be performed on data corresponding to the rule to be repaired, so that when a subsequent fault occurs, a scroll-up operation is performed, or a repair fault analysis processing is performed on the fault occurring in the abnormal repair operation.

Based on the foregoing embodiments, in other embodiments of the present application, step 214c may be implemented by the following steps: grouping at least one abnormal data rule and a target association rule through a relation that a target engine is divided into a group according to rules belonging to the same service area to obtain a rule to be repaired.

In the embodiment of the application, the target engine groups at least one abnormal data rule and the target association rule according to a grouping relation that the rules belonging to the same service area are divided into a group, so as to obtain the rule to be repaired. When at least one abnormal data kneels and a repeated rule exists in the target association rule, the duplicate removal processing is required.

Based on the foregoing embodiments, in other embodiments of the present application, step 214d may be implemented by steps b 11-b 12, or steps b 13-b 14, or steps b 15-b 16:

step b11, if the preset abnormal repairing mode is a number repairing mode, obtaining the preset repairing rule corresponding to the rule to be repaired through the target engine.

In this embodiment of the present application, when the preset abnormal repairing mode is a number repairing mode, the target engine obtains a preset repairing rule corresponding to a rule to be repaired, where the preset repairing rule is a rule for updating (Update) data corresponding to each rule preset in the rule to be repaired.

And b12, executing a preset repair rule through the target engine to realize the abnormal repair operation.

In the embodiment of the application, the target engine executes the preset restoration rule corresponding to the rule to be restored, and performs corresponding restoration processing on data corresponding to the rule to be restored. For example, the abnormal data corresponding to the rule to be repaired is replaced by the data specified in the preset repair rule.

And b13, if the preset abnormal repairing mode is a cleaning mode, executing data cleaning operation corresponding to the rule to be repaired through the target engine.

Wherein the exception recovery operation comprises a data scrubbing operation.

In this embodiment of the application, when the preset abnormal repairing mode is a cleaning mode, the target engine executes an operation for cleaning data corresponding to the rule to be repaired, for example, an operation for deleting data corresponding to the rule to be repaired.

Step b14, if the preset abnormal repairing mode comprises a number repairing mode and a cleaning mode, obtaining a preset repairing rule corresponding to the rule to be repaired through the target engine.

And b15, executing the preset repair rule through the target engine, and then executing the data cleaning operation.

In this embodiment of the present application, when the preset abnormal recovery manner includes a number recovery manner and a cleaning manner, the two recovery manners usually have a processing priority order to ensure accuracy of finally processed data, and the processing priority of the number recovery manner is usually set to be higher than that of the cleaning manner, that is, in an execution process, a recovery process corresponding to the number recovery manner needs to be executed for a rule to be recovered, and then the recovery process corresponding to the cleaning manner needs to be executed. However, in some application scenarios, the priority of the cleaning mode may be set to be higher than the priority of the modifying mode, and may be determined according to the actual application scenario.

Based on the foregoing embodiment, in other embodiments of the present application, after the monitoring device performs step 214d, the monitoring device is further configured to perform step 214 e:

and 214e, if the target product is detected to be abnormal, repeatedly executing the step 'if the target product is determined to be abnormal based on the execution result, determining at least one abnormal data rule by the target engine' until the target product is detected to be normal by the target engine, and ending the abnormal repairing operation.

In this embodiment of the application, if the monitoring device detects that the target product is normal after performing step 214d, that is, if there is no abnormal condition for monitoring the target product, it indicates that the abnormality is eliminated, the abnormality repairing operation is ended, otherwise, if it is detected that the target product still has the abnormality, the contents corresponding to steps 213 to 214 need to be repeatedly executed, and the abnormality repairing operation is ended until it is detected that the target product is not abnormal.

Based on the foregoing embodiments, in other embodiments of the present application, an embodiment of the present application provides a monitoring method, including:

and c11, judging whether a cross-service area exists according to the stock rule, if so, executing step c12, otherwise, executing step c 14.

An application scenario schematic diagram across service areas may be as shown in fig. 7, and a corresponding inventory rule may be denoted as Rac — Ta1 join Tc 1; the service area 2 is connected with the service area 1 by Ta1 in the service area 1 and by Tc1 in the service area 2, so that cross-area access between the service area 1 and the service area 2 is realized, and similarly, the service area 1 is connected with the service area 2 by Tb1 in the service area 1 and by Tc1 in the service area 2 by rbcc. When the stock rule is determined to be a rule crossing the service area, the stock rule can be identified by adopting preset identification information so as to be analyzed subsequently.

Step c12, engine initialization.

After determining each database corresponding to the stock rule, traversing the test environment of each database to acquire database (Data Base, DB) information. The DB information at least comprises information such as a test environment, a DCN area, an IP, a Port, a User, a Password and the like. In general, such information may be acquired from a Configuration center of a Configuration Management Database (CMDB), or may be directly acquired from the local when stored locally.

For example, the inventory rule across service areas relates to 1 Application Data Management (ADM) area and 2 DCN areas, and when deployed on different service areas DBSTAT 1 and DBSTAT 2, the corresponding DB information can be recorded as DBSTAT 1: Source Environment K, area ADM, library glpdb, IP:10.1.1.1, PORT:3301, USER: USER1, PASWD: xxx, and Table T1; DBSET 2: source Environment K, region DCN, library cpsdb, IP:10.1.1.2, PORT:3302, USER: USER2, PASSSWD: xxx, Table T2.

Step c13, executing the inventory rules by the engine.

Firstly, a fed engine is created according to environment information and DCN area information, wherein the fed engine comprises a master (master) node for decision making and slave (slave) nodes corresponding to a plurality of service areas, and simultaneously, a fed table corresponding to the fed engine is created according to the environment information and the DCN information, and a mapping rule of the fed table can be a source library name _ source table name _ area name. In the process, a mapping relation is established between the fed engine and each database corresponding to the stock rule.

Secondly, the Federated engine generates different operation strategies according to the stock rule and the Federated table, and the steps are as follows:

1) and analyzing the stock rule to generate a rule area list to obtain a fed table. For example, the inventory rules relate to the fed table middle zone database as shown in the table below; wherein, T1 table: represented in virtual area a0, which has only 1 logical area; t2 table: represented in virtual area B0, which has 2 logical areas; table T3 and table T4: represented in the same virtual area C0, there are two different service databases.

Logical area	Virtual area	Library	Watch (A)
				A1	A0	db1	T1
B1	B0	db2	T2
				B2	B0	db2	T2
C1	C0	db3	T3
				C2	C0	db3	T3
C1	C0	db4	T4
				C2	C0	db4	T4

2) And generating the operation strategy according to the fed table. Wherein generating the operation policy comprises: policy 1, policy 2, and policy 3; wherein, the policy 1 is a combination of the same virtual area and the same logical area under different service areas, for example, when join is performed in the T3 and T4 tables, the combinations generated are (C1, C1) and (C2, C2); policy 2 is different virtual areas, and different service areas need to be combined with each other, for example, when join is performed in T2 and T3 tables, (B1, C1), (B1, C2), (B2, C1), and (B2, C2) are generated as combinations; policy 3 is that if there are only 1 logical regions in the virtual region corresponding to the table, only 1 combination needs to be performed, and if the table is queried alone in T1, the combination is generated (a 1).

3) And iteratively replacing the stock rule based on the operation strategy to generate an engine executable statement.

And when the stock rule is replaced iteratively and the engine executable statement is generated, the iterative replacement mode is consistent with the fed table creation principle. Exemplary, the stock rules before replacement are as follows:

select count(1)from

(select a,b from db1.T1)t1

join

(select a,b from db2.T2)t2

on t1.a＝t2.a

where t1.c＝‘xx’；

correspondingly, after replacement, 2 combinations (a1, B1) and (a1, B2) are split according to the policy combination, and engine executable statements corresponding to the two rules are as follows:

select count(1)from

(select a,b from fedlink_db1_A1.T1)t1

join

(select a,b from fedlink_db2_B1.T2)t2

on t1.a＝t2.a

where t1.c＝‘xx’；

select count(1)from

(select a,b from fedlink_db1_A1.T1)t1

join

(select a,b from fedlink_db2_B2.T2)t2

on t1.a＝t2.a

where t1.c＝‘xx’；

it should be noted that, when generating the engine executable statements, the base tables of each virtual area related to the stock rule need to be traversed until the engine executable statements corresponding to all policy combinations are generated.

And finally, the fed engine runs the executable statements of the engine in parallel to realize the execution process of the stock rule.

And step c14, executing the stock rule.

Based on the foregoing embodiments, in other embodiments of the present application, after the fed engine performs step c13, the fed engine is further configured to perform the following steps:

and c15, detecting whether the stock rule is updated at preset time intervals, if the stock rule is updated, synchronously updating the fed engine and the fed table, and then executing the updated stock rule.

Based on the foregoing embodiments, in other embodiments of the present application, after the fed engine performs step c13 or step c15, the fed engine is further configured to perform the following steps:

and c16, outputting the rule operation result.

And the Federated engine performs regional cross presentation according to the generated strategy combination, and if any combination operation result does not meet the abnormal check condition, the rule operation result is determined to be the rule check failure. Illustratively, the two rule combinations (a1, B1) and (a1, B2) split from the above are still determined to be failed in checking the (a1, B1) operation result, and the (a1, B2) operation is successfully checked.

And step c17, if the rule operation result is that the verification fails, determining at least one abnormal data rule.

First, a KEY (KEY) corresponding to the rule is found out through step-by-step iterative anti-association of table fields related to abnormal rules in the stock rule.

Secondly, extracting multiple KEY in a single library and searching similar association rules in the single library. For example, assuming that the stock rule across the service area is R1, the corresponding 1) is to extract the table list T _ num referred to by R1; 2) if the number of T _ num is 1, and if the table list T _ num is T1, searching the associated primary KEY and index of the table T1, and then reversely searching all KEY KEYs, associated tables and abnormal field value list sets associated under the database to which the table list T1 belongs to obtain the association rule in the single library; if the number of T _ num is greater than 1, the rule R1 is split into a plurality of sub-queries, for example, when R1 refers to tables T1, T2 and T3, it can be split into 3 sub-query rules for tables T1, T2 and T3, and the process of obtaining the association rule for each sub-query is the same as that when the number of T _ num is 1.

For example, the specific process of looking up the associated primary KEY and index of the table T1, and then looking up all KEY, association table and abnormal field value list sets associated under the database to which the table list T1 belongs to obtain the association rule in the single library may be represented as follows: 1) the keyword set T is searched according to a T1 table and is [ k1, k2, k3 ]; 2) according to the keyword k set and the original field set sc [ c1, c2 and c3] in the R1, the abnormal field set c [ c1 and c2], searching association table sets related to similar association rules in the rule base [ t1, t2 and t3], wherein the judgment method of the similar association rule table is as follows: firstly, generating a table linked list, taking T1 of the current rule R1 as a vertex, searching a first-layer association rule node list containing a T1 table

L1R [ R2, R3, … ], and the associated rule table thereof, take the intersection set L1t [ t1, t2, t3], it should be noted that, for

The method avoids scanning the full rule base, the number of the nodes which are downwards taken for searching the association rule linked list is equal to the initial T _ num, the steps are repeated until the nodes of the generated list linked list are traversed, the association rule judges that the searching is completed, and the association rule corresponding to the final single base T1 is obtained: { k: [ k1, k2, k3], t: [ t1, t2, t3], c: [ c1, c2] }.

And finally, taking a union set of the association rules of the plurality of databases corresponding to the stock rule to obtain at least one final abnormal data rule.

And c18, determining that the abnormal repairing mode comprises a number repairing mode and a cleaning mode.

And c19, determining a preset repair rule corresponding to the at least one abnormal data rule.

And searching a preset repair rule set Rk matched with the k value and a preset repair rule set Rt matched with the T value in the rule base, and filtering out the preset repair rule of the single table T1 in the Rk and the Rt according to the abnormal field set c { c1, c2 }. For example, the R1 rule is written as: SELECT COUNT (1) FROM cpsddb, t1where c1 IS NULL AND c2 ═ 0AND c3 ═ a'; correspondingly, the preset repair rule corresponding to the R1 rule is: UPDATE cpsddb, t1 set c1 ═ 0, c2 ═ 1where c1 IS NULL AND c2 ═ 0AND c3 ═ a'; the preset repair rule corresponding to the R1 rule indicates that c1 is updated to 0and c2 is updated to 1 when c1 is detected to be empty, c2 is detected to be 0and c3 is detected to be a.

And c20, before executing the preset repair rule, backing up the abnormal data corresponding to at least one abnormal data rule in order to keep the scene or trace the abnormal scene.

And c21, executing the preset repair rule corresponding to the at least one abnormal data rule.

When the preset repair rule corresponding to at least one abnormal data rule is executed, it is determined whether the other data tables are affected by the association, and if the abnormality is found, the step c17 is continuously repeated. The preset repairing rules determined in the step c19 are preferentially executed until the detection result is successful after all the preset repairing rules finally determined are executed.

And c22, after the number correcting mode is executed, the cleaning mode is continuously and automatically executed until the detection and the monitoring are successful after the cleaning mode is executed, the abnormal data is repaired, and otherwise, the steps c17 to c22 are repeatedly executed.

It should be noted that, for the descriptions of the same steps and the same contents in this embodiment as those in other embodiments, reference may be made to the descriptions in other embodiments, which are not described herein again.

In the embodiment of the application, after a to-be-run rule for performing exception monitoring on a target product is determined, the to-be-run rule is analyzed to obtain a to-be-accessed database identifier and a to-be-accessed data table identifier, then the type of the to-be-run rule is determined based on the to-be-accessed database identifier and the to-be-accessed data table identifier, if the type of the to-be-run rule is the target type, database attribute information of at least one corresponding target database is obtained based on the to-be-accessed database identifier and the to-be-accessed data table identifier, a target engine is generated based on the database attribute information, and the to-be-run rule is executed aiming at the at least one target database through the target engine to achieve monitoring on the target product. Therefore, the type of the to-be-operated rule to be monitored is determined after the to-be-operated rule is analyzed, and when the type of the to-be-operated rule is the target type, the target engine is generated according to the database attribute information of at least one target database related to the to-be-operated rule, so that the to-be-operated rule is executed through the target engine, the monitoring operation is realized, the problem that the monitoring analysis efficiency of the cross-server and cross-database monitoring process is low at present is solved, the cross-server and cross-database monitoring method is realized, data does not need to be extracted from different databases to the hive, the consumption of operation resources is reduced, and the efficiency of the monitoring analysis process is improved.

Based on the foregoing embodiments, an embodiment of the present application provides a monitoring device, which may be applied to the monitoring method provided in the embodiments corresponding to fig. 1 to 6, and as shown in fig. 8, the monitoring device 3 may include: a processor 31, a memory 32, and a communication bus 33, wherein:

a memory 32 for storing executable instructions;

a communication bus 33 for implementing a communication connection between the processor 31 and the memory 32;

the processor 31 is configured to execute the monitoring program stored in the memory 32 to implement the method implementation process provided in the embodiments corresponding to fig. 1 to 6, which is not described herein again.

Based on the foregoing embodiments, embodiments of the present application provide a computer-readable storage medium, which is referred to as a storage medium for short, where the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the implementation processes of the monitoring methods provided in the embodiments corresponding to fig. 1 to 6, and details are not described here again.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above description is only a preferred embodiment of the present application, and is not intended to limit the scope of the present application.

Claims

1. A method of monitoring, the method comprising:

if the type of the database to be accessed is a target type, acquiring database attribute information of at least one corresponding target database based on the identifier of the database to be accessed and the identifier of the data table to be accessed;

generating a target engine based on the database attribute information;

2. The method according to claim 1, wherein if the type to which the database belongs is a target type, acquiring database attribute information of the corresponding at least one target database based on the identifier of the database to be accessed and the identifier of the data table to be accessed includes:

if the type is a target type, determining at least one target database to be accessed based on the identifier of the database to be accessed and the identifier of the data table to be accessed;

and traversing the test environment corresponding to at least one target database to obtain the attribute information of the database.

3. The method of claim 1, wherein after generating a target engine based on the database attribute information, the method further comprises:

generating an engine data table corresponding to the target engine based on the database attribute information; the engine data table is used for recording the characteristic information of the data table corresponding to the rule to be operated;

correspondingly, the executing, by the target engine, the to-be-run rule for at least one target database includes:

executing, by the target engine, the rules to be run based on the engine data table.

4. The method of claim 3, wherein after the executing, by the target engine, the rules to be run against at least one of the target databases, the method further comprises:

if the condition that the rule to be operated is updated is detected according to a preset time interval, updating the engine data table based on the updated rule to be operated through the target engine;

and recording the updating time for updating the engine data table through the target engine.

5. The method according to claim 4, wherein if it is detected that the rule to be run is updated at a preset time interval, after updating the engine data table based on the updated rule to be run, the method further comprises:

executing the updated rule to be run based on the updated engine data table through the target engine.

6. The method of claim 3, wherein executing, by the target engine, the rules to be run based on the engine data table comprises:

generating an operation strategy by the target engine based on the rule to be operated and the engine data table;

generating, by the target engine, a target execution statement based on the operating policy;

and executing the target execution statement through the target engine so as to realize the execution of the rule to be run.

7. The method of claim 6, wherein generating an operation policy based on the rules to be operated and the engine data table comprises:

determining at least one first data table belonging to at least one database corresponding to the same virtual area under the same service area from the engine data table;

combining at least one first data table based on the rule to be operated to obtain a first strategy;

determining at least one second data table of at least one database corresponding to different virtual areas under different service areas from the engine data table;

combining at least one second data table based on the rule to be operated to obtain a second strategy;

determining that the service areas corresponding to the virtual areas only have 1 third data table from the engine data tables;

determining the third data table as a third strategy based on the rule to be operated; wherein the operating policies include the first policy, the second policy, and the third policy.

8. The method according to any one of claims 1 to 7, wherein after executing, by the target engine, the rules to be run against at least one of the target databases, the method further comprises:

outputting, by the target engine, an execution result; the execution result is obtained by the target engine executing the rule to be run;

if the target product is determined to have abnormity based on the execution result, determining at least one abnormal data rule through the target engine;

and executing an exception recovery operation by the target engine based on at least one exception data rule and a preset exception recovery mode.

9. The method of claim 8, wherein performing, by the target engine, an exception fix operation based on at least one of the exception data rules and a predetermined exception fix pattern comprises:

determining, by the target engine, at least one list object to which each of the anomalous data rules relates;

determining association rules of association relations of each list object from at least one target database through the target engine to obtain target association rules;

obtaining a rule to be repaired based on at least one abnormal data rule and the target association rule through the target engine;

and executing an abnormal repairing operation based on the rule to be repaired and the preset abnormal repairing mode through the target engine.

10. The method of claim 9, wherein obtaining, by the target engine, a rule to be repaired based on at least one of the anomalous data rule and the target association rule comprises:

grouping at least one abnormal data rule and the target association rule through the relation that the target engine is divided into a group according to rules belonging to the same service area, and obtaining the rule to be repaired.

11. The method according to claim 9 or 10, wherein the performing, by the target engine, an exception repair operation based on the rule to be repaired and the preset exception repair manner includes:

if the preset abnormal repairing mode is a number repairing mode, acquiring a preset repairing rule corresponding to the rule to be repaired through the target engine;

executing the preset repair rule through the target engine to realize the abnormal repair operation;

if the preset abnormal repairing mode is a cleaning mode, executing data cleaning operation corresponding to the rule to be repaired through the target engine; wherein the exception recovery operation comprises the data scrubbing operation;

if the preset abnormal repairing mode comprises the number repairing mode and the cleaning mode, acquiring the preset repairing rule corresponding to the rule to be repaired through the target engine;

and executing the data cleaning operation after the preset repair rule is executed by the target engine.

12. The method of claim 8, wherein after performing, by the target engine, an exception recovery operation based on at least one of the exception data rules and a predetermined exception recovery pattern, the method further comprises:

if the target product is monitored to be abnormal, the target engine repeatedly executes the step, namely, if the target product is determined to be abnormal based on the execution result, at least one abnormal data rule is determined by the target engine until the target product is detected to be normal, and the abnormal repairing operation is finished.

13. A monitoring device, characterized in that the device comprises: a memory, a processor, and a communication bus; wherein:

the memory to store executable instructions;

the processor, configured to execute the monitoring program stored in the memory, and implement the steps of the monitoring method according to any one of claims 1 to 12.

14. A storage medium, characterized in that the storage medium has stored thereon a monitoring program which, when executed by a processor, implements the steps of the monitoring method according to any one of claims 1 to 12.