CN113157671A

CN113157671A - Data monitoring method and device

Info

Publication number: CN113157671A
Application number: CN202110531825.6A
Authority: CN
Inventors: 吴华壮
Original assignee: WeBank Co Ltd
Current assignee: WeBank Co Ltd
Priority date: 2021-05-17
Filing date: 2021-05-17
Publication date: 2021-07-23

Abstract

The invention relates to the field of financial technology (Fintech), and discloses a data monitoring method and a device, which are used for inquiring a data table where a field to be monitored is located; the field to be monitored is obtained in a user configuration mode; acquiring data to be monitored corresponding to the field to be monitored from any data table containing the field to be monitored; acquiring a monitoring rule corresponding to a field to be monitored from a corresponding relation between a preset field to be monitored and the monitoring rule; and determining whether each data to be monitored is abnormal or not according to the monitoring rule corresponding to the field to be monitored. The method can automatically determine whether the data to be monitored is abnormal or not, and the monitored data is located in all data tables in the database, so the method is universal.

Description

Data monitoring method and device

Technical Field

The embodiment of the application relates to the field of financial technology (Fintech), in particular to a data monitoring method and device.

Background

With the development of computer technology, more and more technologies (such as big data, cloud computing or block chain) are applied in the financial field, the traditional financial industry is gradually shifting to financial technology, and in the context of big data technology, business data need to be stored based on a database.

Under the condition that the traffic is not large, namely when the business data stored in the database is not much, whether the business data are abnormal or not can be checked in a manual mode; however, the manual inspection of the abnormal business data is inefficient when the amount of business increases, and thus the inspection of the business data is not satisfactory.

In this regard, the service data to be checked may be monitored by developing and running a monitoring code of a specific rule. However, in this way of checking whether the service data is abnormal by running the monitoring code, only the data table and the field for the stock are valid, and when the data table in the system database is updated, the monitoring code is required to be updated synchronously, which is high in maintenance cost; in addition, due to the difference in development habits of developers, the modes of presentation of the same object in different business systems are not completely consistent, for example, the field names are different, so that the monitoring code developed for a certain object to be checked is difficult to be directly used in different business systems, that is, the monitoring code needs to be adapted and modified for use.

In summary, a general and automatic method for inspecting data is needed.

Disclosure of Invention

The application provides a data monitoring method and device, which are used for automatically checking the correctness of data in all tables of any object needing to be checked in a system database.

In a first aspect, an embodiment of the present application provides a data monitoring method, where the method includes: inquiring a data table where the field to be monitored is located; the field to be monitored is obtained in a user configuration mode; aiming at any data table containing the field to be monitored, acquiring the data to be monitored corresponding to the field to be monitored from the data table; acquiring a monitoring rule corresponding to a field to be monitored from a corresponding relation between a preset field to be monitored and the monitoring rule; and determining whether each data to be monitored is abnormal or not according to the monitoring rule corresponding to the field to be monitored.

Based on the scheme, the field to be monitored, the correctness of which needs to be checked, is set in a user configuration mode, when a data table where the field to be monitored is located is queried, the data to be monitored corresponding to the field to be monitored can be obtained, and then whether the data to be monitored is abnormal or not can be determined through the matched monitoring rule. The method can automatically determine whether the data to be monitored is abnormal or not, and the monitored data is located in all data tables in the database, so the method is universal.

In a possible implementation method, the determining whether each data to be monitored is abnormal according to a monitoring rule corresponding to the field to be monitored includes: determining whether the data to be monitored exists in a local cache or not aiming at any data to be monitored in the data to be monitored; if not, detecting the data to be monitored according to a monitoring rule corresponding to the field to be monitored to obtain a monitoring result; and storing the monitoring result of the data to be monitored into the local cache.

Based on the scheme, the data to be monitored is matched with the local cache, and when the data to be monitored exists in the local cache, a conclusion whether the data to be monitored is normal data can be quickly output; when the local cache does not store the data to be monitored, the data to be monitored is brand new data, the data to be monitored needs to be detected by applying a corresponding monitoring rule to obtain a monitoring result, and finally, the monitoring result is stored into the local cache, so that the conclusion that whether the data is normal data or not is rapidly output by matching the monitoring result in the local cache during subsequent detection of the data. According to the method, the monitoring result of the data to be monitored is stored in the local cache, so that the detection efficiency of the data to be monitored in the subsequent data detection process can be improved.

In a possible implementation method, the storing the monitoring result of the data to be monitored in the local cache includes: determining a hash value of the data to be monitored; storing the hash value of the data to be monitored and the monitoring result of the data to be monitored into the local cache in a key value pair mode; determining whether the data to be monitored exists in a local cache, including: determining a hash value of the data to be monitored; and determining whether the data to be monitored exists in a local cache or not according to the hash value of the data to be monitored.

Based on the scheme, after the data to be monitored is matched with the monitoring rule to obtain a corresponding monitoring result, the hash value of the data to be monitored is determined, and the hash value and the corresponding monitoring result can be stored in the local cache in a key value pair mode.

In a possible implementation method, determining whether the data to be monitored exists in a local cache, and storing a monitoring result of the data to be monitored in the local cache includes: and determining whether the data to be monitored exists in a local cache or not by using an @ Cacheable labeling method, and storing a monitoring result of the data to be monitored into the local cache.

Based on the scheme, when determining whether the data to be monitored exists in the local cache, the data to be monitored can be determined by the @ Cacheable labeling method, and the monitoring result is stored in the local cache.

In one possible implementation, the fields to be monitored include different field names for the same object; the data sheet where the field to be monitored is inquired comprises the following steps: and aiming at any field name, determining the table name of the data table containing the field name by inquiring the data table in the information _ schema mode.

Based on the scheme, the data table is inquired in the information _ schema mode, all data tables containing field names to be inquired in the current database can be inquired, the data tables not only comprise the stock data table, but also comprise the new data table, and the aim of inquiring all the data tables in the database can be achieved.

In a possible implementation method, the obtaining data to be monitored corresponding to the field to be monitored from the data table includes: constructing an SQL query statement according to the table name and the field name of the data table through a dynamic SQL function; and acquiring the data to be monitored corresponding to the field name from the data table according to the SQL query statement.

Based on the scheme, based on the dynamic SQL function, aiming at the table names and the field names of any group of data tables with corresponding relations, the corresponding SQL query statements can be dynamically constructed, so that the data to be monitored corresponding to the indicated fields can be obtained from the data tables indicated by the SQL statements by operating the SQL statements. The method is based on the dynamic SQL function, when any group of table names and field names of the data table with corresponding relations exists, the SQL sentences for acquiring the data to be monitored can be dynamically generated without dead writing of the query sentences, and the efficiency for acquiring the data to be monitored is improved.

In a possible implementation method, the obtaining a monitoring rule corresponding to the field to be monitored includes: and acquiring the monitoring rule corresponding to the field to be monitored in a getBean mode.

Based on the scheme, for a field to be monitored, the corresponding monitoring rule can be obtained in a getBean mode; in the subsequent data detection process, if the monitoring rule is adjusted, the updating requirement on the monitoring rule can be met by modifying the configuration mode, and the flexibility of the program is improved.

In a second aspect, an embodiment of the present application provides a data monitoring apparatus, including: the data table query unit is used for querying a data table where the field to be monitored is located; the field to be monitored is obtained in a user configuration mode; a to-be-monitored data acquisition unit, configured to acquire, from any data table including the to-be-monitored field, to-be-monitored data corresponding to the to-be-monitored field; the monitoring rule determining unit is used for acquiring a monitoring rule corresponding to a field to be monitored from a corresponding relation between a preset field to be monitored and the monitoring rule; and the judging unit is used for determining whether each data to be monitored is abnormal or not according to the monitoring rule corresponding to the field to be monitored.

In a possible implementation method, the determining unit is specifically configured to: determining whether the data to be monitored exists in a local cache or not aiming at any data to be monitored in the data to be monitored; if not, detecting the data to be monitored according to a monitoring rule corresponding to the field to be monitored to obtain a monitoring result; and storing the monitoring result of the data to be monitored into the local cache.

In a possible implementation of the method, the determining unit is further configured to: determining a hash value of the data to be monitored; storing the hash value of the data to be monitored and the monitoring result of the data to be monitored into the local cache in a key value pair mode; determining a hash value of the data to be monitored; and determining whether the data to be monitored exists in a local cache or not according to the hash value of the data to be monitored.

In a possible implementation of the method, the determining unit is further configured to: and determining whether the data to be monitored exists in a local cache or not by using an @ Cacheable labeling method, and storing a monitoring result of the data to be monitored into the local cache.

In one possible implementation, the fields to be monitored include different field names for the same object; the data table query unit is specifically configured to: and aiming at any field name, determining the table name of the data table containing the field name by inquiring the data table in the information _ schema mode.

In a possible implementation method, the unit for acquiring data to be monitored is specifically configured to: constructing an SQL query statement according to the table name and the field name of the data table through a dynamic SQL function; and acquiring the data to be monitored corresponding to the field name from the data table according to the SQL query statement.

In a possible implementation method, the monitoring rule determining unit is specifically configured to: and acquiring the monitoring rule corresponding to the field to be monitored in a getBean mode.

In a third aspect, an embodiment of the present application provides a computing device, including:

a memory for storing program instructions;

and the processor is used for calling the program instructions stored in the memory and executing the implementation method of the first aspect according to the obtained program.

In a fourth aspect, the present application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the method according to any one of the first aspects.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.

Fig. 1 is a data monitoring method according to an embodiment of the present disclosure;

fig. 2 is a schematic diagram illustrating a result of obtaining a data table in an information _ schema mode according to an embodiment of the present application;

fig. 3 is a schematic diagram of a dynamic SQL pieced query statement provided in an embodiment of the present application;

fig. 4 is a schematic diagram of an acquisition process of data to be monitored according to an embodiment of the present application;

fig. 5 is a data monitoring apparatus according to an embodiment of the present application;

fig. 6 is a computing device provided in an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be described in further detail with reference to the accompanying drawings, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

At present, the business data stored in the database is mainly checked manually, and obviously, the method is extremely inefficient when the business data continuously rises. The method is not universal, and cannot be applied to all data tables in a database, and the monitoring code of the same object is difficult to be directly applied to different business systems.

In view of the above technical problem, an embodiment of the present application provides a data monitoring method, as shown in fig. 1, the method includes the following steps:

step 101, inquiring a data table where a field to be monitored is located; the field to be monitored is obtained in a user configuration mode.

In this step, for an enterprise, different business departments often intersect with each other for recording the content of the user information, for example, different business departments may record an object, i.e., a mobile phone number of a user, and for example, different business departments may record an object, i.e., an identity number of a user, at the same time. In the embodiment of the present application, different business departments record the user identification number as an example. In the operation process of an enterprise, since technicians responsible for different business departments are different, it is difficult to ensure consistency of expression modes of the same object in data tables respectively corresponding to the business departments, for example, regarding an object of a user identification number, it may have different expression modes in different data tables, for example, it may be recorded as various forms, i.e. fields, such as 'left _ cert _ id', 'cert _ id', etc. For the object of the user identity card number, the representation modes of each business department in the enterprise about the user identity card number can be collected in advance, and then the various representation modes of the object of the user identity card number are configured, for example, the fields of 'left _ cert _ id' and 'cert _ id' which are configured in advance can be used for representing the user identity card number. As another example, the object related to the user's mobile phone number may have different representations in different data tables, such as various forms, i.e. fields, which may be described as ' phone ', ' phone no ', ' mobile '. Based on the same configuration logic as the user identity card number, for the object of the user mobile phone number, the representation modes of each business department in the enterprise can be collected in advance, and then the various representation modes of the object of the user mobile phone number are configured, for example, a 'phone' field, a 'phone _ no' field and a 'mobile' field which are configured in advance can be used for representing the user mobile phone number.

In the embodiment of the present application, an example of monitoring an object, which is a user identification number, is described, and a field to be monitored is an expression manner of the object, which is the user identification number, in a data table corresponding to each business department in an enterprise. Based on the foregoing example, when monitoring an object, i.e. a user identification number, the field to be monitored may include a 'legacy _ cert _ id' field and a 'cert _ id' field, which are configured in advance by a user. After obtaining the field to be monitored, the data table with the field to be monitored may be queried based on the field to be monitored, such as querying the data table with the field 'legacy _ cert _ id' and querying the data table with the field 'cert _ id'.

And 102, acquiring data to be monitored corresponding to the field to be monitored from any data table containing the field to be monitored.

In this step, based on the data table with the field to be monitored being queried in step 101, the data corresponding to the field to be monitored can be obtained from the data table, that is, the data to be monitored is obtained.

Next, in the foregoing example of monitoring the user identity card number, for example, when the user identity card number is queried based on the 'legacy _ cert _ id' field, the data table a and the data table B in the system database both have the field, which indicates that the data to be monitored corresponding to the 'legacy _ cert _ id' field can be obtained from the data table a and the data table B, respectively; meanwhile, when the user identity card number is queried based on the 'cert _ id' field, only the data table C in the system database has the field, which indicates that the data to be monitored corresponding to the 'cert _ id' field can be obtained from the data table C.

Step 103, acquiring a monitoring rule corresponding to the field to be monitored from a preset corresponding relation between the field to be monitored and the monitoring rule.

Due to the difference of the objects of interest, there will be a difference in the monitoring rules for monitoring the objects of interest. Specifically, different fields to be monitored may also have different monitoring rules for monitoring the fields to be monitored. For example, when the concerned object is an enterprise certificate number, that is, when each field to be monitored of the object, namely the enterprise certificate number, needs to be monitored, the monitoring rule can be whether the format is correct or not; when the concerned object is the user identification number, namely when each field to be monitored of the object of the user identification number needs to be monitored, the monitoring rule can be whether the length is correct or not; when the concerned object is the borrowing amount, the borrowing interest rate and the borrowing period, namely when the object of the borrowing amount, the borrowing interest rate and the borrowing period needs to be monitored, the monitoring rule can be whether the monitoring rule accords with the specified monitoring interval or not; when the object of interest is a product number, that is, when each field to be monitored of the object of the product number needs to be monitored, the monitoring rule may be whether the range of enumeration is met. The above monitoring of the field to be monitored may be referred to as a general check, wherein the general check may include checking the format, length, size, enumeration, and so on.

In this step, when the concerned object is the user identity card number, that is, the field to be monitored is the 'legacy _ cert _ id' field and the 'cert _ id' field, the corresponding monitoring rule may be to check whether the length of the identity card number belongs to 15-18 bits. Furthermore, different monitoring rule identifications can be set for different monitoring rules, so that the monitoring rules are distinguished. For example, in the embodiment of the present application, any monitoring rule and a monitoring rule identifier corresponding to the monitoring rule may be configured in a rule configuration table in advance, so as to be used for querying the monitoring rule by the data monitoring process. If the monitoring rule identifier of the monitoring rule corresponding to each field to be monitored, which is the object of the user identity card number, can be set as: ruleId 00001. For another example, when the attention object is a mobile phone number of the user, that is, when the field to be monitored is a 'phone' field, a 'phone _ no' field, and a 'mobile' field, the corresponding monitoring rule may be to check whether the mobile phone number is a pure number, and to record the monitoring rule in the rule configuration table; in addition, the rule configuration table can also set the monitoring rule identification of the monitoring rule corresponding to each field to be monitored, which is the object of the user mobile phone number, as: ruleId 00002. It is noted that, the setting processes of the monitoring rule identifiers corresponding to other objects of interest are not listed one by one here.

And step 104, determining whether each data to be monitored is abnormal or not according to the monitoring rule corresponding to the field to be monitored.

In this step, based on the monitoring rule obtained in step 103, the data to be monitored obtained in step 102 may be checked to determine whether the data to be monitored is abnormal.

Next, in the foregoing example of monitoring the user identification number, since the object that needs the user identification number is monitored this time, the corresponding monitoring rule identifier is ruleId of 00001; therefore, by querying the preset rule configuration table, it can be known from the corresponding relationship between the ruleId 00001 recorded therein and the monitoring rule that whether the length of each user identity card number corresponding to each of the 'Legal _ cert _ id' field in the data table A, the 'Legal _ cert _ id' field in the data table B, and the 'cert _ id' field in the data table C is 15-18 bits is required to be checked this time. Wherein, the verification result may include:

case 1: if the length of the user identification number meets 15-18 bits, determining that the record of the user identification number is normal, and executing the verification of the next user identification number;

case 2: if the length of the user identity card number does not meet 15-18 bits, determining that the record of the user identity number is abnormal, and notifying service personnel to check the situation based on a preset alarm mode. The alarm mode can be obtained through a mode configured by a user, and comprises various modes such as an email mode, a short message mode, a WeChat mode and the like.

Some of the above steps will be described in detail with reference to examples.

In one implementation of the above step 101, the fields to be monitored include different field names for the same object; the data sheet where the field to be monitored is inquired comprises the following steps: and aiming at any field name, determining the table name of the data table containing the field name by inquiring the data table in the information _ schema mode.

In the embodiment of the application, the time point for monitoring each to-be-monitored data corresponding to the to-be-monitored field may be set, for example, monitoring may be performed every 1 minute, or monitoring may be performed every 10 minutes, and the specific time may be set according to the actual requirements of technicians. After the monitoring system is started, by reading the monitoring range configured by the user this time, for example, the monitoring range configured by the user this time monitors an object, which is the user identification number, and when the object, which is the user identification number, can be represented as 'legacy _ cert _ id' and 'cert _ id', it indicates that the fields to be monitored can be read as 'legacy _ cert _ id' and 'cert _ id'.

The monitoring system can dynamically acquire all related data tables including the latest online data table by inquiring the COLUMNS table in the MYSQL database information _ schema mode according to the field to be monitored. Fig. 2 is a schematic diagram illustrating a result of obtaining a data table in information _ schema according to an embodiment of the present application. Wherein, data tables with table names "entry _ info" and "white _ info" can be obtained according to the 'left _ start _ id' field, and data tables with table names "entry _ staff _ bind" can be obtained according to the 'start _ id' field.

Illustratively, the information _ schema mode: the MYSQL system self-contained database provides a method for accessing metadata of the database. COLUMNS Table: in the information _ schema mode, a table of all entity objects in the database is stored. All TABLE NAMEs TABLE _ NAME and fields COLUMN _ NAME of the current database can be obtained.

In some implementations of the present application, the obtaining the data to be monitored corresponding to the field to be monitored from the data table includes: constructing an SQL query statement according to the table name and the field name of the data table through a dynamic SQL function; and acquiring the data to be monitored corresponding to the field name from the data table according to the SQL query statement.

Following the foregoing example, data tables with table names "respectively" entry _ info "and" white _ info "may be queried based on the 'left _ start _ id' field, and data tables with table names" entry _ start _ bind "may be queried based on the 'left _ id' field, so that according to the dynamic SQL hash query statement of the mybases framework, a record corresponding to the 'left _ start _ id' field in the" entry _ info "data table, a record corresponding to the 'left _ start _ id' field in the" white _ info "data table, and a record corresponding to the 'center _ id' field in the" entry _ start _ bind "data table, that is, data to be monitored, may be obtained respectively. As shown in fig. 3, a schematic diagram of a dynamic SQL makeup query statement provided in this embodiment of the application is provided, wherein a variable of { } includes < 'left _ start _ id', entry _ info >, < 'start _ id', entry _ status _ bind >, < 'left _ start _ id', and white _ info > for correspondence between the aforementioned 3 sets of table fields and table names, and can be substituted for the table fields and table names of each set one by one; therefore, for each group of table fields and table names with corresponding relations, a group of data to be monitored can be obtained, and then the data to be monitored can be output to a file to be checked and wait for checking. As shown in fig. 4, a schematic diagram of an obtaining process of data to be monitored according to an embodiment of the present application is provided, where for a 'legacy _ start _ id' field in an "legacy _ info" data table, a 'start _ id' field in an "legacy _ staff _ bind" data table, and a 'legacy _ start _ id' field in a "white _ info" data table, the fields are respectively replaced into a 'SELECT $ { columns } FROM $ { table }' statement, and three data tables can be respectively obtained, where the obtained three data tables are respectively shown as excel tables in fig. 4.

In an implementation of the foregoing step 104, the determining whether each data to be monitored is abnormal according to the monitoring rule corresponding to the field to be monitored includes: determining whether the data to be monitored exists in a local cache or not aiming at any data to be monitored in the data to be monitored; if not, detecting the data to be monitored according to a monitoring rule corresponding to the field to be monitored to obtain a monitoring result; and storing the monitoring result of the data to be monitored into the local cache.

Based on the foregoing example, the records corresponding to the 'left _ cert _ id' field in the "entry _ info" data table, the records corresponding to the 'left _ cert _ id' field in the "white _ info" data table, and the records corresponding to the 'cert _ id' field in the "entry _ staff _ bind" data table, that is, three pieces of data to be monitored, are obtained, each piece of data to be monitored is made into one piece of file to be checked, and then the data to be monitored in the file to be checked is monitored. In the embodiment of the present application, the files to be verified corresponding to the three pieces of data to be monitored can be referred to as file a, file B, and file C in sequence. Imagine the following: setting a file A which comprises 1000W identification numbers; setting a file B which comprises 800W identification numbers; let it be assumed that file C includes 500W identification numbers.

The current method for monitoring data to be monitored comprises the following steps: and analyzing and acquiring the user identity card number of any file, monitoring the user identity card numbers in the file one by applying a pre-written identity card number monitoring rule, and outputting a corresponding monitoring result. Obviously, for the three pieces of data to be monitored, in total 2300W user identification numbers, 2300W identification number monitoring rules need to be executed.

With the above-described method of monitoring data to be monitored at present, there will be at least two problems:

problem 1: performance issues. A user identification number is in the same system and may exist in a plurality of data tables. For a scene with a large amount of repeated data, if the monitoring rule is executed repeatedly, the system resource is wasted greatly.

Problem 2: monitoring is carried out by using a pre-compiled identity card number monitoring rule according to the identity card number of the user, and if the monitoring rule is adjusted or expanded in the subsequent process, the monitoring rule is not flexible enough.

In view of the above problem 1, in the embodiment of the present application, a monitoring result caching manner is used, that is, a monitoring rule compiled in advance is applied for the first time to monitor to generate data to be monitored of a monitoring result and cache the monitoring result; then, in the subsequent data monitoring process, for any piece of data to be monitored, the piece of data to be monitored can be firstly compared with the cached monitoring result, and if the piece of data to be monitored can be matched in the cache, the monitoring result of the piece of data to be monitored is directly output; if the data to be monitored cannot be matched in the cache, the data to be monitored is a brand-new data to be monitored relative to the cache, the pre-programmed monitoring rule can be used for monitoring to obtain a monitoring result, and the data to be monitored and the monitoring result are recorded into the cache.

In some implementations of the present application, storing the monitoring result of the data to be monitored in the local cache includes: determining a hash value of the data to be monitored; storing the hash value of the data to be monitored and the monitoring result of the data to be monitored into the local cache in a key value pair mode; determining whether the data to be monitored exists in a local cache, including: determining a hash value of the data to be monitored; and determining whether the data to be monitored exists in a local cache or not according to the hash value of the data to be monitored.

In some implementations of the present application, the determining whether the data to be monitored exists in a local cache, and storing the monitoring result of the data to be monitored in the local cache includes: and determining whether the data to be monitored exists in a local cache or not by using an @ Cacheable labeling method, and storing a monitoring result of the data to be monitored into the local cache.

For example, in the embodiment of the present application, Caffeine may be used as the local cache, where Caffeine is a high-performance Java cache library, and the data structure of the cache is a MAP structure, that is, a form of < key, value >. For any piece of data to be monitored, the hash value of the data to be monitored and the monitoring rule identifier of the monitoring rule corresponding to the data to be monitored is used as a key value, and a monitoring result obtained after the data to be monitored is processed by the monitoring rule is used as a value.

Specifically, the description can be made by taking the examples of the file a, the file B, and the file C in the foregoing examples:

firstly, setting the first file to be a file A, and acquiring the user identification number of each piece. Aiming at any one user identity card number, the user identity card number and the hash value of the monitoring rule identifier are determined, and whether the hash value hits a local cache or not can be automatically judged through the annotation @ Cacheable method of the Spring Boot framework. Since the file a is the first file in this verification, all the user identification numbers in the file a appear for the first time, and obviously do not hit the local cache (it can be understood that the local cache is empty at this time), the user identification number of each of the files a needs to enter a specific business method, that is, any user identification number in the file a needs to be processed by applying a pre-programmed monitoring rule, and the user identification number and the monitoring result thereof are stored in the local cache, more specifically, the user identification number and the ruleId are subjected to hash calculation, the hash value obtained by the calculation is stored in the local cache as a key value, and the corresponding monitoring result is stored in the local cache as a value.

Then, the second file to be analyzed is set as file B, and the user identification number of each piece is obtained. Aiming at any user identity card number, determining the user identity card number and a hash value identified by a monitoring rule, and automatically judging whether the hash value hits a local cache or not through an annotation @ Cacheable method of a Spring Boot framework. Before monitoring the user identity card number in the file B, monitoring the user identity card number in the file A, and storing all the user identity card numbers and corresponding monitoring results in the file A into a local cache, so that whether the user identity card number in the file B hits the local cache can be determined by an annotation @ cache method of a Spring book frame for the user identity card number in the file B, if so, directly taking the monitoring result corresponding to the user identity card number in the local cache as the monitoring result of the user identity card number in the file B and outputting the monitoring result without using a pre-programmed monitoring rule for processing; if the user identity card number in the file B is not hit, the user identity card number in the file B is a brand-new user identity card number relative to the local cache, at this time, the user identity card number needs to be processed by applying a pre-programmed monitoring rule, so that a monitoring result about the user identity card number is output, and the user identity card number and the monitoring result are stored in the local cache.

Finally, the method for monitoring the user identification number in the file C may refer to the file B, which is not described herein.

In some implementations of the present application, the obtaining of the monitoring rule corresponding to the field to be monitored includes: and acquiring the monitoring rule corresponding to the field to be monitored in a getBean mode.

Next, in the foregoing example of monitoring the user identity number, the Service method may query the rule configuration table according to a monitoring rule identifier, for example, rule id is 0001, obtain a rule Service corresponding to rule id 0001, and obtain a specific Service object in a getpeer manner of an ApplicationContext object in a Spring framework of JAVA.

It is noted that, for the foregoing problem 2, through the configuration rule configuration table and the dynamic getteam mode, if the subsequent monitoring rule is adjusted or expanded, the configuration of the getteam mode may be dynamically increased to meet the changing requirement of the monitoring rule. For example, as the number of the population increases, the existing user identity numbers with the length of 15-18 bits cannot meet the requirement of population growth at a certain time point in the future, and the user identity numbers with the length of 19-22 bits can be started for some new population to overcome the problem of insufficient distribution of the user identity numbers. Therefore, the monitoring rule for monitoring the data to be monitored needs to be updated adaptively, that is, a new monitoring rule needs to be added on the basis of the existing monitoring rule. At this time, in order not to cause erroneous monitoring of the data to be monitored, a new monitoring rule may be added on the basis of the existing monitoring rule retained in the rule configuration table, and the configuration of the getteam mode is adapted to be added. Therefore, when the monitoring rule changes, the method only needs to modify the configuration, and the flexibility of the program is improved.

Based on the same concept, the embodiment of the present application provides a data monitoring apparatus, as shown in fig. 5, the apparatus includes a data table query unit 501, a data to be monitored acquisition unit 502, a monitoring rule determination unit 503, and a determination unit 504;

a data table query unit 501, configured to query a data table where a field to be monitored is located; the field to be monitored is obtained in a user configuration mode.

A to-be-monitored data obtaining unit 502, configured to obtain, from any data table including the to-be-monitored field, to-be-monitored data corresponding to the to-be-monitored field.

The monitoring rule determining unit 503 is configured to obtain a monitoring rule corresponding to a field to be monitored from a preset corresponding relationship between the field to be monitored and the monitoring rule.

A determining unit 504, configured to determine whether each to-be-monitored data is abnormal according to a monitoring rule corresponding to the to-be-monitored field.

Further, for the apparatus, the determining unit 504 is specifically configured to: determining whether the data to be monitored exists in a local cache or not aiming at any data to be monitored in the data to be monitored; if not, detecting the data to be monitored according to a monitoring rule corresponding to the field to be monitored to obtain a monitoring result; and storing the monitoring result of the data to be monitored into the local cache.

Further to this apparatus, the determining unit 504 is further configured to: determining a hash value of the data to be monitored; storing the hash value of the data to be monitored and the monitoring result of the data to be monitored into the local cache in a key value pair mode; determining a hash value of the data to be monitored; and determining whether the data to be monitored exists in a local cache or not according to the hash value of the data to be monitored.

Further to this apparatus, the determining unit 504 is further configured to: and determining whether the data to be monitored exists in a local cache or not by using an @ Cacheable labeling method, and storing a monitoring result of the data to be monitored into the local cache.

Further, for the apparatus, the fields to be monitored comprise different field names for the same object; the data table querying unit 501 is specifically configured to: and aiming at any field name, determining the table name of the data table containing the field name by inquiring the data table in the information _ schema mode.

Further, for the apparatus, the data to be monitored obtaining unit 502 is specifically configured to: constructing an SQL query statement according to the table name and the field name of the data table through a dynamic SQL function; and acquiring the data to be monitored corresponding to the field name from the data table according to the SQL query statement.

Further, for the apparatus, the monitoring rule determining unit 503 is specifically configured to: and acquiring the monitoring rule corresponding to the field to be monitored in a getBean mode.

The embodiment of the present application further provides a computing device, which may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, a Personal Digital Assistant (PDA), and the like. The computing device may include a Central Processing Unit (CPU), memory, input/output devices, etc., the input devices may include a keyboard, mouse, touch screen, etc., and the output devices may include a Display device, such as a Liquid Crystal Display (LCD), a Cathode Ray Tube (CRT), etc.

Memory, which may include Read Only Memory (ROM) and Random Access Memory (RAM), provides the processor with program instructions and data stored in the memory. In an embodiment of the present application, the memory may be configured to store program instructions of a data monitoring method;

and the processor is used for calling the program instructions stored in the memory and executing the data monitoring method according to the obtained program.

As shown in fig. 6, a schematic diagram of a computing device provided in an embodiment of the present application includes:

a processor 601, a memory 602, a transceiver 603, a bus interface 604; the processor 601, the memory 602 and the transceiver 603 are connected by a bus 605;

the processor 601 is configured to read a program in the memory 602 and execute the data monitoring method;

the processor 601 may be a Central Processing Unit (CPU), a Network Processor (NP), or a combination of a CPU and an NP. But also a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a field-programmable gate array (FPGA), a General Array Logic (GAL), or any combination thereof.

The memory 602 is used for storing one or more executable programs, and may store data used by the processor 601 in performing operations.

In particular, the program may include program code including computer operating instructions. The memory 602 may include a volatile memory (volatile memory), such as a random-access memory (RAM); the memory 602 may also include a non-volatile memory (non-volatile memory), such as a flash memory (flash memory), a Hard Disk Drive (HDD) or a solid-state drive (SSD); the memory 602 may also comprise a combination of memories of the kind described above.

The memory 602 stores the following elements, executable modules or data structures, or a subset thereof, or an expanded set thereof:

and (3) operating instructions: including various operational instructions for performing various operations.

Operating the system: including various system programs for implementing various basic services and for handling hardware-based tasks.

The bus 605 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 6, but this is not intended to represent only one bus or type of bus.

The bus interface 604 may be a wired communication access port, a wireless bus interface, or a combination thereof, wherein the wired bus interface may be, for example, an ethernet interface. The ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless bus interface may be a WLAN interface.

The embodiment of the application also provides a computer-readable storage medium, which stores computer-executable instructions for enabling a computer to execute the data monitoring method.

It will be apparent to those skilled in the art that embodiments of the present application may be provided as a method, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims

1. A method for monitoring data, comprising:

inquiring a data table where the field to be monitored is located; the field to be monitored is obtained in a user configuration mode;

aiming at any data table containing the field to be monitored, acquiring the data to be monitored corresponding to the field to be monitored from the data table;

acquiring a monitoring rule corresponding to a field to be monitored from a corresponding relation between a preset field to be monitored and the monitoring rule;

and determining whether each data to be monitored is abnormal or not according to the monitoring rule corresponding to the field to be monitored.

2. The method of claim 1,

determining whether each data to be monitored is abnormal according to the monitoring rule corresponding to the field to be monitored, including:

determining whether the data to be monitored exists in a local cache or not aiming at any data to be monitored in the data to be monitored; if not, detecting the data to be monitored according to a monitoring rule corresponding to the field to be monitored to obtain a monitoring result; and storing the monitoring result of the data to be monitored into the local cache.

3. The method of claim 2,

the storing the monitoring result of the data to be monitored into the local cache includes:

determining a hash value of the data to be monitored;

storing the hash value of the data to be monitored and the monitoring result of the data to be monitored into the local cache in a key value pair mode;

determining whether the data to be monitored exists in a local cache, including:

determining a hash value of the data to be monitored;

and determining whether the data to be monitored exists in a local cache or not according to the hash value of the data to be monitored.

4. The method of claim 2,

the determining whether the data to be monitored exists in a local cache and storing the monitoring result of the data to be monitored into the local cache includes:

and determining whether the data to be monitored exists in a local cache or not by using an @ Cacheable labeling method, and storing a monitoring result of the data to be monitored into the local cache.

5. The method of any of claims 1-4, wherein the fields to be monitored include different field names for the same object;

the data sheet where the field to be monitored is inquired comprises the following steps:

and aiming at any field name, determining the table name of the data table containing the field name by inquiring the data table in the information _ schema mode.

6. The method of claim 5,

the acquiring the data to be monitored corresponding to the field to be monitored from the data table includes:

constructing an SQL query statement according to the table name and the field name of the data table through a dynamic SQL function;

and acquiring the data to be monitored corresponding to the field name from the data table according to the SQL query statement.

7. The method of claim 1,

the acquiring of the monitoring rule corresponding to the field to be monitored includes:

and acquiring the monitoring rule corresponding to the field to be monitored in a getBean mode.

8. A data monitoring device, comprising:

the data table query unit is used for querying a data table where the field to be monitored is located; the field to be monitored is obtained in a user configuration mode;

a to-be-monitored data acquisition unit, configured to acquire, from any data table including the to-be-monitored field, to-be-monitored data corresponding to the to-be-monitored field;

the monitoring rule determining unit is used for acquiring a monitoring rule corresponding to a field to be monitored from a corresponding relation between a preset field to be monitored and the monitoring rule;

and the judging unit is used for determining whether each data to be monitored is abnormal or not according to the monitoring rule corresponding to the field to be monitored.

9. A computer device, comprising:

a memory for storing a computer program;

a processor for calling a computer program stored in said memory, for executing the method according to any one of claims 1-7 in accordance with the obtained program.

10. A computer-readable storage medium having stored thereon computer-executable instructions for causing a computer to perform the method of any one of claims 1-7.