CN111367754A - Data monitoring method and device, computer equipment and storage medium - Google Patents
Data monitoring method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN111367754A CN111367754A CN202010092311.0A CN202010092311A CN111367754A CN 111367754 A CN111367754 A CN 111367754A CN 202010092311 A CN202010092311 A CN 202010092311A CN 111367754 A CN111367754 A CN 111367754A
- Authority
- CN
- China
- Prior art keywords
- monitoring
- data
- monitoring result
- rule
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
- G06F11/0775—Content or structure details of the error report, e.g. specific table structure, specific error fields
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/079—Root cause analysis, i.e. error or fault diagnosis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0793—Remedial or corrective actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of system data monitoring, and discloses a data monitoring method, a device, computer equipment and a storage medium, wherein the method comprises the following steps: configuring a monitoring rule in an Oracle database, wherein the monitoring rule is used for monitoring a specified data index; sending the monitoring rule to a Hive database; respectively acquiring monitoring results in an Oracle database and a Hive database according to monitoring rules; calculating the difference degree according to the monitoring result; if the difference degree is larger than a preset alarm threshold value, inputting a monitoring result into a preset monitoring result processing model; acquiring error reasons output by a preset monitoring result processing model; acquiring a data recovery script adapted to the error reason; and repairing the data of the Oracle database and/or Hive database according to the data repairing script. The invention can improve the processing efficiency and timeliness of cross-database data monitoring between the Hive database and the Oracle database, and can realize automatic repair of error data.
Description
Technical Field
The present invention relates to the field of system data monitoring, and in particular, to a data monitoring method, apparatus, computer device, and storage medium.
Background
Data monitoring has very important significance for data operation. The data monitoring can find the abnormity existing in the data calculation process in time and can also find the abnormity existing in the data calculation result in time, so that the intervention can be performed in time when the data abnormity occurs, and the existing abnormity can be repaired.
Traditional data monitoring can only monitor data on a single database or simply compare two structurally similar databases. If the database with large structural difference is monitored by crossing databases for operation data, such as a Hive database and an Oracle database, the monitoring data of the databases need to be respectively counted and then exported for processing comparison, more manual work is needed, and the processing efficiency and the timeliness are poor. If the abnormal data among the databases cannot be processed in time, serious adverse consequences can occur.
Disclosure of Invention
Therefore, it is necessary to provide a data monitoring method, an apparatus, a computer device and a storage medium for solving the above technical problems, so as to improve the processing efficiency and timeliness of cross-database data monitoring between the Hive database and the Oracle database.
A method of data monitoring, comprising:
configuring a monitoring rule in an Oracle database, wherein the monitoring rule is used for monitoring a specified data index;
sending the monitoring rule to a Hive database;
acquiring a first monitoring result in a specified time range in the Oracle database according to the monitoring rule; acquiring a second monitoring result in the appointed time range in the Hive database according to the monitoring rule;
importing the second monitoring result into the Oracle database, and calculating the difference degree between the first monitoring result and the second monitoring result;
if the difference degree is larger than a preset alarm threshold value, inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model;
acquiring error reasons output by the preset monitoring result processing model;
acquiring a data recovery script adapted to the error reason;
and repairing the data of the Oracle database and/or the Hive database according to the data repairing script.
A data monitoring apparatus, comprising:
the rule configuration module is used for configuring a monitoring rule in an Oracle database, and the monitoring rule is used for monitoring a specified data index;
the rule sending module is used for sending the monitoring rule to a Hive database;
the monitoring result acquisition module is used for acquiring a first monitoring result in a specified time range in the Oracle database according to the monitoring rule; acquiring a second monitoring result in the appointed time range in the Hive database according to the monitoring rule;
the difference degree calculation module is used for importing the second monitoring result into the Oracle database and calculating the difference degree of the first monitoring result and the second monitoring result;
the model analysis module is used for inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model if the difference degree is larger than a preset alarm threshold value;
the model output module is used for acquiring the preset monitoring result and processing the error reason output by the model;
the acquisition and repair script module is used for acquiring the data repair script matched with the error reason;
and the data repairing module is used for repairing the data of the Oracle database and/or the Hive database according to the data repairing script.
A computer device comprising a memory, a processor and a computer program stored in said memory and executable on said processor, said processor implementing the above data monitoring method when executing said computer program.
A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the above-mentioned data monitoring method.
According to the data monitoring method, the data monitoring device, the computer equipment and the storage medium, monitoring rules are configured in the Oracle database, and the monitoring rules are used for monitoring specified data indexes so as to define parameters to be monitored in a user-defined mode. And sending the monitoring rule to a Hive database to synchronize the monitoring rule. Acquiring a first monitoring result in a specified time range in the Oracle database according to the monitoring rule; and acquiring a second monitoring result in the appointed time range in the Hive database according to the monitoring rule so as to acquire the monitoring results of the two databases. And importing the second monitoring result into the Oracle database, and calculating the difference degree of the first monitoring result and the second monitoring result so as to compare the difference of the two monitoring results. And if the difference degree is greater than a preset alarm threshold value, inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model so as to automatically analyze abnormal monitoring results. And acquiring the error reasons output by the preset monitoring result processing model so as to determine the reasons of data errors in the database. And acquiring a data repair script adapted to the error reason so as to automatically match a repair scheme and improve the data repair efficiency. And repairing the data of the Oracle database and/or the Hive database according to the data repairing script so as to realize automatic repairing of error data. The data monitoring method provided by the invention improves the processing efficiency and timeliness of cross-database data monitoring between the Hive database and the Oracle database, and can realize automatic repair of error data.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments of the present invention will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive labor.
FIG. 1 is a diagram illustrating an application environment of a data monitoring method according to an embodiment of the present invention;
FIG. 2 is a flow chart of a data monitoring method according to an embodiment of the invention;
FIG. 3 is a flow chart of a data monitoring method according to an embodiment of the invention;
FIG. 4 is a flow chart of a data monitoring method according to an embodiment of the invention;
FIG. 5 is a flow chart of a data monitoring method according to an embodiment of the invention;
FIG. 6 is a flow chart of a data monitoring method according to an embodiment of the invention;
FIG. 7 is a flow chart of a data monitoring method according to an embodiment of the invention;
FIG. 8 is a schematic diagram of a data monitoring apparatus according to an embodiment of the present invention;
FIG. 9 is a schematic diagram of a computer device according to an embodiment of the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The data monitoring method provided by the embodiment can be applied to the application environment shown in fig. 1, in which the client communicates with the server through a network. The client includes, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of a plurality of servers.
In an embodiment, as shown in fig. 2, a data monitoring method is provided, which is described by taking the application of the method to the server in fig. 1 as an example, and includes the following steps:
s10, configuring a monitoring rule in an Oracle database, wherein the monitoring rule is used for monitoring a specified data index;
s20, sending the monitoring rule to a Hive database;
s30, acquiring a first monitoring result in a specified time range in the Oracle database according to the monitoring rule; acquiring a second monitoring result in the appointed time range in the Hive database according to the monitoring rule;
s40, importing the second monitoring result into the Oracle database, and calculating the difference degree between the first monitoring result and the second monitoring result;
s50, if the difference degree is larger than a preset alarm threshold value, inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model;
s60, acquiring the error reason output by the preset monitoring result processing model;
s70, acquiring a data recovery script adapted to the error reason;
and S80, repairing the data of the Oracle database and/or the Hive database according to the data repairing script.
The data monitoring method provided by the embodiment of the invention is used for monitoring the data of the databases with different structure types. Particularly, the method can be used for data monitoring between the Oracle database and the Hive database. Oracle is short for Oracle Database, also known as Oracle RDBMS, a relational Database management system of Oracle Inc. Hive is a data warehouse tool based on Hadoop, can map structured data files into a database table, provides a simple sql query function, and can convert sql statements into MapReduce tasks for operation. The data storage structures of the Oracle database and the Hive database are greatly different.
The monitoring rule is used for determining a data object to be monitored, a specified data index and a monitoring mode. For example, the monitoring rules may set data tables on key process links as the monitored data objects. The data object can refer to service data to be monitored by an Oracle database and a Hive database. The specified data index may refer to parameters of the data object, such as a library name, a table name, a business rule name, a dimension name, an index name, and the like. The monitoring mode may be monitoring according to a certain time period, such as daily, every three days, etc. Here, on the server running the Oracle database, a corresponding monitoring platform may be provided, which is used to configure the monitoring rules. For the same data table, different monitoring rules can be set, or the monitoring rule combination can be used for monitoring the data of the Oracle database.
After the monitoring rules are configured, the monitoring rules can be synchronized to a server running the Hive database through the monitoring platform.
After synchronization is completed, a timing task can be set, the monitoring rule is read periodically, a corresponding monitoring task is generated, the monitoring task is executed, and a corresponding monitoring result is obtained. For example, a first monitoring result may be generated in an Oracle database, and the first monitoring result may be monitoring data obtained after monitoring a plurality of specified data indexes on the day of 8/7/2019; a second monitoring result may be generated in the Hive database, and the second monitoring result may be monitoring data obtained after monitoring the plurality of specified data indicators on the day of 8/7 in 2019.
After the Hive database generates a second monitoring result, the second monitoring result can be sent to a server running an Oracle database, then the second monitoring result is imported into the Oracle database, the first monitoring result and the second monitoring result are compared, and the difference degree is calculated. In a normal case, the first monitoring result and the second monitoring result are the same, and thus the degree of difference is zero. And under the abnormal condition, the first monitoring result and the second monitoring result have difference, and the difference degree is not zero.
The preset alarm threshold may be set according to actual needs, for example, the preset alarm threshold may be set to zero. And comparing the first monitoring result with the second monitoring result, calculating the difference between the first monitoring result and the second monitoring result, and taking the absolute value as the difference. Thus, the degree of difference between the first monitoring result and the second monitoring result may be a value not less than zero.
And when the difference degree is greater than a preset alarm threshold value, inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model so as to judge the error reason of the monitoring result. The preset monitoring result processing model can be constructed based on historical monitoring result data, and the data error reason and the importance of specific data can be accurately positioned. After the processing of the preset monitoring result processing model, the error reason and the error grade (related to the importance of the data with errors) of the monitoring result corresponding to the current monitoring behavior can be obtained.
A plurality of data recovery scripts can be preset and adapted to different error reasons. When the error reason output by the preset monitoring result processing model has the data repair script matched with the error reason, the matched data repair script can be used for automatically modifying the data of the database so as to ensure the correctness of the data of the database. Here, the data in error may be data of an Oracle database, may be a Hive database, or may have different error data in both databases.
In one example, the predetermined monitoring result processing model outputs the error sources as: settlement amount inconsistency due to exchange rate settlement factors. At this time, the error data is a settlement amount converted from the initial transaction amount. And storing a data repair script adapted to the error reason on the server. When the data of the database is modified by using the data repair script, the following steps are executed: acquiring settlement time of the initial transaction amount; and inquiring the standard exchange rate of the settlement time, calculating a settlement amount standard value matched with the initial transaction amount according to the standard exchange rate, and replacing the settlement amount in the database with the settlement amount standard value. In this way, the data repair script automatically corrects the error data in the database.
When the error reason output by the preset monitoring result processing model does not have the data recovery script matched with the error reason, whether the reminding information is sent to the appointed client side or not can be determined according to the error grade output by the preset monitoring result processing model. The error level is low, and the reminding can be not sent or sent in a delayed manner; and if the error level is high, immediately sending a prompt. Herein, the designated client may refer to a computer, a mobile phone or other communication devices used by an administrator, and may also be an alarm device of a data monitoring center. The reminding information can be pop-up window information, short messages, light reminding signals, reminding mails and the like.
In steps S10-S80, monitoring rules are configured in an Oracle database, and the monitoring rules are used for monitoring specified data indexes so as to define parameters needing to be monitored in a self-defining mode. And sending the monitoring rule to a Hive database to synchronize the monitoring rule. Acquiring a first monitoring result in a specified time range in the Oracle database according to the monitoring rule; and acquiring a second monitoring result in the appointed time range in the Hive database according to the monitoring rule so as to acquire the monitoring results of the two databases. And importing the second monitoring result into the Oracle database, and calculating the difference degree of the first monitoring result and the second monitoring result so as to compare the difference of the two monitoring results. And if the difference degree is greater than a preset alarm threshold value, inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model so as to automatically analyze abnormal monitoring results. And acquiring the error reasons output by the preset monitoring result processing model so as to determine the reasons of data errors in the database. And acquiring a data repair script adapted to the error reason so as to automatically match a repair scheme and improve the data repair efficiency. And repairing the data of the Oracle database and/or the Hive database according to the data repairing script so as to realize automatic repairing of error data.
Optionally, as shown in fig. 3, step S10 includes:
s11, receiving a first input instruction, wherein the first input instruction is used for selecting a plurality of specified data indexes and setting monitoring parameters;
and S12, receiving a second input instruction, wherein the second input instruction is used for generating the monitoring rule, and the monitoring rule is used for collecting the monitoring data of the specified data index according to the monitoring parameter so as to generate the first monitoring result or the second monitoring result.
In this embodiment, the first input instruction may be an input instruction generated by a database administrator performing operations of selecting a specific data index and inputting a monitoring parameter on a server running an Oracle database. The administrator can select important data indexes as specified data indexes according to actual needs. In an application example, the specified data index may be a library name, a table name, a business rule name, a dimension name, an index name, etc. to be monitored. Monitoring parameters include, but are not limited to, monitoring intervals, monitoring time ranges, and the like. In some cases, the incremental data can be monitored only, so that the number of times of reading and writing the data of the database is reduced, and the data monitoring efficiency is improved.
The second input instruction may be an input instruction generated by a database administrator performing a validation operation on a server running the Oracle database. In the monitoring platform, after the database administrator selects the designated data index and sets the monitoring parameter, a picture or a button for confirming the operation appears, and the data administrator performs the confirming operation on the picture for confirming the operation according to the guidance of the picture or clicks the confirming button to generate the second input instruction. After receiving the second input instruction, the server can generate a corresponding monitoring rule according to the first input instruction provided by the database administrator in the previous step.
And S11-S12, receiving a first input instruction, wherein the first input instruction is used for selecting a plurality of designated data indexes and setting monitoring parameters, and greatly improving the efficiency of parameter configuration due to the fact that the selectable designated data indexes and the setting monitoring parameters are preset. And receiving a second input instruction, wherein the second input instruction is used for generating the monitoring rule, and the monitoring rule is used for collecting the monitoring data of the specified data index according to the monitoring parameter to generate the first monitoring result or the second monitoring result.
Alternatively, as shown in fig. 4, step S30 includes:
s301, creating a first monitoring result table, and writing the specified data index into the first monitoring result table;
s302, generating a first query statement according to the monitoring rule, wherein the first query statement comprises the specified time range;
s303, executing the first query statement in the Oracle database, and acquiring corresponding first monitoring data of the specified data index in the specified time range;
s304, importing the first monitoring data into the first monitoring result table to generate a first monitoring result.
In this embodiment, a first monitoring result table may be created on a server running an Oracle database, and the specified data index may be written in the first monitoring result table. The first monitoring result table may be a blank table when the first monitoring result table is first created. In some cases, a first monitoring result table may be newly created every time a monitoring operation is performed. In other cases, the last first monitoring result table may be used if the monitoring rule has not changed. The first monitoring result table comprises a plurality of information columns, such as a library name column, a list name column, a business rule column, a dimension column, an index column and the like. Each information column corresponds to a specified data index.
A query task of the Oracle database can be created according to the monitoring rule, and a first query statement capable of executing query operation on the Oracle database is generated. The first query statement includes a specified data index and a specified time range that require querying. After the Oracle database completes the query operation, a corresponding query result, namely the first monitoring data, can be obtained. And then writing the first monitoring data into a first monitoring result table to obtain a first monitoring result. For example, in a monitoring task of an Oracle database, a first monitoring result of 20190807 japanese chart a can be obtained, and the data amount recorded in the first monitoring result is 1000.
In steps S301 to S304, a first monitoring result table is created, and the specified data index is written in the first monitoring result table, so that the specified data index included in the monitoring rule is written in the first monitoring result table. And generating a first query statement according to the monitoring rule, wherein the first query statement comprises the specified time range so as to generate the first query statement capable of executing query operation in an Oracle database. And executing the first query statement in the Oracle database, and acquiring corresponding first monitoring data of the specified data index in the specified time range to acquire the monitoring data of the Oracle database. And importing the first monitoring data into the first monitoring result table to generate a first monitoring result so as to obtain a monitoring result of the Oracle database.
Optionally, as shown in fig. 5, step S30 further includes:
s305, creating a second monitoring result table, and writing the specified data index into the second monitoring result table;
s306, generating a MapReduce task according to the monitoring rule, wherein the MapReduce task comprises the specified time range;
s307, the MapReduce task is executed in the Hive database, and corresponding second monitoring data of the specified data index in the specified time range are obtained;
and S308, importing the second monitoring data into the second monitoring result table to generate a second monitoring result.
In this embodiment, a second monitoring result table may be created on a server running the Hive database, and the specified data index may be written in the second monitoring result table. The second monitoring result table may be a blank table when the second monitoring result table is created for the first time. In some cases, a second monitoring result table may be newly created every time a monitoring operation is performed. In other cases, the last second monitoring result table may be used if the monitoring rule has not changed. The second monitoring result table comprises a plurality of information columns, such as a library name column, a list name column, a business rule column, a dimension column, an index column and the like. Each information column corresponds to a specified data index.
And creating a query task of the Hive database according to the monitoring rule, and generating a MapReduce task which can execute query operation on the Hive database. The MapReduce task includes a specified data index and a specified time range that need to be queried. And obtaining a corresponding query result, namely second monitoring data, after the Hive database completes the query operation. And then writing the second monitoring data into a second monitoring result table to obtain a second monitoring result. For example, in a task of monitoring a Hive database, a second monitoring result of 20190807 japanese chart a may be obtained, and the data amount recorded in the second monitoring result is 1001.
In steps S305 to S308, a second monitoring result table is created, and the specified data index is written in the second monitoring result table, so that the specified data index included in the monitoring rule is written in the second monitoring result table. And generating a MapReduce task according to the monitoring rule, wherein the MapReduce task comprises the specified time range so as to generate the MapReduce task which can be executed in a Hive database. And executing the MapReduce task in the Hive database, and acquiring corresponding second monitoring data of the specified data index in the specified time range to acquire the monitoring data of the Hive database. And importing the second monitoring data into the second monitoring result table to generate a second monitoring result so as to obtain a monitoring result of the Hive database.
Alternatively, as shown in fig. 6, step S40 includes:
s401, acquiring a preset processing script;
s402, analyzing the first monitoring result and the second monitoring result according to the preset processing script, and calculating the index difference of each specified data index;
and S403, generating the difference degree of the first monitoring result and the second monitoring result based on the index difference degree of the specified data index.
In this embodiment, the preset processing script may be used to compare the difference of each specified data index in the first monitoring result table and the second monitoring result table, that is, the index difference. For example, if the value of the data index X in the first monitoring result table is 1002 and the value in the second monitoring result table is 1002, the index difference degree of the data index X is 0. Each index difference may be an absolute value of a difference between respective values in the monitoring result table. For example, the value of the data index Y in the first monitoring result table is 1005, the value in the second monitoring result table is 1006, and the index difference degree of the data index Y is 1; if the value of the data index Z in the first monitoring result table is 1006 and the value in the second monitoring result table is 1004, the index difference degree of the data index Z is 2.
The degree of difference of the first monitoring result and the second monitoring result may be a weighted sum of all index degrees of difference. If the weight of each specified data index is equal, the difference between the first monitoring result and the second monitoring result may be the sum of the difference between all indexes.
In steps S401-S403, a preset processing script is obtained to obtain a script that can be used to process the monitoring result. And analyzing the first monitoring result and the second monitoring result according to the preset processing script, and calculating the index difference of each specified data index so as to calculate the index difference of the first monitoring result and the second monitoring result on each specified data index. And generating the difference degree of the first monitoring result and the second monitoring result based on the index difference degree of the specified data index so as to obtain the difference degree of the first monitoring result and the second monitoring result on the whole.
Optionally, as shown in fig. 7, before step S50, the method further includes:
s51, acquiring a preset alarm judgment rule, wherein the preset alarm judgment rule comprises an alarm threshold and an importance of each specified data index;
s52, calculating a threshold comparison value of the difference degree according to the warning threshold and the importance degree;
s53, judging whether the threshold comparison value is larger than the preset alarm threshold value;
and S54, if the threshold comparison value is larger than the preset alarm threshold, judging that the difference degree is larger than the preset alarm threshold.
In this embodiment, the preset alarm evaluation rule may set an alarm threshold and an importance of each specified data index. Specifically, the warning threshold in the preset warning evaluation rule may be represented by an array, such as { p }1，p2，p3，……pn(where p represents the warning threshold and n is the total number of warning thresholds). The importance of the preset alarm evaluation rule can be represented by an array, such as { q }1，q2，q3，……qn(where q represents importance and n is the total number of warning thresholds).
In some cases, the degree of difference between the first monitoring result and the second monitoring result may beExpressed by an array, e.g. { c }1，c2，c3，……cnAnd (wherein c represents index difference, and n is the total number of index differences).
Thus, the threshold comparison value K of the degree of difference can be calculated as follows:
and when the threshold comparison value is larger than the preset alarm threshold, judging that the difference degree is larger than the preset alarm threshold. Here, the preset alarm threshold may refer to an overall threshold, and the value of the overall threshold may be zero. In some cases, different alarm levels may be set, with the alarm level varying with the difference between the threshold comparison value and the preset alarm threshold.
In the steps S51-S54, a preset alarm evaluation rule is obtained, where the preset alarm evaluation rule includes the alarm threshold and the importance of each specified data index, so that the alarm evaluation rule is preset to facilitate evaluation of the difference. And calculating a threshold comparison value of the difference degree according to the warning threshold and the importance degree so as to convert the difference degree into an intermediate value which can be compared with a preset warning threshold. And judging whether the threshold comparison value is larger than the preset alarm threshold value or not so as to compare the magnitude of the threshold comparison value and the preset alarm threshold value. If the threshold comparison value is larger than the preset alarm threshold, judging that the difference degree is larger than the preset alarm threshold so as to determine the comparison result of the difference degree.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
In an embodiment, a data monitoring apparatus is provided, and the data monitoring apparatus corresponds to the data monitoring method in the above embodiments one to one. As shown in fig. 8, the data monitoring apparatus includes a rule configuration module 10, a rule transmission module 20, a monitoring result obtaining module 30, a difference calculation module 40, a model analysis module 50, a model output module 60, a repair script obtaining module 70, and a data repair module 80. The functional modules are explained in detail as follows:
the rule configuration module 10 is used for configuring a monitoring rule in an Oracle database, wherein the monitoring rule is used for monitoring a specified data index;
a rule sending module 20, configured to send the monitoring rule to a Hive database;
the monitoring result obtaining module 30 is used for obtaining a first monitoring result within a specified time range in the Oracle database according to the monitoring rule; acquiring a second monitoring result in the appointed time range in the Hive database according to the monitoring rule;
the difference calculation module 40 is configured to import the second monitoring result into the Oracle database, and calculate a difference between the first monitoring result and the second monitoring result;
the model analysis module 50 is configured to input the first monitoring result and the second monitoring result into a preset monitoring result processing model if the difference is greater than a preset alarm threshold;
the model output module 60 is used for acquiring the preset monitoring result and processing the error reason of the model output;
a get repair script module 70, configured to get a data repair script adapted to the error cause;
and the data repairing module 80 is used for repairing the data of the Oracle database and/or the Hive database according to the data repairing script.
Optionally, the rule configuring module 10 includes:
the first input unit is used for receiving a first input instruction, and the first input instruction is used for selecting a plurality of specified data indexes and setting monitoring parameters;
the second input unit is configured to receive a second input instruction, where the second input instruction is used to generate the monitoring rule, and the monitoring rule is used to collect monitoring data of the specified data index according to the monitoring parameter to generate the first monitoring result or the second monitoring result.
Optionally, the module 30 for obtaining a monitoring result includes:
the first table creating unit is used for creating a first monitoring result table and writing the specified data index into the first monitoring result table;
the query statement unit is used for generating a first query statement according to the monitoring rule, wherein the first query statement comprises the specified time range;
a first monitoring data acquisition unit, configured to execute the first query statement in the Oracle database, and acquire corresponding first monitoring data of the specified data indicator in the specified time range;
and the first monitoring result generating unit is used for importing the first monitoring data into the first monitoring result table to generate a first monitoring result.
Optionally, the module 30 for obtaining a monitoring result further includes:
the second table creating unit is used for creating a second monitoring result table and writing the specified data index into the second monitoring result table;
the task generating unit is used for generating a MapReduce task according to the monitoring rule, and the MapReduce task comprises the specified time range;
a second monitoring data acquisition unit, configured to execute the MapReduce task in the Hive database, and acquire corresponding second monitoring data of the specified data index within the specified time range;
and the second monitoring result generating unit is used for importing the second monitoring data into the second monitoring result table to generate a second monitoring result.
Optionally, the difference calculating module 40 includes:
the script acquiring unit is used for acquiring a preset processing script;
the index difference calculation unit is used for analyzing the first monitoring result and the second monitoring result according to the preset processing script and calculating the index difference of each specified data index;
a generation difference degree unit configured to generate a difference degree of the first monitoring result and the second monitoring result based on an index difference degree of the specified data index.
Optionally, the data monitoring apparatus further includes a difference degree comparison module, where the difference degree comparison module includes:
the acquisition rule unit is used for acquiring a preset alarm judgment rule, and the preset alarm judgment rule comprises an alarm threshold value and an importance degree of each specified data index;
a calculation comparison value unit for calculating a threshold comparison value of the degree of difference according to the warning threshold and the degree of importance;
the size comparison unit is used for judging whether the threshold comparison value is larger than the preset alarm threshold value or not;
and the determining alarm unit is used for judging that the difference degree is greater than the preset alarm threshold value if the threshold comparison value is greater than the preset alarm threshold value.
For specific limitations of the data monitoring apparatus, reference may be made to the above limitations of the data monitoring method, which will not be described herein again. The modules in the data monitoring device can be wholly or partially implemented by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 9. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing data related to the data monitoring method. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of data monitoring.
In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
configuring a monitoring rule in an Oracle database, wherein the monitoring rule is used for monitoring a specified data index;
sending the monitoring rule to a Hive database;
acquiring a first monitoring result in a specified time range in the Oracle database according to the monitoring rule; acquiring a second monitoring result in the appointed time range in the Hive database according to the monitoring rule;
importing the second monitoring result into the Oracle database, and calculating the difference degree between the first monitoring result and the second monitoring result;
if the difference degree is larger than a preset alarm threshold value, inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model;
acquiring error reasons output by the preset monitoring result processing model;
acquiring a data recovery script adapted to the error reason;
and repairing the data of the Oracle database and/or the Hive database according to the data repairing script.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
configuring a monitoring rule in an Oracle database, wherein the monitoring rule is used for monitoring a specified data index;
sending the monitoring rule to a Hive database;
acquiring a first monitoring result in a specified time range in the Oracle database according to the monitoring rule; acquiring a second monitoring result in the appointed time range in the Hive database according to the monitoring rule;
importing the second monitoring result into the Oracle database, and calculating the difference degree between the first monitoring result and the second monitoring result;
if the difference degree is larger than a preset alarm threshold value, inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model;
acquiring error reasons output by the preset monitoring result processing model;
acquiring a data recovery script adapted to the error reason;
and repairing the data of the Oracle database and/or the Hive database according to the data repairing script.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.
Claims (10)
1. A method for monitoring data, comprising:
configuring a monitoring rule in an Oracle database, wherein the monitoring rule is used for monitoring a specified data index;
sending the monitoring rule to a Hive database;
acquiring a first monitoring result in a specified time range in the Oracle database according to the monitoring rule; acquiring a second monitoring result in the appointed time range in the Hive database according to the monitoring rule;
importing the second monitoring result into the Oracle database, and calculating the difference degree between the first monitoring result and the second monitoring result;
if the difference degree is larger than a preset alarm threshold value, inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model;
acquiring error reasons output by the preset monitoring result processing model;
acquiring a data recovery script adapted to the error reason;
and repairing the data of the Oracle database and/or the Hive database according to the data repairing script.
2. The data monitoring method of claim 1, wherein the configuring of the monitoring rule in the Oracle database, the monitoring rule being used for monitoring the specified data index, comprises:
receiving a first input instruction, wherein the first input instruction is used for selecting a plurality of specified data indexes and setting monitoring parameters;
receiving a second input instruction, where the second input instruction is used to generate the monitoring rule, and the monitoring rule is used to collect monitoring data of the specified data index according to the monitoring parameter, so as to generate the first monitoring result or the second monitoring result.
3. The data monitoring method of claim 1, wherein the obtaining a first monitoring result in a specified time range in the Oracle database according to the monitoring rule comprises:
creating a first monitoring result table, and writing the specified data index into the first monitoring result table;
generating a first query statement according to the monitoring rule, wherein the first query statement comprises the specified time range;
executing the first query statement in the Oracle database, and acquiring corresponding first monitoring data of the specified data index in the specified time range;
and importing the first monitoring data into the first monitoring result table to generate a first monitoring result.
4. The data monitoring method as claimed in claim 1, wherein the obtaining of the second monitoring result in the Hive database within a specified time range according to the monitoring rule comprises:
creating a second monitoring result table, and writing the specified data index into the second monitoring result table;
generating a MapReduce task according to the monitoring rule, wherein the MapReduce task comprises the specified time range;
executing the MapReduce task in the Hive database, and acquiring corresponding second monitoring data of the specified data index in the specified time range;
and importing the second monitoring data into the second monitoring result table to generate a second monitoring result.
5. The data monitoring method of claim 1, wherein the importing the second monitoring result into the Oracle database and calculating a degree of difference between the first monitoring result and the second monitoring result comprises:
acquiring a preset processing script;
analyzing the first monitoring result and the second monitoring result according to the preset processing script, and calculating the index difference degree of each specified data index;
generating a degree of difference of the first monitoring result and the second monitoring result based on an index degree of difference of the specified data index.
6. The data monitoring method of claim 1, wherein before inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model if the difference is greater than a preset alarm threshold, the method comprises:
acquiring a preset alarm judgment rule, wherein the preset alarm judgment rule comprises an alarm threshold and an importance of each specified data index;
calculating a threshold comparison value of the difference degree according to the warning threshold and the importance degree;
judging whether the threshold comparison value is greater than the preset alarm threshold value;
and if the threshold comparison value is greater than the preset alarm threshold, judging that the difference degree is greater than the preset alarm threshold.
7. A data monitoring device, comprising:
the rule configuration module is used for configuring a monitoring rule in an Oracle database, and the monitoring rule is used for monitoring a specified data index;
the rule sending module is used for sending the monitoring rule to a Hive database;
the monitoring result acquisition module is used for acquiring a first monitoring result in a specified time range in the Oracle database according to the monitoring rule; acquiring a second monitoring result in the appointed time range in the Hive database according to the monitoring rule;
the difference degree calculation module is used for importing the second monitoring result into the Oracle database and calculating the difference degree of the first monitoring result and the second monitoring result;
the model analysis module is used for inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model if the difference degree is larger than a preset alarm threshold value;
the model output module is used for acquiring the preset monitoring result and processing the error reason output by the model;
the acquisition and repair script module is used for acquiring the data repair script matched with the error reason;
and the data repairing module is used for repairing the data of the Oracle database and/or the Hive database according to the data repairing script.
8. The data monitoring apparatus of claim 7, wherein the rule configuration module comprises:
the first input unit is used for receiving a first input instruction, and the first input instruction is used for selecting a plurality of specified data indexes and setting monitoring parameters;
the second input unit is configured to receive a second input instruction, where the second input instruction is used to generate the monitoring rule, and the monitoring rule is used to collect monitoring data of the specified data index according to the monitoring parameter to generate the first monitoring result or the second monitoring result.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the data monitoring method according to any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out a data monitoring method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010092311.0A CN111367754A (en) | 2020-02-14 | 2020-02-14 | Data monitoring method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010092311.0A CN111367754A (en) | 2020-02-14 | 2020-02-14 | Data monitoring method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111367754A true CN111367754A (en) | 2020-07-03 |
Family
ID=71206248
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010092311.0A Pending CN111367754A (en) | 2020-02-14 | 2020-02-14 | Data monitoring method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111367754A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111832661A (en) * | 2020-07-28 | 2020-10-27 | 平安国际融资租赁有限公司 | Classification model construction method and device, computer equipment and readable storage medium |
| CN112463785A (en) * | 2020-12-08 | 2021-03-09 | 中国人寿保险股份有限公司 | Data quality monitoring method and device, electronic equipment and storage medium |
| CN112527661A (en) * | 2020-12-17 | 2021-03-19 | 平安消费金融有限公司 | Data script verification method and device and computer equipment |
-
2020
- 2020-02-14 CN CN202010092311.0A patent/CN111367754A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111832661A (en) * | 2020-07-28 | 2020-10-27 | 平安国际融资租赁有限公司 | Classification model construction method and device, computer equipment and readable storage medium |
| CN111832661B (en) * | 2020-07-28 | 2024-04-02 | 平安国际融资租赁有限公司 | Classification model construction method, device, computer equipment and readable storage medium |
| CN112463785A (en) * | 2020-12-08 | 2021-03-09 | 中国人寿保险股份有限公司 | Data quality monitoring method and device, electronic equipment and storage medium |
| CN112463785B (en) * | 2020-12-08 | 2024-04-16 | 中国人寿保险股份有限公司 | Data quality monitoring method and device, electronic equipment and storage medium |
| CN112527661A (en) * | 2020-12-17 | 2021-03-19 | 平安消费金融有限公司 | Data script verification method and device and computer equipment |
| CN112527661B (en) * | 2020-12-17 | 2024-04-26 | 平安消费金融有限公司 | Verification method and device of data script and computer equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108874644B (en) | Data monitoring method and device, computer equipment and storage medium | |
| CN111367754A (en) | Data monitoring method and device, computer equipment and storage medium | |
| CN110019116B (en) | Data tracing method, device, data processing equipment and computer storage medium | |
| EP2713288B1 (en) | Foreign key identification in database management systems | |
| CN110737594A (en) | Database standard conformance testing method and device for automatically generating test cases | |
| CN110750443A (en) | Webpage testing method and device, computer equipment and storage medium | |
| CN111240976A (en) | Software testing method and device, computer equipment and storage medium | |
| CN112101584A (en) | Nuclear power station spare part systematic maintenance method, device, equipment and storage medium | |
| CN113762914A (en) | Early warning auditing method and related equipment | |
| CN115879748A (en) | Enterprise informatization management integration platform based on big data | |
| CN114943383A (en) | Prediction method and device based on time series, computer equipment and storage medium | |
| CN112596723B (en) | Database script generation method, device, equipment and medium | |
| CN111475526B (en) | Sequential data conversion method based on oracle data and related equipment thereof | |
| CN110765148B (en) | Service data processing method and device | |
| CN112235358A (en) | Data acquisition method and device, electronic equipment and computer readable storage medium | |
| CN115525660A (en) | Data table verification method, device, equipment and medium | |
| CN109324963A (en) | The method and terminal device of automatic test profitable result | |
| CN113495831B (en) | Method, system, equipment and medium for generating test case based on keywords | |
| CN114860759A (en) | Data processing method, device and equipment and readable storage medium | |
| CN114721945A (en) | Graph database-based distribution method and device, electronic equipment and storage medium | |
| CN116401177B (en) | DDL correctness detection method, device and medium | |
| CN114401494B (en) | Short message issuing abnormality detection method, device, computer equipment and storage medium | |
| CN117034036A (en) | Method and device for detecting timeliness of data processing and computer equipment | |
| CN116881164A (en) | Verification and correction method, device and equipment for test data in financial information system | |
| CN117785853A (en) | Data processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |