CN108874644B - Data monitoring method and device, computer equipment and storage medium - Google Patents
Data monitoring method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN108874644B CN108874644B CN201810574141.2A CN201810574141A CN108874644B CN 108874644 B CN108874644 B CN 108874644B CN 201810574141 A CN201810574141 A CN 201810574141A CN 108874644 B CN108874644 B CN 108874644B
- Authority
- CN
- China
- Prior art keywords
- data
- monitored
- table structure
- structure data
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a data monitoring method, a data monitoring device, computer equipment and a storage medium, wherein the method comprises the following steps: creating a data relation table according to the incidence relation among different data tables in the database under each application environment; receiving table information of a data table to be monitored; according to the table information, obtaining table structure data of the data table to be monitored by using a preset script; if detecting that the table structure data of the data table to be monitored is changed, recording change information; acquiring an associated data table associated with a data table to be monitored, of which the table structure data is changed, from the data relation table, and checking whether the table structure data of the associated data table is correct or not according to the change information; and if the table structure data of the associated data table is incorrect, outputting abnormal information. According to the technical scheme, the condition that the code is not completely updated manually can be monitored in the development process, and the abnormal information is output, so that related personnel can maintain the code in time, and the development quality and efficiency are improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data monitoring method and apparatus, a computer device, and a storage medium.
Background
At present, in order to facilitate smooth development work of developers, developers having Data Definition Language (DDL) authority can perform Data development work such as table Definition or modification of table structures, Data types, and links and constraints between tables. However, in the same development project, the data of the development environment is often inconsistent with the data of the test environment or the production environment due to the respective data development work of different developers.
For example, as the amount of data in a database increases, production problems may arise if a developer modifies the definition of database attributes, and the test and production phase workers do not know this information and do not modify the database in the test and production environments in synchronization. If a developer modifies a field attribute, and the field attribute related to the field in the data tables such as the application library, the report library, or the service table and the backup table is not modified synchronously, this situation easily causes a problem in the production stage, and a large amount of human resources are needed to search for the cause of the problem, thereby reducing the working efficiency of development.
Disclosure of Invention
Therefore, it is necessary to provide a data monitoring method, device, computer device and storage medium that can monitor data of multiple databases, find abnormal data information in time, and improve development quality and efficiency, in order to solve the above technical problems.
A method of data monitoring, comprising:
acquiring an incidence relation between different data tables in a database under each application environment, and creating a data relation table according to the incidence relation;
receiving table information of a data table to be monitored;
according to the table information, obtaining table structure data of the data table to be monitored by using a preset script;
if detecting that the table structure data of the data table to be monitored is changed, recording change information;
acquiring an associated data table associated with a data table to be monitored, of which the table structure data is changed, from the data relation table, and checking whether the table structure data of the associated data table is correct or not according to the change information;
and if the table structure data of the associated data table is incorrect, outputting abnormal information that the table structure data of the associated data table is not matched with the table structure data of the data table to be monitored, of which the table structure data is changed.
A data monitoring apparatus, comprising:
the data relation table creating module is used for acquiring the incidence relation among different data tables in the database under each application environment and creating a data relation table according to the incidence relation;
the table information receiving module is used for receiving the table information of the data table to be monitored;
the table structure data acquisition module is used for acquiring the table structure data of the data table to be monitored by using a preset script according to the table information;
the change information recording module is used for recording change information if the table structure data of the data table to be monitored is detected to be changed;
the associated data table checking module is used for acquiring an associated data table associated with a data table to be monitored, of which the table structure data is changed, from the data relation table and checking whether the table structure data of the associated data table is correct or not according to the change information;
and the abnormal information output module is used for outputting abnormal information that the table structure data of the associated data table is not matched with the table structure data of the data table to be monitored, of which the table structure data is changed, if the table structure data of the associated data table is incorrect.
A computer device comprising a memory, a processor and a computer program stored in said memory and executable on said processor, said processor implementing the steps of the above data monitoring method when executing said computer program.
A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned data monitoring method.
The data monitoring method, the device, the computer equipment and the storage medium establish the data relation table by acquiring the incidence relation among different data tables in the database under each application environment, show the mapping relation among different data tables in the database, clarify the incidence relation among the table structure data of each data table, acquire the table structure data of the data table to be monitored according to the table information of the data table to be monitored, record the change information if the change of the table structure data of the data table to be monitored is detected, acquire the incidence data table correlated with the data table to be monitored, which has the changed table structure data, from the data relation table, and check whether the table structure data of the incidence data table is correct according to the change information, thereby outputting the abnormal information of the incorrect table structure data of the incidence data table for the timely maintenance of related personnel, the situation that production problems occur due to incomplete manual code updating in the development process is avoided, and development quality and development work efficiency are improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments of the present invention will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive labor.
FIG. 1 is a diagram illustrating an application environment of a data monitoring method according to an embodiment of the present invention;
FIG. 2 is a flow chart of a data monitoring method according to an embodiment of the present invention;
FIG. 3 is a diagram of a data relationship table according to an embodiment of the invention.
FIG. 4 is a detailed flowchart of step S4 in FIG. 2;
FIG. 5 is a detailed flowchart of step S5 in FIG. 2;
fig. 6 is a specific flowchart for checking change information in the data monitoring method according to an embodiment of the present invention;
FIG. 7 is a functional block diagram of a data monitoring device in accordance with an embodiment of the present invention;
FIG. 8 is a schematic diagram of a computer device according to an embodiment of the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The data monitoring method provided by the application can be applied to an application environment shown in fig. 1, wherein the application environment comprises a server and a client, the server is connected with the client through a network, a user sets a data table to be monitored through the client, the server monitors the data table to be monitored set by the user, checks an associated data table associated with the changed data table, and outputs abnormal information to the client. The client may specifically be but not limited to various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices, and the server may specifically be implemented by an independent server or a server cluster formed by a plurality of servers. The data monitoring method provided by the embodiment of the invention is applied to the server side.
In an embodiment, fig. 2 shows a flowchart of a data monitoring method in this embodiment, which is applied to the server side in fig. 1 for monitoring table structure data of a data table in a database. As shown in fig. 2, the data monitoring method includes steps S1 to S6, which are detailed as follows:
s1: acquiring the incidence relation among different data tables in the database under each application environment, and creating a data relation table according to the incidence relation.
In this embodiment, the application environment includes, but is not limited to, a development environment, a test environment, a production environment, and the like, the database includes, but is not limited to, an application library, a report library, a production library, a backup library, and the like, and the data table includes, but is not limited to, a base table, a derivative table, and a materialized view, where the base table is a data table including table names, fields, and table records, the materialized view is a data table storing operation results such as table connections or aggregations, and the derivative table is a data table generated by obtaining information of other data tables.
Further, in the process of project development, if the server detects that another data table is generated based on one or more data tables in the database, or copies one or more data tables into another database, a corresponding association relationship is created for the one or more data tables and the newly-created data table to represent association information between the newly-created data table and the one or more data tables, where the association relationship is a specific relationship that identifies table structure data between the data tables through different identification information, and the table structure data includes, but is not limited to, structure data such as fields, types, primary keys, foreign keys, and indexes of the data tables. Specifically, a data relationship table is created according to the association relationship between different data tables in the multiple databases in each application environment, and the data relationship table is used for identifying the interdependence and influence relationship of data between different data tables in the databases in each application environment.
It should be noted that the association relationship between different data tables in the databases in each application environment includes at least one of or a combination of two or more of the association relationships between different data tables in the same database, or the association relationships between data tables in different databases in the same application environment, or the association relationships between data tables in different application environments.
In order to better understand the data relationship table in the embodiment of the present invention, a specific data relationship table between different data tables is illustrated. The details are as follows:
referring to fig. 3, fig. 3 shows data relationship tables between different data tables in a plurality of databases in respective application environments. The application environment comprises an application environment 1 and an application environment 2, the application environment 1 comprises a database 1 and a database 2, the application environment 2 comprises a database 3, the database 1 comprises a base table 1, a base table 2, a derivative table 1 and a materialized view 1, the database 2 comprises a base table 3, a base table 4 and a derivative table 2, and the database 3 comprises a base table 5, a base table 6, a derivative table 3 and a materialized view 2. The association relationship is represented by identification information and a connection relationship, identification information a1 represents that the table structure data of the data table is a call relationship, and identification information a2 represents that the table structure data of the data table is the same relationship. In the data relationship expression intention, the table structure data of the base table 1 is called by the materialized view 1 and the derivative table 1, and the table structure data of the base table 1 is the same as that of the base table 5; the table structure data of the base table 2 is called by the materialized view 1 and the derivative table 1, and the table structure data of the base table 2 is the same as that of the base table 6; the table structure data of the base table 3 is called by the base table 2, the base table 4 and the derivative table 2; the table structure data of base table 4 is called by derivative table 2; the table structure data of the base table 5 is called by the materialized view 2 and the derivative table 3; the table structure data of the base table 6 is called by the materialized view 2 and the derivative table 3, so that the data relation table is created according to the incidence relation among different data tables in a plurality of databases under each application environment.
S2: and receiving the table information of the data table to be monitored.
In this embodiment, the table information includes an application environment of the data table, a database name, and a data table name, the server receives the table information of the data table to be monitored, and the table information is used as the table information of the data table to be monitored, the data table to be monitored may be a data table which is default by the system and needs to be monitored, or may be a data table which is custom-set by the user at the client and needs to be monitored, and the table information may be specifically set according to an actual situation, which is not limited here.
S3: and acquiring the table structure data of the data table to be monitored by using a preset script according to the table information.
Specifically, according to the application environment, the database name and the data table name of the data table to be monitored, a preset script is connected to a server of the application environment through a network, according to the database connection string, a user name and a user password are used for logging in a database corresponding to the database name, and table structure data of the corresponding data table are obtained.
The preset script includes a data source, a user name, a user password, and the like, where the data source stores connection information of all databases, for example, connection information such as an IP address of a server and a database connection port.
S4: and if detecting that the table structure data of the data table to be monitored is changed, recording change information.
In this embodiment, the data table to be monitored is monitored, and if it is detected that the table structure data of the data table to be monitored is changed, the change information of the changed table structure data in the data table to be monitored is acquired and stored in the history operation record.
The change information includes the application environment where the changed table structure data is located, the database name, the data table name, the change record, and the like, and the change record includes the timestamp, the table structure data before the change, the table structure data after the change, and the like.
Further, a specific method for monitoring the Data table to be monitored may be that a DDL trigger or a DML (Data management Language) trigger is added to the database, when a user adds, deletes, or modifies the Data table, the DDL trigger or the DML trigger may acquire change information that the table structure Data of the Data table is changed, or may refresh the table structure Data of the Data table to be monitored at regular time, and determine whether the table structure Data after being refreshed is inconsistent with the table structure Data acquired before being refreshed, and if so, acquire the change information that the table structure Data of the Data table is changed, store the change information in the history operation record, and facilitate searching the change information.
S5: and acquiring an associated data table associated with the data table to be monitored, of which the table structure data is changed, from the data relation table, and checking whether the table structure data of the associated data table is correct or not according to the change information.
In this embodiment, according to the association relationship, an association data table associated with a data table to be monitored, in which table structure data is changed, is obtained from a data relationship table, where the association data table includes data tables of different databases in each application environment, a preset script is used to obtain table structure data of the association data table, and according to change information, the table structure data of the association data table is traversed and compared, and whether the table structure data of the association data table is correct is checked.
For example, if the user modifies the field type of a certain base table, the server checks the base table, the materialized view, or the derivative table associated with the base table according to the association relationship in the data relationship table, and detects whether all the types of the field in the associated data table are modified synchronously, whether the modified field types are consistent, and the like.
S6: and if the table structure data of the associated data table is incorrect, outputting abnormal information that the table structure data between the associated data table and the data table to be monitored, of which the table structure data is changed, is not matched.
In this embodiment, after the table structure data of the associated data table is verified according to the change information, if the table structure data of the associated data table is not matched with the table structure data of the data table to be monitored after the table structure data is changed, it is determined that the table structure data of the associated data table is incorrect, and abnormal information that the table structure data of the associated data table is not matched with the table structure data of the data table to be monitored after the table structure data is changed is output and provided to relevant personnel for processing.
Further, the specific output mode of the abnormal information may be to output the verification result to an interactive interface of the client, or to generate a detection report according to the verification result, and send the detection report to an information receiving address of the relevant worker, where the information receiving address includes but is not limited to a mailbox, a short message receiving address, or an instant messaging tool address, for example, a messaging tool such as a WeChat and a QQ, and the specific output mode of the abnormal information may be set according to an actual application, and is not limited here, so that the relevant worker can process the occurred problem in time according to the obtained abnormal information, and the efficiency of processing the problem is improved.
In this embodiment, by obtaining the association relationship between different data tables in the database under each application environment, creating a data relationship table, displaying the mapping relationship between different data tables in the database, defining the association between the table structure data of each data table, obtaining the table structure data of the data table to be monitored according to the table information of the data table to be monitored, if it is detected that the table structure data of the data table to be monitored is changed, recording the change information, obtaining the association data table associated with the changed data table from the data relationship table, and checking whether the table structure data of the association data table is correct according to the change information, so as to output the abnormal information of the table structure data of the association data table, so that the related personnel can maintain the data in time, and avoid the occurrence of production problems due to incomplete update of codes manually in the development process, the development quality and the development work efficiency are improved.
In an embodiment, as shown in fig. 4, in step S4, that is, if it is detected that the table structure data of the data table to be monitored is changed, the recording of the change information specifically includes the following steps:
s41: and storing the table structure data of the data table to be monitored into a first file.
In this embodiment, according to the table information of the data table to be monitored, the table structure data of the data table to be monitored is obtained by using a preset script, and is stored in the first file, where the table structure data of the data table to be monitored includes, but is not limited to, structure data of a field, a type, a primary key, a foreign key, an index, and the like of the data table to be monitored, and the first file may specifically be a table or a document, and may also be set according to the needs of actual applications, which is not limited herein.
S42: and starting from the storage time of the first file, when a preset time interval is reached, acquiring the current table structure data of the data table to be monitored by using a preset script, and storing the current table structure data in a second file.
In this embodiment, starting from the time point when the table structure data of the data table to be monitored is stored in the first file, when the preset time interval is reached, the current table structure data of the data table to be monitored is obtained by using the preset script, and the current table structure data is stored in the second file, where the second file is the same as the first file in type.
It should be noted that the preset time interval may specifically be 30 minutes, but is not limited thereto, and it may specifically be set according to the needs of the practical application, and is not limited herein.
S43: and comparing the first file with the second file, if the first file and the second file have difference data, acquiring change information of the change of the table structure data of the data table to be monitored according to the difference data, and recording the change information.
In this embodiment, a first file and a second file are subjected to traversal comparison, whether current table structure data is consistent with table structure data acquired at a last preset time interval is detected, if the first file and the second file have difference data, change information of a change of the table structure data of the data table to be monitored is acquired according to the difference data, and change information including an application environment, a database name, a data table name and a change record where the changed table structure data is located is recorded.
It should be noted that the first file and the second file are temporary files, after the comparison between the table structure data of the first file and the table structure data of the second file is completed, the first file is deleted, only the current table structure data of the data table to be monitored, that is, the second file, is reserved, and when the preset time interval is reached, the second file is used for the next comparison.
In the embodiment, the table structure data of the data table to be monitored is stored in one temporary file, from the storage time of the first file, when the preset time interval is reached, the current table structure data of the data table to be monitored is acquired and stored in the other temporary file, the two temporary files are subjected to traversal comparison, the change information of the table structure data of the data table to be monitored is accurately acquired according to the difference data, and only the current table structure data is reserved after the comparison is completed, so that the system resources are saved.
In an embodiment, as shown in fig. 5, in step S5, that is, acquiring an associated data table associated with a data table to be monitored, in which table structure data is changed, from a data relationship table, and checking whether the table structure data of the associated data table is correct according to the change information, specifically includes the following steps:
s51: and acquiring a data table to be monitored corresponding to the change information, and taking the data table to be monitored as a reference table.
In this embodiment, according to the change information recorded in step S4, the data table to be monitored corresponding to the change information is acquired, and the data table to be monitored is used as the reference table.
For example, if one piece of change information stored in the history operation record is: and on 1 month and 1 day of 2018, modifying the G type of the application environment C, the database D, the data table E and the field F into the H type, and taking the data table E in the database D under the application environment C as a reference table.
S52: and acquiring the table structure data of the associated data table associated with the reference table according to the data relation table.
Specifically, based on the reference table determined in step S51, in the created data relationship table, table information of the associated data table associated with the reference table is acquired, and according to the table information, table structure data of the associated data table is acquired using a preset script.
For example, if the data table E in the database D in the application environment C is used as the reference table and it is determined that the data table L in the database D in the application environment C calls the table structure data of the data table E according to the data relationship table, the data table E is the related data table of the reference table and the table structure data of the data table E is acquired.
S53: and if the table structure data of the associated data table is not matched with the table structure data changed in the reference table, determining that the table structure data of the associated data table is correct.
In this embodiment, the table structure data of the associated data table is checked based on the table structure data changed in the reference table, and it is detected whether the table structure data of the associated data table matches the table structure data changed in the reference table.
Specifically, whether the table structure data in the associated data table is associated with the table structure data is detected according to the association relationship, if the table structure data is modified synchronously according to the table structure data changed in the reference table, whether the modified table structure data is consistent, or whether the modified associated data table maintains the same association relationship with the reference table, and the like, if the table structure data of the associated data table is matched with the table structure data changed in the reference table, the table structure data of the associated data table is determined to be correct, otherwise, the table structure data of the associated data table is determined to be incorrect.
In this embodiment, the data table to be monitored corresponding to the change information is used as the reference table, and the table structure data of the associated data table associated with the reference table is obtained according to the data relationship table for verification, so that whether the table structure data of the associated data table matches with the table structure data changed in the reference table can be detected, the table structure data of the associated data table in the data relationship table is comprehensively monitored, and the problem that the code is not completely updated manually can be timely found.
In an embodiment, as shown in fig. 6, after step S5, the data monitoring method may further verify the change information, which may be specifically implemented in the following manner, which is detailed as follows:
s7: and checking whether the change information meets the preset data element standard or not according to the preset data element standard.
In this embodiment, the data element standard is a constraint condition for defining a data element, the data element is a basic data element of the data table, the data element standard is configured in advance according to actual application requirements of the development project, for example, a type of a configuration field or a threshold value of the field is configured, and according to the data element standard configured in advance, the table structure data that is changed is checked, and whether the change information meets the preset data element standard is checked.
S8: and if the change information does not meet the preset data element standard, outputting the change information and error information.
In this embodiment, if it is detected that the changed table structure data in the data table to be monitored does not meet the pre-configured data metadata standard, a constraint condition between the changed table structure data and the preset data metadata standard is determined as error information, and the change information and the error information are output to an interactive interface of a client or sent to an information receiving address of a relevant worker and processed by the relevant worker.
For example, the configured data element standard is that a field length threshold value in the data table is 20, if the field length of the changed table structure data detected in the data table to be monitored exceeds 20, change information including the application environment where the changed table structure data is located, the database name, the data table name and the change record is acquired, the field is marked with red to indicate that the field length cannot exceed 20, and the change information and the error information are output.
In the embodiment, the data element standard is configured in advance, and whether the change information meets the preset data element standard or not is verified according to the data element standard, so that the change information which does not meet the data element standard can be found in time, and the change information and the error information are output, so that related personnel can find and solve problems in time, the production problem caused by human errors in the development process is avoided, and the development quality is improved.
In an embodiment, after step S6, the data monitoring method may further maintain table structure data of the associated data table, which may be specifically implemented in the following manner, which is detailed as follows:
and maintaining the table structure data of the associated data table according to the maintenance mode input by the user.
In this embodiment, the server receives a maintenance mode input by a user at the client, and maintains the table structure data of the associated data table according to the maintenance mode, where the maintenance mode may specifically be modified according to a default modification mode of the system, for example, taking the table structure data of a reference table in a development environment as a standard, and automatically synchronizing the table structure data of the associated data table under the test environment and the production environment according to the association relationship; the actual situation obtained by analyzing the verification result may also be modified by the relevant personnel, for example, all NUMBER types are directly converted into an inter type, all string types are converted into a variance 2 type, or a field naming mode needs to be started with COL _ and the like, and a specific maintenance mode may be selected by the user according to the actual situation, which is not limited herein.
In this embodiment, the table structure data of the associated data table is maintained according to the maintenance mode input by the user, so that the problem of abnormal information can be quickly solved, timely and comprehensive maintenance is performed, greater loss is avoided, and the development work efficiency is improved.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
In an embodiment, a data monitoring apparatus is provided, and the data monitoring apparatus corresponds to the data monitoring method in the above embodiments one to one. As shown in fig. 7, the data monitoring apparatus includes: a data relation table creating module 71, a table information receiving module 72, a table structure data acquiring module 73, a change information recording module 74, an associated data table verifying module 75, and an abnormal information outputting module 76. The functional modules are explained in detail as follows:
a data relation table creating module 71, configured to obtain an association relation between different data tables in the database in each application environment, and create a data relation table according to the association relation;
a table information receiving module 72, configured to receive table information of a data table to be monitored;
a table structure data obtaining module 73, configured to obtain the table structure data of the data table to be monitored by using a preset script according to the table information;
a change information recording module 74, configured to record change information if it is detected that the table structure data of the data table to be monitored is changed;
the associated data table checking module 75 is configured to obtain an associated data table associated with the data table to be monitored, of which the table structure data is changed, from the data relationship table, and check whether the table structure data of the associated data table is correct according to the change information;
and an exception information output module 76, configured to output, if the table structure data of the associated data table is incorrect, exception information that the table structure data of the associated data table is not matched with the table structure data of the data table to be monitored, of which the table structure data is changed.
Further, the change information recording module 74 includes:
the first storage submodule 741 is configured to store table structure data of a data table to be monitored into a first file;
a second storage submodule 742, configured to, from the storage time of the first file, when a preset time interval is reached, obtain current table structure data of the data table to be monitored by using a preset script, and store the current table structure data in a second file;
the file comparison submodule 743 is configured to compare the first file with the second file, and if there is difference data between the first file and the second file, obtain change information indicating that the table structure data of the data table to be monitored is changed according to the difference data, and record the change information.
Further, the associated data table checking module 75 includes:
the reference table setting submodule 751 is used for acquiring a data table to be monitored corresponding to the change information and taking the data table to be monitored as a reference table;
the associated data obtaining submodule 752 is configured to obtain, according to the data relation table, table structure data of an associated data table associated with the reference table;
the data matching sub-module 753 is configured to determine that the table structure data of the associated data table is correct if the table structure data of the associated data table matches the table structure data changed in the reference table, and determine that the table structure data of the associated data table is incorrect if the table structure data of the associated data table does not match the table structure data changed in the reference table.
Further, the data monitoring apparatus further includes:
a modified information checking module 77, configured to check whether the modified information conforms to a preset data metadata standard according to the preset data metadata standard;
and an error information output module 78, configured to output the change information and the error information if the change information does not meet the preset data element standard.
Further, the data monitoring apparatus further includes:
and a table structure data maintenance module 79, configured to maintain the table structure data of the associated data table according to the maintenance mode input by the user.
For specific limitations of the data monitoring apparatus, reference may be made to the above limitations of the data monitoring method, which will not be described herein again. The modules in the data monitoring device can be wholly or partially implemented by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 8. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of data monitoring.
In one embodiment, a computer device is provided, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the processor implements the steps of the data monitoring method in the above embodiments, such as the steps S1 to S6 shown in fig. 2, or when the processor executes the computer program, the processor implements the functions of the modules of the data monitoring apparatus in the above embodiments, such as the functions of the modules 71 to 76 shown in fig. 7. To avoid repetition, further description is omitted here.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program is executed by a processor to implement the steps in the data monitoring method provided by the above-mentioned embodiment, for example, the steps S1 to S6 shown in fig. 2, or the processor executes the computer program to implement the functions of the modules of the data monitoring apparatus in the above-mentioned embodiment, for example, the functions of the modules 71 to 76 shown in fig. 7. To avoid repetition, further description is omitted here.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.
Claims (9)
1. A method for data monitoring, the data monitoring comprising:
acquiring an incidence relation between different data tables in a database under each application environment, and creating a data relation table according to the incidence relation; the incidence relation comprises at least one of incidence relations between different data tables in the same database, or incidence relations between data tables of different databases in the same application environment, or incidence relations between data tables in different application environments, or a combination of more than two of the incidence relations;
receiving table information of a data table to be monitored;
according to the table information, obtaining table structure data of the data table to be monitored by using a preset script;
if detecting that the table structure data of the data table to be monitored is changed, recording change information;
acquiring an associated data table associated with a data table to be monitored, of which the table structure data is changed, from the data relation table, and checking whether the table structure data of the associated data table is correct or not according to the change information;
if the table structure data of the associated data table is incorrect, outputting abnormal information that the table structure data of the associated data table is not matched with the table structure data of the data table to be monitored, of which the table structure data is changed;
the table information comprises an application environment of a data table, a name of a database and a name of the data table;
the obtaining of the table structure data of the data table to be monitored by using a preset script according to the table information includes:
according to the application environment, the database name and the data table name of the data table to be monitored, connecting a preset script to a server corresponding to the application environment through a network; the preset script is used for describing a data source, a user name and a user password, and the data source at least comprises IP addresses of all databases and/or database connection ports;
logging in a database corresponding to the database name by using the user name and the user password, and acquiring the table structure data of the data table corresponding to the data table name;
if the table structure data of the data table to be monitored is detected to be changed, recording change information, including:
storing the table structure data of the data table to be monitored into a first file;
starting from the storage time of the first file, when a preset time interval is reached, acquiring the current table structure data of the data table to be monitored by using the preset script, and storing the current table structure data in a second file;
and comparing the first file with the second file, if the first file and the second file have difference data, acquiring change information of the change of the table structure data of the data table to be monitored according to the difference data, and recording the change information.
2. The data monitoring method according to claim 1, wherein the obtaining an associated data table associated with a data table to be monitored, in which table structure data is changed, from the data relationship table, and verifying whether the table structure data of the associated data table is correct according to the change information comprises:
acquiring a data table to be monitored corresponding to the change information, and taking the data table to be monitored as a reference table;
according to the data relation table, obtaining table structure data of an associated data table associated with the reference table;
and if the table structure data of the associated data table is not matched with the table structure data changed in the reference table, determining that the table structure data of the associated data table is incorrect.
3. The data monitoring method according to claim 1 or 2, wherein after recording change information if it is detected that the table structure data of the data table to be monitored is changed, the data monitoring method further comprises:
according to a preset data element standard, checking whether the change information meets the preset data element standard;
and if the change information does not accord with the preset data element standard, outputting the change information and error information.
4. The data monitoring method according to claim 1, wherein after outputting abnormal information that the table structure data of the associated data table does not match with the table structure data of the data table to be monitored, of which the table structure data is changed, if the table structure data of the associated data table is incorrect, the data monitoring method further comprises:
and maintaining the table structure data of the associated data table according to a maintenance mode input by a user.
5. A data monitoring device, the data monitoring device comprising:
the data relation table creating module is used for acquiring the incidence relation among different data tables in the database under each application environment and creating a data relation table according to the incidence relation; the incidence relation comprises at least one of incidence relations between different data tables in the same database, or incidence relations between data tables of different databases in the same application environment, or incidence relations between data tables in different application environments, or a combination of more than two of the incidence relations;
the table information receiving module is used for receiving the table information of the data table to be monitored;
the table structure data acquisition module is used for acquiring the table structure data of the data table to be monitored by using a preset script according to the table information;
the change information recording module is used for recording change information if detecting that the table structure data of the data table to be monitored is changed;
the associated data table checking module is used for acquiring an associated data table associated with a data table to be monitored, of which the table structure data is changed, from the data relation table and checking whether the table structure data of the associated data table is correct or not according to the change information;
the abnormal information output module is used for outputting abnormal information that the table structure data of the associated data table is not matched with the table structure data of the data table to be monitored, of which the table structure data is changed, if the table structure data of the associated data table is incorrect;
the table information comprises an application environment of a data table, a name of a database and a name of the data table;
the table structure data acquisition module is specifically configured to:
according to the application environment, the database name and the data table name of the data table to be monitored, connecting a preset script to a server corresponding to the application environment through a network; the preset script is used for describing a data source, a user name and a user password, and the data source at least comprises IP addresses of all databases and/or database connection ports; logging in a database corresponding to the database name by using the user name and the user password, and acquiring the table structure data of the data table corresponding to the data table name;
the change information recording module includes:
the first storage submodule is used for storing the table structure data of the data table to be monitored into a first file;
the second storage submodule is used for acquiring the current table structure data of the data table to be monitored by using the preset script from the storage time of the first file when a preset time interval is reached, and storing the current table structure data in a second file;
and the file comparison submodule is used for comparing the first file with the second file, and if the first file and the second file have difference data, acquiring change information of the change of the table structure data of the data table to be monitored according to the difference data and recording the change information.
6. The data monitoring apparatus of claim 5, wherein the associated data table check module comprises:
the reference table setting submodule is used for acquiring a data table to be monitored corresponding to the change information and taking the data table to be monitored as a reference table;
the associated data acquisition submodule is used for acquiring the table structure data of the associated data table associated with the reference table according to the data relation table;
and the data matching submodule is used for confirming that the table structure data of the associated data table is correct if the table structure data of the associated data table is matched with the table structure data changed in the reference table, or else, confirming that the table structure data of the associated data table is incorrect if the table structure data of the associated data table is not matched with the table structure data changed in the reference table.
7. The data monitoring device of claim 5 or 6, further comprising:
the change information checking module is used for checking whether the change information meets the preset data element standard or not according to the preset data element standard;
and the error information output module is used for outputting the change information and the error information if the change information does not accord with the preset data element standard.
8. A computer arrangement comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the data monitoring method according to any one of claims 1 to 4 when executing the computer program.
9. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the data monitoring method according to any one of claims 1 to 4.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810574141.2A CN108874644B (en) | 2018-06-06 | 2018-06-06 | Data monitoring method and device, computer equipment and storage medium |
| PCT/CN2018/092647 WO2019232832A1 (en) | 2018-06-06 | 2018-06-25 | Data monitoring method and device, computer device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810574141.2A CN108874644B (en) | 2018-06-06 | 2018-06-06 | Data monitoring method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108874644A CN108874644A (en) | 2018-11-23 |
| CN108874644B true CN108874644B (en) | 2021-12-24 |
Family
ID=64337108
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810574141.2A Active CN108874644B (en) | 2018-06-06 | 2018-06-06 | Data monitoring method and device, computer equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108874644B (en) |
| WO (1) | WO2019232832A1 (en) |
Families Citing this family (38)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111275846B (en) * | 2018-12-04 | 2022-01-07 | 比亚迪股份有限公司 | Data record generation method and device, electronic equipment and storage medium |
| CN109815111A (en) * | 2018-12-13 | 2019-05-28 | 深圳壹账通智能科技有限公司 | Gray scale test method, device, computer equipment and storage medium |
| CN110209521B (en) * | 2019-02-22 | 2022-03-18 | 腾讯科技（深圳）有限公司 | Data verification method and device, computer readable storage medium and computer equipment |
| CN110059071B (en) * | 2019-03-13 | 2024-05-31 | 深圳市大梦龙途网络科技有限公司 | Data comparison monitoring method, system, equipment and medium based on storage process |
| CN111752956B (en) * | 2019-03-26 | 2023-12-08 | 华为技术有限公司 | Entity association change method and device and electronic equipment |
| CN110263032B (en) * | 2019-05-13 | 2023-08-29 | 平安科技（深圳）有限公司 | Method, device, computer equipment and storage medium for comparing table structures in database |
| CN110347672A (en) * | 2019-05-27 | 2019-10-18 | 深圳壹账通智能科技有限公司 | Verification method and device, the electronic equipment and storage medium of tables of data related update |
| CN110377569B (en) * | 2019-06-19 | 2023-07-28 | 中国平安人寿保险股份有限公司 | Log monitoring method, device, computer equipment and storage medium |
| CN110647526B (en) * | 2019-08-15 | 2023-06-16 | 平安健康保险股份有限公司 | Batch data processing method, device, computer equipment and storage medium |
| CN111221918A (en) * | 2019-11-04 | 2020-06-02 | 深圳力维智联技术有限公司 | Data updating method, device, product and medium based on relational database |
| CN111125101B (en) * | 2019-12-16 | 2023-10-13 | 杭州涂鸦信息技术有限公司 | Data center table structure consistency monitoring method and system |
| CN113051265A (en) * | 2019-12-27 | 2021-06-29 | 中信百信银行股份有限公司 | Method, device, computer equipment and readable storage medium for reducing loss caused by relational database table structure change |
| CN111327680B (en) * | 2020-01-21 | 2023-01-03 | 数字广东网络建设有限公司 | Authentication data synchronization method, device, system, computer equipment and storage medium |
| CN111259104A (en) * | 2020-01-22 | 2020-06-09 | 山东浪潮通软信息科技有限公司 | Report data synchronization method and device capable of defining rules |
| CN111324593A (en) * | 2020-02-17 | 2020-06-23 | 广州虎牙科技有限公司 | Data table detection method and device, electronic equipment and computer readable storage medium |
| CN111400294B (en) * | 2020-03-12 | 2023-08-01 | 时时同云科技(成都)有限责任公司 | Data anomaly monitoring method, device and system |
| CN111581191B (en) * | 2020-04-10 | 2023-10-13 | 岭东核电有限公司 | Nuclear safety data verification method, device, computer equipment and storage medium |
| CN111488363B (en) * | 2020-06-28 | 2020-10-02 | 平安国际智慧城市科技股份有限公司 | Data processing method, device, electronic equipment and medium |
| CN111752925B (en) * | 2020-06-29 | 2024-05-14 | 浪潮通用软件有限公司 | Visual data management method |
| CN111754131A (en) * | 2020-06-30 | 2020-10-09 | 苏州朗动网络科技有限公司 | Enterprise information dynamic monitoring method, equipment and medium |
| CN111782489B (en) * | 2020-07-16 | 2024-05-10 | 上海乾臻信息科技有限公司 | Data table monitoring method and device |
| CN112001710A (en) * | 2020-09-07 | 2020-11-27 | 山东钢铁集团日照有限公司 | Big data reading and integrating system in steel product production process |
| CN112423032B (en) * | 2020-10-21 | 2022-11-11 | 当趣网络科技(杭州)有限公司 | Data monitoring method and device based on smart television, electronic equipment and medium |
| CN112487282A (en) * | 2020-11-04 | 2021-03-12 | 山东中创软件商用中间件股份有限公司 | Business processing condition monitoring method and device, electronic equipment and storage medium |
| CN112347103B (en) * | 2020-11-05 | 2024-04-12 | 深圳市极致科技股份有限公司 | Data synchronization method, device, electronic equipment and storage medium |
| CN112396332B (en) * | 2020-11-25 | 2024-09-06 | 上汽通用五菱汽车股份有限公司 | Data synchronization method, device, equipment and medium based on multi-vehicle type part change |
| CN113760891B (en) * | 2020-12-02 | 2024-07-16 | 北京沃东天骏信息技术有限公司 | Data table generation method, device, equipment and storage medium |
| CN112579398B (en) * | 2020-12-28 | 2024-07-16 | 中国建设银行股份有限公司 | Monitoring deployment method and device |
| CN112768039A (en) * | 2020-12-31 | 2021-05-07 | 平安国际智慧城市科技股份有限公司 | Information monitoring method and device based on artificial intelligence, computer equipment and medium |
| CN112947973A (en) * | 2021-04-08 | 2021-06-11 | 天津天地伟业智能安全防范科技有限公司 | Configuration updating method and device of middleware service program and electronic equipment |
| CN113051330B (en) * | 2021-04-13 | 2024-07-16 | 福州外语外贸学院 | Real-time environment monitoring method and device based on big data and computer equipment |
| CN113420026B (en) * | 2021-06-22 | 2023-10-31 | 康键信息技术（深圳）有限公司 | Database table structure changing method, device, equipment and storage medium |
| CN113360513A (en) * | 2021-06-24 | 2021-09-07 | 平安普惠企业管理有限公司 | Report influence analysis method and device, computer equipment and storage medium |
| CN113986907A (en) * | 2021-12-24 | 2022-01-28 | 深圳市明源云科技有限公司 | Graphic maintenance method, device and equipment for abnormal data and storage medium |
| CN114915643B (en) * | 2022-05-09 | 2024-01-26 | 北京交大微联科技有限公司 | Configuration method, device, equipment and medium of railway signal centralized monitoring system |
| CN115655757A (en) * | 2022-11-04 | 2023-01-31 | 中车长春轨道客车股份有限公司 | Method and system for determining vehicle-mounted superconducting magnet monitoring system and storage medium |
| CN116069775B (en) * | 2023-04-06 | 2023-08-22 | 上海二三四五网络科技有限公司 | Data quality verification system and method for data warehouse |
| CN116401239A (en) * | 2023-04-12 | 2023-07-07 | 广州绿石碳科技股份有限公司 | Data extraction method and system of carbon emission data storage analysis platform |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7127449B2 (en) * | 2003-08-21 | 2006-10-24 | International Business Machines Corporation | Data query system load optimization |
| CN104572672B (en) * | 2013-10-15 | 2018-10-02 | 北大方正集团有限公司 | The synchronous method and application system of heterogeneous database |
| CN104636483A (en) * | 2015-02-16 | 2015-05-20 | 广东省公安厅 | Data monitoring method |
| CN105095056B (en) * | 2015-08-14 | 2018-01-12 | 焦点科技股份有限公司 | A kind of method of data warehouse data monitoring |
| CN106649378B (en) * | 2015-11-02 | 2020-07-14 | 北大方正集团有限公司 | Data synchronization method and device |
| CN105335527A (en) * | 2015-12-04 | 2016-02-17 | 浪潮(北京)电子信息产业有限公司 | Method and system for monitoring database operation |
| CN107015991B (en) * | 2016-01-28 | 2020-10-30 | 创新先进技术有限公司 | Self-checking method, device and system for data consistency and service device |
| CN106326376A (en) * | 2016-08-15 | 2017-01-11 | 东软集团股份有限公司 | Method and device for copying information after table structure modification |
| CN107220301A (en) * | 2017-05-10 | 2017-09-29 | 北京小度信息科技有限公司 | The data monitoring method and device of a kind of configurableization |
-
2018
- 2018-06-06 CN CN201810574141.2A patent/CN108874644B/en active Active
- 2018-06-25 WO PCT/CN2018/092647 patent/WO2019232832A1/en active Application Filing
Also Published As
| Publication number | Publication date |
|---|---|
| CN108874644A (en) | 2018-11-23 |
| WO2019232832A1 (en) | 2019-12-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108874644B (en) | Data monitoring method and device, computer equipment and storage medium | |
| CN109032611B (en) | Script deployment method and device, computer equipment and storage medium | |
| CN109474578B (en) | Message checking method, device, computer equipment and storage medium | |
| CN109032824B (en) | Database verification method, database verification device, computer equipment and storage medium | |
| CN109446068B (en) | Interface test method, device, computer equipment and storage medium | |
| CN110569035B (en) | Code compiling method, device, equipment and storage medium of software development project | |
| CN110941546A (en) | Automatic test method, device, equipment and storage medium for WEB page case | |
| CN107688664B (en) | Chart generation method and device, computer equipment and storage medium | |
| CN110088744B (en) | Database maintenance method and system | |
| WO2020253122A1 (en) | Data verification method and device, computer equipment and storage medium | |
| CN111538779A (en) | Incremental data synchronization method and device, computer equipment and storage medium | |
| CN109284125B (en) | Method, device, equipment and medium for configuring dependency package in big data platform | |
| CN110063042B (en) | Database fault response method and terminal thereof | |
| CN110688305B (en) | Test environment synchronization method, device, medium and electronic equipment | |
| CN110737594A (en) | Database standard conformance testing method and device for automatically generating test cases | |
| CN111611009A (en) | Database script management method and device, computer equipment and storage medium | |
| CN110750443A (en) | Webpage testing method and device, computer equipment and storage medium | |
| CN114610635A (en) | Interface test case generation method and device, computer equipment and storage medium | |
| CN111367754A (en) | Data monitoring method and device, computer equipment and storage medium | |
| CN112612706A (en) | Automated testing method, computer device and storage medium | |
| CN112069223A (en) | Data acquisition demand processing method and device, computer equipment and storage medium | |
| CN111427864A (en) | Batch archiving method, device, equipment and storage medium for data | |
| CN111522881A (en) | Service data processing method, device, server and storage medium | |
| CN109240906B (en) | Database configuration information adaptation method and device, computer equipment and storage medium | |
| CN113918594A (en) | Method and device for generating structured query statement and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |