CN116361391A - Method and device for detecting and repairing structural abnormality of data synchronization table - Google Patents

Method and device for detecting and repairing structural abnormality of data synchronization table Download PDF

Info

Publication number
CN116361391A
CN116361391A CN202310328337.4A CN202310328337A CN116361391A CN 116361391 A CN116361391 A CN 116361391A CN 202310328337 A CN202310328337 A CN 202310328337A CN 116361391 A CN116361391 A CN 116361391A
Authority
CN
China
Prior art keywords
data synchronization
field
data
synchronization task
abnormal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310328337.4A
Other languages
Chinese (zh)
Inventor
彭壮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
CLP Cloud Digital Intelligence Technology Co Ltd
Original Assignee
CLP Cloud Digital Intelligence Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by CLP Cloud Digital Intelligence Technology Co Ltd filed Critical CLP Cloud Digital Intelligence Technology Co Ltd
Priority to CN202310328337.4A priority Critical patent/CN116361391A/en
Publication of CN116361391A publication Critical patent/CN116361391A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2282Tablespace storage structures; Management thereof
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24552Database cache management
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24553Query execution of query operations
    • G06F16/24558Binary matching operations
    • G06F16/2456Join operations
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The invention relates to the technical field of data processing, and provides a method and a device for detecting and repairing structural abnormality of a data synchronization table, wherein the method comprises the following steps: configuring operation parameters of the data synchronization task, and delaying the data synchronization task according to the configured operation parameters; acquiring the latest table structure and configuration information corresponding to the data synchronization task which is delayed to be issued, comparing the table structure cached in the configuration information with the latest table structure, acquiring an abnormal result and grading the abnormal result; transmitting corresponding abnormal early warning information according to the abnormal result to the responsible person of the data synchronization task; and repairing the table structure of the data synchronization task according to the early warning information and the abnormal result classification. The invention can realize the active detection of the table structure abnormality of the total quantity and task level of the data synchronization system, improve the processing efficiency of the table structure abnormality synchronization task of personnel, and improve the operation success rate and the data accuracy of the data synchronization task.

Description

Method and device for detecting and repairing structural abnormality of data synchronization table
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and an apparatus for detecting and repairing a structural abnormality of a data synchronization table.
Background
The data synchronization system is used for data exchange and data synchronization among various heterogeneous data sources under large data volume, provides a solution for data exchange, transmission and cleaning among various users and different service systems, can simplify complex data synchronization configuration work, and easily realizes data integration and sharing.
In practical application, because of complex service system, numerous databases and data tables, high maintenance and management difficulty of the data tables, and in the process of using the data synchronization system to synchronize data, abnormal changes of the source end table and the target end table structure of various data synchronization can be faced, and the abnormal changes can cause operation failure of a large number of data synchronization tasks or generate a large number of dirty data, so that great difficulty is brought to the data synchronization working belt. The traditional method mainly carries out post-processing on the appeared change abnormality, after the operation failure of the data synchronization task due to the change abnormality of the table structure, the method uses manual investigation, repairs the appeared table structure abnormality one by one in a targeted way, and then rerun the data synchronization task.
Therefore, how to provide a more accurate and efficient method for detecting and repairing abnormal table structure is a technical problem to be solved.
Disclosure of Invention
In view of the above, the present invention aims to overcome the defects of the prior art, and to provide a method and an apparatus for detecting and repairing structural anomalies of a data synchronization table.
In one aspect, the present invention provides a method for detecting and repairing structural anomalies of a data synchronization table, including:
step S1: configuring operation parameters of the data synchronization task, and delaying the data synchronization task according to the configured operation parameters;
step S2: acquiring the latest table structure and configuration information corresponding to the data synchronization task which is delayed to be issued, comparing the table structure cached in the configuration information with the latest table structure, acquiring an abnormal result and grading the abnormal result;
step S3: transmitting corresponding abnormal early warning information according to the abnormal result to the responsible person of the data synchronization task;
step S4: and repairing the table structure of the data synchronization task according to the early warning information and the abnormal result classification.
As a preferred embodiment of the present invention, in the method for detecting and repairing a structural abnormality of a data synchronization table, step S1 includes:
step S11: configuring the running time, the running frequency and the running interval time of the data synchronization task;
step S12: monitoring the running state of the data synchronization task, and obtaining the time point when the running of the data synchronization task is completed;
step S13: obtaining a time point when the next data synchronization task is scheduled according to the running time and the running frequency of the data synchronization task and the obtained time point when the running of the data synchronization task is completed;
step S14: obtaining a time point of delayed issuing of the data synchronization task according to the running interval time of the data synchronization task and the time point of the next scheduled data synchronization task;
step S15: and carrying out delay issuing on the data synchronization task according to the obtained time point of delay issuing of the data synchronization task.
In the method for detecting and repairing abnormal structure of data synchronization table, step S1, delay issuing of data synchronization task is performed according to configured operation parameters, including:
setting delay time by adopting a RDelayedQueue queue and storing a data synchronization task which is issued in a delayed manner;
when the set delay time is reached, the RDelayedQueue queue sends the stored data synchronization task to the RBlongungQueue queue;
and monitoring event changes in the RBlongQueue queue by adopting a monitoring thread, and taking out a data synchronization task from the RBlongQueue queue for issuing when a new event is generated.
As a preferred embodiment of the present invention, in the method for detecting and repairing a structural abnormality of a data synchronization table, step S2 includes:
step S21: acquiring an latest table structure corresponding to a data synchronization task in real time through JDBC connection;
step S22: acquiring configuration information of a data synchronization task, and comparing a table structure cached in the configuration information with the latest table structure in the step S21 to acquire an abnormal result;
step S23: and classifying the abnormal results according to the influence of the abnormal results on the operation of the data synchronization task.
As a preferred aspect of the present invention, in the method for detecting and repairing a structural abnormality of a data synchronization table, step S21 includes: acquiring source end and target end database names, data table names, corresponding data source connection information and an up-to-date table structure of a data table of a data synchronization task in real time through JDBC connection, wherein the data source connection information comprises a connection address, a user name and a password, the up-to-date table structure stores field information in a list form, and the field information comprises a field name, a field length, precision and a field type; and when the source end and the target end of the data synchronization task do not exist, acquiring an empty table structure.
As a preferred aspect of the present invention, in the method for detecting and repairing a structural abnormality of a data synchronization table, step S22 includes: and acquiring configuration information of the stored data synchronization task from a database or a storage device, and acquiring a cached table structure in the configuration information by acquiring a full field list of the cached source-end and target-end data tables and a field mapping relation between the source-end and target-end data tables from the configuration information.
As a preferred aspect of the present invention, in the method for detecting and repairing a structural abnormality of a data synchronization table, step S23 includes:
deleting a source end table or a target end table of the data synchronization task, acquiring an up-to-date table structure list in real time as empty, judging an abnormal result as a table deletion, and dividing the abnormal result into first-level abnormality;
when a field is missing in a field of which the mapping relation is configured in the source end or the target end data table, judging an abnormal result as field missing, and dividing the abnormal result into two-level anomalies;
when a field type change exists in a field of the configured mapping relation of the source end or the target end data table, judging an abnormal result as the field type change, and dividing the abnormal result into two-level anomalies;
when the field length change exists in the field of the configured mapping relation of the source end or the target end data table, judging the abnormal result as the field length change, and dividing the abnormal result into two-level abnormality;
when the field of the mapping relation is not configured in the source end or the target end data table, judging the abnormal result as field missing, and dividing the abnormal result into three levels of abnormal;
when a field is newly added in the source end or target end data table, judging an abnormal result as the field is newly added, and dividing the abnormal result into three levels of abnormal;
when a field type change exists in a field of which the mapping relation is not configured in the source end or the target end data table, judging an abnormal result as the field type change, and dividing the abnormal result into three levels of abnormal;
when the field length change exists in the field of which the mapping relation is not configured in the source end or the target end data table, judging the abnormal result as the field length change, and dividing the abnormal result into three levels of abnormal.
In the method for detecting and repairing the structural abnormality of the data synchronization table, the abnormality pre-warning information in the step S3 includes the data synchronization task number, the database name, the data table name and the abnormality result type.
As a preferred embodiment of the present invention, in the method for detecting and repairing a structural abnormality of a data synchronization table, step S4 includes:
when the abnormal result is a table missing, acquiring a full field list of a stored data table from configuration information of a data synchronization task, assembling and generating a table construction statement DDL corresponding to the data table according to the data source type of the data table, and creating a data table by the table construction statement DDL;
when the abnormal result is that the field is missing, acquiring stored field information of the missing field from configuration information of the data synchronization task, assembling and generating an SQL statement of a newly added field according to the data source type of the data table, and recovering the missing field through the SQL statement of the newly added field;
when the abnormal result is field type change, the mapping relation of the type change field is readjusted through a graphical interface, and the new mapping relation is persisted to a database or related storage equipment;
when the abnormal result is that the field length is changed, the mapping relation of the length change field is readjusted through the graphical interface, and the new mapping relation is persisted in a database or related storage equipment;
when the abnormal result is that the field is newly added, the mapping relation of the newly added field is readjusted through the graphical interface, and the new mapping relation is persisted to a database or related storage equipment.
On the other hand, the invention provides a device for detecting and repairing structural abnormality of a data synchronization table, which comprises the following components:
the scheduling module is used for configuring the operation parameters of the data synchronization task and delaying the data synchronization task according to the configured operation parameters;
the abnormality detection module is used for acquiring the latest table structure and configuration information corresponding to the data synchronization task issued in a delayed manner, comparing the table structure cached in the configuration information with the latest table structure, acquiring an abnormality result and grading the abnormality result;
the abnormal early warning module is used for sending corresponding abnormal early warning information to a responsible person of the data synchronization task according to an abnormal result;
and the abnormality repairing module is used for repairing the table structure of the data synchronization task according to the early warning information and the abnormal result classification.
The method and the device for detecting and repairing the structural abnormality of the data synchronization table have the following beneficial effects:
1. the front processing scheme is adopted to detect the table structure, the two detection modes of timing detection and active detection of the scheduling module are combined to realize active detection of the table structure abnormality, early warning is carried out in advance, operation failure conditions of a data synchronization task due to the table structure abnormality are reduced, or dirty data conditions due to the table structure abnormality are reduced, active detection of the table structure abnormality of the whole data synchronization system and the task level is realized, and early perception of the table structure abnormality can be realized.
2. Based on the detection result of the table structure abnormality, the abnormality types are divided according to the influence on the task operation degree, a corresponding graphic table structure abnormality repairing function is provided, the processing efficiency of the table structure abnormality synchronous task by personnel is improved, the workload of manual investigation is reduced, and the operation success rate and the data accuracy of the data synchronous task are improved.
3. Aiming at the analyzed table structure abnormal results, the table structure abnormal conditions such as table deletion, field type change, field length change, field addition and the like are conveniently repaired through a graphical interface, and the original data synchronization task configuration is not required to be additionally adjusted, so that the operation is simplified.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a method for detecting and repairing an anomaly of a data synchronization table according to an exemplary first embodiment of the present invention.
Fig. 2 is a flowchart of a method for detecting and repairing an anomaly of a data synchronization table according to a second embodiment of the present invention.
Fig. 3 is a flowchart of a method for detecting and repairing structural anomalies of a data synchronization table according to a fourth embodiment of the present invention.
Fig. 4 is a schematic diagram of a data synchronization table structure abnormality detection and repair device according to a seventh embodiment of the present invention.
Fig. 5 is a schematic structural diagram of the apparatus provided by the present invention.
Detailed Description
Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
It should be noted that, without conflict, the following embodiments and features in the embodiments may be combined with each other; and, based on the embodiments in this disclosure, all other embodiments that may be made by one of ordinary skill in the art without inventive effort are within the scope of the present disclosure.
It is noted that various aspects of the embodiments are described below within the scope of the following claims. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and/or function described herein is merely illustrative. Based on the present disclosure, one skilled in the art will appreciate that one aspect described herein may be implemented independently of any other aspect, and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method practiced using any number of the aspects set forth herein. In addition, such apparatus may be implemented and/or such methods practiced using other structure and/or functionality in addition to one or more of the aspects set forth herein.
The technical principle of the invention is as follows:
the invention provides a method and a device for detecting and repairing the structural abnormality of a data synchronization table, which adopts a scheduling module to monitor and identify a data synchronization task to be scheduled after 30 minutes (time is adjustable) and actively issue the task to a table structural abnormality detection module; the abnormality detection module adopts a timing detection and active detection mode, acquires latest table structure information from the tables of the source end and the target end of the data synchronization task through a JDBC mode, classifies and classifies the latest table structure information according to the change condition of the table structure after comparing the latest table structure information with the configuration information, and generates a table structure abnormality detection result; the abnormal early warning module sends early warning information to related responsible persons according to the generated table structure abnormal detection result; in addition, the exception handling module provides a graphical interface for repairing the table structure exception according to different types of table structure exceptions.
The invention adopts a pre-processing scheme to detect the table structure, combines two detection modes of timing detection and active detection of the scheduling module to realize active detection of the table structure abnormality, early warning is carried out in advance, and operation failure conditions of the data synchronization task caused by the table structure abnormality are reduced or dirty data conditions caused by the table structure abnormality are reduced; based on the detection result of the table structure abnormality, the abnormality types are divided according to the influence on the task operation degree, a corresponding graphic table structure abnormality repairing function is provided, the processing efficiency of the table structure abnormality synchronous task by personnel is improved, the workload of manual investigation is reduced, and the operation success rate and the data accuracy of the data synchronous task are improved.
The terms in the following examples are explained as follows:
rediston: an advanced distributed coordination Redis client.
RDelayedQueue queue: RDelayedQueue of Redis-based Redis-son distributed delay Queue (Delayed Queue) structure, java objects provide the function of delaying adding items to the Queue as required on the basis of realizing a RQUE interface.
Rblongungqueue: redis-based RBlockingQueue of Redis distributed unbounded Blocking double-end Queue (Blocking Queue) structure, java object realizes Java.
DDL: the database schema definition language DDL (Data Definition Language) is a language for describing real world entities to be stored in a database.
SQL: SQL (Structured Query Language) is a database language with multiple functions such as data manipulation and data definition, which has the characteristic of interactivity and can provide great convenience to users.
Fig. 1 is a flowchart of a method for detecting and repairing an anomaly of a data synchronization table structure according to an exemplary first embodiment of the present invention. As shown in fig. 1, the method of the present embodiment is implemented as follows:
step S1: configuring operation parameters of the data synchronization task, and delaying the data synchronization task according to the configured operation parameters;
step S2: acquiring the latest table structure and configuration information corresponding to the data synchronization task which is delayed to be issued, comparing the table structure cached in the configuration information with the latest table structure, acquiring an abnormal result and grading the abnormal result;
step S3: transmitting corresponding abnormal early warning information according to the abnormal result to the responsible person of the data synchronization task;
step S4: and repairing the table structure of the data synchronization task according to the early warning information and the abnormal result classification.
The abnormality early warning information in step S3 of the method of this embodiment includes a data synchronization task number, a database name, a data table name, and an abnormality result type. Because in the field of data synchronization, the source end table and the target end table are all of a third party, if an automatic repair scheme is adopted after an abnormality is detected by a program, the security risk is presented to the third party database, and the method of the embodiment focuses on realizing the automatic detection of the table structure of the data synchronization task, and for the occurrence of the abnormality, the corresponding repair is required after the confirmation of a data operator, thereby avoiding the unknown risk brought by the automatic repair. In addition, after the detection is completed, a visual graphical interface is provided for an operator to check the abnormal condition of the task list structure.
Fig. 2 is a flowchart of a method for detecting and repairing an anomaly of a data synchronization table structure according to a second embodiment of the present invention, which is a preferred embodiment of the method shown in fig. 1, as shown in fig. 2, step S1 of the method of the present embodiment is implemented as follows:
step S11: configuring the running time, the running frequency and the running interval time of the data synchronization task;
step S12: monitoring the running state of the data synchronization task, and obtaining the time point when the running of the data synchronization task is completed;
step S13: obtaining a time point when the next data synchronization task is scheduled according to the running time and the running frequency of the data synchronization task and the obtained time point when the running of the data synchronization task is completed;
step S14: obtaining a time point of delayed issuing of the data synchronization task according to the running interval time of the data synchronization task and the time point of the next scheduled data synchronization task;
step S15: and carrying out delay issuing on the data synchronization task according to the obtained time point of delay issuing of the data synchronization task.
In practical application, the data synchronization task to be scheduled is issued to perform table structure abnormality detection by monitoring and identifying the data synchronization task to be scheduled after 30 minutes (the time can be correspondingly configured according to the scheduling interval of the practical task).
After each time of data synchronization task operation is completed, calculating a time point when the next data synchronization task is to be scheduled, and pushing back for 30 minutes forward through the time point to calculate that the data synchronization task is to be issued to perform abnormality detection, for example, the time point t1 when the data synchronization task operation is completed is 2022-10-20:02:00:00, the time point when the data synchronization task is to be scheduled is 2022-10-2010:00:00, and the time point t2 when the data synchronization task is issued to the table structure abnormality detection module is 2022-10-2009:30:00.
An exemplary third embodiment of the present invention provides a method for detecting and repairing structural anomalies of a data synchronization table, where the method is a preferred embodiment of the method shown in fig. 1, and in step S1 of the method of the present embodiment, the method delays issuing a data synchronization task according to configured operation parameters, including:
setting delay time by adopting a RDelayedQueue queue and storing a data synchronization task which is issued in a delayed manner;
when the set delay time is reached, the RDelayedQueue queue sends the stored data synchronization task to the RBlongungQueue queue;
and monitoring event changes in the RBlongQueue queue by adopting a monitoring thread, and taking out a data synchronization task from the RBlongQueue queue for issuing when a new event is generated.
In order to ensure timeliness of task issuing, the method of the embodiment combines with a Redison delay queue to accurately realize delay issuing of each data synchronization task. Redisson provides two queues RBlongQueue and RDelayedQueue to realize the function of a delay queue, wherein the RDelayedQueue queue is used for storing delay tasks and setting delay time, and the RblocingQueue queue is used for acquiring the delay tasks. When the set delay time is reached, the RDelayedQuue queue puts the stored delay task into the RBlongungQuue queue, starts a thread to monitor event change in the RBlongungQuue queue, and when a new event is generated, takes out the delay task from the RBlongungQuue queue for processing.
Fig. 3 is a flowchart of a method for detecting and repairing an anomaly of a data synchronization table structure according to a fourth embodiment of the present invention, which is a preferred embodiment of the method shown in fig. 1, as shown in fig. 3, step S2 of the method of the present embodiment includes:
step S21: acquiring an latest table structure corresponding to a data synchronization task in real time through JDBC connection;
step S22: acquiring configuration information of a data synchronization task, and comparing a table structure cached in the configuration information with the latest table structure in the step S21 to acquire an abnormal result;
step S23: and classifying the abnormal results according to the influence of the abnormal results on the operation of the data synchronization task.
An exemplary fifth embodiment of the present invention provides a method for detecting and repairing structural anomalies of a data synchronization table, which is a preferred embodiment of the methods shown in fig. 1 and 3. Specifically, step S2 of the method of the present embodiment is implemented in the following manner:
the table structure abnormality detection is mainly to identify the change condition of the table by comparing the difference between the configuration information of the data synchronization task and the real-time table structure information, so that the latest table structure needs to be queried in real time. Because the data synchronization is mainly applied to various relational databases and data warehouses, in order to better adapt and expand various data sources, the source end and target end database names, data table names, corresponding data source connection information and the latest table structure of a data table of a data synchronization task are obtained in real time through JDBC connection, the data source connection information comprises a connection address, a user name and a password, the latest table structure stores field information in a list form, and the field information comprises a field name, a field length, precision and a field type; and when the source end and the target end of the data synchronization task do not exist, acquiring an empty table structure.
And acquiring configuration information of the stored data synchronization task from a database or a storage device, and acquiring a cached table structure in the configuration information by acquiring a full field list of the cached source-end and target-end data tables and a field mapping relation between the source-end and target-end data tables from the configuration information. Comparing the cached table structure in the configuration information with the real-time latest table structure, classifying and grading the compared abnormal results according to the influence degree on the synchronous task operation:
deleting a source end table or a target end table of the data synchronization task, acquiring an up-to-date table structure list in real time as empty, judging an abnormal result as a table deletion, and dividing the abnormal result into first-level abnormality; the primary exception has very serious influence on the normal operation of the data synchronization task, and the table missing can cause the direct operation failure of the data synchronization task.
The secondary exception has serious influence on the normal operation of the data synchronization task, and the secondary exception comprises field deletion, field type change and field length change.
When a field is missing in a field of which the mapping relation is configured in the source end or the target end data table, judging an abnormal result as field missing, and dividing the abnormal result into two-level anomalies; the field missing can cause the number of fields of the source end and the target end of the data synchronization task to be out of pairs, thereby causing the data synchronization task to run failure.
When the field type change exists in the field of the configured mapping relation of the source end or the target end data table, the abnormal result is judged to be the field type change, and the abnormal result is divided into two-level abnormal. For example, when the field configuration is performed, the data type is Date, and when the latest field type acquired during detection is changed to Int, the field data types of the source end and the target end of the data synchronization task are not matched, so that the data synchronization task data cannot be inserted into the target end to generate dirty data.
When the field length change exists in the field of the configured mapping relation of the source end or the target end data table, judging the abnormal result as the field length change, and dividing the abnormal result into two-level abnormality. For example, the data length in the field configuration is 20, and the latest field length acquired in the detection is changed to 10. This situation may result in mismatch of the source and destination field data lengths of the data sync task, which may result in dirty data being generated by the inability of the data sync task data to be inserted into the destination.
Three-level anomalies do not affect the normal operation of the data sync task, but need to be of interest, including: a field miss, a field addition, a field type change, and a field length change. In order to avoid the fact that the system erroneously recognizes that the field type is abnormal, the basic field type mapping configuration allows the correspondence relationship to exist, so that the field type is not recognized as abnormal, and only when a scene which cannot be converted occurs, such as conversion of a numerical type and a date type, the field type is recognized.
When the field of the mapping relation is not configured in the source end or the target end data table, judging the abnormal result as field missing, and dividing the abnormal result into three levels of abnormal;
when a field is newly added in the source end or target end data table, judging an abnormal result as the field is newly added, and dividing the abnormal result into three levels of abnormal;
when the field type change exists in the field of the source end or target end data table which is not configured with the mapping relation, judging the abnormal result as the field type change, and dividing the abnormal result into three levels of abnormal. For example, the data type in field configuration is Date, and the latest field type acquired in detection is changed to Int.
When the field length change exists in the field of which the mapping relation is not configured in the source end or the target end data table, judging the abnormal result as the field length change, and dividing the abnormal result into three levels of abnormal. For example, the data length in the field configuration is 20, and the latest field length acquired in the detection is changed to 10.
An exemplary sixth embodiment of the present invention provides a method for detecting and repairing structural anomalies of a data synchronization table, which is a preferred implementation of the method shown in fig. 1, and step S4 of the method of the present embodiment is implemented in the following manner:
when the abnormal result is a table missing, acquiring a full field list of a stored data table from configuration information of a data synchronization task, assembling and generating a table construction statement DDL corresponding to the data table according to the data source type of the data table, and creating a data table by the table construction statement DDL;
when the abnormal result is that the field is missing, acquiring stored field information of the missing field from configuration information of the data synchronization task, assembling and generating an SQL statement of a newly added field according to the data source type of the data table, and recovering the missing field through the SQL statement of the newly added field; in addition, for the abnormal situation of field missing, the mapping relation of the fields of the source end and the target end can be displayed through a graphical interface, the mapping relation of the missing fields is readjusted by means of the graphical interface, and then the new mapping relation is persisted in the related storage equipment.
When the abnormal result is field type change, the mapping relation between the source end and the target end is displayed through the graphical interface, the mapping relation of the field of the type change is readjusted through the graphical interface, and the new mapping relation is persisted in a database or related storage equipment.
When the abnormal result is that the field length is changed, the mapping relation between the source end and the target end is displayed through a graphical interface, the mapping relation of the field with the length changed is readjusted through the graphical interface, and the new mapping relation is persisted in a database or related storage equipment;
when the abnormal result is that the field is newly added, the mapping relation between the source end and the target end field is displayed through the graphical interface, the mapping relation of the newly added field is readjusted through the graphical interface, and the new mapping relation is persisted to a database or related storage equipment.
After the table structure abnormality of the data synchronization task is repaired by the abnormality repair module, the missing data table and field can be quickly recovered, or after the field mapping is adjusted, the new field mapping relation can be saved again, the original data synchronization task configuration does not need to be additionally modified, and the subsequent data synchronization task can automatically acquire the new field mapping relation from the database or related storage equipment to perform normal task operation.
Fig. 4 is a schematic diagram of a data synchronization table structure anomaly detection and repair device according to a seventh embodiment of the present invention, as shown in fig. 4, the device of the present embodiment includes:
the scheduling module is used for configuring the operation parameters of the data synchronization task and delaying the data synchronization task according to the configured operation parameters;
the abnormality detection module is used for acquiring the latest table structure and configuration information corresponding to the data synchronization task issued in a delayed manner, comparing the table structure cached in the configuration information with the latest table structure, acquiring an abnormality result and grading the abnormality result;
the abnormal early warning module is used for sending corresponding abnormal early warning information to a responsible person of the data synchronization task according to an abnormal result;
and the abnormality repairing module is used for repairing the table structure of the data synchronization task according to the early warning information and the abnormal result classification.
The abnormality detection module of the device of the embodiment adopts two modes of timing detection and active detection. In practical application, there may be hundreds or thousands of data synchronization tasks in a state to be operated, and the operation periods of the data synchronization tasks are different, for example, year, month, day, hour, and minute.
The abnormality detection module of the device of this embodiment supports an active detection mode at the same time. The data synchronization task to be operated is actively issued to the table structure abnormality detection module by depending on the delay issuing function of the scheduling module, so that the active table structure abnormality detection before the data synchronization task is operated is realized, the active detection belongs to the task level active detection, and the active detection mode can ensure that the table structure abnormality detection is carried out once before each data synchronization task is operated, thereby discovering the table structure abnormality condition in advance and reducing the task operation failure condition.
The abnormality repair module of the device of the embodiment provides a clear and simple repair mode, the original configuration task is not required to be edited and modified again, and the repair of the table structure abnormality can be realized through a graphical interface.
As shown in FIG. 5, the present invention also provides an apparatus comprising a processor 110, a communication interface 120, a memory 130 for storing a processor executable computer program, and a communication bus 140. The processor 110, the communication interface 120, and the memory 130 perform communication with each other through the communication bus 140. The processor 110 implements the data synchronization table structure anomaly detection and repair method described above by running an executable computer program.
Wherein the computer program in the memory 130 may be implemented in the form of software functional units and may be stored in a computer readable storage medium when sold or used as a separate product. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The system embodiments described above are merely illustrative, in which elements illustrated as separate elements may or may not be physically separate, and elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected based on actual needs to achieve the purpose of the embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on such understanding, the foregoing technical solutions may be embodied essentially or in part in the form of a software product, which may be stored in a computer-readable storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the various embodiments or methods of some parts of the embodiments.
The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any changes or substitutions easily contemplated by those skilled in the art within the scope of the present invention should be included in the present invention. Therefore, the protection scope of the invention is subject to the protection scope of the claims.

Claims (10)

1. The method for detecting and repairing the structural abnormality of the data synchronization table is characterized by comprising the following steps:
step S1: configuring operation parameters of the data synchronization task, and delaying the data synchronization task according to the configured operation parameters;
step S2: acquiring the latest table structure and configuration information corresponding to the data synchronization task which is delayed to be issued, comparing the table structure cached in the configuration information with the latest table structure, acquiring an abnormal result and grading the abnormal result;
step S3: transmitting corresponding abnormal early warning information according to the abnormal result to the responsible person of the data synchronization task;
step S4: and repairing the table structure of the data synchronization task according to the early warning information and the abnormal result classification.
2. The method for detecting and repairing abnormal structure of data synchronization table according to claim 1, wherein the step S1 comprises:
step S11: configuring the running time, the running frequency and the running interval time of the data synchronization task;
step S12: monitoring the running state of the data synchronization task, and obtaining the time point when the running of the data synchronization task is completed;
step S13: obtaining a time point when the next data synchronization task is scheduled according to the running time and the running frequency of the data synchronization task and the obtained time point when the running of the data synchronization task is completed;
step S14: obtaining a time point of delayed issuing of the data synchronization task according to the running interval time of the data synchronization task and the time point of the next scheduled data synchronization task;
step S15: and carrying out delay issuing on the data synchronization task according to the obtained time point of delay issuing of the data synchronization task.
3. The method for detecting and repairing abnormal structures of data synchronization table according to claim 1, wherein in step S1, the data synchronization task is issued in a delayed manner according to the configured operation parameters, comprising:
setting delay time by adopting a RDelayedQueue queue and storing a data synchronization task which is issued in a delayed manner;
when the set delay time is reached, the RDelayedQueue queue sends the stored data synchronization task to the RBlongungQueue queue;
and monitoring event changes in the RBlongQueue queue by adopting a monitoring thread, and taking out a data synchronization task from the RBlongQueue queue for issuing when a new event is generated.
4. The method for detecting and repairing abnormal structure of data synchronization table according to claim 1, wherein step S2 comprises:
step S21: acquiring an latest table structure corresponding to a data synchronization task in real time through JDBC connection;
step S22: acquiring configuration information of a data synchronization task, and comparing a table structure cached in the configuration information with the latest table structure in the step S21 to acquire an abnormal result;
step S23: and classifying the abnormal results according to the influence of the abnormal results on the operation of the data synchronization task.
5. The method for detecting and repairing a structural anomaly of a data synchronization table according to claim 4, wherein the step S21 comprises: acquiring source end and target end database names, data table names, corresponding data source connection information and an up-to-date table structure of a data table of a data synchronization task in real time through JDBC connection, wherein the data source connection information comprises a connection address, a user name and a password, the up-to-date table structure stores field information in a list form, and the field information comprises a field name, a field length, precision and a field type; and when the source end and the target end of the data synchronization task do not exist, acquiring an empty table structure.
6. The method for detecting and repairing a structural anomaly of a data synchronization table according to claim 4, wherein the step S22 comprises: and acquiring configuration information of the stored data synchronization task from a database or a storage device, and acquiring a cached table structure in the configuration information by acquiring a full field list of the cached source-end and target-end data tables and a field mapping relation between the source-end and target-end data tables from the configuration information.
7. The method for detecting and repairing a structural anomaly of a data synchronization table according to claim 4, wherein the step S23 comprises:
deleting a source end table or a target end table of the data synchronization task, acquiring an up-to-date table structure list in real time as empty, judging an abnormal result as a table deletion, and dividing the abnormal result into first-level abnormality;
when a field is missing in a field of which the mapping relation is configured in the source end or the target end data table, judging an abnormal result as field missing, and dividing the abnormal result into two-level anomalies;
when a field type change exists in a field of the configured mapping relation of the source end or the target end data table, judging an abnormal result as the field type change, and dividing the abnormal result into two-level anomalies;
when the field length change exists in the field of the configured mapping relation of the source end or the target end data table, judging the abnormal result as the field length change, and dividing the abnormal result into two-level abnormality;
when the field of the mapping relation is not configured in the source end or the target end data table, judging the abnormal result as field missing, and dividing the abnormal result into three levels of abnormal;
when a field is newly added in the source end or target end data table, judging an abnormal result as the field is newly added, and dividing the abnormal result into three levels of abnormal;
when a field type change exists in a field of which the mapping relation is not configured in the source end or the target end data table, judging an abnormal result as the field type change, and dividing the abnormal result into three levels of abnormal;
when the field length change exists in the field of which the mapping relation is not configured in the source end or the target end data table, judging the abnormal result as the field length change, and dividing the abnormal result into three levels of abnormal.
8. The method for detecting and repairing an anomaly in a data synchronization table according to claim 1, wherein the anomaly early warning information in step S3 includes a data synchronization task number, a database name, a data table name, and an anomaly result type.
9. The method for detecting and repairing abnormal structure of data synchronization table according to claim 1, wherein step S4 comprises:
when the abnormal result is a table missing, acquiring a full field list of a stored data table from configuration information of a data synchronization task, assembling and generating a table construction statement DDL corresponding to the data table according to the data source type of the data table, and creating a data table by the table construction statement DDL;
when the abnormal result is that the field is missing, acquiring stored field information of the missing field from configuration information of the data synchronization task, assembling and generating an SQL statement of a newly added field according to the data source type of the data table, and recovering the missing field through the SQL statement of the newly added field;
when the abnormal result is field type change, the mapping relation of the type change field is readjusted through a graphical interface, and the new mapping relation is persisted to a database or related storage equipment;
when the abnormal result is that the field length is changed, the mapping relation of the length change field is readjusted through the graphical interface, and the new mapping relation is persisted in a database or related storage equipment;
when the abnormal result is that the field is newly added, the mapping relation of the newly added field is readjusted through the graphical interface, and the new mapping relation is persisted to a database or related storage equipment.
10. A data synchronization table structure anomaly detection and repair device, the device comprising:
the scheduling module is used for configuring the operation parameters of the data synchronization task and delaying the data synchronization task according to the configured operation parameters;
the abnormality detection module is used for acquiring the latest table structure and configuration information corresponding to the data synchronization task issued in a delayed manner, comparing the table structure cached in the configuration information with the latest table structure, acquiring an abnormality result and grading the abnormality result;
the abnormal early warning module is used for sending corresponding abnormal early warning information to a responsible person of the data synchronization task according to an abnormal result; and the abnormality repairing module is used for repairing the table structure of the data synchronization task according to the early warning information and the abnormal result classification.
CN202310328337.4A 2023-03-30 2023-03-30 Method and device for detecting and repairing structural abnormality of data synchronization table Pending CN116361391A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310328337.4A CN116361391A (en) 2023-03-30 2023-03-30 Method and device for detecting and repairing structural abnormality of data synchronization table

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310328337.4A CN116361391A (en) 2023-03-30 2023-03-30 Method and device for detecting and repairing structural abnormality of data synchronization table

Publications (1)

Publication Number Publication Date
CN116361391A true CN116361391A (en) 2023-06-30

Family

ID=86929973

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310328337.4A Pending CN116361391A (en) 2023-03-30 2023-03-30 Method and device for detecting and repairing structural abnormality of data synchronization table

Country Status (1)

Country Link
CN (1) CN116361391A (en)

Citations (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105095056A (en) * 2015-08-14 2015-11-25 焦点科技股份有限公司 Method for monitoring data in data warehouse
CN109885581A (en) * 2019-03-14 2019-06-14 苏州达家迎信息技术有限公司 Synchronous method, device, equipment and the storage medium of database
CN110704475A (en) * 2019-09-29 2020-01-17 中国银行股份有限公司 Method and system for comparing ETL loading table structures
CN111177165A (en) * 2019-12-23 2020-05-19 拉扎斯网络科技(上海)有限公司 Method, device and equipment for detecting data consistency
CN111309790A (en) * 2018-11-27 2020-06-19 千寻位置网络有限公司 Automatic ETL configuration method and device based on maxcomputer database and control system
CN112306994A (en) * 2020-11-10 2021-02-02 北京沃东天骏信息技术有限公司 Database data migration method and device and storage medium
CN112463883A (en) * 2020-11-20 2021-03-09 广东电网有限责任公司广州供电局 Reliability monitoring method, device and equipment based on big data synchronization platform
CN112800150A (en) * 2021-02-25 2021-05-14 浪潮云信息技术股份公司 Automatic mapping and table building method for heterogeneous database
CN113139017A (en) * 2021-04-23 2021-07-20 武汉优品楚鼎科技有限公司 Data synchronization method and device, electronic equipment and storage medium
CN113407551A (en) * 2020-03-17 2021-09-17 北京同邦卓益科技有限公司 Data consistency determining method, device, equipment and storage medium
CN113609119A (en) * 2021-08-11 2021-11-05 平安国际智慧城市科技股份有限公司 Data account checking and repairing method and device, storage medium and electronic equipment
CN113609230A (en) * 2021-09-16 2021-11-05 平安国际智慧城市科技股份有限公司 Data synchronization abnormity warning method and device, computer equipment and storage medium
CN113946627A (en) * 2021-10-27 2022-01-18 北京科杰科技有限公司 Data accuracy detection early warning system and method under data real-time synchronization scene
CN114579664A (en) * 2022-01-29 2022-06-03 中通服咨询设计研究院有限公司 Remote real-time incremental synchronization method and system for SQLite database
CN115114284A (en) * 2022-05-27 2022-09-27 中银金融科技有限公司 Table change processing method and system
CN115599870A (en) * 2022-12-15 2023-01-13 云筑信息科技(成都)有限公司(Cn) Data synchronization method based on fusion of stock data and incremental data of message queue

Patent Citations (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105095056A (en) * 2015-08-14 2015-11-25 焦点科技股份有限公司 Method for monitoring data in data warehouse
CN111309790A (en) * 2018-11-27 2020-06-19 千寻位置网络有限公司 Automatic ETL configuration method and device based on maxcomputer database and control system
CN109885581A (en) * 2019-03-14 2019-06-14 苏州达家迎信息技术有限公司 Synchronous method, device, equipment and the storage medium of database
CN110704475A (en) * 2019-09-29 2020-01-17 中国银行股份有限公司 Method and system for comparing ETL loading table structures
CN111177165A (en) * 2019-12-23 2020-05-19 拉扎斯网络科技(上海)有限公司 Method, device and equipment for detecting data consistency
CN113407551A (en) * 2020-03-17 2021-09-17 北京同邦卓益科技有限公司 Data consistency determining method, device, equipment and storage medium
CN112306994A (en) * 2020-11-10 2021-02-02 北京沃东天骏信息技术有限公司 Database data migration method and device and storage medium
CN112463883A (en) * 2020-11-20 2021-03-09 广东电网有限责任公司广州供电局 Reliability monitoring method, device and equipment based on big data synchronization platform
CN112800150A (en) * 2021-02-25 2021-05-14 浪潮云信息技术股份公司 Automatic mapping and table building method for heterogeneous database
CN113139017A (en) * 2021-04-23 2021-07-20 武汉优品楚鼎科技有限公司 Data synchronization method and device, electronic equipment and storage medium
CN113609119A (en) * 2021-08-11 2021-11-05 平安国际智慧城市科技股份有限公司 Data account checking and repairing method and device, storage medium and electronic equipment
CN113609230A (en) * 2021-09-16 2021-11-05 平安国际智慧城市科技股份有限公司 Data synchronization abnormity warning method and device, computer equipment and storage medium
CN113946627A (en) * 2021-10-27 2022-01-18 北京科杰科技有限公司 Data accuracy detection early warning system and method under data real-time synchronization scene
CN114579664A (en) * 2022-01-29 2022-06-03 中通服咨询设计研究院有限公司 Remote real-time incremental synchronization method and system for SQLite database
CN115114284A (en) * 2022-05-27 2022-09-27 中银金融科技有限公司 Table change processing method and system
CN115599870A (en) * 2022-12-15 2023-01-13 云筑信息科技(成都)有限公司(Cn) Data synchronization method based on fusion of stock data and incremental data of message queue

Similar Documents

Publication Publication Date Title
US20180129579A1 (en) Systems and Methods with a Realtime Log Analysis Framework
US11288231B2 (en) Reproducing datasets generated by alert-triggering search queries
US7895167B2 (en) System and method for analysis and management of logs and events
CN107302469B (en) Monitoring device and method for data update of distributed service cluster system
US20180285184A1 (en) Apparatus, system, and method for analyzing logs
CN107643956B (en) Method and apparatus for locating the origin of an anomaly in anomaly data
CN105868521A (en) Data information processing method and apparatus
CN115374102A (en) Data processing method and system
US20180032567A1 (en) Method and device for processing data blocks in a distributed database
CN111177244A (en) Data association analysis method for multiple heterogeneous databases
CN112527620A (en) Database performance analysis method and device, electronic equipment, medium and product
CN116483831B (en) Recommendation index generation method for distributed database
CN116361391A (en) Method and device for detecting and repairing structural abnormality of data synchronization table
EP3953831A1 (en) Enterprise resource planning system, server and supervision method of sql queries in such a system or server
US20160179866A1 (en) Method and system to search logs that contain a massive number of entries
US20140136274A1 (en) Providing multiple level process intelligence and the ability to transition between levels
CN114531338A (en) Monitoring alarm and tracing method and system based on call chain data
US8631391B2 (en) Method and a system for process discovery
CN113568804A (en) Web application-oriented performance bottleneck accurate positioning system
CN112667469A (en) Method, system and readable medium for automatically generating diversified big data statistical report
CN112182025A (en) Log analysis method, device, equipment and computer readable storage medium
US20150193865A1 (en) Improper Financial Activity Detection Tool
CN114281902B (en) Python-based customer data monitoring method and system
CN117609315B (en) Data processing method, device, equipment and readable storage medium
CN112561368B (en) Visual performance calculation method and device for OA approval system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information

Address after: 430058 No. n3013, 3rd floor, R & D building, building n, Artificial Intelligence Science Park, economic and Technological Development Zone, Caidian District, Wuhan City, Hubei Province

Applicant after: Zhongdian Cloud Computing Technology Co.,Ltd.

Address before: 430058 No. n3013, 3rd floor, R & D building, building n, Artificial Intelligence Science Park, economic and Technological Development Zone, Caidian District, Wuhan City, Hubei Province

Applicant before: CLP cloud Digital Intelligence Technology Co.,Ltd.

CB02 Change of applicant information