Disclosure of Invention
Aiming at the problems in the prior art, the embodiment of the invention provides a source-end Oracle database DDL synchronization method and device based on log analysis.
In a first aspect, an embodiment of the present invention provides a source Oracle database DDL synchronization method based on log parsing, including: deploying a synchronization system on a source Oracle database and a target end database, initializing an Oracle system dictionary table by the source Oracle database synchronization system, constructing a data type mapping table from the source Oracle database to the target end database, and reading an Oracle log from the source Oracle database synchronization system; acquiring DDL operation log information and a transaction number according to an operation code of an Oracle log, collecting and obtaining a DML operation log according to the DDL operation log information and the transaction number, if the transaction number of the DML operation log is the same as the transaction number of the DDL operation log, collecting the DDL operation log, and if the type of the DML operation log is transaction submission, finishing collecting the DDL operation log; analyzing the collected DDL operation log and the corresponding DML operation log, restoring DDL sentences according to the Oracle system dictionary table, converting the restored DDL sentences into DDL sentences suitable for being executed by the target end database by combining with the data type mapping table from the source end Oracle database to the target end database, and sending the DDL sentences to the target end database to realize DDL sentence synchronization.
Further, the deploying and synchronizing system in the source-end Oracle database and the target-end database includes: a synchronous system management module, a log reading and analyzing module and a sending module are deployed in a source Oracle database.
Further, the deploying and synchronizing system in the source-end Oracle database and the target-end database includes: and deploying a synchronous system management module, a receiving module and an execution module in the target-end database.
Further, the target data includes: an Oracle database or a non-Oracle database.
Further, the Oracle system dictionary table includes: sys.obj $, sys.col $, sys.icol $, and/or sys.lob $.
In a second aspect, an embodiment of the present invention provides a source Oracle database DDL synchronization apparatus based on log parsing, including:
the system comprises an Oracle log acquisition module, a source end Oracle database synchronization system and a target end database synchronization system, wherein the Oracle database synchronization system is used for deploying a synchronization system on the source end Oracle database and the target end database, initializing an Oracle system dictionary table, constructing a data type mapping table from the source end Oracle database to the target end database, and reading an Oracle log from the source end Oracle database synchronization system;
the DDL operation log collection module is used for acquiring DDL operation log information and a transaction number according to an operation code of the Oracle log, collecting and obtaining a DML operation log according to the DDL operation log information and the transaction number, if the transaction number of the DML operation log is the same as the transaction number of the DDL operation log, collecting the DDL operation log, and if the type of the DML operation log is transaction submission, finishing collecting the DDL operation log;
and the DDL statement synchronization module is used for analyzing the collected DDL operation logs and the corresponding DML operation logs, restoring the DDL statements according to the Oracle system dictionary table, converting the restored DDL statements into DDL statements suitable for being executed by the target end database by combining with the data type mapping table from the source end Oracle database to the target end database, and sending the DDL statements to the target end database to realize DDL statement synchronization.
In a third aspect, an embodiment of the present invention provides an electronic device, including:
at least one processor; and
at least one memory communicatively coupled to the processor, wherein:
the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the log parsing based source Oracle database DDL synchronization method provided by any one of the various possible implementations of the first aspect.
In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores computer instructions, where the computer instructions cause a computer to execute a method for source-end Oracle database DDL synchronization based on log parsing provided in any one of various possible implementations of the first aspect.
According to the source Oracle database DDL synchronization method and device based on log analysis, provided by the embodiment of the invention, by utilizing the fact that DML operation objects are all Oracle system dictionary tables, DDL operation is decomposed into a plurality of DML operations to be executed in the Oracle database, the corresponding DDL operation is obtained by reversely tracking the DML operations, the obtained DDL operation is mapped to the target end database to be executed, and the DDL operation can be synchronized between the source Oracle database and the target end database without adopting an event trigger to capture the DDL operation.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention. In addition, technical features of various embodiments or individual embodiments provided by the invention can be arbitrarily combined with each other to form a feasible technical solution, but must be realized by a person skilled in the art, and when the technical solution combination is contradictory or cannot be realized, the technical solution combination is not considered to exist and is not within the protection scope of the present invention.
The user's definitions for objects of the Oracle database management system are all stored in the system dictionary table, which the user can view but cannot modify, i.e., the system dictionary table is read-only by the user. The invention tracks DML operation of the corresponding system dictionary table in the log according to the change of the record of the Oracle database system dictionary table, and deduces DDL operation of creating, updating and deleting the user object. Therefore, the method for obtaining the DDL operation of the Oracle source end database is realized under the condition that no DDL operation record is provided in the Oracle source end database and no event trigger is allowed to be created on the Oracle source end database.
Based on the above reality, an embodiment of the present invention provides a source Oracle database DDL synchronization method based on log parsing, and referring to FIG. 1, the method includes:
101. deploying a synchronization system on a source Oracle database and a target end database, initializing an Oracle system dictionary table by the source Oracle database synchronization system, constructing a data type mapping table from the source Oracle database to the target end database, and reading an Oracle log from the source Oracle database synchronization system;
102. acquiring DDL operation log information and a transaction number according to an operation code of an Oracle log (in another embodiment, the operation code of the log is the log with the number of 24.1), collecting and obtaining a DML operation log according to the DDL operation log information and the transaction number, if the transaction number of the DML operation log is the same as the transaction number of the DDL operation log, collecting the DDL operation log, and if the type of the DML operation log is transaction submission, finishing collecting the DDL operation log;
103. analyzing the collected DDL operation log and the corresponding DML operation log, restoring DDL sentences according to the Oracle system dictionary table, converting the restored DDL sentences into DDL sentences suitable for being executed by the target end database by combining with the data type mapping table from the source end Oracle database to the target end database, and sending the DDL sentences to the target end database to realize DDL sentence synchronization.
The basic steps of the above embodiment can be explained as follows:
firstly, before a real-time synchronization system of a database with an Oracle source end is started, a source end Oracle storage object is defined, and a related system dictionary table is loaded with a table structure (such as SYS. OBJ $, SYS. COL $andSYS. ICOL $), namely, a table ID and a table definition of the system dictionary table are recorded. If the system dictionary table (table definition and table ID) is not initialized, the related operations cannot be restored to the table structure after the related dictionary table read from the log is operated because the real-time synchronization system has no definition of the system dictionary table structure. And the database real-time synchronization system tracks whether the Oracle has DDL operation, restores the DDL operation according to the related log corresponding to the system dictionary table ID in the DDL operation log and by combining the system dictionary table structure stored in the real-time synchronization system, and executes the DDL operation in a target library.
And judging the DDL operation according to the log operation code, and acquiring a transaction number (ID) of the DDL operation. And monitoring whether the DDL operation occurs in the database log, and if the DDL operation occurs, collecting a synchronous system dictionary table at the corresponding moment in the log according to the related information in the log, and restoring DDL statements. In the ORACLE database log, a specific log format is defined when the DDL operation of a user is recorded; the log operation code is 24.1, which represents the DDL operation, and after the operation code, the DDL operation type, the operation object name and the user to which the operation object belongs are recorded respectively.
And collecting DML operation logs related to the DDL operation logs according to the DDL operation transaction numbers, and realizing log screening. In an Oracle database, a DDL operation of a user is usually decomposed into a plurality of DMLs to be executed in the database, DML operation objects are a system dictionary table of Oracle, and the transaction number of each DML operation is the same as the transaction number of the DDL operation; collecting the DML operation with the same number as the DDL operation number, and taking the operation type as a submission as an end mark, thereby completing the collection work of the DDL operation record.
And establishing a data type mapping table, and solving the problem of incompatibility when Oracle is synchronized to a non-Oracle database. And when the destination database is a heterogeneous database, establishing a data type mapping table, and performing keyword mapping on the reduced DDL statement to ensure the correct execution of the DDL statement in the destination database management system. Keywords in a database management system include data types, functions, etc., such as char, varchar, int … …. If the mapping is not performed, the destination database management system may not support similar data types or functions, and the DDL statement execution fails.
On the basis of the foregoing embodiment, the source Oracle database DDL synchronization method based on log parsing provided in the embodiment of the present invention deploys a synchronization system in a source Oracle database and a target database, and includes: a synchronous system management module, a log reading and analyzing module and a sending module are deployed in a source Oracle database.
On the basis of the foregoing embodiment, the source Oracle database DDL synchronization method based on log parsing provided in the embodiment of the present invention deploys a synchronization system in a source Oracle database and a target database, and includes: and deploying a synchronous system management module, a receiving module and an execution module in the target-end database.
On the basis of the foregoing embodiment, in the source Oracle database DDL synchronization method based on log parsing provided in the embodiment of the present invention, the target end data includes: an Oracle database or a non-Oracle database.
On the basis of the foregoing embodiment, in the source Oracle database DDL synchronization method based on log parsing provided in the embodiment of the present invention, the Oracle system dictionary table includes: sys.obj $, sys.col $, sys.icol $, and/or sys.lob $.
In the source-end Oracle database DDL synchronization method based on log analysis provided by the embodiment of the invention, by using the fact that DML operation objects are all Oracle system dictionary tables, the DDL operation is decomposed into a plurality of DML operations to be executed in the Oracle database, the corresponding DDL operation is obtained by reversely tracking the DML operations, and the obtained DDL operation is mapped to the target-end database to be executed, so that the DDL operation can be synchronized between the source-end Oracle database and the target-end database without adopting an event trigger to capture the DDL operation.
The implementation basis of the various embodiments of the present invention is realized by programmed processing performed by a device having a processor function. Therefore, in engineering practice, the technical solutions and functions thereof of the embodiments of the present invention can be packaged into various modules. Based on this reality, on the basis of the foregoing embodiments, embodiments of the present invention provide a source-side Oracle database DDL synchronization apparatus based on log parsing, where the apparatus is configured to execute the source-side Oracle database DDL synchronization method based on log parsing in the foregoing method embodiments. Referring to fig. 2, the apparatus includes:
the Oracle log acquisition module 201 is used for deploying a synchronization system in a source Oracle database and a target end database, initializing an Oracle system dictionary table by the source Oracle database synchronization system, constructing a data type mapping table from the source Oracle database to the target end database, and reading an Oracle log from the source Oracle database synchronization system;
the DDL operation log collection module 202 is configured to obtain DDL operation log information and a transaction number according to an operation code of an Oracle log, collect and obtain a DML operation log according to the DDL operation log information and the transaction number, collect the DDL operation log if the transaction number of the DML operation log is the same as the transaction number of the DDL operation log, and end collection of the DDL operation log if the type of the DML operation log is transaction commit;
and the DDL statement synchronization module 203 is used for analyzing the collected DDL operation logs and the corresponding DML operation logs, restoring the DDL statements according to the Oracle system dictionary table, converting the restored DDL statements into DDL statements suitable for being executed by the target end database by combining with the data type mapping table from the source end Oracle database to the target end database, and sending the DDL statements to the target end database to realize DDL statement synchronization.
The source Oracle database DDL synchronization device based on log analysis provided by the embodiment of the invention adopts an Oracle log acquisition module, a DDL operation log collection module and a DDL statement synchronization module, decomposes DDL operation into a plurality of DML operations to be executed in an Oracle database by using that DML operation objects are all Oracle system dictionary tables, obtains the corresponding DDL operation from DML operation reverse tracking, maps the obtained DDL operation to a target end database to be executed, and can synchronize the DDL operation between the source Oracle database and the target end database under the condition that the DDL operation is captured without adopting an event trigger.
The method of the embodiment of the invention is realized by depending on the electronic equipment, so that the related electronic equipment is necessarily introduced. To this end, an embodiment of the present invention provides an electronic apparatus, as shown in fig. 3, including: at least one processor (processor)301, a communication Interface (Communications Interface)304, at least one memory (memory)302 and a communication bus 303, wherein the at least one processor 301, the communication Interface 304 and the at least one memory 302 are configured to communicate with each other via the communication bus 303. The at least one processor 301 may call logic instructions in the at least one memory 302 to perform the following method: deploying a synchronization system on a source Oracle database and a target end database, initializing an Oracle system dictionary table by the source Oracle database synchronization system, constructing a data type mapping table from the source Oracle database to the target end database, and reading an Oracle log from the source Oracle database synchronization system; acquiring DDL operation log information and a transaction number according to an operation code of an Oracle log, collecting and obtaining a DML operation log according to the DDL operation log information and the transaction number, if the transaction number of the DML operation log is the same as the transaction number of the DDL operation log, collecting the DDL operation log, and if the type of the DML operation log is transaction submission, finishing collecting the DDL operation log; analyzing the collected DDL operation log and the corresponding DML operation log, restoring DDL sentences according to the Oracle system dictionary table, converting the restored DDL sentences into DDL sentences suitable for being executed by the target end database by combining with the data type mapping table from the source end Oracle database to the target end database, and sending the DDL sentences to the target end database to realize DDL sentence synchronization.
Furthermore, the logic instructions in the at least one memory 302 may be implemented in software functional units and stored in a computer readable storage medium when sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. Examples include: deploying a synchronization system on a source Oracle database and a target end database, initializing an Oracle system dictionary table by the source Oracle database synchronization system, constructing a data type mapping table from the source Oracle database to the target end database, and reading an Oracle log from the source Oracle database synchronization system; acquiring DDL operation log information and a transaction number according to an operation code of an Oracle log, collecting and obtaining a DML operation log according to the DDL operation log information and the transaction number, if the transaction number of the DML operation log is the same as the transaction number of the DDL operation log, collecting the DDL operation log, and if the type of the DML operation log is transaction submission, finishing collecting the DDL operation log; analyzing the collected DDL operation log and the corresponding DML operation log, restoring DDL sentences according to the Oracle system dictionary table, converting the restored DDL sentences into DDL sentences suitable for being executed by the target end database by combining with the data type mapping table from the source end Oracle database to the target end database, and sending the DDL sentences to the target end database to realize DDL sentence synchronization. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.