CN114077601A - Operation method of data synchronization system suitable for different databases - Google Patents
Operation method of data synchronization system suitable for different databases Download PDFInfo
- Publication number
- CN114077601A CN114077601A CN202111420555.8A CN202111420555A CN114077601A CN 114077601 A CN114077601 A CN 114077601A CN 202111420555 A CN202111420555 A CN 202111420555A CN 114077601 A CN114077601 A CN 114077601A
- Authority
- CN
- China
- Prior art keywords
- database
- original
- script
- module
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides an operation method of a data synchronization system suitable for different databases, which comprises the steps of configuring relevant connection information of an original database and a target database, and configuring operation parameters of the system and differential correspondence between the original database and the target database according to actual conditions; the starting system automatically reads the configuration information, initializes the system operation environment and the starting operation of each module; the address and the port of the target database connected with the original system are the address of the server and the port in the configuration; analyzing the operation to be executed through an analysis module to form a script to be executed corresponding to the target database; executing an original database sent by an original script, adding a to-be-executed script of a modified target database into a to-be-executed queue of a target database, and waiting for the queue to be automatically executed to ensure that the script is sent to the target database and executed; acquiring an execution result of an original database, and sending the result to an original application system through a feedback module; and determining whether to record the system operation log according to the configuration.
Description
Technical Field
The invention belongs to the technical field of computers, relates to database operation, and particularly relates to an operation method of a data synchronization system suitable for different databases.
Background
Along with the emphasis of the country on the informatization construction and the enhancement on the information security construction, each important system requires the use of a safe and reliable domestic system, wherein the domestic replacement of software and hardware is used; after replacement, in order to ensure that the work of system users is not affected after the system is migrated, the systems before and after migration are generally required to run for a period of time through double tracks simultaneously; because the system runs on different database systems, system users use the system under the non-localization environment and the system under the localization environment, which requires to ensure the data consistency in the two systems in the running process; after the dual-track operation of the system is finished, the system can be directly switched to a domestic environment under the conditions of no operation and no data loss after the original system is stopped, and the whole switching process needs to achieve the purpose of being fast and not influencing the normal operation of the original service.
The existing solution is to perform data synchronization on a database level, and it is necessary to perform data synchronization operation on an original database and a migrated database, and the existing solution has disadvantages that there is an intrusion operation on a database of an original system, and in a system with a large data volume and frequent update, the performance of the original database is affected, which further causes system operation abnormality, and there is a possibility that synchronization is not timely, and there is no measure to ensure re-execution after synchronization failure. The method has the conditions of data loss, incapability of real-time synchronization and incapability of really realizing dual-rail operation of a dual system.
Disclosure of Invention
The invention aims to solve the problems in the prior art, and provides an operation method of a data synchronization system which is suitable for different databases and only solves the problem of new data synchronization in dual-track operation by adding a database middleware system between an application system and a database.
The purpose of the invention can be realized by the following technical scheme: an operation method of a data synchronization system suitable for different databases comprises a configuration module, a connection module, an analysis module, a distribution module, a feedback module and a log module, and comprises the following steps:
s1, configuring the related connection information of the original database and the target database in the configuration file, and configuring the operation parameters of the system and the differential correspondence between the original database and the target database according to the actual situation;
s2, starting the system, wherein the system automatically reads the configuration information, initializes the system running environment and the starting operation of each module;
s3, modifying the database connection part of the original system to make the address and port of the target database connected with the original system be the address of the server and the port in the configuration;
s4, when the original system executes the operations of adding, deleting, modifying and checking the database, the original system analyzes the operation to be executed through an analysis module to form a script to be executed corresponding to the target database;
s5, executing the original database sent by the original script, adding the to-be-executed script of the modified target database into a to-be-executed queue of the target database, waiting for automatic execution of the queue, and ensuring that the script is sent to the target database and executed by the queue;
s6, obtaining the execution result of the original database, and sending the result to the original application system by the feedback module;
and S7, determining whether to record the system operation log according to the configuration.
In the foregoing method for operating a data synchronization system applicable to different databases, the configuration module is configured to: configuring the connection of an original database and a target database; configuring differences of functions in an original database and a target database; configuring relevant parameters of a connection pool in an original database and a target database; other system-related operating parameters are configured.
In the above operation method of the data synchronization system applicable to different databases, the other operation parameters include a system cache parameter, a system garbage collection mechanism parameter, and a system heartbeat parameter.
In the above operation method of the data synchronization system applicable to different databases, the connection module is configured to receive a database connection of an original system, and connect the system to a target database; the system supports the configuration of a connection pool, keeps a plurality of connections to the database, and controls the conversion of the idle state and the busy state of each connection; the system periodically detects the configuration parameters and the current normal connection, simultaneously rejects the unavailable connection and supplements the new connection according to the current system operation status and the configuration information, and determines whether to increase the connection number according to the current connection number and the configuration information.
In the above operating method of the data synchronization system applicable to different databases, the parsing module is configured to parse an SQL script to be executed, distinguish differences between an original library and a target library, and transform the target library script; the analysis module is mainly used for analyzing the command, the field and the used function in the received SQL script, replacing the related incompatible SQL command according to the configuration information, and splitting the original SQL statement into two SQL statements in the analysis module.
In the above operating method of the data synchronization system applicable to different databases, the distribution module is configured to distribute the target library to the original SQL script, and distribute the modified SQL script to the target library execution queue; and distributing the analyzed SQL script to a main library and a target library, distributing the original SQL script to the main library, distributing the analyzed SQL script to an internal message queue, and not feeding back an execution result after the queue executes the analyzed script.
In the above operation method of the data synchronization system applicable to different databases, the feedback module is used for the execution of the SQL script to feed back information to the application system, the feedback information includes simple information or structured data, and the feedback information is analyzed and encapsulated according to the execution result and fed back to the data structure required by the original system; for the execution of the SQL script of the target library, the part only makes log records for recording whether the execution of the SQL script is completed and correct.
In the above operation method of the data synchronization system applicable to different databases, the log module is used for recording the operation log of the system and the operation condition of the SQL script of the system. The system log is divided into two parts, namely a system running state log and an SQL script log, and is used for tracking the running stability of the system and carrying out error correction processing in operation and maintenance.
In the above operation method of the data synchronization system suitable for different databases, during dual-track operation, the system after migration is configured in step S1, where the original database corresponds to the localization database after migration, the target database corresponds to the non-localization database before migration, and the system after migration executes steps S2 to S7.
Compared with the prior art, the operation method of the data synchronization system applicable to different databases has the following beneficial effects:
the consistency of the data in the original system database and the data in the target database can be ensured, the performances of the original database and the target database are not influenced, the operation of the original database and the target database does not occupy the running resources of the original system database and the target system database, and the performances of the original system database and the target system database are not influenced.
Drawings
FIG. 1 is a schematic diagram of the operation of the data synchronization system for different databases.
Detailed Description
The following description of the embodiments of the present invention will be made with reference to the accompanying drawings:
the invention is mainly used for solving the problem of real-time synchronization of data of a heterogeneous database during dual-track operation of a system, and provides a method and a system for real-time synchronization of data during dual-track operation after system migration in a localization trusted environment.
Because different databases are different in data types, database functions and the like, the synchronization of heterogeneous databases cannot be directly performed on the database level, and the real-time synchronization is performed by means of the capability of an application system. When the system operates the database before and after the migration, the invention receives the operation script of the database, and carries out the reconstruction of the script aiming at the target database, and the script before and after the reconstruction is distributed to the original and target databases of the system by the distribution module to be executed, thereby having less invasion to the database and no performance loss.
As shown in fig. 1, the operation method of the data synchronization system applicable to different databases includes a configuration module, a connection module, an analysis module, a distribution module, a feedback module, and a log module, and includes the following steps:
s1, configuring the related connection information of the original database and the target database in the configuration file, and configuring the operation parameters of the system and the differential correspondence between the original database and the target database according to the actual situation;
s2, starting the system, wherein the system automatically reads the configuration information, initializes the system running environment and the starting operation of each module;
s3, modifying the database connection part of the original system to make the address and port of the target database connected with the original system be the address of the server and the port in the configuration;
s4, when the original system executes the operations of adding, deleting, modifying and checking the database, the original system analyzes the operation to be executed through an analysis module to form a script to be executed corresponding to the target database;
s5, executing the original database sent by the original script, adding the to-be-executed script of the modified target database into a to-be-executed queue of the target database, waiting for automatic execution of the queue, and ensuring that the script is sent to the target database and executed by the queue;
s6, obtaining the execution result of the original database, and sending the result to the original application system by the feedback module;
and S7, determining whether to record the system operation log according to the configuration.
The configuration module is to accomplish: configuring the connection of an original database and a target database; configuring differences of functions in an original database and a target database; configuring relevant parameters of a connection pool in an original database and a target database; other system-related operating parameters are configured.
Other operating parameters include system cache parameters, system garbage sorting mechanism parameters, and system heartbeat parameters.
The connection module is used for receiving the database connection of the original system and connecting the system to a target database; the system supports the configuration of a connection pool, keeps a plurality of connections to the database, and controls the conversion of the idle state and the busy state of each connection; the system periodically detects the configuration parameters and the current normal connection, simultaneously rejects the unavailable connection and supplements the new connection according to the current system operation status and the configuration information, and determines whether to increase the connection number according to the current connection number and the configuration information.
The analysis module is used for analyzing the SQL script to be executed, distinguishing the difference between the original library and the target library and transforming the target library script; the analysis module is mainly used for analyzing the command, the field and the used function in the received SQL script, replacing the related incompatible SQL command according to the configuration information, and splitting the original SQL statement into two SQL statements in the analysis module.
The distribution module is used for distributing the target library to the original SQL script and distributing the modified SQL script to the target library execution queue; and distributing the analyzed SQL script to a main library and a target library, distributing the original SQL script to the main library, distributing the analyzed SQL script to an internal message queue, and not feeding back an execution result after the queue executes the analyzed script.
The feedback module is used for feeding back information to the application system when the SQL script is executed, wherein the feedback information comprises simple information or structured data, and the feedback information is analyzed and packaged according to the execution result and fed back to a data structure required by the original system; for the execution of the SQL script of the target library, the part only makes log records for recording whether the execution of the SQL script is completed and correct.
The log module is used for recording the running log of the system and the running condition of the SQL script of the system. The system log is divided into two parts, namely a system running state log and an SQL script log, and is used for tracking the running stability of the system and carrying out error correction processing in operation and maintenance.
The system is used on the premise that original data in the two databases are consistent through steps before and after migration, and only the system solves the problem of synchronization of new data in dual-track operation. The system adds a database middleware system between an application system and a database, the application system is directly connected to the middleware, the middleware divides an SQL script to be executed, the script is modified to be suitable for a target database according to system configuration information after the SQL script to be executed is analyzed and modified, and then the original script and the modified script are respectively sent to the two databases; and after the original database is executed, the returned data is received and fed back to the application, and the reconstructed script is only executed in the backup database and does not receive the returned data.
During dual-track operation, the post-migration system is configured in step S1, the original database corresponds to the localization database after migration, the target database corresponds to the non-localization database before migration, and the post-migration system performs steps S2 to S7.
After the system is subjected to localization migration transformation, the localization progress of peripheral equipment and third-party software involved in the system is inconsistent, so that the system can run two sets of systems simultaneously under localization and non-localization, and the problem of data inconsistency can be solved.
Compared with the prior art, the operation method of the data synchronization system applicable to different databases has the following beneficial effects:
the consistency of the data in the original system database and the data in the target database can be ensured, the performances of the original database and the target database are not influenced, the operation of the original database and the target database does not occupy the running resources of the original system database and the target system database, and the performances of the original system database and the target system database are not influenced.
It is to be understood that the above description is not intended to limit the present invention, and the present invention is not limited to the above examples, and those skilled in the art may make modifications, alterations, additions or substitutions within the spirit and scope of the present invention.
Claims (9)
1. An operation method of a data synchronization system suitable for different databases is provided, the system comprises a configuration module, a connection module, an analysis module, a distribution module, a feedback module and a log module, and the operation method comprises the following steps:
s1, configuring the related connection information of the original database and the target database in the configuration file, and configuring the operation parameters of the system and the differential correspondence between the original database and the target database according to the actual situation;
s2, starting the system, wherein the system automatically reads the configuration information, initializes the system running environment and the starting operation of each module;
s3, modifying the database connection part of the original system to make the address and port of the target database connected with the original system be the address of the server and the port in the configuration;
s4, when the original system executes the operations of adding, deleting, modifying and checking the database, the original system analyzes the operation to be executed through an analysis module to form a script to be executed corresponding to the target database;
s5, executing the original database sent by the original script, adding the to-be-executed script of the modified target database into a to-be-executed queue of the target database, waiting for automatic execution of the queue, and ensuring that the script is sent to the target database and executed by the queue;
s6, obtaining the execution result of the original database, and sending the result to the original application system by the feedback module;
and S7, determining whether to record the system operation log according to the configuration.
2. The method of claim 1, wherein the configuration module is configured to perform the following steps: configuring the connection of an original database and a target database; configuring differences of functions in an original database and a target database; configuring relevant parameters of a connection pool in an original database and a target database; other system-related operating parameters are configured.
3. The method according to claim 2, wherein the other operation parameters include a system cache parameter, a system garbage correction mechanism parameter, and a system heartbeat parameter.
4. The method as claimed in claim 1, wherein the connection module is configured to accept a database connection from an original system and connect the system to a target database; the system supports the configuration of a connection pool, keeps a plurality of connections to the database, and controls the conversion of the idle state and the busy state of each connection; the system periodically detects the configuration parameters and the current normal connection, simultaneously rejects the unavailable connection and supplements the new connection according to the current system operation status and the configuration information, and determines whether to increase the connection number according to the current connection number and the configuration information.
5. The method according to claim 1, wherein the parsing module is configured to parse an SQL script to be executed, distinguish differences between the original library and the target library, and modify the target library script; the analysis module is mainly used for analyzing the command, the field and the used function in the received SQL script, replacing the related incompatible SQL command according to the configuration information, and splitting the original SQL statement into two SQL statements in the analysis module.
6. The method according to claim 1, wherein the distribution module is configured to distribute the target library to the original SQL script and distribute the modified SQL script to the target library execution queue; and distributing the analyzed SQL script to a main library and a target library, distributing the original SQL script to the main library, distributing the analyzed SQL script to an internal message queue, and not feeding back an execution result after the queue executes the analyzed script.
7. The method according to claim 1, wherein the feedback module is used for the SQL script execution requiring feedback information to the application system, the feedback information including simple information or structured data, and the feedback information is parsed and encapsulated according to the execution result and fed back to the data structure required by the original system; for the execution of the SQL script of the target library, the part only makes log records for recording whether the execution of the SQL script is completed and correct.
8. The method according to claim 1, wherein the log module is used for recording the operation log of the system and the operation condition of the SQL script of the system; the system log is divided into two parts, namely a system running state log and an SQL script log, and is used for tracking the running stability of the system and carrying out error correction processing in operation and maintenance.
9. The method as claimed in claim 1, wherein during dual-track operation, the post-migration system is configured in step S1, the original database corresponds to the localization database after migration, the target database corresponds to the non-localization database before migration, and the post-migration system performs steps S2 to S7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111420555.8A CN114077601A (en) | 2021-11-26 | 2021-11-26 | Operation method of data synchronization system suitable for different databases |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111420555.8A CN114077601A (en) | 2021-11-26 | 2021-11-26 | Operation method of data synchronization system suitable for different databases |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114077601A true CN114077601A (en) | 2022-02-22 |
Family
ID=80284298
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111420555.8A Pending CN114077601A (en) | 2021-11-26 | 2021-11-26 | Operation method of data synchronization system suitable for different databases |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114077601A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104317921A (en) * | 2014-10-29 | 2015-01-28 | 北京思特奇信息技术股份有限公司 | Cross-database data synchronizing system and method |
| CN110471688A (en) * | 2019-08-21 | 2019-11-19 | 深圳蓝贝科技有限公司 | Operation system processing method, device, equipment and storage medium |
| CN111651428A (en) * | 2020-05-22 | 2020-09-11 | 上海浪擎信息科技有限公司 | Double-track operation method based on domestic database migration environment |
| CN112231384A (en) * | 2020-10-30 | 2021-01-15 | 中国建设银行股份有限公司 | Distributed application localization transformation method and device adaptive to multiple data sources |
| CN113094434A (en) * | 2021-04-08 | 2021-07-09 | 中国工商银行股份有限公司 | Database synchronization method, system, device, electronic equipment and medium |
| CN113127447A (en) * | 2021-04-22 | 2021-07-16 | 山东福生佳信科技股份有限公司 | Data synchronization device and method for application system conversion |
-
2021
- 2021-11-26 CN CN202111420555.8A patent/CN114077601A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104317921A (en) * | 2014-10-29 | 2015-01-28 | 北京思特奇信息技术股份有限公司 | Cross-database data synchronizing system and method |
| CN110471688A (en) * | 2019-08-21 | 2019-11-19 | 深圳蓝贝科技有限公司 | Operation system processing method, device, equipment and storage medium |
| CN111651428A (en) * | 2020-05-22 | 2020-09-11 | 上海浪擎信息科技有限公司 | Double-track operation method based on domestic database migration environment |
| CN112231384A (en) * | 2020-10-30 | 2021-01-15 | 中国建设银行股份有限公司 | Distributed application localization transformation method and device adaptive to multiple data sources |
| CN113094434A (en) * | 2021-04-08 | 2021-07-09 | 中国工商银行股份有限公司 | Database synchronization method, system, device, electronic equipment and medium |
| CN113127447A (en) * | 2021-04-22 | 2021-07-16 | 山东福生佳信科技股份有限公司 | Data synchronization device and method for application system conversion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110069572B (en) | HIVE task scheduling method, device, equipment and storage medium based on big data platform | |
| CN108076098B (en) | Service processing method and system | |
| CN102591964B (en) | Implementation method and device for data reading-writing splitting system | |
| EP3958126A1 (en) | Micro-service component-based database system and related method | |
| US11928089B2 (en) | Data processing method and device for distributed database, storage medium, and electronic device | |
| CN101707399A (en) | Method and system for acquiring electric energy information | |
| US20060053183A1 (en) | System and method for providing increased database fault tolerance | |
| CN110196859A (en) | Data base read-write based on JDBC distributor separates cluster real-time consistency method | |
| CN111026573B (en) | Watchdog system of multi-core processing system and control method | |
| CN113157411B (en) | Celery-based reliable configurable task system and device | |
| US20230137119A1 (en) | Method for replaying log on data node, data node, and system | |
| CN111427963A (en) | Data access method and system for distributed database | |
| US20210373945A1 (en) | Method and device for processing distributed data solving problem of manual intervention by data analysts | |
| CN110427427B (en) | Method for realizing global transaction distributed processing through pin bridging | |
| WO2021031583A1 (en) | Method and apparatus for executing statements, server and storage medium | |
| CN113254275A (en) | MySQL high-availability architecture method based on distributed block device | |
| CN115562676A (en) | Triggering method of graph calculation engine | |
| CN107357809A (en) | Magnanimity platform Hisoon real-time database metric data access system | |
| CN114077601A (en) | Operation method of data synchronization system suitable for different databases | |
| WO2024066635A1 (en) | Method and system for intelligently optimizing database performance | |
| CN112395308A (en) | Data query method based on HDFS database | |
| CN115630121A (en) | Method and equipment for realizing data synchronization service | |
| CN115905413A (en) | Data synchronization platform based on Python corotation and DataX | |
| CN114050989A (en) | Distributed test execution method based on cloud computing technology | |
| CN113010229A (en) | Distributed heterogeneous software cluster configuration management framework |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220222 |
|
| RJ01 | Rejection of invention patent application after publication |