CN110837535A - Data synchronization method, device, equipment and medium - Google Patents
Data synchronization method, device, equipment and medium Download PDFInfo
- Publication number
- CN110837535A CN110837535A CN201810935532.2A CN201810935532A CN110837535A CN 110837535 A CN110837535 A CN 110837535A CN 201810935532 A CN201810935532 A CN 201810935532A CN 110837535 A CN110837535 A CN 110837535A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- key
- java object
- value type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a method, a device, equipment and a medium for data synchronization. The method comprises the following steps: obtaining an archive log file generated by a relational database; analyzing and processing the filing log file to obtain an updated operation log; constructing the updated operation log as a JAVA object dedicated to the key-value type database; pushing the JAVA object to a key value type database; and reconstructing the JAVA object in the key value type database according to the service logic requirement, and executing the reconstructed JAVA object. By the scheme of the invention, the timeliness of data synchronization can be improved, and meanwhile, the data unification is greatly maintained.
Description
Technical Field
The present invention relates to the field of computing, and in particular, to a method, an apparatus, a device, and a medium for data synchronization.
Background
A large amount of information between a large data platform and a traditional IT system needs to be synchronized, and the existing data synchronization technology comprises distributed transaction interaction through a database chain; restoring SQL sentences of a target database by the archived log file for interaction; processing with a daily/monthly batch file interface, and so on. Although these data synchronization technologies can achieve data synchronization between databases at present, the timeliness of data synchronization is low, thereby causing inconsistency of system data.
Disclosure of Invention
The embodiment of the invention provides a data synchronization method, a data synchronization device, data synchronization equipment and a data synchronization medium, which can improve the timeliness of data synchronization and greatly keep the unification of data.
According to a first aspect of the embodiments of the present invention, there is provided a method for data synchronization, the method including:
obtaining an archive log file generated by a relational database;
analyzing and processing the filing log file to obtain an updated operation log;
constructing the updated operation log as a JAVA object dedicated to the key-value type database;
pushing the JAVA object to a key value type database;
and reconstructing the JAVA object in the key value type database according to the service logic requirement, and executing the reconstructed JAVA object.
According to a second aspect of the embodiments of the present invention, there is provided an apparatus for data synchronization, the apparatus comprising: the system comprises a relational database data monitoring module, a relational database data processing module, an encrypted data transmission module and a key value type database data loading module;
the relational database data monitoring module is used for acquiring an archiving log file generated by the relational database; the system is used for analyzing and processing the filing log file to obtain an updated operation log;
the relational database data processing module is used for constructing the updated operation log into a JAVA object special for the key value type database;
the encrypted data transmission module is used for pushing the JAVA object to the key value type database data loading module;
and the key value type database data loading module is used for reconstructing the JAVA object according to the service logic requirement and executing the reconstructed JAVA object.
According to a third aspect of the embodiments of the present invention, there is provided a terminal device, including:
a memory, a processor, a communication interface, and a bus;
the memory, the processor and the communication interface are connected through a bus and complete mutual communication;
the memory is used for storing program codes;
the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory for performing the method as in the first aspect.
According to a fourth aspect of embodiments of the present invention, there is provided a computer storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of the first aspect.
According to the data synchronization method, device, equipment and medium in the embodiment of the invention, when the relational database is used for data synchronization to the key value type database commonly used by a large data platform, the system overhead and the program maintenance work are reduced, meanwhile, the timeliness of the data synchronization is greatly improved, and the unification of the data is greatly kept.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described below, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart illustrating a method of data synchronization of an embodiment of the present invention;
FIG. 2 is a detailed flow chart illustrating a method of data synchronization of an embodiment of the present invention;
fig. 3 is a schematic diagram illustrating a structure of pushing a JAVA object according to an embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating an apparatus for data synchronization according to an embodiment of the present invention;
FIG. 5 is a block diagram illustrating an exemplary hardware architecture of a computing device capable of implementing the method and apparatus for data synchronization according to embodiments of the present invention.
Detailed Description
Features and exemplary embodiments of various aspects of the present invention will be described in detail below, and in order to make objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not to be construed as limiting the invention. It will be apparent to one skilled in the art that the present invention may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present invention by illustrating examples of the present invention.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
At present, most of technologies solve the problem that Structured Query Languages (SQL) between different database systems cannot be completely mutually universal, and achieve synchronization between a relational database and a key-value database by restoring SQL statements.
Such as: when the data table to be synchronized changes, converting the data table to be synchronized into an SQL statement; SQL statements are input to a key-value database. Therefore, according to the change of the data table to be synchronized in the database, the SQL sentence causing the data change can be obtained and input into the key value type database, and the synchronization of the data among the heterogeneous databases can be realized. The relational database is a database established on the basis of a relational model, and data in the database is processed by means of mathematical concepts and methods such as set algebra and the like. A relational database refers to a database that organizes a data structure using a relational model, i.e., a two-dimensional table. The database management system may be mysql, oracle, DB2, sql server, Sqlite, etc.
For another example: when a data operation request is received, the data operation request is analyzed to obtain a corresponding data operation command and corresponding data operation information, whether the data operation request comes from the outside of the database system or a second main database in the database system is judged, if the data operation request comes from the second main database in the database system, the number of synchronous data entries which are synchronized from the second main database at present is recorded according to the data operation information, and the data operation command is executed.
Therefore, real-time data synchronization between every two main databases is realized by establishing a database system of the double main databases, and after communication interruption and recovery of the two main databases are ensured, the unsynchronized data can be supplemented in time, so that accidental data loss is avoided. But at present, the technologies cannot support the problems of high concurrency, high access, unified inventory removal business and data and the like.
In the embodiment of the invention, the relational database of the traditional IT system and the key value type database of the big data platform are pulled through, so that the relational database based on the traditional IT system is constructed into a high-concurrency high-access stock-removing business system, the timeliness of data synchronization can be improved, and the data unification is greatly kept.
For a better understanding of the present invention, methods, apparatuses, devices, and media for data synchronization according to embodiments of the present invention will be described in detail below with reference to the accompanying drawings, and it should be noted that these embodiments are not intended to limit the scope of the present disclosure.
Fig. 1 is a flow chart illustrating a method of data synchronization according to an embodiment of the present invention.
As shown in fig. 1, the method 100 for data synchronization in the present embodiment includes the following steps:
in step S110, an archive log file generated by the relational database is obtained.
In this step, the archive log file may be composed of a plurality of data tables, and all the data for database recovery are stored in the archive log file, and all the DML statements for each change of the database structure, that is, for the database operation are recorded. The reading of the filing log file of the database is developed based on the current mature LogMiner technology, is mature and stable, has small influence on source data, and usually occupies a 2-core Central Processing Unit (CPU) and an 8GB memory.
In the embodiment of the present invention, the LogMiner is a practical and very useful analysis tool provided by Oracle corporation from product 8i, and the specific content in the Oracle archive log file can be easily obtained by using the tool, and the LogMiner analysis tool actually consists of a set of PL or SQL packages and some dynamic views, which are released as part of the Oracle database, and is a completely free tool provided by Oracle corporation. Data growth patterns may be analyzed by analyzing data in the archive log file.
Step S120, the archive log file is analyzed to obtain an updated operation log.
In this step, the updated operation log means a log in which there are update operations such as addition and deletion. Such as: analyzing a plurality of data tables in the filing log file, judging the data tables with the updating operation, analyzing and extracting the data tables with the updating operation, and the like.
Step S130, the updated operation log is constructed as a JAVA object dedicated by the key-value type database.
In this step, because the relational database and the key-value database employ different data structures, the key-value database stores data in a key-value pair manner, i.e., a dictionary-like manner, such as: redis. The updated oplog needs to be constructed as a JAVA object dedicated to the key-value type database.
Step S140, pushing the JAVA object to a key value type database.
And S150, reconstructing the JAVA object according to the service logic requirement, and executing the reconstructed JAVA object in the key value type database.
In this step, the key-value database needs to specify the field of the query as a key, and the obtained field as a value, and the key needs to be unique. However, the content of the relational database has no such requirement, and the JAVA object needs to be reconstructed to form a service-oriented JAVA object according to different service logic requirements, and the reconstructed JAVA object is executed in the key-value type database.
By the method of the embodiment of the invention, when the relational database is synchronized to the key value type database commonly used by a large data platform, the system overhead and the program maintenance work are reduced, meanwhile, the timeliness of data synchronization is greatly improved, and the data unification is kept.
In an embodiment, before performing step S110, the method may further include:
and acquiring the account number and the password of the relational database, and configuring the information of the data table to be synchronized.
The information of the data table to be synchronized comprises at least one of the following four types:
table name, table field authority, network transmission authority for transmitting the data table to be synchronized.
It can be understood that, by configuring the information of the data tables to be synchronized, when the tables to be synchronized are judged subsequently, whether to perform synchronization is determined according to the configured information of the data tables to be synchronized.
In an embodiment, step S120, analyzing the archive log file to obtain an updated operation log, includes:
step S121, obtaining the filing table information corresponding to the data table to be synchronized in the filing log file.
In this step, after the account number related authority of the relational database is configured, the archive log file generated by the database is acquired in real time, and simultaneously archive table information corresponding to any data table in the archive log file is acquired, such as: the data table A is compared with the filing table information A; judging whether the information in the data table to be synchronized has information matched with the information A of the filing table; when present, the archive log for this data table A is sorted out.
And step S122, when the information of the table to be synchronized has the information matched with the information of the filing table, analyzing the filing log file to obtain an updated operation log.
By judging whether the information in the data table to be synchronized has the information matched with the filing table information A or not, only the filing log corresponding to the matched data table is sorted out. Therefore, the pressure on data synchronization processing can be reduced, and the efficiency of data synchronization is improved.
In one embodiment, the step S130 of constructing the updated operation log as a JAVA object dedicated to the key-value type database may include:
step S131, a first data attribute is acquired from the updated operation log.
In this step, the data block address, object number, block inner row number, field value, data operation type, transaction number are mainly obtained from the sorted updated operation log. Such as: "there are 10 data blocks of the updated operation log, and the data block address corresponding to each database is determined", etc.
Step S132, obtaining a second data attribute corresponding to the updated operation log according to the first data attribute and the data table to be synchronized.
In the step, according to the data block address, the object number, the row number in the block, the field value, the data operation type and the data table to be synchronized configured in advance by the user, the table name, the primary key field value, the modified field name, the modified field value and the like corresponding to the updated operation log are obtained.
Step S133, according to the first data attribute and the second data attribute, a JAVA object dedicated to the key-value type database is constructed.
In this step, a JAVA object dedicated to the key-value type database is constructed according to the sorted data operation type, table name, primary key field value, modified field name, and modified field value.
According to the embodiment of the invention, the JAVA object special for the key value type database is constructed according to the first data attribute and the second data attribute, so that the accuracy of data synchronization can be ensured, the accuracy of data synchronization is improved, and the phenomenon of omission in data synchronization is avoided.
Fig. 2 is a detailed flowchart illustrating a method of data synchronization according to an embodiment of the present invention, and the same reference numerals are used for the same or equivalent steps in fig. 2 as those in fig. 1.
As shown in fig. 2, in an embodiment, the step S140 may specifically include:
in step S141, a log file including the sequence number SQN information in the updated operation log is extracted.
Step S142, determining a data control type corresponding to the generated SQL statement in the log file including the sequence number SQN information according to the transaction number.
And step S143, when the data control type is submission, encrypting the JAVA object to form an Msg database file.
And step S144, pushing the Msg database file to a key value type database through a Java message service application program interface in a point-to-point mode.
In this step, a Message body encryption technology is developed and integrated based on a mature Java Message Service (JMS) specification technology, which not only integrates the maturity and stability of the original interface, but also realizes data transmission encryption. The JMS application program interface is an API for Message Oriented Middleware (MOM) in a Java platform for asynchronous communication by sending messages between two applications or in a distributed system. JMS is an API independent of the specific platform, and most MOM providers support JMS.
It can be understood that the log file including the sequence number SQN information is mainly extracted from the updated operation log; judging whether the log file containing the SQN information is a data control type corresponding to the generated SQL statement or not according to the transaction number; when the judgment is yes and the execution of the type of the Database Control Language (DCL) is rollback, discarding the standard SQL statement; and when the judgment result is yes and the DCL type execution is submission, pushing the constructed JAVA object to the key-value type database through a JMS application program interface in a point-to-point mode.
The database control language DCL is different from the programming language, and the SQL language comprises the DCL database control language. The SQL language includes: three main programming language classes of statements are Data Definition Language (DDL), Data Manipulation Language (DML), and Data Control Language (DCL).
Fig. 3 is a schematic diagram illustrating a structure of pushing a JAVA object according to an embodiment of the present invention.
In one embodiment, after pushing the JAVA object to the key-value type database, the method further includes:
decrypting the Msg database file according to the secret key;
when the decryption is successful, the key value type database feeds back confirmation information to the relational database;
and when the decryption fails, the relational database pushes the Msg database file to the key-value type database again through a Java message service application program interface in a point-to-point mode.
It can be understood that as shown in fig. 3, the JAVA object is developed based on JMS api specification, the constructed JAVA object is encrypted to form an Msg database file, the Msg database file is placed in a linear table Queue, the Msg database file is sent out through a relational database, the key-value database receives the "constructed JAVA object" from the linear table Queue, and then the Msg database file is decrypted according to a secret key to restore the "constructed JAVA object". And when the decryption reduction is judged to be successful, sending 'the message is accepted' to the linear table Queue, confirming the message reception, and finishing the data transmission. And when the decryption reduction is judged to be failed, not sending 'message accepted' to the linear table Queue, and retransmitting the source data segment overtime.
In an embodiment, step S150, reconstructing the JAVA object according to the service logic requirement, and executing the reconstructed JAVA object in the key-value type database includes:
and step S151, determining the key value pair of the target service and the unique aperture of the key according to the service logic requirement.
And S152, reconstructing the JAVA object according to the key value pair of the target service and the unique aperture of the key.
Step S153, executing the reconstructed JAVA object in the key-value type database.
It can be understood that the key value pair required by the target service is combed out according to the service logic requirement, and the unique aperture of the key is determined. And configuring a reconstruction mode of the JAVA object when key values conflict under different scenes according to the unique aperture of the key. The JAVA object reconstruction method is mainly divided into: three cases of "replace, ignore, integrate".
Wherein, the replacement is to replace the original key-value pair with the new key-value pair; if the new key value pair is ignored, the original key value pair is reserved; the integration is to integrate the new key-value pair with the original key-value pair, write the new and old values into the values at the same time, and separate them with commas for processing by the application.
And the key value type database executes the reconstructed JAVA object, judges whether the reconstructed JAVA object is successfully executed or not, generates error information when the reconstructed JAVA object is not successfully executed, and transmits the error information to the relational database.
By the method, the technical problem that synchronization cannot be performed between the heterogeneous databases of the abnormal shape is solved by triggering reconstruction and executing the reconstructed JAVA object in real time, and timeliness and universality of data synchronization are improved. Secondly, through the sorted filing log files and the JAVA object formed by key value type database reconstruction, syntax conversion from traditional SQL to JAVA object is realized, and therefore cross-technology line pull-through is realized. In addition, a message body of plaintext transmission of the existing JMS message service is encrypted, so that the message body is prevented from being stolen, data communication between the relational database and the key value type database is realized, and the data security is improved.
The following describes in detail a data synchronization apparatus according to an embodiment of the present invention with reference to the accompanying drawings
Fig. 4 is a schematic structural diagram illustrating an apparatus for data synchronization according to an embodiment of the present invention.
As shown in fig. 4, the apparatus 400 for data synchronization may include:
relational database data monitoring module 410, relational database data processing module 420, encrypted data transmission module 430, and key-value-type database data loading module 440.
A relational database data monitoring module 410 for obtaining archive log files generated by the relational database; and the method is used for analyzing and processing the filing log file to obtain an updated operation log.
A relational database data processing module 420 for constructing the updated oplog as JAVA objects dedicated to the key-value database.
And the encrypted data transmission module is configured to push the JAVA object to the key-value type database data loading module 440.
And the key value type database data loading module 440 is configured to reconstruct the JAVA object according to the service logic requirement, and execute the reconstructed JAVA object.
By the device provided by the embodiment of the invention, when the relational database is subjected to data synchronization to the key value type database commonly used by a large data platform, the system overhead and the program maintenance work are reduced, meanwhile, the timeliness of data synchronization is greatly improved, and the data unification is greatly kept.
In an embodiment, the relational database data monitoring module 410 is further configured to obtain an account and a password of the relational database, and configure information of the table to be synchronized;
the information of the table to be synchronized comprises at least one of the following four types:
table names, table fields, table field permissions and network transmission permissions for transmitting the table to be synchronized.
In an embodiment, the relational database data monitoring module 410 is specifically configured to obtain archive table information corresponding to a table to be synchronized in an archive log file; and for
And when the information of the table to be synchronized has the information matched with the information of the filing table, analyzing the filing log file to obtain an updated operation log.
In one embodiment, the relational database data processing module 420 comprises: a first acquisition unit 421, a second acquisition unit 422, and a construction unit 423.
A first obtaining unit 421, configured to obtain the first data attribute from the updated operation log.
The second obtaining unit 422 is configured to obtain a second data attribute corresponding to the updated oplog according to the first data attribute and the table to be synchronized.
A constructing unit 423, configured to construct, according to the first data attribute and the second data attribute, a JAVA object dedicated to the key-value type database.
In an embodiment, the first data attribute comprises at least one of the following six: data block address, object number, row number within data block, field value, data operation type, and transaction number.
In an embodiment, the second data attribute comprises at least one of the following five: table name, primary key field value, modified field name, and modified field value.
In one embodiment, the encrypted data transmission module 430 includes:
an encryption extraction unit 431, a type determination unit 432, an encryption unit 433, and a push unit 434.
And the encryption extraction unit 431 is used for extracting the log file which comprises the sequence number SQN information in the updated operation log.
The type determining unit 432 is configured to determine, according to the transaction number, a data control type corresponding to a generated SQL statement in a log file that includes the sequence number SQN information.
And the encryption unit 433 is used for encrypting the JAVA object to form an Msg database file when the data control type execution is submitted.
A pushing unit 434, configured to push the Msg database file to the key-value type database data loading module 440 through a Java message service application program interface in a point-to-point mode.
In an embodiment, the encrypted data transmission module 430 is further configured to decrypt the Msg database file according to the key. And is used for feeding back confirmation information to the relational database by the key-value database data loading module 440 when the decryption is successful; and when the decryption fails, the relational database pushes the Msg database file to the key-value database data loading module 440 through the Java message service application program interface in the point-to-point mode again.
In an embodiment, the key-value database data loading module 440 is specifically configured to determine the key-value pair of the target service and the unique aperture of the key according to the service logic requirement.
And the method is used for reconstructing the JAVA object according to the key value pair of the target service and the unique aperture of the key.
And for executing the reconstructed JAVA object.
By the aid of the device, the technical problem that synchronization cannot be performed between the heterogeneous databases is solved by triggering reconstruction and executing the reconstructed JAVA objects in real time, and timeliness and universality of data synchronization are improved. Secondly, through the sorted filing log files and the JAVA object formed by key value type database reconstruction, syntax conversion from traditional SQL to JAVA object is realized, and therefore cross-technology line pull-through is realized. In addition, a message body of plaintext transmission of the existing JMS message service is encrypted, so that the message body is prevented from being stolen, data communication between the relational database and the key value type database is realized, and the data security is improved.
Other details of the apparatus for data synchronization according to the embodiment of the present invention are similar to the method according to the embodiment of the present invention described above with reference to fig. 1 to 4, and are not repeated herein.
FIG. 5 is a block diagram illustrating an exemplary hardware architecture of a computing device capable of implementing the method and apparatus for data synchronization according to embodiments of the present invention.
As shown in fig. 5, computing device 500 includes an input device 501, an input interface 502, a central processor 503, a memory 504, an output interface 505, and an output device 506. The input interface 502, the central processing unit 503, the memory 504, and the output interface 505 are connected to each other through a bus 510, and the input device 501 and the output device 506 are connected to the bus 510 through the input interface 502 and the output interface 505, respectively, and further connected to other components of the computing device 500. Specifically, the input device 501 receives input information from the outside and transmits the input information to the central processor 503 through the input interface 502; the central processor 503 processes input information based on computer-executable instructions stored in the memory 504 to generate output information, temporarily or permanently stores the output information in the memory 504, and then transmits the output information to the output device 506 through the output interface 505; output device 506 outputs the output information outside of computing device 500 for use by a user.
In one embodiment, the computing device 500 shown in fig. 5 may be implemented as a terminal device comprising: a memory and a processor; the memory is used for storing executable program codes; the processor is used for reading the executable program codes stored in the memory to execute the processing method of the above embodiment.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product or computer-readable storage medium. The computer program product or computer-readable storage medium includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is to be understood that the invention is not limited to the specific arrangements and instrumentality described above and shown in the drawings. A detailed description of known methods is omitted herein for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present invention are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications and additions or change the order between the steps after comprehending the spirit of the present invention.
As described above, only the specific embodiments of the present invention are provided, and it can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the system, the module and the unit described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. It should be understood that the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present invention, and these modifications or substitutions should be covered within the scope of the present invention.
Claims (15)
1. A method of data synchronization, the method comprising:
obtaining an archive log file generated by a relational database;
analyzing and processing the filing log file to obtain an updated operation log;
constructing the updated oplog as a JAVA object dedicated to a key-value type database;
pushing the JAVA object to the key value type database;
and reconstructing the JAVA object in the key value type database according to the service logic requirement, and executing the reconstructed JAVA object.
2. The method of claim 1, further comprising, prior to said obtaining an archive log file generated by a relational database:
acquiring an account number and a password of the relational database and configuring information of a table to be synchronized;
wherein, the information of the table to be synchronized comprises at least one of the following four types:
table name, table field authority and network transmission authority for transmitting the table to be synchronized.
3. The method of claim 2, wherein analyzing the archive log file to obtain an updated oplog comprises:
acquiring archiving table information corresponding to the table to be synchronized in the archiving log file;
and when the information of the table to be synchronized has the information matched with the information of the filing table, analyzing the filing log file to obtain the updated operation log.
4. The method of claim 2, wherein constructing the updated oplog as a JAVA object specific to a key-value type database comprises:
obtaining a first data attribute from the updated operation log;
acquiring a second data attribute corresponding to the updated operation log according to the first data attribute and the table to be synchronized;
and constructing the JAVA object dedicated by the key-value type database according to the first data attribute and the second data attribute.
5. The method of claim 4, wherein the first data attribute comprises at least one of the following six: data block address, object number, row number within data block, field value, data operation type, and transaction number.
6. The method of claim 4, wherein the second data attribute comprises at least one of the following five:
table name, primary key field value, modified field name, and modified field value.
7. The method of claim 1, wherein pushing the JAVA object into the key-value type database comprises:
extracting a log file of the updated operation log, wherein the log file comprises sequence number SQN information;
determining a data control type corresponding to a generated SQL statement in the log file comprising the sequence number SQN information according to the transaction number;
when the data control type is submitted, encrypting the JAVA object to form an Msg database file;
and pushing the Msg database file to the key value type database through a Java message service application program interface in a point-to-point mode.
8. The method of claim 7, further comprising, after said pushing the JAVA object into the key-value type database:
decrypting the Msg database file according to a secret key;
when the decryption is successful, the key value type database feeds back confirmation information to the relational database;
and when decryption fails, the relational database pushes the Msg database file to the key-value type database again through a Java message service application program interface in a point-to-point mode.
9. The method of claim 1, wherein reconstructing the JAVA object in the key-value type database according to business logic requirements and executing the reconstructed JAVA object comprises:
determining a key value pair of a target service and a unique aperture of a key according to service logic requirements;
reconstructing the JAVA object according to the key value pair of the target service and the unique aperture of the key;
and executing the reconstructed JAVA object in the key value type database.
10. An apparatus for data synchronization, the apparatus comprising: the system comprises a relational database data monitoring module, a relational database data processing module, an encrypted data transmission module and a key value type database data loading module;
the relational database data monitoring module is used for acquiring an archiving log file generated by the relational database; the archive log file is used for analyzing and processing the archive log file to obtain an updated operation log;
the relational database data processing module is used for constructing the updated operation log into a JAVA object special for a key value type database;
the encrypted data transmission module is used for pushing the JAVA object to the key value type database data loading module;
and the key value type database data loading module is used for reconstructing the JAVA object according to the service logic requirement and executing the reconstructed JAVA object.
11. The device according to claim 10, wherein the relational database data monitoring module is further configured to obtain an account and a password of the relational database, and configure information of a table to be synchronized;
wherein, the information of the table to be synchronized comprises at least one of the following four types:
table name, table field authority and network transmission authority for transmitting the table to be synchronized.
12. The apparatus according to claim 11, wherein the relational database data monitoring module is specifically configured to obtain archive table information corresponding to the table to be synchronized in the archive log file; and for
And when the information of the table to be synchronized has the information matched with the information of the filing table, analyzing the filing log file to obtain the updated operation log.
13. The apparatus of claim 11, wherein the relational database data processing module comprises: the device comprises a first acquisition unit, a second acquisition unit and a construction unit;
the first obtaining unit is used for obtaining a first data attribute from the updated operation log;
the second obtaining unit is configured to obtain a second data attribute corresponding to the updated operation log according to the first data attribute and the table to be synchronized;
the constructing unit is configured to construct the JAVA object dedicated to the key-value type database according to the first data attribute and the second data attribute.
14. A terminal device, comprising:
a memory, a processor, a communication interface, and a bus;
the memory, the processor and the communication interface are connected through the bus and complete mutual communication;
the memory is used for storing program codes;
the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory for performing the method of any one of claims 1 to 9.
15. A computer storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of any of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810935532.2A CN110837535A (en) | 2018-08-16 | 2018-08-16 | Data synchronization method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810935532.2A CN110837535A (en) | 2018-08-16 | 2018-08-16 | Data synchronization method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110837535A true CN110837535A (en) | 2020-02-25 |
Family
ID=69573301
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810935532.2A Pending CN110837535A (en) | 2018-08-16 | 2018-08-16 | Data synchronization method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110837535A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111949627A (en) * | 2020-08-14 | 2020-11-17 | 北京每日优鲜电子商务有限公司 | Method, apparatus, electronic device, and medium for tabulating log files |
| CN112732839A (en) * | 2021-01-21 | 2021-04-30 | 云账户技术(天津)有限公司 | Data synchronization method and device |
| CN113312351A (en) * | 2021-06-01 | 2021-08-27 | 北京沃东天骏信息技术有限公司 | Data processing method and device |
| CN113486019A (en) * | 2021-07-27 | 2021-10-08 | 中国银行股份有限公司 | Method and device for automatically triggering real-time batch synchronization of remote multi-database data |
| CN113760845A (en) * | 2020-08-17 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Log processing method, system, device, client and storage medium |
| WO2022009162A1 (en) * | 2020-07-10 | 2022-01-13 | International Business Machines Corporation | Archiving accelerator-only database tables |
| CN115098537A (en) * | 2021-10-19 | 2022-09-23 | 腾讯科技（深圳）有限公司 | Transaction execution method and device, computing equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7356550B1 (en) * | 2001-06-25 | 2008-04-08 | Taiwan Semiconductor Manufacturing Company | Method for real time data replication |
| CN103761318A (en) * | 2014-01-27 | 2014-04-30 | 中国工商银行股份有限公司 | Method and system for data synchronization of relational heterogeneous databases |
| CN105243067A (en) * | 2014-07-07 | 2016-01-13 | 北京明略软件系统有限公司 | Method and apparatus for realizing real-time increment synchronization of data |
| CN105447111A (en) * | 2015-11-13 | 2016-03-30 | 铜陵有色金属集团股份有限公司金冠铜业分公司 | Mysql and MongoDB data synchronization method and system |
| CN107818431A (en) * | 2016-09-14 | 2018-03-20 | 北京京东尚科信息技术有限公司 | A kind of method and system that order track data is provided |
| CN107958010A (en) * | 2016-10-18 | 2018-04-24 | 北京京东尚科信息技术有限公司 | Method and system for online data migration |
-
2018
- 2018-08-16 CN CN201810935532.2A patent/CN110837535A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7356550B1 (en) * | 2001-06-25 | 2008-04-08 | Taiwan Semiconductor Manufacturing Company | Method for real time data replication |
| CN103761318A (en) * | 2014-01-27 | 2014-04-30 | 中国工商银行股份有限公司 | Method and system for data synchronization of relational heterogeneous databases |
| CN105243067A (en) * | 2014-07-07 | 2016-01-13 | 北京明略软件系统有限公司 | Method and apparatus for realizing real-time increment synchronization of data |
| CN105447111A (en) * | 2015-11-13 | 2016-03-30 | 铜陵有色金属集团股份有限公司金冠铜业分公司 | Mysql and MongoDB data synchronization method and system |
| CN107818431A (en) * | 2016-09-14 | 2018-03-20 | 北京京东尚科信息技术有限公司 | A kind of method and system that order track data is provided |
| CN107958010A (en) * | 2016-10-18 | 2018-04-24 | 北京京东尚科信息技术有限公司 | Method and system for online data migration |
Non-Patent Citations (1)
| Title |
|---|
| 邓以克 等: "基于JMS的安全通信模型研究与设计", 《计算机工程与设计》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022009162A1 (en) * | 2020-07-10 | 2022-01-13 | International Business Machines Corporation | Archiving accelerator-only database tables |
| US11500833B2 (en) | 2020-07-10 | 2022-11-15 | International Business Machines Corporation | Archiving accelerator-only database tables |
| GB2611968A (en) * | 2020-07-10 | 2023-04-19 | Ibm | Archiving accelerator-only database tables |
| CN111949627A (en) * | 2020-08-14 | 2020-11-17 | 北京每日优鲜电子商务有限公司 | Method, apparatus, electronic device, and medium for tabulating log files |
| CN111949627B (en) * | 2020-08-14 | 2023-12-15 | 深圳市智汇云网络科技有限公司 | Method, device, electronic equipment and medium for tabulating log files |
| CN113760845A (en) * | 2020-08-17 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Log processing method, system, device, client and storage medium |
| CN112732839A (en) * | 2021-01-21 | 2021-04-30 | 云账户技术(天津)有限公司 | Data synchronization method and device |
| CN113312351A (en) * | 2021-06-01 | 2021-08-27 | 北京沃东天骏信息技术有限公司 | Data processing method and device |
| CN113486019A (en) * | 2021-07-27 | 2021-10-08 | 中国银行股份有限公司 | Method and device for automatically triggering real-time batch synchronization of remote multi-database data |
| CN113486019B (en) * | 2021-07-27 | 2024-02-23 | 中国银行股份有限公司 | Automatic triggering real-time batch synchronization method and device for remote multi-database data |
| CN115098537A (en) * | 2021-10-19 | 2022-09-23 | 腾讯科技（深圳）有限公司 | Transaction execution method and device, computing equipment and storage medium |
| CN115098537B (en) * | 2021-10-19 | 2023-03-10 | 腾讯科技（深圳）有限公司 | Transaction execution method and device, computing equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110837535A (en) | Data synchronization method, device, equipment and medium | |
| US11604804B2 (en) | Data replication system | |
| CN107958010B (en) | Method and system for online data migration | |
| US11036754B2 (en) | Database table conversion | |
| CN107220142B (en) | Method and device for executing data recovery operation | |
| CN107818431B (en) | Method and system for providing order track data | |
| CN107391635A (en) | Data synchronous system and method | |
| CN109144785B (en) | Method and apparatus for backing up data | |
| EP3722973B1 (en) | Data processing method and device for distributed database, storage medium, and electronic device | |
| US20170161291A1 (en) | Database table conversion | |
| CN106960007A (en) | A kind of data-updating method, device and electronic equipment | |
| US11275760B2 (en) | Online schema and data transformations | |
| US12079202B2 (en) | Parallel stream processing of change data capture | |
| CN109086382B (en) | Data synchronization method, device, equipment and storage medium | |
| CN111737227B (en) | Data modification method and system | |
| CN113934792B (en) | Processing method and device of distributed database, network equipment and storage medium | |
| US20230144072A1 (en) | Data storage server and client devices for securely storing data | |
| CN110781197A (en) | Hive offline synchronous verification method and device and electronic equipment | |
| EP2904520B1 (en) | Reference data segmentation from single to multiple tables | |
| US11093525B2 (en) | Transaction merging for offline applications | |
| CN116501700B (en) | APP formatted file offline storage method, device, equipment and storage medium | |
| CN115858493A (en) | Data migration method and device, electronic equipment and readable storage medium | |
| CN114328464A (en) | Data maintenance method, device, equipment and readable medium for distributed storage equipment | |
| US11175905B2 (en) | Optimizing uploads for an offline scenario by merging modification operations | |
| CN118410106B (en) | Cross-data source real-time synchronization method based on time line mapping |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200225 |
|
| RJ01 | Rejection of invention patent application after publication |