CN116166739A - Method, device, electronic equipment and storage medium for synchronizing data in real time - Google Patents

Method, device, electronic equipment and storage medium for synchronizing data in real time Download PDF

Info

Publication number
CN116166739A
CN116166739A CN202111407866.0A CN202111407866A CN116166739A CN 116166739 A CN116166739 A CN 116166739A CN 202111407866 A CN202111407866 A CN 202111407866A CN 116166739 A CN116166739 A CN 116166739A
Authority
CN
China
Prior art keywords
data
database
current
log file
new log
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111407866.0A
Other languages
Chinese (zh)
Inventor
展波
刘超
陈合
王征
姜传德
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qingdao Haier Technology Co Ltd
Haier Smart Home Co Ltd
Original Assignee
Qingdao Haier Technology Co Ltd
Haier Smart Home Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qingdao Haier Technology Co Ltd, Haier Smart Home Co Ltd filed Critical Qingdao Haier Technology Co Ltd
Priority to CN202111407866.0A priority Critical patent/CN116166739A/en
Publication of CN116166739A publication Critical patent/CN116166739A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815—Journaling file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application relates to the technical field of digital information transmission and discloses a method for synchronizing data in real time. The method for synchronizing data in real time includes: under the condition that a new log file is generated by the source database is monitored, current incremental data is obtained by analyzing the new log file; reversely checking the source database to obtain field information of the source database before generating a new log file; determining current database data after the operation behavior contained in the new log file according to the field information and the current increment data; and sending the current database data to the target database, so that the target database and the source database are synchronized. The method for synchronizing data in real time can provide reliability of data synchronization. The application also discloses a device for synchronizing data in real time, electronic equipment and a storage medium.

Description

Method, device, electronic equipment and storage medium for synchronizing data in real time
Technical Field
The present invention relates to the technical field of digital information transmission, and for example, to a method, an apparatus, an electronic device, and a storage medium for synchronizing data in real time.
Background
Currently, more and more enterprises rely on databases and other infrastructures, and data transmission is frequently performed in daily work scenes of the application enterprises. In the single transmission process of data, the data flows from a source end to a target end, for example, when an information release platform pushes advertisements to company staff, a working mailbox of each staff is needed, staff mailbox data are not input again in general, the data are pulled from a staff information table of a database of a manpower department, the database of the manpower is usually called as the source end, and the database of the information release platform is called as the target end.
The conventional data transmission generally uses a java database connection (Java Database Connectivity, JDBC) to pump data, a source database is connected in the pumping process, the data is called by a development interface mode, packaged and sent to a target terminal, and the data is transmitted in a certain time rule by setting a timing task. Thus, when data is transmitted each time, the source database resource needs to be called, and the source database resource is occupied more.
In this regard, in some prior arts, a log file corresponding to the source database may be obtained, in which each operation behavior and operation information of the source database within a specified duration before the current time are recorded, and corresponding operations are performed on the target database according to each record of the obtained log file records corresponding to the operation behavior and operation information. And the data synchronization process occupies the source database resources.
In the process of implementing the embodiments of the present disclosure, it is found that at least the following problems exist in the related art:
the data synchronization of the source database and the target database is realized only according to the operation behaviors and the operation information in the log file, and the reliability is poor.
Disclosure of Invention
The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed embodiments. This summary is not an extensive overview, and is intended to neither identify key/critical elements nor delineate the scope of such embodiments, but is intended as a prelude to the more detailed description that follows.
The embodiment of the disclosure provides a method, a device and electronic equipment for synchronizing data in real time, so as to improve the reliability of data synchronization of a source database and a target database.
In some embodiments, a method for synchronizing data in real time includes: under the condition that a new log file is generated by the source database is monitored, current incremental data is obtained by analyzing the new log file; reversely checking the source database to obtain field information of the source database before the new log file is generated; determining current database data after the operation behavior contained in the new log file according to the field information and the current increment data; and sending the current database data to a target database, so that the target database and the source database are synchronized.
Optionally, determining current database data after the operation behavior contained in the new log file according to the field information and the current increment data includes: one-to-one correspondence is carried out on the first type field information in the field information and the current increment data; performing field completion by using the second type field information and the value of the second type field information in the field information; and determining the data after one-to-one correspondence and the data after complement as the current database data.
Optionally, the second type field information includes at least primary key information and name information.
Optionally, sending the current database data to a target database includes: packaging the current database data into character strings according to a preset data exchange format; and sending the character string to a message queue middleware, and pushing the character string to the target database by the message queue middleware.
Optionally, according to a preset data exchange format, packaging the field information of the source database and the current database data into a character string, including: and packaging the corresponding primary key information, the field information and the current database data into character strings in sequence.
Optionally, obtaining the current incremental data by parsing the new log file includes: simulating an interaction protocol of a slave database to send a dump request to the source database; after the source database receives the dump request, obtaining a binary log corresponding to the new log file sent by the source database; and analyzing the binary log to obtain the current increment data.
Optionally, before parsing the new log file to obtain the current incremental data, the method for synchronizing data in real time further includes: obtaining new log data of the source database; synthesizing the new log data into a new log file; and storing the new log file.
In some embodiments, an apparatus for synchronizing data in real time includes: the system comprises an analysis module, a query module, a determination module and a sending module, wherein the analysis module is configured to obtain current incremental data by analyzing a new log file under the condition that the new log file is generated by a source database; the query module is configured to review the source database to obtain historical data before the new log file is generated; the determining module is configured to determine current database data after the operation behavior contained in the new log file according to the historical data and the current increment data; the sending module is configured to send the current database data to a target database, so that the target database and the source database are synchronized.
In some embodiments, an electronic device comprises a processor and a memory storing program instructions configured to, when executed, perform the method for synchronizing data in real time provided by the previous embodiments.
In some embodiments, a storage medium stores program instructions that, when executed, perform the method for synchronizing data in real time provided by the previous embodiments.
The method, the device, the electronic device and the storage medium for synchronizing data in real time provided by the embodiment of the disclosure can realize the following technical effects:
under the condition that the log file is generated by the source database is monitored, the log file is analyzed to obtain current increment data, meanwhile, the historical data before the log file is generated is obtained through checking the database, finally, the current database data is determined according to the historical data and the current increment data, the determined current database data is matched with the historical data and the current increment data, and compared with the data determined only according to the operation information in the log file, the method is more accurate and perfect, and the reliability of data synchronization of the source database and the target database is improved.
The foregoing general description and the following description are exemplary and explanatory only and are not restrictive of the application.
Drawings
One or more embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements, and in which:
FIG. 1 is a schematic diagram of a data transmission model of a source database and a target database according to an embodiment of the present disclosure;
FIG. 2 is a schematic diagram of a method for synchronizing data in real time provided by an embodiment of the present disclosure;
FIG. 3 is a schematic diagram of a method for synchronizing data in real time provided by an embodiment of the present disclosure;
FIG. 4 is a schematic diagram of a method for synchronizing data in real time provided by an embodiment of the present disclosure;
FIG. 5 is a schematic diagram of an apparatus for synchronizing data in real time according to an embodiment of the present application;
FIG. 6 is a schematic diagram of an apparatus for synchronizing data in real time according to an embodiment of the present application;
fig. 7 is a schematic diagram of an apparatus for synchronizing data in real time according to an embodiment of the present application.
Detailed Description
So that the manner in which the features and techniques of the disclosed embodiments can be understood in more detail, a more particular description of the embodiments of the disclosure, briefly summarized below, may be had by reference to the appended drawings, which are not intended to be limiting of the embodiments of the disclosure. In the following description of the technology, for purposes of explanation, numerous details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, one or more embodiments may still be practiced without these details. In other instances, well-known structures and devices may be shown simplified in order to simplify the drawing.
The terms first, second and the like in the description and in the claims of the embodiments of the disclosure and in the above-described figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe embodiments of the present disclosure. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion.
The term "plurality" means two or more, unless otherwise indicated.
In the embodiment of the present disclosure, the character "/" indicates that the front and rear objects are an or relationship. For example, A/B represents: a or B.
The term "and/or" is an associative relationship that describes an object, meaning that there may be three relationships. For example, a and/or B, represent: a or B, or, A and B.
Fig. 1 is a schematic diagram of a data transmission model of a source database and a target database according to an embodiment of the disclosure.
After the source database generates a new log file, the new log file is extracted, parsed and packaged, sent to the message queue middleware, and then pushed to the target database by the message queue middleware as shown in fig. 1.
More specifically, referring to fig. 2, the method for synchronizing data in real time includes:
s201, under the condition that a new log file is generated by the source database, the current incremental data is obtained by analyzing the new log file.
Here, the new log file may be parsed to obtain current delta data using a cananal technique. For example, obtaining current delta data by parsing a new log file may include: simulating sending dump requests from an interaction protocol of the database to the source database; after the source database receives the dump request, binary logs corresponding to the new log files and sent by the source database are obtained; and analyzing the binary log to obtain the current increment data.
In the process of analyzing the new log file to obtain the current incremental data, the source database is used as a master, and the slave database is used as a slave. Based on an interaction protocol, after sending a dump request from a database to a source database, the source database sends a binary log corresponding to a new log file to the slave database; after receiving the binary log from the database, the binary log can be parsed and the current incremental data obtained.
In a conventional application of canal, after the current delta data is obtained, the delta data is sent to a target database. In the embodiment provided in the application, after the current incremental data is obtained, the current incremental data is subjected to related processing to improve the reliability of the data, and then the processed data is sent to the target database.
In order to facilitate data recovery, problem diagnosis and problem tracking, after the data in the source database is changed, the operation record and the operation data are stored as log files. Compared with the method that the log file is read after the specified time length, the method has the advantages that the source database is monitored, the data in the source database is changed after the generation of the new log file is monitored, the log file is analyzed at the moment, the current increment data are obtained, and therefore the real-time performance of the synchronous data can be improved.
S202, reversely checking the source database to obtain field information of the source database before generating the new log file.
The query database can be divided into forward query and reverse query (reverse query), wherein the party with the foreign key or the multi-to-multi-field is called the forward query, and the party with the foreign key or the multi-to-multi-field is called the reverse query.
In the process of synchronizing the source database and the target database, compared with the source data resources occupied by data pumping of the source database, the source data resources occupied by the reverse checking source database are greatly reduced, and waste of the source database resources is reduced.
The field information of the source database may include first type field information and second type field information, where the first type field information corresponds to the current incremental data, and the second type field information may complement the fields to make the data in the target database consistent with the data format in the source data.
Specifically, the first type of field information may be field information corresponding to field information in the current incremental data, and the first type of field information may be fields such as "employee number (Identity Document, ID)", "employee mailbox", etc.; the second type field information is field information corresponding to each of the first type field information, and includes at least primary key information and name information, for example, primary key information corresponding to the field "employee ID" (i.e., whether the field "employee number" is a primary key), and primary key information corresponding to the field "employee mailbox" (i.e., whether the field "employee number" is a primary key). Thus, the data in the target database and the primary key in the source database are convenient to keep consistent, and the reliability of data synchronization is improved.
The first type of field information may be different in different application scenarios, for example, in the aforementioned scenario in which advertisements are pushed to company employees by the information distribution platform, the first type of field information may include "employee ID"; in the file storage database, the first type of field information may include a "file name". According to the application scenarios, the field suitable for the scenario is determined as the first field information by the person skilled in the art, and will not be described in detail here.
Further, the back-looking up the source database to obtain field information of the source database before generating the new log file may include:
and reversely checking the source database to obtain the first field information and the second field information of the source database and the corresponding values thereof.
By the method, the source database is reversely checked, the content is less, the occupied resources of the source database are less, and the waste of the resources of the source database is reduced.
S203, determining current database data after the operation behaviors contained in the new log file according to the field information and the current increment data.
For example, the current database data may be determined as follows: one-to-one correspondence is carried out on the first type field information in the field information and the current increment data; performing field completion by using the values of the second type field information and the second type field information in the field information; and determining the data after one-to-one correspondence and the data after complement as current database data.
The technical scheme is used for determining the current database data, so that the data in the target database and the data in the source database are consistent in format, and the accuracy of data synchronization is improved.
The second type field information has a value corresponding to the source database or, alternatively, to the current delta data.
Specifically, the second type field information may further include "sequence number (index)", "null value (null)", "change (update)", and the like.
Here, an "id" is added to the source database: 28. file_name: science, file_size:600M "is exemplary. After the information is added in the source database, a new log file is generated, and the new log file records: "id" is added: 28. file_name: science, file_size:600M ", after parsing the new log file, the current incremental data obtained is: "id: 28. file_name: science, file_size:600M ", reversely checking the field information of the source database, wherein the first type of field information in the field information is" id "," file_name "and" file_size ", and the second type of field information and the value thereof can comprise" id, key: true "," file_name, key: false "and" file_size, key: false ", in which case the field" id "is the primary key; the second type field information and its value may further include: "id, index:0"," file_name, index:1 "and" file_size, index: 2'; in the second-type field information, a value of the second-type field information corresponds to the source database.
Alternatively, the second type field information and its value may further include: "id, null: false "," file_name, null: false "and" file_size, null: the field "includes values of fields" id "," file_name "and" file_size ", so that the three first-type field information in the second-type field information corresponds to" null ": false "; in the second type field information, a value of the second type field information corresponds to the current delta data.
Further, the first field information corresponds to the current increment data one by one, and includes: the first type of field information includes each field corresponding to the current delta data. For example, in the case where the first type field information is "id", "file_name", and "file_size", if the field corresponding to the current delta data includes "id", "file_name", and "file_size", it is determined that the first type field information corresponds to the current delta data one by one; if the field corresponding to the current increment data comprises an 'id' and a 'file_name', determining that the first type field corresponds to the current increment data one by one; if the field corresponding to the current increment data comprises an 'id' and a 'file_names', determining that the first type field information and the current increment data do not correspond one by one because the first type field information does not comprise the 'file_names'.
Under the condition that the first type field information is determined to correspond to the current increment data one by one, the second type field information in the field information and the value of the second type field information are utilized to carry out field completion; and determining the data after one-to-one correspondence and the data after complement as current database data.
S204, sending the current database data to the target database, and synchronizing the target database and the source database.
Optionally, sending the current database data to the target database includes: packaging the current database data into character strings according to a preset data exchange format; and sending the character string to the message queue middleware, and pushing the character string to the target database by the message queue middleware.
And after the target database receives the character string pushed by the message queue middleware, updating according to the character string. Thus, the updating of the source database and the target database is realized.
Optionally, according to a preset data exchange format, packaging field information of the source database and current database data into a character string, including: and packaging the corresponding primary key information, field information and current database data into character strings in turn. Therefore, the data format and sequence of the target database and the source database are consistent, and the reliability of data synchronization is improved.
In some application scenarios, the "id" is added to the source database: 28. file_name: science, file_size:600M "for example, the packaged string may include the following format:
Figure BDA0003373044610000081
/>
Figure BDA0003373044610000091
in some application scenarios, the string also includes an operation type, e.g., the string also includes an "eventType" that indicates that the operation type is INSERT.
Under the condition that the log file is generated by the source database is monitored, the log file is analyzed to obtain current increment data, meanwhile, the historical data before the log file is generated is obtained through checking the database, finally, the current database data is determined according to the historical data and the current increment data, the determined current database data is matched with the historical data and the current increment data, and compared with the data determined only according to the operation information in the log file, the method is more accurate and perfect, and the reliability of data synchronization of the source database and the target database is improved.
Fig. 3 is a schematic diagram of a method for synchronizing data in real time provided by an embodiment of the present disclosure.
As shown in connection with fig. 3, the method for synchronizing data in real time includes:
s301, monitoring a source database, and executing S302 if the source database is monitored to generate a new log file; otherwise, S301 is continued.
S302, obtaining current incremental data by analyzing the new log file.
S303, reversely checking the source database to obtain field information of the source database before the new log file is generated.
S304, determining the current database data after the operation behavior contained in the new log file according to the field information and the current increment data.
S305, sending the current database data to the target database, and synchronizing the target database and the source database.
Fig. 4 is a schematic diagram of a method for synchronizing data in real time provided by an embodiment of the present disclosure.
Referring to fig. 4, the method for synchronizing data in real time includes:
s401, monitoring a source database.
If the source database does not generate new log data, continuing to monitor the source database; if it is monitored that the source database generates new log data, S402 is performed.
S402, obtaining new log data of the source database.
S403, synthesizing the new log data into a new log file.
S404, storing the new log file.
S405, obtaining current incremental data by analyzing the new log file.
S406, reversely checking the source database to obtain field information of the source database before the new log file is generated.
S407, determining the current database data after the operation behavior contained in the new log file according to the field information and the current increment data.
S408, the current database data is sent to the target database, so that the target database and the source database are synchronized.
Fig. 5 is a schematic diagram of an apparatus for synchronizing data in real time according to an embodiment of the present application.
As shown in connection with fig. 5, the apparatus for synchronizing data in real time includes an parsing module 51, a querying module 52, a determining module 53, and a transmitting module 54. The parsing module 51 is configured to obtain current incremental data by parsing the new log file in case that it is heard that the source database generates the new log file; the query module 52 is configured to review the source database to obtain field information of the source database prior to generating the new log file; the determining module 53 is configured to determine current database data following the operation behavior contained in the new log file according to the field information and the current delta data; the sending module 54 is configured to send the current database data to the target database to synchronize the target database with the source database.
Optionally, the determining module 53 includes a corresponding unit, a complementing unit, and a determining unit; the corresponding unit is configured to perform one-to-one correspondence between the first type field information in the field information and the current increment data; the completion unit is configured to perform field completion by using the second type field information and the value of the second type field information in the field information; the determining unit is configured to determine the one-to-one correspondence data and the complemented data as current database data.
Optionally, the second type field information includes at least primary key information and name information.
Optionally, the transmitting module 54 includes a packaging unit and a first transmitting unit; the packaging unit is configured to package the current database data into character strings according to a preset data exchange format; the first sending unit is configured to send the character string to the message queue middleware, which pushes the character string to the target database.
Optionally, the packaging unit is specifically configured to: and packaging the corresponding primary key information, field information and current database data into character strings in sequence.
Alternatively, the parsing module 51 includes a second transmitting unit, an obtaining unit, and a parsing unit; the second sending unit is configured to simulate sending a dump request from an interaction protocol of the database to the source database; the obtaining unit is configured to obtain a binary log corresponding to the new log file sent by the source database after the source database receives the dump request; the parsing unit is configured to parse the binary log to obtain current incremental data.
Fig. 6 is a schematic diagram of an apparatus for synchronizing data in real time according to an embodiment of the present disclosure.
As shown in connection with fig. 6, the apparatus for synchronizing data in real time further comprises an obtaining module 55, a synthesizing module 56 and a saving module 57; the obtaining module 55 is configured to obtain new log data of the shown source database; the synthesizing module 56 is configured to synthesize the new log data into a new log file; the save module 57 is configured to save the new log file.
In some embodiments, an electronic device includes a processor and a memory storing program instructions, the processor being configured, when executing the program instructions, to perform the method for synchronizing data in real time provided by the foregoing embodiments.
As shown in fig. 7, a schematic diagram of an electronic device is provided in an embodiment of the disclosure.
As shown in connection with fig. 7, the electronic device includes:
a processor (processor) 71 and a memory (memory) 72, and may also include a communication interface (Communication Interface) 73 and a bus 74. The processor 71, the communication interface 73, and the memory 72 may communicate with each other via the bus 74. Communication interface 73 may be used for information transfer. The processor 71 may invoke logic instructions in the memory 72 to perform the method for providing data in real time provided by the previous embodiments.
Further, the logic instructions in the memory 72 described above may be implemented in the form of software functional units and stored in a computer readable storage medium when sold or used as a stand alone product.
The memory 72 serves as a computer readable storage medium for storing a software program, a computer executable program, such as program instructions/modules corresponding to the methods in the embodiments of the present disclosure. The processor 71 executes functional applications and data processing by running software programs, instructions and modules stored in the memory 72, i.e. implements the methods of the method embodiments described above.
Memory 72 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created according to the use of the terminal device, etc. In addition, memory 72 may include high-speed random access memory, and may also include non-volatile memory.
The embodiment of the disclosure provides an electronic device, which comprises the device for synchronizing data in real time provided by the embodiment.
The disclosed embodiments provide a storage medium storing computer-executable instructions configured to perform the method for synchronizing data in real time provided by the foregoing embodiments.
The disclosed embodiments provide a computer program product comprising a computer program stored on a computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the method for synchronizing data in real time provided by the previous embodiments.
The computer readable storage medium may be a transitory computer readable storage medium or a non-transitory computer readable storage medium.
The aspects of the disclosed embodiments may be embodied in a software product stored on a storage medium, including one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of a method in an embodiment of the disclosure. And the aforementioned storage medium may be a non-transitory storage medium including: a plurality of media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or a transitory storage medium.
The above description and the drawings illustrate embodiments of the disclosure sufficiently to enable those skilled in the art to practice them. Other embodiments may involve structural, logical, electrical, process, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the sequence of operations may vary. Portions and features of some embodiments may be included in, or substituted for, those of others. Moreover, the terminology used in the present application is for the purpose of describing embodiments only and is not intended to limit the claims. As used in the description of the embodiments and the claims, the singular forms "a," "an," and "the" (the) are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, when used in this application, the terms "comprises," "comprising," and/or "includes," and variations thereof, mean that the stated features, integers, steps, operations, elements, and/or components are present, but that the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof is not precluded. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method or apparatus comprising such elements. In this context, each embodiment may be described with emphasis on the differences from the other embodiments, and the same similar parts between the various embodiments may be referred to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method sections disclosed in the embodiments, the description of the method sections may be referred to for relevance.
Those of skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. The skilled person may use different methods for each particular application to achieve the described functionality, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. It will be clearly understood by those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not described herein again.
In the embodiments disclosed herein, the disclosed methods, articles of manufacture (including but not limited to devices, apparatuses, etc.) may be practiced in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of elements may be merely a logical functional division, and there may be additional divisions when actually implemented, e.g., multiple elements or components may be combined or integrated into another system, or some features may be omitted or not performed. In addition, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interface, device or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form. The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to implement the present embodiment. In addition, each functional unit in the embodiments of the present disclosure may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. Each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Claims (10)

1. A method for synchronizing data in real time, comprising:
under the condition that a new log file is generated by the source database is monitored, current incremental data is obtained by analyzing the new log file;
reversely checking the source database to obtain field information of the source database before the new log file is generated;
determining current database data after the operation behavior contained in the new log file according to the field information and the current increment data;
and sending the current database data to a target database, so that the target database and the source database are synchronized.
2. The method of claim 1, determining current database data following the operational behavior contained in the new log file based on the field information and the current delta data, comprising:
one-to-one correspondence is carried out on the first type field information in the field information and the current increment data;
performing field completion by using the second type field information and the value of the second type field information in the field information;
and determining the data after one-to-one correspondence and the data after complement as the current database data.
3. The method of claim 2, wherein the second type of field information includes at least primary key information and name information.
4. The method of claim 1, wherein transmitting the current database data to a target database comprises:
packaging the current database data into character strings according to a preset data exchange format;
and sending the character string to a message queue middleware, and pushing the character string to the target database by the message queue middleware.
5. The method of claim 2, wherein packaging the field information of the source database and the current database data as a string according to a preset data exchange format, comprises:
and packaging the corresponding primary key information, the field information and the current database data into character strings in sequence.
6. The method of claim 1, wherein obtaining current delta data by parsing the new log file comprises:
simulating an interaction protocol of a slave database to send a dump request to the source database;
after the source database receives the dump request, obtaining a binary log corresponding to the new log file sent by the source database;
and analyzing the binary log to obtain the current increment data.
7. The method of any of claims 1 to 6, further comprising, prior to parsing the new log file to obtain current delta data:
obtaining new log data of the source database;
synthesizing the new log data into a new log file;
and storing the new log file.
8. An apparatus for synchronizing data in real time, comprising:
the analysis module is configured to obtain current incremental data by analyzing the new log file under the condition that the new log file is generated by the source database;
the query module is configured to reversely check the source database to obtain historical data before the new log file is generated;
a determining module configured to determine current database data after the operation behavior contained in the new log file according to the history data and the current increment data;
and the sending module is configured to send the current database data to a target database so as to synchronize the target database with the source database.
9. An electronic device comprising a processor and a memory storing program instructions, wherein the processor is configured to perform the method for synchronizing data in real time as claimed in any one of claims 1 to 7 when executing the program instructions.
10. A storage medium storing program instructions which, when executed, perform the method for synchronizing data in real time as claimed in any one of claims 1 to 7.
CN202111407866.0A 2021-11-24 2021-11-24 Method, device, electronic equipment and storage medium for synchronizing data in real time Pending CN116166739A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111407866.0A CN116166739A (en) 2021-11-24 2021-11-24 Method, device, electronic equipment and storage medium for synchronizing data in real time

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111407866.0A CN116166739A (en) 2021-11-24 2021-11-24 Method, device, electronic equipment and storage medium for synchronizing data in real time

Publications (1)

Publication Number Publication Date
CN116166739A true CN116166739A (en) 2023-05-26

Family

ID=86415015

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111407866.0A Pending CN116166739A (en) 2021-11-24 2021-11-24 Method, device, electronic equipment and storage medium for synchronizing data in real time

Country Status (1)

Country Link
CN (1) CN116166739A (en)

Similar Documents

Publication Publication Date Title
CN109933630B (en) Real-time database data synchronization method and device
CN107818120B (en) Data processing method and device based on big data
CN111694840A (en) Data synchronization method, device, server and storage medium
CN110162526B (en) Method, device and equipment for inquiring data records in block chain type account book
CN105447046A (en) Distributed system data consistency processing method, device and system
CN112559475B (en) Data real-time capturing and transmitting method and system
CN114911830B (en) Index caching method, device, equipment and storage medium based on time sequence database
CN114647698A (en) Data synchronization method and device and computer storage medium
CN114328566A (en) Relationship graph updating method, device, medium, equipment and generating method
CN113672692B (en) Data processing method, data processing device, computer equipment and storage medium
CN115858488A (en) Parallel migration method and device based on data governance and readable medium
CN112416557B (en) Method and device for determining call relation, storage medium and electronic device
CN111209266B (en) Audit method and device based on Redis database and electronic equipment
CN112699183A (en) Data processing method, system, readable storage medium and computer equipment
CN116166739A (en) Method, device, electronic equipment and storage medium for synchronizing data in real time
CN115858322A (en) Log data processing method and device and computer equipment
CN115344633A (en) Data processing method, device, equipment and storage medium
CN114911872A (en) Intranet and extranet data synchronization method, device and system, extranet server and storage medium
CN112597119A (en) Method and device for generating processing log and storage medium
CN114221883B (en) Message testing method, device, server and storage medium
CN111654410B (en) Gateway request monitoring method, device, equipment and medium
CN112835991B (en) System, method, device and storage medium for monitoring data
CN113992436B (en) Local information generating method, device, equipment and storage medium
CN117892355B (en) Multiparty data joint analysis method and system based on privacy protection
CN111581289B (en) Data management method, system, device and medium for double-head block chain

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination