CN111460038A - Quasi-real-time data synchronization method and device - Google Patents
Quasi-real-time data synchronization method and device Download PDFInfo
- Publication number
- CN111460038A CN111460038A CN202010263463.2A CN202010263463A CN111460038A CN 111460038 A CN111460038 A CN 111460038A CN 202010263463 A CN202010263463 A CN 202010263463A CN 111460038 A CN111460038 A CN 111460038A
- Authority
- CN
- China
- Prior art keywords
- data
- synchronization
- database table
- synchronous
- synchronized
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 230000001360 synchronised effect Effects 0.000 claims abstract description 168
- 238000004590 computer program Methods 0.000 claims description 18
- 238000010586 diagram Methods 0.000 description 18
- 238000004891 communication Methods 0.000 description 11
- 238000012545 processing Methods 0.000 description 7
- 238000012217 deletion Methods 0.000 description 5
- 230000037430 deletion Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 238000007792 addition Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 239000000463 material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention provides a data quasi-real-time synchronization method and a device, wherein the method comprises the following steps: receiving a data subscription request sent by a client; generating a data synchronization configuration file according to the data subscription request and determining a synchronization data table; determining a database table to be synchronized according to the database log; and if the database table to be synchronized is judged to comprise the synchronous database table, transmitting the data to be synchronized of the synchronous database table comprised by the database table to be synchronized to the client according to the data synchronization configuration file. The device is used for executing the method. The data quasi-real-time synchronization method and device provided by the embodiment of the invention improve the data synchronization efficiency.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a data quasi-real-time synchronization method and device.
Background
In an enterprise, there is a wide variety of data that is typically stored in a database for recall when needed.
The asset management platform is used for carrying out unified management display on the bank asset data such as the architecture assets, the design assets and the like. At present, an asset management platform mainly acquires asset related data from a service component through a scheduled FTP file transmission task according to a scheduled time through a scheduled data interface format. The data acquisition mode has time delay in data processing and display. In addition, some data need to be acquired by the asset management platform actively calling a data service interface provided by a data provider, and a polling mode is needed to always access a data server of the data provider, so that the access pressure of the data server is increased, and the resource consumption is also increased.
Disclosure of Invention
To solve the problems in the prior art, embodiments of the present invention provide a method and an apparatus for synchronizing data in a near real-time manner, which can at least partially solve the problems in the prior art.
In one aspect, the present invention provides a data quasi-real-time synchronization method, including:
receiving a data subscription request sent by a client;
generating a data synchronization configuration file according to the data subscription request and determining a synchronization data table;
determining a database table to be synchronized according to the database log;
and if the database table to be synchronized is judged to comprise the synchronous database table, transmitting the data to be synchronized of the synchronous database table comprised by the database table to be synchronized to the client according to the data synchronization configuration file.
In another aspect, the present invention provides a data quasi-real-time synchronization apparatus, including:
the receiving unit is used for receiving a data subscription request sent by a client;
the generating unit is used for generating a data synchronization configuration file according to the data subscription request and determining a synchronization data table;
the determining unit is used for determining a database table to be synchronized according to the database log;
and the synchronization unit is used for sending the data to be synchronized of the synchronous data table included in the database table to be synchronized to the client according to the data synchronization configuration file after judging that the database table to be synchronized includes the synchronous database table.
In another aspect, the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data quasi-real-time synchronization method according to any of the above embodiments when executing the computer program.
In yet another aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of the data quasi-real-time synchronization method according to any one of the above embodiments.
The data quasi-real-time synchronization method and device provided by the embodiment of the invention can receive a data subscription request sent by a client, generate a data synchronization configuration file and determine a synchronization data table according to the data subscription request, determine a database table to be synchronized according to a database log, send data to be synchronized of the synchronization database table included in the database table to be synchronized to the client according to the data synchronization configuration file after judging that the database table to be synchronized includes the synchronization database table, complete the synchronization of the data through the data synchronization configuration file, and can synchronize the data to the client in time when the data changes, thereby improving the efficiency of data synchronization.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:
fig. 1 is a schematic architecture diagram of a data synchronization system according to an embodiment of the present invention.
Fig. 2 is a schematic structural diagram of a data synchronization system according to another embodiment of the present invention.
Fig. 3 is a flowchart illustrating a data near real-time synchronization method according to an embodiment of the present invention.
Fig. 4 is a schematic flow chart of a data near real-time synchronization method according to another embodiment of the present invention.
Fig. 5 is a schematic structural diagram of a data near real-time synchronization apparatus according to an embodiment of the present invention.
Fig. 6 is a schematic structural diagram of a data near real-time synchronization apparatus according to another embodiment of the present invention.
Fig. 7 is a schematic physical structure diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention. It should be noted that the embodiments and features of the embodiments in the present application may be arbitrarily combined with each other without conflict.
In order to facilitate understanding of the technical solutions provided in the present application, the following first describes relevant contents of the technical solutions in the present application.
Fig. 1 is a schematic architecture diagram of a data synchronization system according to an embodiment of the present invention, and as shown in fig. 1, the data synchronization system according to the embodiment of the present invention includes a service terminal 101, a database server 102, a data exchange platform 103, and a plurality of clients 104, where:
the service terminal 101 is respectively connected with the database server 102 and the data exchange platform 103 in a communication way, and the data exchange platform 103 is connected with each client 104 in a communication way.
The data generated by the service terminal 101 is sent to the database server 102 to be stored in the database, and simultaneously sent to the data exchange platform 103, and then sent to the required client 104 by the data exchange platform 103.
In the data synchronization system, a lot of codes for sending messages need to be maintained, and adding or updating messages brings about a lot of maintenance cost.
Fig. 2 is a schematic architecture diagram of a data synchronization system according to another embodiment of the present invention, and as shown in fig. 2, the data synchronization system according to the embodiment of the present invention includes a service terminal 201, a database server 202, a data exchange platform 203, and a plurality of clients 204, where:
the service terminal 201 is connected with the database server 202 in a communication mode, the database server 202 is connected with the data exchange platform 203 in a communication mode, and the data exchange platform 203 is connected with each client 204 in a communication mode.
Data generated by the service terminal 201 is sent to the database server 202 and stored in the database, the database log, such as binlog, can be monitored by the data exchange platform 203, changed data is found through the database log, and the database server 202 sends the changed data to the data exchange platform 203 and then to the required client 204. The data exchange platform 203 may adopt a Kafka server cluster, and the monitoring of the database log may be realized by using synchronization tools such as Canel, Databus, Puma and the like.
The data synchronization system provided by the embodiment of the invention directly accesses the database server into the data synchronization system, monitors the data change through the data exchange platform, avoids that each client monitors the data change of the database in the database server when synchronizing the data, and reduces the pressure of the database server.
In order to enable the client 204 to obtain the changed data from the data exchange platform 203, the embodiment of the present invention provides a data quasi-real-time synchronization method, which achieves data synchronization through a data synchronization configuration file, can synchronize the changed data in time, and improves the efficiency of data synchronization. In order to clearly illustrate the technical solution of the embodiment of the present invention, a data exchange platform is used as an execution subject to illustrate the implementation process of the data quasi-real-time synchronization method provided by the embodiment of the present invention. It can be understood that the data near real-time synchronization method provided by the embodiment of the invention includes, but is not limited to, a data exchange platform.
Fig. 3 is a schematic flow chart of a data near-real-time synchronization method according to an embodiment of the present invention, and as shown in fig. 3, the data near-real-time synchronization method according to the embodiment of the present invention includes:
s301, receiving a data subscription request sent by a client;
specifically, a client sends a data subscription request to a data exchange platform, and the data exchange platform receives the data subscription request. Wherein the data subscription request is used for informing the data exchange platform that the client needs synchronized data.
S302, generating a data synchronization configuration file according to the data subscription request and determining a synchronization data table;
specifically, after receiving the data subscription request, the data exchange platform generates a data synchronization configuration file corresponding to the client according to the data subscription request, and determines a synchronization data table, where the synchronization data table is used to determine data that the client needs to synchronize.
When the data synchronization configuration file is generated, message types (Topic) can be determined according to the classification of the synchronous data, and a URI list is set for each message type, wherein the URI list comprises micro-service RestAPI which needs to be called when the data is synchronized.
The data pipelining type can comprise adding, modifying and deleting and is used for indicating the operation type of the classification of the synchronous data, the adding indicates that a new synchronous data table is established, and the new synchronous data table comprises the classification of the synchronous data and the content of the synchronous data; the modification indicates that the synchronous data table is modified, and the classification of the synchronous data in the data subscription request and the content of the synchronous data are added in the synchronous data table; deleting the classification of the synchronous data and the content of the synchronous data included in the data subscription request in the synchronous data table. The corresponding relation between the type of the synchronous data and the type of the message is preset. The URI list corresponding to each message category is preset. Each message category corresponds to a synchronization data table.
S303, determining a database table to be synchronized according to the database log;
specifically, the database server sends the database log to the data exchange platform, the data exchange platform receives the database log, and determines a database table with changed data according to the database log, wherein the database table with changed data is a database table to be synchronized.
For example, the database log employs a binlog, which is a binary log for recording database data changes. When data in the database of the database server is changed, the changed database table and the changed content are recorded into the binlog log, the database server sends the binlog log to the data exchange platform, the data exchange platform can determine which data in which database tables are changed by analyzing the binlog log, and the database table with the changed data is the database table to be synchronized.
S304, if the database table to be synchronized is judged to comprise the synchronous data table, sending the data to be synchronized of the synchronous data table comprised by the database table to be synchronized to the client according to the data synchronization configuration file.
Specifically, after obtaining the database tables to be synchronized, the data exchange platform compares the synchronization database tables with the database tables to be synchronized one by one, and if the synchronization database tables are included in the database tables to be synchronized, it indicates that data synchronization needs to be performed on the synchronization database tables. And the data exchange platform sends the data to be synchronized of the synchronous data table included in the database table to be synchronized to the client according to the data synchronization configuration file.
For example, the data synchronization configuration file includes a message category and a URI list corresponding to the message category. The data exchange platform determines the data to be synchronized of the synchronous data table according to the change data of the synchronous data table included by the database table to be synchronized, determines a URI list required to be called by the data to be synchronized according to the configuration information including the message type, the URI list corresponding to the message type and the message type of the data to be synchronized, and sends the data to be synchronized to the client by calling the URI list to call the corresponding RestAPI.
The data quasi-real-time synchronization method provided by the embodiment of the invention can receive a data subscription request sent by a client, generate a data synchronization configuration file and determine a synchronization data table according to the data subscription request, determine a database table to be synchronized according to a database log, send data to be synchronized of the synchronization database table included by the database table to be synchronized to the client according to the data synchronization configuration file after judging that the database table to be synchronized includes the synchronization database table, complete data synchronization through the data synchronization configuration file, and can synchronize the data to the client in time when the data changes, thereby improving the efficiency of data synchronization. In addition, because the code for realizing data synchronization does not need to be repeatedly written for each program in the development of the client application program, the written code amount is reduced, and the development efficiency of the client application program can be improved.
Fig. 4 is a schematic flow chart of a data near-real-time synchronization method according to another embodiment of the present invention, and as shown in fig. 4, on the basis of the foregoing embodiments, further, the data near-real-time synchronization method according to the embodiment of the present invention further includes:
s401, receiving an offline data file at regular time and updating a local database table;
specifically, the database server may send an offline data file to the data exchange platform at regular time, and the data exchange platform may update the local database table according to the received offline data file. The timing is set according to actual needs, and the embodiment of the present invention is not limited.
For example, the database server regularly pushes an offline data file to the data exchange platform every day, the offline data file is a data file needing synchronization, the offline data file can be stored in MySQ L and hive.hive tables of the data exchange platform in a mode of configuring a timing task through a shell script to be divided into ods/mid/ads three-layer storage data, dimension tables and fact tables of ads layers in Hive are loaded through Kylin, a model and a corresponding cube are established, precomputation is performed based on MR, and a result is stored in a distributed database Hbase for subsequent query.
And S402, if the updated local database table is judged to comprise the synchronous database table, sending the offline synchronous data of the synchronous database table comprised by the updated local database table to the client according to the data synchronization configuration file.
Specifically, after the local database table is updated, the data exchange platform compares the updated local database table with the synchronization data table one by one, and if the synchronization data table is included in the updated local database table, it indicates that data synchronization needs to be performed on the synchronization data table. And the data exchange platform sends the off-line synchronous data of the synchronous database table included in the updated local database table to the client according to the data synchronous configuration file. The specific process of sending the offline synchronization data to the client is similar to the process of sending the data to be synchronized in step S304, and is not described herein again.
On the basis of the foregoing embodiments, further, the data subscription request includes a data pipeline type, a classification of synchronous data, and a content of synchronous data; wherein the classification of the synchronization data corresponds to the content of the synchronization data.
Specifically, the data subscription request includes a data pipeline type, a classification of the synchronization data and a content of the synchronization data, and the classification of the synchronization data corresponds to the content of the synchronization data. Wherein the data pipeline type may include add, modify, and delete, for indicating operations on the synchronized database table. The classification of the synchronous data may include a first-level classification and a second-level classification, the second-level classification is a further classification based on the first-level classification, and the specific type of the classification of the synchronous data is set according to actual needs. The content of the synchronization data is set according to actual needs, and the embodiment of the invention is not limited.
For example, as shown in table 1, the data stream type includes addition, modification, and deletion, the newly added attribute value is a, the modified attribute value is U, the deleted attribute value is d, the attribute key of the data stream type is denoted by operType, the classification of the synchronized data includes a first class and a second class, the first class includes requirement class data and item class data, the attribute value of the requirement class data is 01, the attribute value of the item class data is 02, the attribute key of the first class adopts dataType L1, the second class corresponding to the requirement class data includes a requirement item, the attribute value of the requirement item is 0101, the second class corresponding to the item class data may include item basic information, item related departments, item targets, item personnel, item plans and item workloads, the attribute key of the second class adopts dataType L2, the synchronized data content is denoted by the field names and the field names, for example, the field names of the synchronized data content corresponding to the requirement item of the second class may include collaborative identifier, approval dates, service component codes, and the like.
TABLE 1 data Structure definition of data subscription request
On the basis of the foregoing embodiments, further, the determining a synchronous data table according to the data subscription request includes:
and performing corresponding operation on the synchronous data table according to the data pipeline type, the classification of the synchronous data and the content of the synchronous data.
Specifically, the data pipeline types may include addition, modification, and deletion. And if the data pipeline type is newly added, the data exchange platform creates a synchronous data table according to the classification of the synchronous data and the content of the synchronous data. If the data pipeline type is modified, the data exchange platform adds the classification of the synchronous data and the content of the synchronous data in an established synchronous data table according to the classification of the synchronous data and the content of the synchronous data. If the data pipeline type is deletion, the data exchange platform deletes the classification of the synchronous data and the content of the synchronous data in the established synchronous data table.
The following specific example illustrates an implementation process of the data near real-time synchronization method provided by the embodiment of the present invention.
The data exchange platform determines the message type according to the basic information of the item, and sets a URI list of the message type to generate a data synchronization configuration file, wherein the URI list corresponding to the message type and the message type is the configuration information of the data synchronization configuration file, the basic information table of the item corresponds to a topic, the name of the topic is named by a primary classification, a secondary classification, a table name and a URI, and is named as the requirement of the item type, the basic information of the item is 001.URI, wherein the table name is set according to actual needs, the embodiment of the invention is not limited, the basic information table of the item and the data stream water type correspond to each other, the newly added attribute value is A, the corresponding attribute value of the basic information table of the item and the basic information table of the item can be set in the basic information table of the mybatis configuration file, and the corresponding operation identifier of the basic information table of the item is set according to the basic information of the basic information table of the mybatis, such as the attribute identifier of the item, the basic information table of the item, the synchronization data classification and the corresponding item identifier of the basic information table of the item can be set according to the basic information of the mybatis, such as the synchronization information of the basic information corresponding to the basic information of the item, the corresponding item, the synchronization information of the basic information of the corresponding item, such as the attribute of the basic information of the item, and the corresponding item identifier of the item, such as the item, the attribute of the item, the.
The data exchange platform binds a database server and a corresponding topic according to the topic in the topic name file, a client also binds the topic with the corresponding topic, and the binding of the topic can be realized through a configuration file app-consumer-ms, according to the provided default data synchronization logic, based on the data stream type and the identification corresponding to the project basic information table in the mybatis configuration file, the corresponding database is called to access an SQ L statement, and the corresponding data synchronization realized by SQ L is executed, for example, for the new adding operation of the project basic information table, the database corresponding to the mybatis configuration file identification 0201_ A is called to access an SQ L statement.
When the data exchange platform judges that the database table to be synchronized comprises the item basic information table, the data to be synchronized of the item basic information table is delivered to the message queue of the message category of the item basic information according to the secondary classification item basic information, and the data to be synchronized is called in a RestAPI mode according to the URI list corresponding to the message category and is sent to the client side sending the data subscription request.
Fig. 5 is a schematic structural diagram of a data near-real-time synchronization apparatus according to an embodiment of the present invention, and as shown in fig. 5, on the basis of the foregoing embodiments, further, the data near-real-time synchronization apparatus according to the embodiment of the present invention includes a receiving unit 501, a generating unit 502, a determining unit 503, and a synchronizing unit 504, where:
the receiving unit 501 is configured to receive a data subscription request sent by a client; the generating unit 502 is configured to generate a data synchronization configuration file according to the data subscription request and determine a synchronization data table; the determining unit 503 is configured to determine a database table to be synchronized according to the database log; the synchronization unit 504 is configured to send the to-be-synchronized data of the synchronization database table included in the to-be-synchronized database table to the client according to the data synchronization configuration file after it is determined that the to-be-synchronized database table includes the synchronization database table.
Specifically, the client sends a data subscription request to the receiving unit 501, and the receiving unit 501 receives the data subscription request. Wherein the data subscription request is used for informing the data exchange platform that the client needs synchronized data.
After receiving the data subscription request, the generating unit 502 generates a data synchronization configuration file corresponding to the client according to the data subscription request, and determines a synchronization data table, where the synchronization data table is used to determine data that the client needs to synchronize.
The database server sends the database log to the data exchange platform, the determining unit 503 receives the database log, and determines the database table with changed data according to the database log, where the database table with changed data is the database table to be synchronized.
After obtaining the database tables to be synchronized, the synchronization unit 504 compares the synchronization database tables with the database tables to be synchronized one by one, and if the synchronization database tables are included in the database tables to be synchronized, it indicates that data synchronization needs to be performed on the synchronization database tables. And the data exchange platform sends the data to be synchronized of the synchronous data table included in the database table to be synchronized to the client according to the data synchronization configuration file.
The data quasi-real-time synchronization device provided by the embodiment of the invention can receive a data subscription request sent by a client, generate a data synchronization configuration file and determine a synchronization data table according to the data subscription request, determine a database table to be synchronized according to a database log, send data to be synchronized of the synchronization database table included by the database table to be synchronized to the client according to the data synchronization configuration file after judging that the database table to be synchronized includes the synchronization database table, complete data synchronization through the data synchronization configuration file, and can synchronize the data to the client in time when the data changes, thereby improving the efficiency of data synchronization. In addition, because the code for realizing data synchronization does not need to be repeatedly written for each program in the development of the client application program, the written code amount is reduced, and the development efficiency of the client application program can be improved.
Fig. 6 is a schematic structural diagram of a data near real-time synchronization apparatus according to another embodiment of the present invention, and as shown in fig. 6, on the basis of the foregoing embodiments, further, the data near real-time synchronization apparatus according to the embodiment of the present invention further includes an updating unit 505 and an offline synchronization unit 506, where:
the updating unit 505 is configured to receive the offline data file at regular time and update the local database table; the offline synchronization unit 506 is configured to, after it is determined that the updated local database table includes the synchronization database table, send offline synchronization data of the synchronization database table included in the updated local database table to the client according to the data synchronization configuration file.
Specifically, the database server may send an offline data file to the data exchange platform at regular time, and the updating unit 505 may update the local database table according to the received offline data file. The timing is set according to actual needs, and the embodiment of the present invention is not limited.
After updating the local database table, the offline synchronization unit 506 may compare the updated local database table with the synchronization data table one by one, and if the synchronization data table is included in the updated local database table, it indicates that data synchronization needs to be performed on the synchronization data table. The offline synchronization unit 506 sends the offline synchronization data of the synchronization database table included in the updated local database table to the client according to the data synchronization configuration file.
On the basis of the foregoing embodiments, further, the data subscription request includes a data pipeline type, a classification of synchronous data, and a content of synchronous data; wherein the classification of the synchronization data corresponds to the content of the synchronization data.
Specifically, the data subscription request includes a data pipeline type, a classification of the synchronization data and a content of the synchronization data, and the classification of the synchronization data corresponds to the content of the synchronization data. Wherein the data pipeline type may include add, modify, and delete, for indicating operations on the synchronized database table. The classification of the synchronous data may include a first-level classification and a second-level classification, the second-level classification is a further classification based on the first-level classification, and the specific type of the classification of the synchronous data is set according to actual needs. The content of the synchronization data is set according to actual needs, and the embodiment of the invention is not limited.
On the basis of the foregoing embodiments, further, the generating unit 502 is specifically configured to:
and performing corresponding operation on the synchronous data table according to the data pipeline type, the classification of the synchronous data and the content of the synchronous data.
Specifically, the data pipeline types may include addition, modification, and deletion. And if the data pipeline type is newly added, the data exchange platform creates a synchronous data table according to the classification of the synchronous data and the content of the synchronous data. If the data pipeline type is modified, the data exchange platform adds the classification of the synchronous data and the content of the synchronous data in an established synchronous data table according to the classification of the synchronous data and the content of the synchronous data. If the data pipeline type is deletion, the data exchange platform deletes the classification of the synchronous data and the content of the synchronous data in the established synchronous data table.
The embodiment of the apparatus provided in the embodiment of the present invention may be specifically configured to execute the processing flows of the above method embodiments, and the functions of the apparatus are not described herein again, and refer to the detailed description of the above method embodiments.
Fig. 7 is a schematic physical structure diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 7, the electronic device may include: a processor (processor)701, a communication Interface (Communications Interface)702, a memory (memory)703 and a communication bus 704, wherein the processor 701, the communication Interface 702 and the memory 703 complete communication with each other through the communication bus 704. The processor 701 may call logic instructions in the memory 703 to perform the following method: receiving a data subscription request sent by a client; generating a data synchronization configuration file according to the data subscription request and determining a synchronization data table; determining a database table to be synchronized according to the database log; and if the database table to be synchronized is judged to comprise the synchronous database table, transmitting the data to be synchronized of the synchronous database table comprised by the database table to be synchronized to the client according to the data synchronization configuration file.
In addition, the logic instructions in the memory 703 can be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The present embodiment discloses a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the method provided by the above-mentioned method embodiments, for example, comprising: receiving a data subscription request sent by a client; generating a data synchronization configuration file according to the data subscription request and determining a synchronization data table; determining a database table to be synchronized according to the database log; and if the database table to be synchronized is judged to comprise the synchronous database table, transmitting the data to be synchronized of the synchronous database table comprised by the database table to be synchronized to the client according to the data synchronization configuration file.
The present embodiment provides a computer-readable storage medium, which stores a computer program, where the computer program causes the computer to execute the method provided by the above method embodiments, for example, the method includes: receiving a data subscription request sent by a client; generating a data synchronization configuration file according to the data subscription request and determining a synchronization data table; determining a database table to be synchronized according to the database log; and if the database table to be synchronized is judged to comprise the synchronous database table, transmitting the data to be synchronized of the synchronous database table comprised by the database table to be synchronized to the client according to the data synchronization configuration file.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In the description herein, reference to the description of the terms "one embodiment," "a particular embodiment," "some embodiments," "for example," "an example," "a particular example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1.A data quasi-real-time synchronization method is characterized by comprising the following steps:
receiving a data subscription request sent by a client;
generating a data synchronization configuration file according to the data subscription request and determining a synchronization data table;
determining a database table to be synchronized according to the database log;
and if the database table to be synchronized is judged to comprise the synchronous database table, transmitting the data to be synchronized of the synchronous database table comprised by the database table to be synchronized to the client according to the data synchronization configuration file.
2. The method of claim 1, further comprising:
receiving an offline data file at regular time and updating a local database table;
and if the updated local database table is judged to comprise the synchronous database table, sending the offline synchronous data of the synchronous database table comprised by the updated local database table to the client according to the data synchronization configuration file.
3. The method of claim 1 or 2, wherein the data subscription request comprises a data pipeline type, a classification of synchronization data, and a synchronization data content; wherein the classification of the synchronization data corresponds to the content of the synchronization data.
4. The method of claim 3, wherein the determining a synchronized data table according to the data subscription request comprises:
and performing corresponding operation on the synchronous data table according to the data pipeline type, the classification of the synchronous data and the content of the synchronous data.
5. A data near real-time synchronization apparatus, comprising:
the receiving unit is used for receiving a data subscription request sent by a client;
the generating unit is used for generating a data synchronization configuration file according to the data subscription request and determining a synchronization data table;
the determining unit is used for determining a database table to be synchronized according to the database log;
and the synchronization unit is used for sending the data to be synchronized of the synchronous data table included in the database table to be synchronized to the client according to the data synchronization configuration file after judging that the database table to be synchronized includes the synchronous database table.
6. The apparatus of claim 5, further comprising:
the updating unit is used for receiving the offline data file at regular time and updating the local database table;
and the offline synchronization unit is used for sending the offline synchronization data of the synchronization database table included in the updated local database table to the client according to the data synchronization configuration file after judging that the updated local database table includes the synchronization database table.
7. The apparatus of claim 5 or 6, wherein the data subscription request comprises a data pipeline type, a classification of synchronization data, and a synchronization data content; wherein the classification of the synchronization data corresponds to the content of the synchronization data.
8. The apparatus according to claim 7, wherein the generating unit is specifically configured to:
and performing corresponding operation on the synchronous data table according to the data pipeline type, the classification of the synchronous data and the content of the synchronous data.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 4 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010263463.2A CN111460038A (en) | 2020-04-07 | 2020-04-07 | Quasi-real-time data synchronization method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010263463.2A CN111460038A (en) | 2020-04-07 | 2020-04-07 | Quasi-real-time data synchronization method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111460038A true CN111460038A (en) | 2020-07-28 |
Family
ID=71683643
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010263463.2A Pending CN111460038A (en) | 2020-04-07 | 2020-04-07 | Quasi-real-time data synchronization method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111460038A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112507020A (en) * | 2020-11-20 | 2021-03-16 | 平安普惠企业管理有限公司 | Data synchronization method and device, computer equipment and storage medium |
| CN113392161A (en) * | 2021-07-08 | 2021-09-14 | 苏州海管家物流科技有限公司 | Data bidirectional real-time synchronization system |
| CN115328974A (en) * | 2022-10-12 | 2022-11-11 | 南斗六星系统集成有限公司 | Data real-time detection method, device, equipment and readable storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104184822A (en) * | 2014-08-29 | 2014-12-03 | 海视云（北京）科技有限公司 | Data synchronism method and system |
| CN108536743A (en) * | 2018-03-09 | 2018-09-14 | 中国电子科技集团公司第二十八研究所 | A kind of Combat Command System database in phase system |
| CN108628881A (en) * | 2017-03-20 | 2018-10-09 | 北京京东尚科信息技术有限公司 | Method of data synchronization and device |
| CN108763387A (en) * | 2018-05-20 | 2018-11-06 | 湖北九州云仓科技发展有限公司 | Big data fusion method, electronic equipment, storage medium and the system of heterogeneous platform |
| CN108920670A (en) * | 2018-07-06 | 2018-11-30 | 深圳市小牛在线互联网信息咨询有限公司 | Cache synchronization method, device, system and storage medium |
| CN110321387A (en) * | 2019-07-10 | 2019-10-11 | 中国联合网络通信集团有限公司 | Method of data synchronization, equipment and terminal device |
| CN110674213A (en) * | 2019-08-30 | 2020-01-10 | 中国人民财产保险股份有限公司 | Data synchronization method and device |
-
2020
- 2020-04-07 CN CN202010263463.2A patent/CN111460038A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104184822A (en) * | 2014-08-29 | 2014-12-03 | 海视云（北京）科技有限公司 | Data synchronism method and system |
| CN108628881A (en) * | 2017-03-20 | 2018-10-09 | 北京京东尚科信息技术有限公司 | Method of data synchronization and device |
| CN108536743A (en) * | 2018-03-09 | 2018-09-14 | 中国电子科技集团公司第二十八研究所 | A kind of Combat Command System database in phase system |
| CN108763387A (en) * | 2018-05-20 | 2018-11-06 | 湖北九州云仓科技发展有限公司 | Big data fusion method, electronic equipment, storage medium and the system of heterogeneous platform |
| CN108920670A (en) * | 2018-07-06 | 2018-11-30 | 深圳市小牛在线互联网信息咨询有限公司 | Cache synchronization method, device, system and storage medium |
| CN110321387A (en) * | 2019-07-10 | 2019-10-11 | 中国联合网络通信集团有限公司 | Method of data synchronization, equipment and terminal device |
| CN110674213A (en) * | 2019-08-30 | 2020-01-10 | 中国人民财产保险股份有限公司 | Data synchronization method and device |
Non-Patent Citations (1)
| Title |
|---|
| 杨欧等: "《基于物联网平台的智能服务机器人设计》", 西安交通大学出版社, pages: 282 - 285 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112507020A (en) * | 2020-11-20 | 2021-03-16 | 平安普惠企业管理有限公司 | Data synchronization method and device, computer equipment and storage medium |
| CN113392161A (en) * | 2021-07-08 | 2021-09-14 | 苏州海管家物流科技有限公司 | Data bidirectional real-time synchronization system |
| CN115328974A (en) * | 2022-10-12 | 2022-11-11 | 南斗六星系统集成有限公司 | Data real-time detection method, device, equipment and readable storage medium |
| CN115328974B (en) * | 2022-10-12 | 2022-12-13 | 南斗六星系统集成有限公司 | Data real-time detection method, device, equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108874558B (en) | Message subscription method of distributed transaction, electronic device and readable storage medium | |
| CN102682052B (en) | The data query that filtering data stores | |
| CN111460038A (en) | Quasi-real-time data synchronization method and device | |
| KR101919180B1 (en) | Synchronizing conversation structures in web-based email systems | |
| AU2015204742B2 (en) | Methods for generating an activity stream | |
| MX2008012378A (en) | Policy based message aggregation framework. | |
| US8001150B2 (en) | Device management method using nodes having additional attribute and device management client thereof | |
| KR101545626B1 (en) | System for interoperation between dds and dbms | |
| CN102769640B (en) | The update method of user profile, server and system | |
| EP2203840A2 (en) | Exchange of syncronization data and metadata | |
| JP2015522879A (en) | Service device, method and storage medium for providing offline message | |
| CN112597249A (en) | Synchronous distribution and storage method and system for service data | |
| CN110071968B (en) | Block chain based message storage method and device | |
| CN113434312A (en) | Data blood relationship processing method and device | |
| CN115757642A (en) | Data synchronization method and device based on filing log file | |
| JP2006146615A (en) | Object-related information management program, management method and management apparatus | |
| US20020107921A1 (en) | Work-flow cooperation processing apparatus, work-flow cooperation processing system, work-flow-system cooperation method, program therefor, and recording medium therefor | |
| CN114064678A (en) | Event data processing method and device and terminal equipment | |
| CN113076380A (en) | Data synchronization method, device, system, equipment and storage medium | |
| CN110716981A (en) | Method and device for realizing data order synchronization | |
| CN117472617B (en) | Message processing method and device, electronic equipment and readable storage medium | |
| CN115002137B (en) | Offline message processing method and device, computer equipment and storage medium | |
| US20180007130A1 (en) | Peer-to-Peer Assisted Personal Synchronization | |
| CN110837453B (en) | Method and related device for monitoring document exchange platform | |
| CN113672776B (en) | Fault analysis method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220927 Address after: 25 Financial Street, Xicheng District, Beijing 100033 Applicant after: CHINA CONSTRUCTION BANK Corp. Address before: 25 Financial Street, Xicheng District, Beijing 100033 Applicant before: CHINA CONSTRUCTION BANK Corp. Applicant before: Jianxin Financial Science and Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right |