CN114003659A

CN114003659A - Data synchronization method, data synchronization device, electronic equipment, storage medium and program product

Info

Publication number: CN114003659A
Application number: CN202111304930.2A
Authority: CN
Inventors: 梁康奇; 莫奇军; 郑文湛; 彭楠
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2021-11-05
Filing date: 2021-11-05
Publication date: 2022-02-01

Abstract

The present disclosure provides a data synchronization method, apparatus, electronic device, storage medium, and program product, which can be applied to the field of computer technology and the field of finance. The data synchronization method comprises the following steps: generating a task to be synchronized aiming at a first database table under the condition that the incremental first database table exists in an upstream main data source, wherein the task to be synchronized comprises a synchronization task table which at least comprises a table name of the first database table, a table name of a second database table corresponding to the first database table in a downstream auxiliary data source, a synchronization serial number set for the first database table and the second database table and data content to be synchronized; and sending the synchronous task table to the downstream auxiliary data source by using an online interface between the upstream main data source and the downstream auxiliary data source so as to update the second database table by the downstream auxiliary data source.

Description

Data synchronization method, data synchronization device, electronic equipment, storage medium and program product

Technical Field

The present disclosure relates to the field of computer technology and financial technology, and in particular, to a data synchronization method, apparatus, electronic device, storage medium, and program product.

Background

With the development of business, more and more systems are provided in enterprises, and synchronous sharing of data among the systems cannot be avoided. The system responsible for producing and maintaining the data acts as a primary data source, and the application receiving the primary data source acts as a secondary data source. The synchronization of data from the primary data source to the secondary data source may also be referred to as upstream and downstream data synchronization.

The table data of the upstream and downstream application systems are synchronized, generally through an online interface for the case with high timeliness requirement, and generally through end-of-day batch file synchronization for the case with low timeliness requirement. The common practice in the related art is that each time a table is added, a dedicated online interface or file needs to be added, and the development cost is high. When the table structure changes, the upstream and downstream application systems need to be modified synchronously, and the communication and maintenance costs are high.

Disclosure of Invention

To solve some or all of the above technical problems, the present disclosure provides a data synchronization method, apparatus, device, medium, and program product that reduce costs.

According to a first aspect of the present disclosure, there is provided a data synchronization method, including:

generating a task to be synchronized aiming at a first database table under the condition that the incremental first database table exists in an upstream main data source, wherein the task to be synchronized comprises a synchronization task table, and the synchronization task table at least comprises a table name of the first database table, a table name of a second database table corresponding to the first database table in a downstream auxiliary data source, a synchronization serial number set for the first database table and the second database table and data content to be synchronized; and

and sending the synchronous task table to the downstream auxiliary data source by using an online interface between the upstream main data source and the downstream auxiliary data source so that the downstream auxiliary data source updates the second database table.

According to an embodiment of the present disclosure, the method further includes:

acquiring a first record of the first database table, wherein the first record comprises historical updating information and a current updating state of the first database table; and

and updating the first record according to the task to be synchronized.

According to some embodiments of the present disclosure, the obtaining the first record of the first database table includes:

querying whether the first record exists in the upstream primary data source:

if the first record does not exist in the upstream main data source, determining the first database table as a newly added table, and creating the first record according to the synchronous task table;

and if the first record exists in the upstream main data source, determining the first database table as an update table, and extracting the first record of the first database table from the upstream main data source.

According to some embodiments of the disclosure, the current update status comprises one of: a synchronization waiting state, a synchronization success state and a synchronization failure state; the first record further includes a number of update failures.

According to some embodiments of the present disclosure, the updating the first record according to the task to be synchronized includes: and periodically scanning the current updating state of the first record.

If the scanning result is in a state to be synchronized, calling the online interface to send the synchronization task table to the downstream auxiliary data source; if the online interface is successful, updating the state to be synchronized of the first record to a synchronization successful state; and if the calling of the online interface fails, adding one to the update failure frequency, if the update failure frequency is smaller than a preset threshold, keeping the current update state of the first record unchanged, and if the update failure frequency is larger than or equal to the preset threshold, updating the current update state into a synchronous failure state.

If the scanning result is a synchronization success state, the current updating state is kept unchanged, and the first record is not scanned any more.

If the scanning result is a synchronization failure state, the current updating state is kept unchanged, and the first record is not scanned any more.

According to some embodiments of the disclosure, the method further comprises:

and periodically sending all the first database tables with current update states as synchronization failures to the downstream auxiliary data source in a file transmission mode so as to update the second database tables with the current update states as synchronization success states.

According to some embodiments of the present disclosure, the structure of the synchronous task table is the same as that of the second database table, and the structure at least includes: table structure field type, length, order and primary key;

the synchronous task table and the second database table comprise at least two public fields, synchronous sequence numbers and field states, and the synchronous sequence numbers comprise time stamps and sequence numbers.

According to some embodiments of the disclosure, the updating the second database table by the downstream secondary data source comprises:

generating a query structured query language according to the table name of the first database table, the table name of the second database table, the primary key and the content of the data to be synchronized; and

querying a second record in the downstream auxiliary data source according to the query structured query language, wherein the second record comprises a history update record of a second database table, and the history update record comprises a history synchronization serial number;

if the second record has no history updating record, generating an insertion structured query language to execute the updating operation of the second database table, and updating the first record according to the execution result.

According to some embodiments of the present disclosure, the sending the synchronization task table to the downstream auxiliary data source includes:

and assembling the table name, the synchronization sequence number and the field content of the synchronization task table into a character string according to the field sequence, and sending the character string to a downstream online interface.

A second aspect of the present disclosure provides a data synchronization apparatus, including: a synchronization task generating module, configured to generate a to-be-synchronized task for a first database table in an upstream main data source when the first database table is incremental, where the to-be-synchronized task includes a synchronization task table, and the synchronization task table at least includes a table name of the first database table, a table name of a second database table corresponding to the first database table in a downstream auxiliary data source, a synchronization sequence number set for the first database table and the second database table, and to-be-synchronized data content;

a flow module, configured to send the synchronous task table to the downstream auxiliary data source by using an online interface between the upstream main data source and the downstream auxiliary data source; and

and the updating module is used for updating the second database table according to the synchronous task table.

According to some embodiments of the present disclosure, the apparatus further comprises a monitoring module, configured to periodically scan a current update status of the first record, where the current update status of the first record includes one of: a synchronization waiting state, a synchronization success state and a synchronization failure state; the first record also includes a number of update failures.

A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the data synchronization method.

The fourth aspect of the present disclosure also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the above-described data synchronization method.

A fifth aspect of the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the above-described data synchronization method.

Drawings

The foregoing and other objects, features and advantages of the disclosure will be apparent from the following description of embodiments of the disclosure, which proceeds with reference to the accompanying drawings, in which:

FIG. 1 schematically illustrates an application scenario diagram of a data synchronization method, apparatus, device, medium, and program product according to embodiments of the disclosure;

FIG. 2 schematically illustrates a flow chart of a method of data synchronization according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow diagram of updating a first record of a data synchronization method according to an embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow chart of a first record state modification of a data synchronization method according to an embodiment of the present disclosure;

FIG. 5 is a flow diagram schematically illustrating updating of a second database table by a downstream secondary data source of a data synchronization method according to an embodiment of the present disclosure;

fig. 6 schematically shows a block diagram of a data synchronization apparatus according to an embodiment of the present disclosure; and

fig. 7 schematically shows a block diagram of an electronic device adapted to implement the data synchronization method according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.

Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

It should be noted that the method and apparatus for data synchronization disclosed in the present disclosure may be used for data synchronization of a general table increment in the financial field, and may also be used in any field other than the financial field.

In the technical scheme of the disclosure, the acquisition, storage, application and the like of the personal information of the related user all accord with the regulations of related laws and regulations, necessary security measures are taken, and the customs of the public order is not violated.

An embodiment of the present disclosure provides a data synchronization method, including: generating a task to be synchronized aiming at a first database table under the condition that the incremental first database table exists in an upstream main data source, wherein the task to be synchronized comprises a synchronization task table which at least comprises a table name of the first database table, a table name of a second database table corresponding to the first database table in a downstream auxiliary data source, a synchronization serial number set for the first database table and the second database table and data content to be synchronized; and sending the synchronous task table to the downstream auxiliary data source by using an online interface between the upstream main data source and the downstream auxiliary data source so as to update the second database table by the downstream auxiliary data source.

Fig. 1 schematically shows an application scenario diagram of a data synchronization method, apparatus, electronic device, storage medium, and program product according to embodiments of the disclosure.

As shown in fig. 1, the application scenario 100 according to this embodiment may include updating and data synchronization of an application program of an electronic product. Network 104 is the medium used to provide communication links between

terminal devices

101, 102, 103, downstream server 106, and upstream server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

A user may use

terminal devices

101, 102, 103 and downstream server 106 to interact with upstream server 105 over network 104 to receive or send messages and the like. The

terminal devices

101, 102, 103 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only). Various programs, databases, etc. are stored within the downstream server 106.

The

terminal devices

101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The upstream server 105 may be a server providing various services, such as a background management server (for example only) providing support for websites browsed by users using the

terminal devices

101, 102, 103. The background management server may analyze and perform other processing on the received data such as the user request, and feed back a processing result (e.g., a webpage, information, or data obtained or generated according to the user request) to the terminal device.

It should be noted that the data synchronization method provided by the embodiment of the present disclosure may be generally executed by the server 105. Accordingly, the data synchronization apparatus provided by the embodiments of the present disclosure may be generally disposed in the server 105. The data synchronization method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105. Accordingly, the data device provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the server 105 and capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

The data synchronization method of the disclosed embodiment will be described in detail below with fig. 2 to 5 based on the scenario described in fig. 1.

Fig. 2 schematically shows a flow chart of a data synchronization method according to an embodiment of the present disclosure.

As shown in fig. 2, the data synchronization method of this embodiment includes operations S201 to S203.

In the present embodiment, operation S201 includes: and generating a task to be synchronized aiming at the first database table.

In the present embodiment, operation S202 includes: and sending the synchronous task table to the downstream auxiliary data source by using an online interface between the upstream main data source and the downstream auxiliary data source.

In this embodiment, operation S203 includes: the second database table is updated.

In the embodiment, the synchronous task table is sent to the downstream auxiliary data source through the online interface by using the synchronous serial number and the table name, so that the synchronous update of the upstream data table to the downstream data table is realized, the need of adding a special online interface or a special file every time a table is added is avoided, and the research and development cost and the maintenance cost are reduced.

In this embodiment, an upstream main data source and a downstream auxiliary data source are provided, where the upstream main data source is used for maintaining and updating data, generating a task to be synchronized and sending the task to the downstream auxiliary data source, and the upstream main data source may be provided on the server 105; the downstream auxiliary data source performs update synchronization of data through a synchronization task, and the downstream auxiliary data source may be disposed on the

terminal devices

101, 102, 103.

According to some embodiments of the present disclosure, operation S201 includes: and under the condition that the incremental first database table exists in the upstream main data source, generating a task to be synchronized aiming at the first database table, wherein the task to be synchronized comprises a synchronization task table, and the synchronization task table at least comprises a table name of the first database table, a table name of a second database table corresponding to the first database table in the downstream auxiliary data source, a synchronization serial number set for the first database table and the second database table and data content to be synchronized.

In this embodiment, a first database table may be generated at an upstream primary data source, such as upstream table 1, table 2 … … table N; a second database table is generated at the downstream secondary database, such as downstream table 1 ', table 2' … … table N ', where the table structure requirements for table M and table M' (M ═ 1 … … N) are the same, and the table structure may include: the table structure comprises a field type, a length, an order and a primary key, wherein the first database table and the second database table comprise at least two public fields, a synchronization sequence number and a field state, and the synchronization sequence number comprises a timestamp and a sequence number. Each table to be synchronized may have two common field synchronization sequence numbers and field states, and the synchronization sequence numbers may be concatenated with a timestamp and a sequence number. The attributes of the upstream and downstream table name comparison table are shown in table 1.

TABLE 1

Upstream application name

Upstream table name

Downstream application name

Downstream table name

Wherein the table M and table M' attributes are shown in table 2.

TABLE 2

Synchronization sequence number

Field state

Field 1

Field 2

……

According to some alternative embodiments of the present disclosure, the synchronization sequence number may be generated according to the following rules: the sync number yyymmdd + LPAD (seq. nextval,15, '0'), 8-bit date concatenates the 15-bit sequence number for a total of 23-bit length.

In this embodiment, the first database table and the second database table include at least two common fields, a synchronization sequence number, and a field status, the synchronization sequence number including a timestamp and a sequence number.

In this embodiment, because the structures of the first database table and the second database table can be accurately matched, the plurality of first database tables can be sent to the downstream auxiliary data source for update synchronization by only calling one online interface, so that a special online interface or file is not required to be added every time one table is added, and the research and development cost and the maintenance cost are reduced.

According to some embodiments of the present disclosure, operation S202 includes: and sending the synchronous task table to the downstream auxiliary data source by using an online interface between the upstream main data source and the downstream auxiliary data source so as to update the second database table by the downstream auxiliary data source.

According to some embodiments of the disclosure, step S203 comprises: and updating the second database table according to the synchronization sequence number and the synchronization content of the synchronization task table.

Fig. 3 schematically shows a flow chart of updating a first record of a data synchronization method according to an embodiment of the present disclosure.

As shown in fig. 3, the method of updating the first record includes operations S301 to S302.

In the present embodiment, operation S301 includes: a first record of a first database table is obtained.

In this embodiment, operation S302 includes: and updating the first record according to the task to be synchronized.

According to some embodiments of the present disclosure, the first record includes historical update information and a current update status of the first database table.

In the embodiment, whether the synchronization task is the latest update task is judged according to the historical update information and the current update state of the first record of the first database table and the content of the synchronization task table.

In this embodiment, a first record corresponding to the first database table is queried in the upstream main data source, and if the first record does not exist in the upstream main data source, the first database table is determined to be a newly added table, and the first record is created according to the synchronous task table.

In this embodiment, a first record corresponding to a first database table is queried in the upstream main data source, and if the first record exists in the upstream main data source, the first database table is determined to be an updated table, and the first record of the first database table is extracted from the upstream main data source.

According to some embodiments of the disclosure, the current update status comprises one of: a synchronization waiting state, a synchronization success state and a synchronization failure state; the first record also includes a number of update failures.

According to some alternative embodiments of the present disclosure, the attributes of the first record are as shown in table 3.

TABLE 3

Synchronization sequence number

Table name

Synchronization status

Time of scan

Number of failures

Batch server ID

Fig. 4 schematically shows a flow chart of a first recording status modification of a data synchronization method according to an embodiment of the present disclosure.

As shown in fig. 4, updating the first record according to the task to be synchronized includes operations S401 to S409.

In the present embodiment, operation S401 includes: the current update status of the first record is periodically scanned and operation S402 is entered.

In this embodiment, operation S402 includes: it is determined whether there is a record of "to-be-synchronized state", and if so, operation S403 is entered, and if not, operation S401 is returned to.

In the present embodiment, operation S403 includes: and calling the online interface to send the synchronous task table to the downstream auxiliary data source, and entering operation S404.

In this embodiment, operation S404 includes: and judging whether the online interface is successfully called, if the online interface is successfully called, entering operation S405, and if the online interface is unsuccessfully called, entering operation S406.

In this embodiment, operation S405 includes: and updating the current updating state of the first record to be synchronous success, and meanwhile, not continuously scanning the first record.

In this embodiment, operation S406 includes: the update failure times +1 recorded in the first record are entered into operation S407.

In the present embodiment, operation S407 includes: determining whether the update failure times recorded in a record exceed a preset threshold, if the update failure times exceed the preset threshold, entering operation S408, and if the update failure times do not exceed the preset threshold, entering operation S409

In the present embodiment, operation S408 includes: and updating the current updating state of the first record into synchronization failure, and meanwhile, not continuously scanning the first record.

In the present embodiment, operation S409 includes: the current update status of the first record is kept unchanged, and operation S401 is returned.

In this embodiment, the upstream primary server is periodically scanned, and the scanning continuously updates the application of the downstream secondary data source with one or more first database tables.

According to some embodiments of the disclosure, the data synchronization method further comprises: and periodically sending all the first database tables with current update states as synchronization failures to a downstream auxiliary data source in a file transmission mode so as to update the second database tables with the current update states as synchronization success states.

In this embodiment, the method of combining the online interface update and the daily end batch update is real-time, mainly calling the online interface for update, and secondarily updating the daily end batch, so as to ensure that the upstream and downstream data are always consistent.

FIG. 5 is a flow chart that schematically illustrates the updating of a second database table by a downstream secondary data source in accordance with a data synchronization method in accordance with an embodiment of the present disclosure.

As shown in fig. 5, updating the first record according to the task to be synchronized includes operations S501 to S508.

In the embodiment, a Query Structured Query Language (SQL) is generated according to the table name of the first database table, the table name of the second database table, the primary key and the data content to be synchronized.

In the present embodiment, operation S501 includes: and generating query SQL, querying the second record in the downstream auxiliary data source according to the query SQL, and entering operation S502.

In this embodiment, operation S502 includes: it is determined whether the second record has a history update record, and if not, the operation proceeds to operation S503, and if so, the operation proceeds to operation S504.

In the present embodiment, operation S503 includes: and generating an insert SQL, updating the second database table and entering operation S507.

In this embodiment, operation S504 includes: it is determined whether the synchronization sequence number of the synchronization task table is greater than the synchronization sequence number recorded in the history update record (the latest update), and if so, the operation proceeds to operation S505, and if not, the operation proceeds to operation S506.

In the present embodiment, operation S505 includes: and generating an update SQL and updating the second database table, and entering operation S507.

In the present embodiment, operation S506 includes: and the online interface is successfully called, and the method is ended.

In this embodiment, operation S507 includes: whether the update is successful is determined, if yes, operation S506 is performed, and if not, operation S508 is performed.

In this embodiment, operation S508 includes: and the online interface fails to call, and the method is finished.

In this embodiment, after the online interface is successfully called, the method further includes recording update information, including the synchronization serial number, the synchronization content, and other information of the current update.

In this embodiment, by comparing the synchronization sequence number in the synchronization task table with the history update record of the second record, it can be accurately determined whether the update is required.

According to some embodiments of the present disclosure, sending the synchronization task table to the downstream secondary data source includes: and assembling the table name, the synchronization sequence number and the field content of the synchronization task table into a character string according to the field sequence, and sending the character string to a downstream online interface.

Through the data synchronization method of the present disclosure, at least one of the following advantages is included:

1. the method has configurability, and if the table for incremental data synchronization needs to be added, the configuration (synchronization task table) only needs to be added, and corresponding special online or files do not need to be developed.

2. The method has the advantages that the method is adaptive, when the table structure is changed, the upstream application and the downstream application do not need to synchronously modify related interfaces, and normal synchronization of data can be guaranteed.

3. And the accuracy is high, the batch file interfaces are synchronized again for the record that the online interfaces cannot be synchronized successfully, and the data consistency is ensured.

Based on the data synchronization method, the disclosure also provides a data synchronization device. The apparatus will be described in detail below with reference to fig. 6.

Fig. 6 schematically shows a block diagram of a data synchronization apparatus according to an embodiment of the present disclosure.

As shown in fig. 6, the data synchronization apparatus 600 of this embodiment includes a synchronization task generation module 601, a circulation module 602, and an update module 603.

The synchronization task generating module 601 generates a to-be-synchronized task for a first database table under the condition that an incremental first database table exists in an upstream main data source, wherein the to-be-synchronized task includes a synchronization task table, and the synchronization task table at least includes a table name of the first database table, a table name of a second database table corresponding to the first database table in a downstream auxiliary data source, a synchronization sequence number set for the first database table and the second database table, and to-be-synchronized data content. In an embodiment, the synchronization task generating module 601 may be configured to perform the operation S201 described above, which is not described herein again.

The flow module 602 is configured to send a synchronization task table to the downstream auxiliary data source by using an online interface between the upstream main data source and the downstream auxiliary data source. In an embodiment, the circulation module 602 may be configured to perform the operation S202 described above, which is not described herein again.

The updating module 603 is configured to update the second database table according to the synchronization task table. In an embodiment, the updating module 603 may be configured to perform the operation S302 described above, which is not described herein again.

According to some embodiments of the present disclosure, the sync task generation module 601 can obtain a first record of a first database table, the first record including historical update information and a current update status of the first database table.

According to some embodiments of the present disclosure, the sync task generation module 601 obtaining the first record of the first database table includes querying whether the first record exists in an upstream primary data source: if the first record does not exist in the upstream main data source, determining the first database table as a newly added table, and creating the first record according to the synchronous task table; and if the first record exists in the upstream main data source, determining the first database table as an update table, and extracting the first record of the first database table from the upstream main data source.

According to some embodiments of the present disclosure, the data synchronization apparatus further includes a listening module for periodically scanning a current update status of the first record, the current update status of the first record including one of: a synchronization waiting state, a synchronization success state and a synchronization failure state; the first record also includes a number of update failures.

According to the embodiment of the present disclosure, any multiple modules of the synchronization task generation module 601, the circulation module 602, the update module 603, and the listening module may be combined and implemented in one module, or any one of the modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the synchronization task generation module 601, the circulation module 602, the update module 603, and the listening module may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or implemented by any one of three implementations of software, hardware, and firmware, or any suitable combination of any of them. Alternatively, at least one of the synchronization task generating module 601, the streaming module 602, the updating module 603 and the listening module may be at least partially implemented as a computer program module, which when executed may perform a corresponding function.

As shown in fig. 7, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. The processor 701 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 701 may also include on-board memory for caching purposes. The processor 701 may comprise a single processing unit or a plurality of processing units for performing the different actions of the method flows according to embodiments of the present disclosure.

In the RAM 703, various programs and data necessary for the operation of the electronic apparatus 700 are stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other by a bus 704. The processor 701 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 702 and/or the RAM 703. It is noted that the programs may also be stored in one or more memories other than the ROM 702 and RAM 703. The processor 701 may also perform various operations of method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

Electronic device 700 may also include input/output (I/O) interface 705, which input/output (I/O) interface 705 is also connected to bus 704, according to an embodiment of the present disclosure. The electronic device 700 may also include one or more of the following components connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.

The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 702 and/or the RAM 703 and/or one or more memories other than the ROM 702 and the RAM 703 described above.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method illustrated in the flow chart. When the computer program product runs in a computer system, the program code is used for causing the computer system to realize the item recommendation method provided by the embodiment of the disclosure.

The computer program performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure when executed by the processor 701. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted in the form of a signal on a network medium, distributed, downloaded and installed via the communication section 709, and/or installed from the removable medium 711. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program, when executed by the processor 701, performs the above-described functions defined in the system of the embodiment of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims

1. A method of data synchronization, comprising:

generating a task to be synchronized aiming at a first database table under the condition that the incremental first database table exists in an upstream main data source, wherein the task to be synchronized comprises a synchronization task table which at least comprises a table name of the first database table, a table name of a second database table corresponding to the first database table in a downstream auxiliary data source, a synchronization sequence number set for the first database table and the second database table and data content to be synchronized; and

and sending the synchronous task table to the downstream auxiliary data source by using an online interface between the upstream main data source and the downstream auxiliary data source so that the downstream auxiliary data source can update the second database table.

2. The method of claim 1, further comprising:

and updating the first record according to the task to be synchronized.

3. The method of claim 2, wherein,

the obtaining the first record of the first database table comprises:

querying whether the first record exists in the upstream primary data source:

4. The method of claim 3, wherein,

the current update status comprises one of: a synchronization waiting state, a synchronization success state and a synchronization failure state; the first record further comprises the number of updating failures;

the updating the first record according to the task to be synchronized comprises:

periodically scanning the current update status of the first record:

if the scanning result is in a state to be synchronized, calling the online interface to send the synchronous task table to the downstream auxiliary data source; if the online interface is successful, updating the state to be synchronized of the first record to a synchronization successful state; if the calling of the online interface fails, adding one to the update failure times, if the update failure times are smaller than a preset threshold, keeping the current update state of the first record unchanged, and if the update failure times are larger than or equal to the preset threshold, updating the current update state into a synchronous failure state;

if the scanning result is in a synchronization success state, keeping the current updating state unchanged and not scanning the first record any more; and

and if the scanning result is in a synchronization failure state, keeping the current updating state unchanged and not scanning the first record any more.

5. The method of claim 4, further comprising:

6. The method of claim 4, wherein,

the first database table and the second database table have the same structure, and the structure at least comprises: table structure field type, length, order and primary key;

wherein the first database table and the second database table include at least two common fields: synchronizing sequence numbers and field states; the synchronization sequence number includes a timestamp and a sequence number.

7. The method of claim 6, wherein,

the updating of the second database table by the downstream secondary data source comprises:

generating a query structured query language according to the table name of the first database table, the table name of the second database table, the primary key and the data content to be synchronized; and

querying a second record in the downstream secondary data source according to the query structured query language, the second record comprising a history update record of a second database table, the history update record comprising a history synchronization sequence number, wherein

If no history updating record exists in the second record, generating an insertion structured query language to execute the updating operation of the second database table, and updating the first record according to the execution result;

if the second record has a history updating record, comparing the synchronization sequence number of the synchronization task table with the history synchronization sequence number in the second record:

if the synchronization sequence number of the synchronization task table is larger than the historical synchronization sequence number in the second record, generating an update structured query language to execute the update operation of the second database table;

and if the synchronization serial number of the synchronization task table is smaller than or equal to the historical synchronization serial number in the second record, not executing the updating operation, and obtaining a successful default updating.

8. The method of claim 1, wherein,

the sending the synchronization task table to the downstream secondary data source includes:

and assembling the table name, the synchronous sequence number and the field content of the synchronous task table into a character string according to the field sequence, and sending the character string to a downstream online interface.

9. A data synchronization apparatus, comprising:

the synchronization task generating module is used for generating a task to be synchronized aiming at a first database table under the condition that the incremental first database table exists in an upstream main data source, wherein the task to be synchronized comprises a synchronization task table, and the synchronization task table at least comprises a table name of the first database table, a table name of a second database table corresponding to the first database table in a downstream auxiliary data source, a synchronization sequence number set for the first database table and the second database table and data content to be synchronized;

the flow module is used for sending the synchronous task table to the downstream auxiliary data source by utilizing an online interface between the upstream main data source and the downstream auxiliary data source; and

10. The apparatus of claim 9, further comprising:

the monitoring module is used for periodically scanning the current updating state of the first record, and the current updating state of the first record comprises one of the following: a synchronization waiting state, a synchronization success state and a synchronization failure state; the first record also includes a number of update failures.

11. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-8.

12. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 8.

13. A computer program product comprising a computer program which, when executed by a processor, implements a method according to any one of claims 1 to 8.