CN117951223A

CN117951223A - Data synchronization method, device, computer equipment and storage medium

Info

Publication number: CN117951223A
Application number: CN202410135024.1A
Authority: CN
Inventors: 吴庭栋
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2024-01-31
Filing date: 2024-01-31
Publication date: 2024-04-30

Abstract

The application relates to a data synchronization method, a data synchronization device, computer equipment and a storage medium, and relates to the technical field of financial science and technology. When the business data updating information of the first data is detected, incremental business data of the first storage position is sent to a cache position corresponding to a target identifier in a preset message middleware cluster, after a target data format corresponding to a second financial database and the target identifier are determined to be in a second storage position corresponding to the second financial database, the incremental business data of the cache position is subjected to format conversion, and the converted target incremental business data is synchronized to the second financial database. Compared with the traditional mode of respectively accessing different databases to acquire service data, the method and the system have the advantages that when the incremental data are updated by the databases, the incremental data are cached by the message middleware cluster and are synchronized to the corresponding positions of other databases by combining with the formats of other databases, and the corresponding data can be acquired by the system only by accessing one position, so that the execution efficiency of banking services is improved.

Description

Data synchronization method, device, computer equipment and storage medium

Technical Field

The present application relates to the technical field of financial science and technology, and in particular, to a data synchronization method, apparatus, computer device, storage medium and computer program product.

Background

As banking business progresses and versions of the banking system are updated iteratively, the banking system generates a large amount of business data which are stored in different databases, so that a state of multiple data sources in parallel exists inside the banking system. To ensure timely response of banking business, data integrity and synchronicity between different databases need to be ensured. The current approach to banking with multiple data sources in parallel is to access each data source separately to obtain the required data. However, by acquiring corresponding data from a plurality of data sources, respectively, the execution efficiency of banking business is reduced.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a data synchronization method, apparatus, computer device, computer readable storage medium, and computer program product that can improve the execution efficiency of banking.

In a first aspect, the present application provides a data synchronization method, the method comprising:

Detecting business data updating information of a first financial database, and acquiring incremental business data in the first financial database and a first storage position of the incremental business data in the first financial database according to the business data updating information;

according to the first storage position, the incremental business data are sent to a cache position corresponding to a target identifier in a preset message middleware cluster; the target identifier corresponds to the first storage position;

Acquiring a target data format corresponding to a second financial database, and acquiring a second storage position corresponding to the target identifier in the second financial database;

And carrying out format conversion on the increment business data in the buffer memory position according to the target data format to obtain target increment business data, and synchronizing the target increment business data to a second storage position in the second financial database.

In one embodiment, the step of obtaining the first storage location of the incremental transaction data in the first financial database comprises:

acquiring a first data table field storing the incremental business data in the first financial database;

and obtaining a first storage position of the incremental business data in the first financial database according to the first data table field.

In one embodiment, the preset message middleware cluster is a kaff card cluster; the step of sending the incremental business data to a cache position corresponding to a target identifier in a preset message middleware cluster according to the first storage position comprises the following steps:

Inquiring a preset storage position corresponding table according to the first data table field to obtain a target identifier corresponding to the first data table field in the Kaff card cluster; the preset storage position corresponding table comprises corresponding relations between preset first data table fields and marks in the card cluster;

And sending the incremental service data to a cache position corresponding to the target identifier in the Kaff card cluster.

In one embodiment, the method further comprises:

For each first data table field in the first financial database, inquiring the second financial database according to the first data table field;

if the second data table field which is the same as the first data table field does not exist in the second financial database, generating the second data table field in the second financial database according to the first data table field and the data format of the second financial database;

And obtaining a preset storage position corresponding table according to the first data table field and the second data table field.

In one embodiment, the obtaining the second storage location corresponding to the target identifier in the second financial database includes:

determining a second data table field corresponding to the first data table field in the second financial database according to a preset storage position corresponding table;

and obtaining a second storage position corresponding to the target identifier in the second financial database according to the corresponding relation between the first data table field and the target identifier and the corresponding relation between the first data table field and the second data table field.

In one embodiment, the performing format conversion on the incremental service data in the buffer location according to the target data format to obtain target incremental service data includes:

Acquiring incremental business data in the cache position through a preset data consumption tool to obtain consumption data;

and carrying out format conversion on the consumption data according to the target data format to obtain target incremental business data.

In a second aspect, the present application provides a data synchronization apparatus, the apparatus comprising:

the detection module is used for detecting business data updating information of a first financial database and acquiring incremental business data in the first financial database and a first storage position of the incremental business data in the first financial database according to the business data updating information;

The caching module is used for sending the incremental business data to a caching position corresponding to a target identifier in a preset message middleware cluster according to the first storage position; the target identifier corresponds to the first storage position;

the acquisition module is used for acquiring a target data format corresponding to a second financial database and acquiring a second storage position corresponding to the target identifier in the second financial database;

And the synchronizing module is used for carrying out format conversion on the increment business data in the buffer memory position according to the target data format to obtain target increment business data, and synchronizing the target increment business data to a second storage position in the second financial database.

In a third aspect, the present application provides a computer device comprising a memory storing a computer program and a processor implementing the steps of the method described above when the processor executes the computer program.

In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the method described above.

In a fifth aspect, the application provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method described above.

According to the data synchronization method, the device, the computer equipment, the storage medium and the computer program product, when the business data updating information of the first data is detected, the incremental business data and the first storage position thereof are determined, the incremental business data are sent to the cache position corresponding to the target mark in the preset message middleware cluster, after the target data format and the target mark corresponding to the second financial database are determined to be at the second storage position corresponding to the second financial database, the incremental business data of the cache position are subjected to format conversion according to the target data format, and the converted target incremental business data are synchronized to the second storage position in the second financial database. Compared with the traditional mode of respectively accessing different databases to acquire corresponding business data, the method and the system have the advantages that when one database updates incremental data, the incremental data is cached through the message middleware cluster and is synchronized to the corresponding positions of other databases by combining with the formats of other databases, and the system can acquire corresponding data only by accessing one position, so that the execution efficiency of banking business is improved.

Drawings

FIG. 1 is an application environment diagram of a data synchronization method in one embodiment;

FIG. 2 is a flow chart of a data synchronization method in one embodiment;

FIG. 3 is a flow chart of a data synchronization method according to another embodiment;

FIG. 4 is a block diagram of a data synchronization device in one embodiment;

Fig. 5 is an internal structural diagram of a computer device in one embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

The data synchronization method provided by the embodiment of the application can be applied to an application environment shown in fig. 1. The terminal communicates with the first database, the second database and the preset message middleware through a network respectively, and the first database, the second database and the preset message middleware can communicate with each other. The preset message middleware may be provided in the terminal. The first database and the second database can both store data which needs to be processed by the banking system. The first database and the second database can be integrated on a server, or can be placed on a cloud or other network servers. The terminal can detect business data updating information of the first financial database, and when the business data updating information is detected, the terminal combines with a preset message middleware to perform incremental synchronization of business data for the first financial database and the second financial database. The terminal may be, but not limited to, various personal computers, notebook computers, smart phones and tablet computers. The server may be implemented as a stand-alone server or as a server cluster composed of a plurality of servers.

In one embodiment, as shown in fig. 2, a data synchronization method is provided, and the method is applied to the terminal in fig. 1 for illustration, and includes the following steps:

Step S202, detecting business data updating information of a first financial database, and acquiring incremental business data in the first financial database and a first storage position of the incremental business data in the first financial database according to the business data updating information.

Wherein the first financial database may be a database for storing corresponding business data in the bank. The bank may include multiple businesses, and since the development time of each business is different, the system in the bank has iterative update in different time, so the business data of each business may be stored in the same or different financial databases, i.e. there are multiple financial databases in the bank system. The service data of each service in the bank are related, namely, the service data of the service is needed for developing one service, and the service data of other services can be used. When each business calls and acquires the data in each financial database in the process of executing each business by the bank system, the terminal needs to ensure the timeliness of the call of the business data. So as to improve the execution efficiency of banking business.

The banking business needs to call various business data when executing, and when the business data are stored in different financial databases, the business has lower efficiency in a mode of respectively inquiring different financial databases to acquire different business data, so that the terminal can synchronize the updated data in a certain financial database to other financial databases in an incremental synchronization mode.

The terminal can detect whether the business data of each financial database is updated, and if the terminal detects the business data updating information of the first financial database, the terminal can acquire the business data updating information, determine the incremental business data in the first financial business database according to the business data updating information, and determine the first storage position of the incremental business data in the first financial business database. The incremental business data represents newly added business data in the first financial database, such as business data manually input by a banking staff for banking corresponding to the first financial database, or business data automatically updated in a banking period, and the like. The business data can be stored in a scattered form at different positions in the first financial database, and the terminal combines the different positions to form the first storage position; these business data may also be stored in the first financial database in its entirety at the same location, based on which the terminal determines the first storage location of the incremental business data.

After determining the incremental business data in the first financial data and the first storage position of the incremental business data in the first financial database, the terminal can acquire the incremental business data in the first financial business database and the first storage position of the incremental business data in the first financial database. The terminal can perform incremental synchronization on the incremental business data among a plurality of financial databases by utilizing the incremental business data and the first storage position corresponding to the incremental business data.

Step S204, according to the first storage position, the incremental business data are sent to a cache position corresponding to a target mark in a preset message middleware cluster; the target identifier corresponds to the first storage location.

The first storage location may be a storage location of incremental business data obtained by querying a first financial database by the terminal in the first financial database. After the terminal obtains the incremental service data and the first storage position of the incremental service data, the incremental service data can be sent to a preset message middleware cluster, and the incremental service data is cached to a cache position corresponding to the target identifier in the preset message middleware cluster. The preset message middleware can be a middleware cluster which is arranged in the terminal and used as a transit container of the two databases. The preset middleware cluster may include a plurality of cache locations with different identifiers, where the identifier of each cache location corresponds to each first storage location in the first financial database, that is, different first storage locations in the first financial database may be cached to cache locations corresponding to different identifiers in the preset message middleware. The terminal can construct the corresponding relation between the target identifier and the storage position in advance aiming at the financial database of the generated incremental business data.

For example, the terminal pre-builds the corresponding relation between each first storage position in the first financial database and each target identifier in the preset message middleware. The first storage positions and the target identifiers can be corresponding to each other through numerical values, for example, the numerical values of the identifiers in the preset message middleware are different, and the terminal correlates and corresponds the first storage positions with the same numerical values with the target identifiers; the correspondence may also be performed by a specific correspondence, for example, a correspondence input by the receiving service personnel for each first storage location and each target identifier may be obtained based on the received correspondence. In some embodiments, if the financial database generating the incremental transaction data is a second financial database, the terminal may also construct a corresponding relationship between the target identifier and the storage location for the second financial database in advance.

Step S206, obtaining a target data format corresponding to a second financial database, and obtaining a second storage position corresponding to the target identifier in the second financial database.

The second financial database may be a synchronization destination of the incremental transaction data. The terminal needs to synchronize the incremental transaction data to the second financial database. In order to achieve incremental synchronization, the terminal may cache the incremental service data to the preset message middleware cluster, and synchronize the cached incremental service data to the second financial database through the preset message middleware. The data format in the second financial database may be different from the data format in the first financial database, and the incremental service data is cached in a cache location corresponding to the target identifier in the preset message middleware cluster in a preset data format, for example, is cached in a cache location corresponding to the target identifier in the preset message middleware cluster in the data format of the first financial database.

The terminal can synchronize the incremental business data cached in the cache position corresponding to the target identifier in the preset message middleware cluster to the second financial database. The terminal may obtain a target data format corresponding to the second financial database, and obtain a second storage location in the second financial database corresponding to the target identifier. The target data format can be provided for a preset message middleware cluster, and the data format of the cached incremental service data is converted based on the target data format in the preset message middleware cluster; and because the first storage position corresponds to the target identifier, determining the second storage position can provide a basis for synchronizing the incremental business data to the position corresponding to the first storage position in the second financial database.

Step S208, according to the target data format, the incremental business data in the buffer position is converted to obtain target incremental business data, and the target incremental business data is synchronized to a second storage position in the second financial database.

And after determining the target data format of the second financial database and a second storage position for storing the incremental business data in the second financial database, the terminal. The terminal can perform format conversion on the incremental business data in each cache position based on the target data format to obtain target incremental business data. The terminal can acquire the incremental business data at the cache position corresponding to the target identifier from the preset message middleware cluster, convert the data format of the incremental business data based on the target data format, and synchronously store the target incremental business data after the data format conversion to the second storage position corresponding to the target identifier based on the corresponding relation between the target identifier and each storage position in the second financial database, so as to realize the incremental synchronization of the first financial database and the second financial data.

In the data synchronization method, when the service data updating information of the first data is detected, the incremental service data and the first storage position thereof are determined, the incremental service data are sent to the cache position corresponding to the target identifier in the preset message middleware cluster, after the target data format corresponding to the second financial database and the second storage position corresponding to the target identifier are determined, the incremental service data of the cache position are subjected to format conversion according to the target data format, and the converted target incremental service data are synchronized to the second storage position in the second financial database. Compared with the traditional mode of respectively accessing different databases to acquire corresponding business data, the method and the system have the advantages that when one database updates incremental data, the incremental data is cached through the message middleware cluster and is synchronized to the corresponding positions of other databases by combining with the formats of other databases, and the system can acquire corresponding data only by accessing one position, so that the execution efficiency of banking business is improved.

In one embodiment, the step of obtaining the incremental transaction data at a first storage location in the first financial database includes: acquiring a first data table field for storing the incremental business data in the first financial database; and obtaining a first storage position of the incremental business data in the first financial database according to the first data table field.

In this embodiment, a plurality of data tables are preset in each of the first financial data and the second financial database. Each data table has a corresponding respective data table field therein, each data table field storing corresponding service data. The storage locations may represent data table fields in a financial database. The delta service data may be stored in a corresponding data table field. After detecting the appearance of the incremental business data, the terminal can acquire a first data table field of the incremental business data in the first financial database. The first data table field may include a plurality of data table fields corresponding to the plurality of incremental service data. The terminal may obtain a first storage location of the incremental transaction data in the first financial database according to the first data table field. Wherein, the first data table field may be named by a number or a letter.

According to the embodiment, the terminal obtains the first storage position of the incremental business data in the first financial database through the data table field based on the incremental business data in the financial database, the terminal performs incremental synchronization based on the first storage position, and banking business is executed through the financial database after the incremental synchronization, so that the banking business execution efficiency is improved.

In one embodiment, according to the first storage location, sending the incremental service data to a cache location corresponding to a target identifier in a preset message middleware cluster includes: inquiring a preset storage position corresponding table according to the first data table field to obtain a target identifier corresponding to the first data table field in the Kaff card cluster; the preset storage position corresponding table comprises corresponding relations between preset first data table fields and marks in the Kaff card cluster; and sending the increment service data to a cache position corresponding to the target identifier in the Kaff card cluster.

In this embodiment, the preset message middleware cluster may be a Kafka (Kafka) cluster, where Kafka is an open-source distributed message middleware, and Kafka is a stream processing platform. Kakfa support the delivery of messages between applications in a publish or subscribe fashion. In a Kafka cluster, messages of one Kafka node are carried by Topic (Topic) and may be stored in one or more Topic partitions. The application for issuing the message is a producer, for example, the first financial database, and the application for consuming the message is a consumer, for example, a preset data consumption tool.

Wherein each topic in the card cluster may represent a respective cache location. The terminal may generate a preset storage location correspondence table including correspondence between each first data table field and each identifier in the kaff card cluster in advance. The identifiers corresponding to the topics in the first data table field and the kav card cluster can be associated through numerical values, for example, the terminal associates the identifiers with the first data table field with the same numerical value, and the association relationship input by service personnel for the identifiers in the first data table field and the kav card cluster can also be obtained. And the terminal can query a preset storage position corresponding table according to the first data table field to obtain a target identifier corresponding to the first data table field in the card cluster. Therefore, the terminal can send the incremental service data to a cache position corresponding to the target identifier in the card cluster, and caching of the incremental service data is realized.

Specifically, the first financial database is set as a source database a, and the first data table field included in the source database a includes table 1 and table 2 … table N. Wherein N is a natural number. The cache positions corresponding to the identifiers in the card cluster are topic1 and topic2 … topicN. Wherein N is a natural number. The terminal may determine, based on the preset storage location correspondence table, a target identifier corresponding to the first data table field, for example, a numerical correspondence corresponding to topicN in table N. The terminal may use a data collector to collect the incremental service data of table 1, table 2 in the source database a in real time, respectively, into topic1, topic2, topicN of the kafka cluster, and buffer the incremental service data, respectively.

According to the embodiment, the terminal can realize the caching of the incremental business data into the card cluster through the corresponding relation between the data table field and the identifier in the card cluster, and the incremental business data is synchronized through the card cluster, so that the banking business can acquire the data more quickly in the business executing process, and the business executing efficiency is improved.

In one embodiment, further comprising: for each first data table field in the first financial database, inquiring the second financial database according to the first data table field; if the second data table field which is the same as the first data table field does not exist in the second financial database, generating the second data table field in the second financial database according to the first data table field and the data format of the second financial database; and obtaining a preset storage position corresponding table according to the first data table field and the second data table field.

In this embodiment, the preset storage location correspondence table further includes a correspondence between a first data table field in the first financial database and a second data table field in the second financial database. When incremental data synchronization is performed, it is required to ensure that a database generated by the incremental service data and a database to which the incremental service data is synchronized both contain corresponding data table fields. The terminal may construct a correspondence between the first data table field and the second data table field in advance. For each first data table field in the first financial database, the terminal may query the second financial database according to the first data table field, detect whether the second data table field with the same first data table field exists in the second financial database, and if the terminal detects that the second data table field with the same first data table field exists in the second financial database, associate the first data table field with the same second data table field, generate a correspondence between the first data table field and the second data table field, and store the correspondence in a preset storage location correspondence table.

If the terminal detects that the second data table field which is the same as the first data table field does not exist in the second financial database, the terminal can generate the second data table field in the second financial database according to the first data table field and the data format of the second financial database. Therefore, the terminal can obtain the preset storage position corresponding table according to the corresponding relation between the first data table field and the second data table field.

Specifically, the second financial database may be plural, and may include a financial database of the same type as the first financial database, or may include a financial database of a different type from the first financial database. Taking the example that the first financial database is a source database a, the second financial database includes a financial database B of the same class as a and a financial database C of a different class from a. When incremental synchronization is performed on the financial databases, the terminal may ensure that the data table fields identical to the source database a exist in each second financial database, where the source database a and the data table fields in the financial database B, C need to be in one-to-one correspondence. If the terminal detects that the data table field which is the same as the source database A does not exist in the financial database B or C, the terminal can newly add the corresponding data table field in the financial database B or C. The data table fields in the source database a may thus be represented as table 1, table 2..table N, the corresponding tables in the financial database B are table 11, table 21..table N1, and the corresponding tables in the financial database C are table 12, table 22..table N2. Wherein N is a natural number.

In addition, when performing incremental synchronization, the terminal needs to synchronously store the cached incremental service data in the preset message middleware to a corresponding second storage position in the second financial database, and then the terminal can determine, according to the preset storage position corresponding table, a cache position corresponding to the target identifier where the incremental service data is located, and corresponds to the position in the second financial database.

For example, in one embodiment, obtaining the second storage location corresponding to the target identifier in the second financial database includes: determining a second data table field corresponding to the first data table field in the second financial database according to a preset storage position corresponding table; and obtaining a second storage position corresponding to the target identifier in the second financial database according to the corresponding relation between the first data table field and the target identifier and the corresponding relation between the first data table field and the second data table field.

In this embodiment, the preset storage location correspondence table includes a correspondence between a first data table field and a second data table field, and may further include a correspondence between a first data table field and a target identifier, so that in order to synchronize incremental service data at a cache location corresponding to the target identifier to a second financial database, the terminal may determine, according to the preset storage location correspondence table, a second data table field corresponding to the first data table field in the second financial database, and obtain, according to a correspondence between the first data table field and the target identifier and a correspondence between the first data table field and the second data table field, a second storage location corresponding to the target identifier in the second financial database. For example, there may be a plurality of the target identifiers, there may be a plurality of the second data table fields, and the terminal obtains each second data table field corresponding to each target identifier according to the corresponding relationship between the first data table field and each target identifier and the corresponding relationship between the first data table field and each second data table field, so that the terminal may send the second data table field to a preset message middleware cluster, so that the terminal synchronizes incremental service data to the corresponding second data table field in the second financial database for storage in combination with each second data table field corresponding to each target identifier

Through the embodiment, the terminal can synchronize the incremental business data by generating the same data table field in the second financial database in advance and the same data table field and the preset storage position corresponding table, so that the banking system executes business based on the data after incremental synchronization, and the business execution efficiency is improved.

In one embodiment, performing format conversion on the incremental service data in the buffer location according to the target data format to obtain target incremental service data, including: acquiring incremental business data in the cache position through a preset data consumption tool to obtain consumption data; and carrying out format conversion on the consumption data according to the target data format to obtain target incremental business data.

In this embodiment, the terminal may synchronize the incremental service data of the buffer location in the card cluster to the second financial database. Wherein the incremental transaction data may be data in a data format of the first financial database. In order to make the terminal meet the data requirement of the second financial database, the terminal can acquire a preset data consumption tool, and acquire the incremental business data in the cache position through the preset data consumption tool to obtain consumption data. The terminal can convert the format of the consumption data according to the target data format corresponding to the second financial database, so as to obtain the target incremental business data.

Specifically, the consumption tool may be a logstash (storage) tool with an open source for consumption, or may be a self-written consumption code for consumption. The data in the first financial database is stored in a certain format in the cache position of the kafka cluster, the terminal fetches the data from the kafka through a consumption tool, and after the data is fetched, the data format of each field is converted to match the financial database when the data is written into the financial database. Taking the example that the cache locations in the kafu card cluster include topic1, topic2.. topicN, the second financial database includes financial databases B and C, where N is a natural number. The terminal may use a corresponding data consumption tool to consume data from topic1, topic2.. topicN in the card cluster, and write the data obtained by consumption into table 11, table 21..table N1 in the financial database B after performing data conversion according to a target data format identifiable by the financial database B, and write into table 12, table 22..table N2 in the financial database C after performing data conversion according to a target data format identifiable by the financial database C. If there are multiple second financial databases, the terminal can perform data increment synchronization by adopting the same method.

According to the embodiment, the terminal can synchronize the data of the cached incremental business data in the card cluster based on the data consumption tool, and the banking system performs business based on the synchronized data, so that the execution efficiency of the banking business is improved.

In one embodiment, as shown in fig. 3, fig. 3 is a flow chart of a data synchronization method in another embodiment. In this embodiment, to ensure data integrity and real-time synchronization between different databases during a parallel period of multiple source databases, the terminal may use the kafka cluster as a transfer container of two databases, collect the change data in real time into the kafka for transfer caching by monitoring the content of the source databases in real time, and the consumer reads the data from the kafka to consume the data in a format of a financial database, so as to realize near real-time synchronization of incremental data, for example, real-time synchronization from mysql to myspl, mysql to oracle, oracle to mysql, and the like. And ensuring the stability of service data when the multi-source database is in parallel period. The method can be used as database migration transformation of the same application; in addition, if the data are among different applications, the data can be synchronized to the financial database by the method according to the table fields in the source database which are required to be read by the target application, and the target application can complete the work by only reading the financial database without repeated reading of the two databases.

The terminal may determine in advance whether the same data table field as in the source database a exists in the financial database B (second financial database) which is the same kind as the source database a (first financial database) and the financial database C (second financial database) which is different from the source database a, and if not, the terminal may newly add the data table field in the corresponding financial database. For stock data, that is, data other than incremental business data in the financial databases, the terminal can synchronize in advance, for example, by exporting SQL, so as to synchronize the stock data among the financial databases.

The tables in the source database a are table 1 and table 2..table N, the corresponding tables in the financial database B are table 11 and table 21..table N1, and the corresponding tables in the financial database C are table 12 and table 22..table N2, N are natural numbers. The cache locations in the card clusters may be represented by topic1, topic2.. topicN, N represents the identity and is a natural number. The terminal may use the data collector to collect the incremental data of table N to topic1, topic2.. topicN of the kafka cluster in real time, respectively, in the source database a. And using a preset data consumption tool to consume data from topic1, topic2.. topicN of kafka, and writing the data after conversion of the data in a format recognizable by the financial database B into corresponding tables 11, 21..table N1 in the financial database B, and writing the data after conversion of the data in a format recognizable by the financial database C into corresponding written tables 12, 22..table N2 in the financial database C. If there are multiple second financial databases, the terminal can use the same method to perform data increment synchronization. The terminal uses the Kaff card cluster as a transfer container, so that the incremental business data synchronization between two similar data sources (A and B) can be realized, and the incremental business data synchronization between two non-similar data sources (A and C) can also be realized.

Through the embodiment, when the incremental data is updated by the terminal through one database, the incremental data is cached by the message middleware cluster and is synchronized to the corresponding positions of other databases by combining with the formats of other databases, so that the databases of different data sources are synchronized, and the quasi-real-time synchronization of the incremental service data is realized. And the system can obtain corresponding data only by accessing one position, thereby improving the execution efficiency of banking business.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides a data synchronization device for realizing the above related data synchronization method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in one or more embodiments of the data synchronization device provided below may refer to the limitation of the data synchronization method hereinabove, and will not be repeated herein.

In one embodiment, as shown in fig. 4, there is provided a data synchronization apparatus including: detection module 500, cache module 502, acquisition module 504, and synchronization module 506, wherein:

the detection module 500 is configured to detect service data update information of a first financial database, and obtain incremental service data in the first financial database and a first storage location of the incremental service data in the first financial database according to the service data update information.

The buffer module 502 is configured to send the incremental service data to a buffer location corresponding to a target identifier in a preset message middleware cluster according to the first storage location; the target identifier corresponds to the first storage location.

The obtaining module 504 is configured to obtain a target data format corresponding to a second financial database, and obtain a second storage location corresponding to the target identifier in the second financial database.

And a synchronizing module 506, configured to perform format conversion on the incremental transaction data in the buffer location according to the target data format, obtain target incremental transaction data, and synchronize the target incremental transaction data to a second storage location in the second financial database.

In one embodiment, the detection module 500 is specifically configured to obtain a first data table field in the first financial database, where the first incremental transaction data is stored; and obtaining a first storage position of the incremental business data in the first financial database according to the first data table field.

In one embodiment, the buffer module 502 is specifically configured to query a preset storage location mapping table according to the first data table field to obtain a target identifier corresponding to the first data table field in the kaff card cluster; the preset storage position corresponding table comprises corresponding relations between preset first data table fields and marks in the Kaff card cluster; and sending the increment service data to a cache position corresponding to the target identifier in the Kaff card cluster.

In one embodiment, the apparatus further comprises: the construction module is used for inquiring the second financial database according to the first data table fields aiming at each first data table field in the first financial database; if the second data table field which is the same as the first data table field does not exist in the second financial database, generating the second data table field in the second financial database according to the first data table field and the data format of the second financial database; and obtaining a preset storage position corresponding table according to the first data table field and the second data table field.

In one embodiment, the obtaining module 504 is specifically configured to determine, according to a preset storage location correspondence table, a second data table field corresponding to the first data table field in the second financial database; and obtaining a second storage position corresponding to the target identifier in the second financial database according to the corresponding relation between the first data table field and the target identifier and the corresponding relation between the first data table field and the second data table field.

In one embodiment, the synchronization module 506 is specifically configured to obtain the incremental service data in the cache location through a preset data consumption tool, so as to obtain consumption data; and carrying out format conversion on the consumption data according to the target data format to obtain target incremental business data.

The modules in the data synchronization device may be implemented in whole or in part by software, hardware, or a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a terminal, and the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a data synchronization method. The display unit of the computer device is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the structure shown in FIG. 5 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

In one embodiment, a computer device is provided that includes a memory having a computer program stored therein and a processor that implements the data synchronization method described above when the computer program is executed by the processor.

In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor implements the data synchronization method described above.

In one embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, implements the data synchronization method described above.

The user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or sufficiently authorized by each party.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magneto-resistive random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (PHASE CHANGE Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.

Claims

1. A method of data synchronization, the method comprising:

2. The method of claim 1, wherein the step of obtaining a first storage location of incremental business data in the first financial database comprises:

3. The method of claim 2, wherein the predetermined message middleware cluster is a kaff card cluster; the step of sending the incremental business data to a cache position corresponding to a target identifier in a preset message middleware cluster according to the first storage position comprises the following steps:

4. The method according to claim 2, wherein the method further comprises:

5. The method of claim 4, wherein the obtaining a second storage location in the second financial database corresponding to the target identifier comprises:

6. The method according to any one of claims 1 to 4, wherein said performing format conversion on the incremental transaction data in the buffer location according to the target data format to obtain target incremental transaction data includes:

7.A data synchronization device, the device comprising:

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.

9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.